UniverseTBD/mmu_desi_edr_sv3

---
configs:
- config_name: default
data_dir: mmu_desi_edr_sv3/dataset
tags:
- astronomy
license: cc-by-4.0
pretty_name: mmu_desi_edr_sv3
size_categories:
- 1M<n<10M
---

<div align="center">
<img src="example_figure.png" width="600">
</div>

# mmu_desi_edr_sv3 HATS Catalog Collection

This is the collection of HATS catalogs representing mmu_desi_edr_sv3.

This dataset is part of the [Multimodal Universe](https://github.com/MultimodalUniverse/MultimodalUniverse),
a large-scale collection of multimodal astronomical data. For full details, see the paper:
[The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TBs of Astronomical Scientific Data](https://arxiv.org/abs/2412.02527).

### Access the catalog

We recommend the use of the [LSDB](https://lsdb.io) Python framework to access HATS catalogs.
LSDB can be installed via `pip install lsdb` or `conda install conda-forge::lsdb`,
see more details [in the docs](https://docs.lsdb.io/).
The following code provides a minimal example of opening this catalog:

```python
import lsdb

# Full sky coverage.
catalog = lsdb.open_catalog("https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3")
# One-degree cone.
catalog = lsdb.open_catalog(
"https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3",
search_filter=lsdb.ConeSearch(ra=247.0, dec=43.0, radius_arcsec=3600.0),
)
```

Each catalog in this collection is represented as a separate [Apache Parquet dataset](https://arrow.apache.org/docs/python/dataset.html) and can be accessed with a variety of tools, including `pandas`, `pyarrow`, `dask`, `Spark`, `DuckDB`.

### File structure

This catalog is represented by the following files and directories:

- [`collection.properties`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/collection.properties) — textual metadata file describing the HATS collection of catalogs
- [`mmu_desi_edr_sv3`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3) — main HATS catalog directory
- [`dataset/`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/dataset/) — Apache Parquet dataset directory for the main catalog
- ... parquet metadata and data files in sub directories ...
- [`hats.properties`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/hats.properties) — textual metadata file describing the main HATS catalog
- [`partition_info.csv`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/partition_info.csv) — CSV file with a list of catalog HEALPix tiles (catalog partitions)
- [`skymap.fits`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/skymap.fits) — HEALPix skymap FITS file with row-counts per HEALPix tile of fixed order 10
- [`mmu_desi_edr_sv3_10arcs/`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3_10arcs) — default margin catalog to ensure data completeness in cross-matching, the margin threshold is 10.0 arcseconds
- ... margin catalog files and directories ...

### Catalog metadata

Metadata of the main HATS catalog, excluding margins and indexes:

| **Number of rows** | **Number of columns** | **Number of partitions** | **Size on disk** | **HATS Builder** |
| --- | --- | --- | --- | --- |
| 1,126,441 | 20 | 306 | 61.8 GiB | hats-import v0.7.1, hats v0.7.1 |

### Catalog columns

The main HATS catalog contains the following columns:

| **Name** | **`_healpix_29`** | **`spectrum.flux`** | **`spectrum.ivar`** | **`spectrum.lsf_sigma`** | **`spectrum.lambda`** | **`spectrum.mask`** | **`Z`** | **`ZERR`** | **`EBV`** | **`FLUX_G`** | **`FLUX_R`** | **`FLUX_Z`** | **`FLUX_IVAR_G`** | **`FLUX_IVAR_R`** | **`FLUX_IVAR_Z`** | **`FIBERFLUX_G`** | **`FIBERFLUX_R`** | **`FIBERFLUX_Z`** | **`FIBERTOTFLUX_G`** | **`FIBERTOTFLUX_R`** | **`FIBERTOTFLUX_Z`** | **`ra`** | **`dec`** | **`ZWARN`** | **`object_id`** |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **Data Type** | int64 | list[float] | list[float] | list[float] | list[float] | list[bool] | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | double | double | bool | string |
| **Nested?** | — | spectrum | spectrum | spectrum | spectrum | spectrum | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| **Value count** | 1,126,441 | 8,764,837,421 | 8,764,837,421 | 8,764,837,421 | 8,764,837,421 | 8,764,837,421 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 |
| **Example row** | 690723642961871770 | [19.97, -4.157, 25.57, -35.53, … (7781 total)] | [0.003643, 0.002593, 0.002245, … (7781 total)] | [0.8615, 0.8615, 0.8615, 0.8615, … (7781 total)] | [3600, 3601, 3602, 3602, 3603, … (7781 total)] | [False, False, False, False, … (7781 total)] | 0.1858 | 5.029e-05 | 0.01178 | 17.86 | 53.7 | 104.7 | 91.37 | 30.44 | 17.35 | 5.629 | 16.93 | 33.01 | 5.629 | 16.93 | 33.01 | 246.7 | 42.85 | False | 39633127718519967 |
| **Minimum value** | 643519964553769984 | -2392754.75 | -0.0 | 0.8452407717704773 | 3600.0 | False | -0.004999999888241291 | -0.0 | -0.0 | -99.0 | -99.0 | -99.0 | -99.0 | -99.0 | -99.0 | -0.0 | -0.0 | -0.0 | -0.0 | -0.0 | -0.0 | 148.36930329196142 | -2.4457088512932574 | False | 1000347531214848 |
| **Maximum value** | 1981012038139459733 | 371762.1875 | 609.637451171875 | 0.8992781639099121 | 9824.0 | True | 5.985368251800537 | 0.007868182845413685 | 0.09311211854219437 | 8527.033203125 | 14788.431640625 | 27467.138671875 | 3400.1640625 | 1222.0245361328125 | 352.811279296875 | 1040.572265625 | 990.2592163085938 | 3370.12158203125 | 1969.8492431640625 | 993.0661010742188 | 3371.087158203125 | 273.9542154942551 | 67.87875977774526 | True | 999860421525504 |

"Nested" indicates whether the column is stored as a nested field inside another "struct" column.

"Value count" may be different from the total number of rows for nested columns: each nested element is counted as a single value.

### Crossmatch with another catalog

HATS catalogs can be efficiently crossmatched using [LSDB](https://lsdb.io),
which leverages the HEALPix partitioning to avoid loading the full datasets into memory:

```python
import lsdb

mmu_desi_edr_sv3 = lsdb.open_catalog("https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3")
other = lsdb.open_catalog("https://huggingface.co/datasets/<org>/<other_catalog>")

crossmatched = mmu_desi_edr_sv3.crossmatch(other, radius_arcsec=1.0)
print(crossmatched)
```

See the [LSDB documentation](https://docs.lsdb.io/) for more details on crossmatching and other operations.

### Dataset-specific context

**Original survey**
This dataset is based on the [Dark Energy Spectroscopic Instrument (DESI)](https://www.desi.lbl.gov/), specifically the Early Data Release (EDR), which represents about 1% of the final survey. DESI collects spectra of millions of galaxies, quasars, and stars to measure the effect of dark energy on the expansion of the universe.

**Data modality**
The dataset consists of spectra with a fixed wavelength range from 3,600 to 9,800 and 7,081 pixels per sample. Each spectrum includes flux, wavelength, and inverse variance (ivar).

**Typical use cases**
The dataset can be used to analyze spectra of galaxies, quasars, and stars. Existing applications include multimodal representation learning (AstroCLIP) and outlier detection using spectrum auto-encoding models.

**Caveats**
The dataset is based on the DESI Early Data Release (EDR), which represents 1% of the final survey. It includes only primary spectra for each object, targets (excluding sky and other object types), and fibers with good status.

**Citation**
Users should cite the DESI Early Data Release and acknowledge the DESI collaboration. The dataset is released under the CC BY 4.0 license, requiring attribution to the original authors.

---
配置项：
- 配置名称：default
数据目录：mmu_desi_edr_sv3/dataset
标签：
- 天文学
许可证：CC BY 4.0
友好名称：mmu_desi_edr_sv3
数据规模类别：100万 < 数据量 < 1000万
---

<div align="center">
<img src="example_figure.png" width="600">
</div>

# mmu_desi_edr_sv3 HATS 星表合集

本数据集为对应mmu_desi_edr_sv3的HATS星表合集。

本数据集隶属于[多模态宇宙（Multimodal Universe）](https://github.com/MultimodalUniverse/MultimodalUniverse)项目，该项目是大规模多模态天文数据合集。如需了解完整细节，请参阅论文：[《多模态宇宙：利用100TB天文科学数据实现大规模机器学习》](https://arxiv.org/abs/2412.02527)。

### 星表访问方式

我们推荐使用[LSDB](https://lsdb.io) Python框架访问HATS星表。LSDB可通过`pip install lsdb`或`conda install conda-forge::lsdb`进行安装，更多细节请参阅[官方文档](https://docs.lsdb.io/)。

以下代码提供了打开该星表的最小示例：

python
import lsdb

# 全天空覆盖范围
catalog = lsdb.open_catalog("https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3")
# 1度天区锥搜索
catalog = lsdb.open_catalog(
"https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3",
search_filter=lsdb.ConeSearch(ra=247.0, dec=43.0, radius_arcsec=3600.0),
)

本合集中的每一份星表均以独立的[Apache Parquet数据集](https://arrow.apache.org/docs/python/dataset.html)形式存储，可通过多种工具访问，包括`pandas`、`pyarrow`、`dask`、`Spark`、`DuckDB`。

### 文件结构

本星表由以下文件与目录组成：

- [`collection.properties`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/collection.properties) — 描述HATS星表合集的文本元数据文件
- [`mmu_desi_edr_sv3`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3) — HATS主星表目录
- [`dataset/`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/dataset/) — 主星表的Apache Parquet数据集目录
- 子目录下的Parquet元数据与数据文件……
- [`hats.properties`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/hats.properties) — 描述HATS主星表的文本元数据文件
- [`partition_info.csv`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/partition_info.csv) — 记录星表HEALPix分区（分块）列表的CSV文件
- [`skymap.fits`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3/skymap.fits) — 固定阶数为10的HEALPix天图FITS文件，记录每个HEALPix分块的行数
- [`mmu_desi_edr_sv3_10arcs/`](https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3/mmu_desi_edr_sv3_10arcs) — 默认边缘星表，用于确保交叉匹配时的数据完整性，边缘阈值为10.0角秒
- 边缘星表文件与子目录……

### 星表元数据

元数据信息（不含边缘星表与索引）：

| **行数** | **列数** | **分区数** | **磁盘占用大小** | **HATS构建工具版本** |
| --- | --- | --- | --- | --- |
| 1,126,441 | 20 | 306 | 61.8 GiB | hats-import v0.7.1、hats v0.7.1 |

### 星表列信息

HATS主星表包含以下列：

| **列名** | `_healpix_29` | `spectrum.flux` | `spectrum.ivar` | `spectrum.lsf_sigma` | `spectrum.lambda` | `spectrum.mask` | `Z` | `ZERR` | `EBV` | `FLUX_G` | `FLUX_R` | `FLUX_Z` | `FLUX_IVAR_G` | `FLUX_IVAR_R` | `FLUX_IVAR_Z` | `FIBERFLUX_G` | `FIBERFLUX_R` | `FIBERFLUX_Z` | `FIBERTOTFLUX_G` | `FIBERTOTFLUX_R` | `FIBERTOTFLUX_Z` | `ra` | `dec` | `ZWARN` | `object_id` |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| **数据类型** | int64 | 浮点型列表 | 浮点型列表 | 浮点型列表 | 浮点型列表 | 布尔型列表 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 浮点型 | 双精度浮点型 | 双精度浮点型 | 布尔型 | 字符串 |
| **是否嵌套** | 无 | spectrum | spectrum | spectrum | spectrum | spectrum | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 | 无 |
| **值计数** | 1,126,441 | 8,764,837,421 | 8,764,837,421 | 8,764,837,421 | 8,764,837,421 | 8,764,837,421 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 | 1,126,441 |
| **示例行** | 690723642961871770 | [19.97, -4.157, 25.57, -35.53, … (7781 total)] | [0.003643, 0.002593, 0.002245, … (7781 total)] | [0.8615, 0.8615, 0.8615, 0.8615, … (7781 total)] | [3600, 3601, 3602, 3602, 3603, … (7781 total)] | [False, False, False, False, … (7781 total)] | 0.1858 | 5.029e-05 | 0.01178 | 17.86 | 53.7 | 104.7 | 91.37 | 30.44 | 17.35 | 5.629 | 16.93 | 33.01 | 5.629 | 16.93 | 33.01 | 246.7 | 42.85 | False | 39633127718519967 |
| **最小值** | 643519964553769984 | -2392754.75 | -0.0 | 0.8452407717704773 | 3600.0 | False | -0.004999999888241291 | -0.0 | -0.0 | -99.0 | -99.0 | -99.0 | -99.0 | -99.0 | -99.0 | -0.0 | -0.0 | -0.0 | -0.0 | -0.0 | -0.0 | 148.36930329196142 | -2.4457088512932574 | False | 1000347531214848 |
| **最大值** | 1981012038139459733 | 371762.1875 | 609.637451171875 | 0.8992781639099121 | 9824.0 | True | 5.985368251800537 | 0.007868182845413685 | 0.09311211854219437 | 8527.033203125 | 14788.431640625 | 27467.138671875 | 3400.1640625 | 1222.0245361328125 | 352.811279296875 | 1040.572265625 | 990.2592163085938 | 3370.12158203125 | 1969.8492431640625 | 993.0661010742188 | 3371.087158203125 | 273.9542154942551 | 67.87875977774526 | True | 999860421525504 |

"是否嵌套"表示该列是否作为嵌套字段存储于另一"struct"列中。

"值计数"可能与嵌套列的总行数不同：每个嵌套元素均被计为单个值。

### 跨星表交叉匹配

HATS星表可通过[LSDB](https://lsdb.io)实现高效交叉匹配，该框架利用HEALPix分区机制避免加载完整数据集至内存中：

python
import lsdb

mmu_desi_edr_sv3 = lsdb.open_catalog("https://huggingface.co/datasets/UniverseTBD/mmu_desi_edr_sv3")
other = lsdb.open_catalog("https://huggingface.co/datasets/<org>/<other_catalog>")

crossmatched = mmu_desi_edr_sv3.crossmatch(other, radius_arcsec=1.0)
print(crossmatched)

如需了解交叉匹配及其他操作的更多细节，请参阅[LSDB官方文档](https://docs.lsdb.io/)。

### 数据集专属背景信息

**原始巡天项目**
本数据集基于[暗能量光谱仪（Dark Energy Spectroscopic Instrument, DESI）](https://www.desi.lbl.gov/)的早期数据发布版（Early Data Release, EDR），该版本约占最终巡天数据的1%。DESI通过获取数百万个星系、类星体与恒星的光谱，以测量暗能量对宇宙膨胀的影响。

**数据模态**
本数据集包含波长范围固定为3600~9800、每个样本含7081个像素的光谱数据。每条光谱均包含流量（flux）、波长与逆方差（ivar）信息。

**典型应用场景**
本数据集可用于分析星系、类星体与恒星的光谱。现有应用包括多模态表征学习（AstroCLIP）以及基于光谱自编码模型的异常值检测。

**使用注意事项**
本数据集基于仅占最终巡天数据1%的DESI早期数据发布版（EDR），仅包含每个目标天体的主光谱、有效观测目标（排除天区及其他天体类型）以及状态良好的光纤观测数据。

**引用要求**
使用者需引用DESI早期数据发布版，并致谢DESI合作团队。本数据集采用CC BY 4.0许可证发布，需注明原作者信息。