mmu_manga

configs: - config_name: default data_dir: mmu_manga/dataset tags: - astronomy license: cc-by-4.0 pretty_name: mmu_manga size_categories: - 10K<n<100K --- # mmu_manga HATS Catalog Collection This is the collection of HATS catalogs representing mmu_manga. This dataset is part of the [Multimodal Universe](https://github.com/MultimodalUniverse/MultimodalUniverse), a large-scale collection of multimodal astronomical data. For full details, see the paper: [The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TBs of Astronomical Scientific Data](https://arxiv.org/abs/2412.02527). ### Access the catalog We recommend the use of the [LSDB](https://lsdb.io) Python framework to access HATS catalogs. LSDB can be installed via `pip install lsdb` or `conda install conda-forge::lsdb`, see more details [in the docs](https://docs.lsdb.io/). The following code provides a minimal example of opening this catalog: ```python import lsdb # Full sky coverage. catalog = lsdb.open_catalog("https://huggingface.co/datasets/hugging-science/mmu_manga") ``` Each catalog in this collection is represented as a separate [Apache Parquet dataset](https://arrow.apache.org/docs/python/dataset.html) and can be accessed with a variety of tools, including `pandas`, `pyarrow`, `dask`, `Spark`, `DuckDB`. ### File structure This catalog is represented by the following files and directories: - [`collection.properties`](https://huggingface.co/datasets/hugging-science/mmu_manga/collection.properties) — textual metadata file describing the HATS collection of catalogs - [`mmu_manga`](https://huggingface.co/datasets/hugging-science/mmu_manga/mmu_manga) — main HATS catalog directory - [`dataset/`](https://huggingface.co/datasets/hugging-science/mmu_manga/mmu_manga/dataset/) — Apache Parquet dataset directory for the main catalog - ... parquet metadata and data files in sub directories ... - [`hats.properties`](https://huggingface.co/datasets/hugging-science/mmu_manga/mmu_manga/hats.properties) — textual metadata file describing the main HATS catalog - [`partition_info.csv`](https://huggingface.co/datasets/hugging-science/mmu_manga/mmu_manga/partition_info.csv) — CSV file with a list of catalog HEALPix tiles (catalog partitions) ### Catalog metadata Metadata of the main HATS catalog: | **Number of rows** | **Number of columns** | **Number of partitions** | **Size on disk** | **HATS Builder** | | --- | --- | --- | --- | --- | | 10,735 | 41 | 3,705 | 1.05 GiB | hats-import v0.9.0, hats v0.9.0 | ### Catalog columns The main HATS catalog contains the following columns: | **Name** | **`_healpix_29`** | **`z`** | **`spaxel_size`** | **`spaxel_size_units`** | **`spaxels.flux`** | **`spaxels.ivar`** | **`spaxels.mask`** | **`spaxels.lsf`** | **`spaxels.lambda`** | **`spaxels.x`** | **`spaxels.y`** | **`spaxels.spaxel_idx`** | **`spaxels.flux_units`** | **`spaxels.lambda_units`** | **`spaxels.skycoo_x`** | **`spaxels.skycoo_y`** | **`spaxels.ellcoo_r`** | **`spaxels.ellcoo_rre`** | **`spaxels.ellcoo_rkpc`** | **`spaxels.ellcoo_theta`** | **`spaxels.skycoo_units`** | **`spaxels.ellcoo_r_units`** | **`spaxels.ellcoo_rre_units`** | **`spaxels.ellcoo_rkpc_units`** | **`spaxels.ellcoo_theta_units`** | **`images.filter`** | **`images.flux`** | **`images.flux_units`** | **`images.psf`** | **`images.psf_units`** | **`images.scale`** | **`images.scale_units`** | **`ra`** | **`dec`** | **`maps.group`** | **`maps.label`** | **`maps.flux`** | **`maps.ivar`** | **`maps.mask`** | **`maps.array_units`** | **`object_id`** | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Data Type** | int64 | double | double | string | fixed_size_list<element: fixed_size_list<element: float>[4563]>[1] | fixed_size_list<element: fixed_size_list<element: float>[4563]>[1] | fixed_size_list<element: fixed_size_list<element: int32>[4563]>[1] | fixed_size_list<element: fixed_size_list<element: float>[4563]>[1] | fixed_size_list<element: fixed_size_list<element: float>[4563]>[1] | int64 | int64 | int64 | string | string | float | float | float | float | float | float | string | string | string | string | string | string | fixed_size_list<element: fixed_size_list<element: float>[96]>[96] | string | fixed_size_list<element: fixed_size_list<element: float>[96]>[96] | string | float | string | double | double | string | string | fixed_size_list<element: fixed_size_list<element: float>[96]>[96] | fixed_size_list<element: fixed_size_list<element: float>[96]>[96] | fixed_size_list<element: fixed_size_list<element: float>[96]>[96] | string | string | | **Nested?** | — | — | — | — | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | spaxels | images | images | images | images | images | images | images | — | — | maps | maps | maps | maps | maps | maps | — | "Nested" indicates whether the column is stored as a nested field inside another "struct" column. ### Crossmatch with another catalog HATS catalogs can be efficiently crossmatched using [LSDB](https://lsdb.io), which leverages the HEALPix partitioning to avoid loading the full datasets into memory: ```python import lsdb mmu_manga = lsdb.open_catalog("https://huggingface.co/datasets/hugging-science/mmu_manga") other = lsdb.open_catalog("https://huggingface.co/datasets/<org>/<other_catalog>") crossmatched = mmu_manga.crossmatch(other, radius_arcsec=1.0) print(crossmatched) ``` See the [LSDB documentation](https://docs.lsdb.io/) for more details on crossmatching and other operations. ### Dataset-specific context **Original survey** This dataset is based on MaNGA (Mapping Nearby Galaxies at Apache Point Observatory), one of the three core programs of SDSS-IV. MaNGA used fiber-bundle integral field units (IFUs) on the SDSS 2.5m telescope to obtain spatially resolved optical spectroscopy for a sample of roughly 10,000 nearby galaxies, selected to give a roughly flat distribution in stellar mass. **Data modality** Each row represents one galaxy and bundles together its integral field spectroscopy and imaging: a `spaxels` array giving, for every spatial element of the IFU footprint, the calibrated flux, inverse variance, mask, line-spread function, and wavelength solution across ~4563 spectral elements, along with its sky and elliptical (deprojected) coordinates; an `images` array with multi-band (4-filter) broadband image cutouts and PSF images on a 96×96 pixel grid; and a `maps` array of derived 96×96 pixel-grid quantities (e.g. kinematic and emission-line maps) grouped by analysis type. Basic catalog-level quantities (`ra`, `dec`, `z`, `spaxel_size`) are also included. **Typical use cases** Spatially-resolved spectroscopic data of this kind has been used for machine learning applications such as estimating galaxy properties (e.g. star formation rate, metallicity, stellar population parameters) from spectra, spectral classification, and learning representations of resolved galaxy structure. **Caveats** The dataset includes only objects with successfully reduced MaNGA data products. No further scientific processing is applied beyond the standard pipeline reduction, so spaxels affected by foreground stars, low signal-to-noise, or instrumental artifacts should be filtered using the provided mask arrays. **Citation** When using this dataset in scientific publications, users should cite the MaNGA survey papers (Bundy et al. 2015) and the relevant SDSS-IV/SDSS-V data release paper, and follow the official SDSS acknowledgment guidelines: https://www.sdss.org/collaboration/citing-sdss/