AstroMLCore/AstroM3Dataset

--- license: mit pretty_name: AstroM3Dataset size_categories: - 10K<n<100K tags: - astronomy - multimodal - classification arxiv: - arXiv:2411.08842 --- # AstroM3Dataset ## Description AstroM3Dataset is a time-series astronomy dataset containing photometry, spectra, and metadata features for variable stars. The dataset was constructed by cross-matching publicly available astronomical datasets, primarily from the ASAS-SN (Shappee et al. 2014) variable star catalog (Jayasinghe et al. 2019) and LAMOST spectroscopic survey (Cui et al. 2012), along with data from WISE (Wright et al. 2010), GALEX (Morrissey et al. 2007), 2MASS (Skrutskie et al. 2006) and Gaia EDR3 (Gaia Collaboration et al. 2021). The dataset includes multiple subsets (`full`, `sub10`, `sub25`, `sub50`) and supports different random seeds (`42`, `66`, `0`, `12`, `123`). Each sample consists of: - **Photometry**: Light curve data of shape `(N, 3)` (time, flux, flux\_error). - **Spectra**: Spectra observations of shape `(M, 3)` (wavelength, flux, flux\_error). - **Metadata**: - `meta_cols`: Dictionary of metadata feature names and values. - `photo_cols`: Dictionary of photometric feature names and values. - **Label**: The class name as a string. ## Corresponding paper and code - Paper: [AstroM³: A self-supervised multimodal model for astronomy](https://arxiv.org/abs/2411.08842) - Code Repository: [GitHub: AstroM³](https://github.com/MeriDK/AstroM3/) - Processed Data: [AstroMLCore/AstroM3Processed](https://huggingface.co/datasets/AstroMLCore/AstroM3Processed/) **Note:** The processed dataset `AstroM3Processed` is created from the original dataset `AstroM3Dataset` by using [preprocess.py](https://huggingface.co/datasets/AstroMLCore/AstroM3Dataset/blob/main/preprocess.py) --- ## Subsets and Seeds AstroM3Dataset is available in different subset sizes: - `full`: Entire dataset - `sub50`: 50% subset - `sub25`: 25% subset - `sub10`: 10% subset Each subset is sampled from the respective train, validation, and test splits of the full dataset. For reproducibility, each subset is provided with different random seeds: - `42`, `66`, `0`, `12`, `123` ## Data Organization The dataset is organized as follows: ``` AstroM3Dataset/ ├── photometry.zip # Contains all photometry light curves ├── utils/ │ ├── parallelzipfile.py # Zip file reader to open photometry.zip ├── spectra/ # Spectra files organized by class │ ├── EA/ │ │ ├── file1.dat │ │ ├── file2.dat │ │ ├── ... │ ├── EW/ │ ├── SR/ │ ├── ... ├── splits/ # Train/val/test splits for each subset and seed │ ├── full/ │ │ ├── 42/ │ │ │ ├── train.csv │ │ │ ├── val.csv │ │ │ ├── test.csv │ │ │ ├── info.json # Contains feature descriptions and preprocessing info │ │ ├── 66/ │ │ ├── 0/ │ │ ├── 12/ │ │ ├── 123/ │ ├── sub10/ │ ├── sub25/ │ ├── sub50/ │── AstroM3Dataset.py # Hugging Face dataset script ``` ## Usage To load the dataset using the Hugging Face `datasets` library: ```python from datasets import load_dataset # Load the default full dataset with seed 42 dataset = load_dataset("AstroMLCore/AstroM3Dataset", trust_remote_code=True) ``` The default configuration is **full_42** (entire dataset with seed 42). To load a specific subset and seed, use {subset}_{seed} as the name: ```python from datasets import load_dataset # Load the 25% subset sampled using seed 123 dataset = load_dataset("AstroMLCore/AstroM3Dataset", name="sub25_123", trust_remote_code=True) ``` --- ## Citation 🤗 If you find this dataset usefull, please cite our paper 🤗 ```bibtex @article{rizhko2024astrom, title={AstroM $\^{} 3$: A self-supervised multimodal model for astronomy}, author={Rizhko, Mariia and Bloom, Joshua S}, journal={arXiv preprint arXiv:2411.08842}, year={2024} } ``` ## References 1. Shappee, B. J., Prieto, J. L., Grupe, D., et al. 2014, ApJ, 788, 48, doi: 10.1088/0004-637X/788/1/48 2. Jayasinghe, T., Stanek, K. Z., Kochanek, C. S., et al. 2019, MNRAS, 486, 1907, doi: 10.1093/mnras/stz844 3. Cui, X.-Q., Zhao, Y.-H., Chu, Y.-Q., et al. 2012, Research in Astronomy and Astrophysics, 12, 1197, doi: 10.1088/1674-4527/12/9/003 4. Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., et al. 2010, AJ, 140, 1868, doi: 10.1088/0004-6256/140/6/1868 5. Morrissey, P., Conrow, T., Barlow, T. A., et al. 2007, ApJS, 173, 682, doi: 10.1086/520512 6. Skrutskie, M. F., Cutri, R. M., Stiening, R., et al. 2006, AJ, 131, 1163, doi: 10.1086/498708 7. Gaia Collaboration, Brown, A. G. A., et al. 2021, AAP, 649, A1, doi: 10.1051/0004-6361/202039657许可证：MIT许可证 规范名称：AstroM3Dataset 样本量范围：10K < n < 100K 标签：天文学、多模态、分类 arXiv编号：arXiv:2411.08842 # AstroM3Dataset ## 描述 AstroM3Dataset是一款时序天文数据集，包含变星的测光（Photometry）、光谱（Spectra）与元数据（Metadata）特征。该数据集通过交叉匹配多组公开天文数据集构建而成，核心数据源包括ASAS-SN变星星表（Shappee等，2014；Jayasinghe等，2019）、LAMOST光谱巡天（Cui等，2012），以及WISE、GALEX、2MASS与Gaia EDR3的配套数据（对应文献分别为Wright等，2010；Morrissey等，2007；Skrutskie等，2006；Gaia合作团队等，2021）。 该数据集包含多个子集（`full`、`sub10`、`sub25`、`sub50`），并支持多种随机种子配置（`42`、`66`、`0`、`12`、`123`）。每个样本包含以下组成部分： - **测光数据（Photometry）**：形状为`(N, 3)`的光变曲线数据，维度依次对应时间、流量与流量误差。 - **光谱数据（Spectra）**：形状为`(M, 3)`的光谱观测数据，维度依次对应波长、流量与流量误差。 - **元数据（Metadata）**： - `meta_cols`：存储元数据特征名称与对应取值的字典。 - `photo_cols`：存储测光特征名称与对应取值的字典。 - **标签（Label）**：以字符串形式表示的样本类别名称。 ## 相关论文与代码 - 论文：[AstroM³：一款面向天文学的自监督多模态模型](https://arxiv.org/abs/2411.08842) - 代码仓库：[GitHub: AstroM³](https://github.com/MeriDK/AstroM3/) - 预处理后数据集：[AstroMLCore/AstroM3Processed](https://huggingface.co/datasets/AstroMLCore/AstroM3Processed/) **注意**：预处理后的数据集`AstroM3Processed`是通过[preprocess.py](https://huggingface.co/datasets/AstroMLCore/AstroM3Dataset/blob/main/preprocess.py)对原始数据集`AstroM3Dataset`进行处理后得到的。 ## 子集与随机种子 AstroM3Dataset提供了多种不同规模的子集： - `full`：完整数据集 - `sub50`：占比50%的子集 - `sub25`：占比25%的子集 - `sub10`：占比10%的子集 每个子集均从完整数据集对应的训练、验证与测试划分中采样得到。为确保实验可复现，每个子集均提供了多组随机种子版本： - `42`、`66`、`0`、`12`、`123` ## 数据组织形式 数据集的目录结构如下： AstroM3Dataset/ ├── photometry.zip # 存储所有测光光变曲线文件 ├── utils/ │ ├── parallelzipfile.py # 用于读取photometry.zip的ZIP文件阅读器 ├── spectra/ # 按类别分组的光谱文件 │ ├── EA/ │ │ ├── file1.dat │ │ ├── file2.dat │ │ ├── ... │ ├── EW/ │ ├── SR/ │ ├── ... ├── splits/ # 各子集与随机种子对应的训练/验证/测试划分文件 │ ├── full/ │ │ ├── 42/ │ │ │ ├── train.csv │ │ │ ├── val.csv │ │ │ ├── test.csv │ │ │ ├── info.json # 包含特征描述与预处理相关信息 │ │ ├── 66/ │ │ ├── 0/ │ │ ├── 12/ │ │ ├── 123/ │ ├── sub10/ │ ├── sub25/ │ ├── sub50/ │── AstroM3Dataset.py # Hugging Face数据集加载脚本 ## 使用方法 通过Hugging Face `datasets`库加载该数据集的示例代码如下： python from datasets import load_dataset # 加载默认配置：使用随机种子42的完整数据集 dataset = load_dataset("AstroMLCore/AstroM3Dataset", trust_remote_code=True) 该数据集的默认配置为**full_42**（即使用随机种子42的完整数据集）。若需加载特定子集与随机种子的版本，可将`{subset}_{seed}`作为配置名称参数： python from datasets import load_dataset # 加载使用随机种子123采样得到的25%规模子集 dataset = load_dataset("AstroMLCore/AstroM3Dataset", name="sub25_123", trust_remote_code=True) ## 引用 🤗 若本数据集对您的研究有所帮助，请引用我们的论文 🤗 bibtex @article{rizhko2024astrom, title={AstroM$^{3}$: A self-supervised multimodal model for astronomy}, author={Rizhko, Mariia and Bloom, Joshua S}, journal={arXiv preprint arXiv:2411.08842}, year={2024} } ## 参考文献 1. Shappee, B. J., Prieto, J. L., Grupe, D., et al. 2014, ApJ, 788, 48, doi: 10.1088/0004-637X/788/1/48 2. Jayasinghe, T., Stanek, K. Z., Kochanek, C. S., et al. 2019, MNRAS, 486, 1907, doi: 10.1093/mnras/stz844 3. Cui, X.-Q., Zhao, Y.-H., Chu, Y.-Q., et al. 2012, Research in Astronomy and Astrophysics, 12, 1197, doi: 10.1088/1674-4527/12/9/003 4. Wright, E. L., Eisenhardt, P. R. M., Mainzer, A. K., et al. 2010, AJ, 140, 1868, doi: 10.1088/0004-6256/140/6/1868 5. Morrissey, P., Conrow, T., Barlow, T. A., et al. 2007, ApJS, 173, 682, doi: 10.1086/520512 6. Skrutskie, M. F., Cutri, R. M., Stiening, R., et al. 2006, AJ, 131, 1163, doi: 10.1086/498708 7. Gaia Collaboration, Brown, A. G. A., et al. 2021, AAP, 649, A1, doi: 10.1051/0004-6361/202039657