SDO_training


# ML-Ready Multi-Modal Image Dataset from SDO

## Overview

This dataset provides machine learning (ML)-ready solar data curated from NASA’s Solar Dynamics Observatory (SDO), covering observations from **May 13, 2010, to Dec 31, 2024**. It includes Level-1.5 processed data from: **Atmospheric Imaging Assembly (AIA)**
and **Helioseismic and Magnetic Imager (HMI)**.

The dataset is designed to facilitate large-scale learning applications in heliophysics, such as space weather forecasting, unsupervised representation learning, and scientific foundation model development.

---

## Dataset Structure

**Data Variables:**

```text
- aia94 (y, x) float32 : AIA 94 Å
- aia131 (y, x) float32 : AIA 131 Å
- aia171 (y, x) float32 : AIA 171 Å
- aia193 (y, x) float32 : AIA 193 Å
- aia211 (y, x) float32 : AIA 211 Å
- aia304 (y, x) float32 : AIA 304 Å
- aia335 (y, x) float32 : AIA 335 Å
- aia1600 (y, x) float32 : AIA 1600 Å (UV continuum)
- hmi_m (y, x) float32 : HMI LOS Magnetogram
- hmi_bx (y, x) float32 : HMI Magnetic Field - x component
- hmi_by (y, x) float32 : HMI Magnetic Field - y component
- hmi_bz (y, x) float32 : HMI Magnetic Field - z component
- hmi_v (y, x) float32 : HMI Doppler Velocity

```

## Dataset Details

| Field | Description |
|------------------------|---------------------------------------------|
| **Temporal Coverage** | May 13, 2010 – Dec 31, 2024 |
| **Data Format** | netCDF (`.nc`), float32 |
| **Temporal Granularity**| 12 minutes |
| **Data Shape** | `[13, 4096, 4096]` per file |
| **Channels** | 13 total (AIA EUV ×8 + HMI magnetograms ×5) |
| **Size per File** | ~570 MB |
| **Total Size** | ~360TB |

---

## Notes

Training Data for 1 month is available on Huggingface parallel to the main branch as Parquet files.

The full dataset is located in AWS S3 buckets. Note that full dataset is over 360 TB. Users can see the full list of files using the below command.

```bash
aws s3 ls s3://nasa-surya-bench --no-sign-request
```
To download the individual files from aws S3 buckets, they can utilize the tutorial in the link [https://docs.aws.amazon.com/AmazonS3/latest/userguide/download-objects.html].
There are different options available for downloading and syncing data from S3.

Authors
Sujit Roy, Dinesha Vasanta Hegde, Johannes Schmude, Amy Lin, Vishal Gaur, Talwinder Singh, Rohit Lal
corr: [e-mail]


# 可直接用于机器学习的SDO多模态图像数据集

## 概述
本数据集从美国国家航空航天局（National Aeronautics and Space Administration, NASA）的太阳动力学观测台（Solar Dynamics Observatory, SDO）中精选整理得到可直接用于机器学习（Machine Learning, ML）的太阳观测数据，观测时间范围覆盖**2010年5月13日至2024年12月31日**，包含经过1.5级（Level-1.5）处理的**大气成像组件（Atmospheric Imaging Assembly, AIA）**与**日震与磁成像仪（Helioseismic and Magnetic Imager, HMI）**观测数据。

本数据集旨在助力日物理学领域的大规模机器学习应用，例如空间天气预报、无监督表征学习以及科学基础模型研发。

---

## 数据集结构

**数据变量：**

text
- aia94 (y, x) float32 : AIA 94埃波段数据
- aia131 (y, x) float32 : AIA 131埃波段数据
- aia171 (y, x) float32 : AIA 171埃波段数据
- aia193 (y, x) float32 : AIA 193埃波段数据
- aia211 (y, x) float32 : AIA 211埃波段数据
- aia304 (y, x) float32 : AIA 304埃波段数据
- aia335 (y, x) float32 : AIA 335埃波段数据
- aia1600 (y, x) float32 : AIA 1600埃波段（紫外连续谱）数据
- hmi_m (y, x) float32 : HMI视线方向磁图（LOS Magnetogram）
- hmi_bx (y, x) float32 : HMI磁场x分量
- hmi_by (y, x) float32 : HMI磁场y分量
- hmi_bz (y, x) float32 : HMI磁场z分量
- hmi_v (y, x) float32 : HMI多普勒速度数据

## 数据集详情

| 字段 | 描述 |
|------------------------|---------------------------------------------|
| **时间覆盖范围** | 2010年5月13日 — 2024年12月31日 |
| **数据格式** | netCDF（`.nc`），float32 精度 |
| **时间粒度** | 12分钟 |
| **数据形状** | 单个文件为 `[13, 4096, 4096]` |
| **总通道数** | 共13通道（8通道AIA极紫外（Extreme Ultraviolet, EUV）数据 + 5通道HMI磁图数据） |
| **单文件大小** | 约570 MB |
| **总数据量** | 约360 TB |

---

## 备注
一个月的训练数据集已以Parquet格式上传至Hugging Face主分支的并行分支中。

完整数据集存储于亚马逊网络服务（Amazon Web Services, AWS）的S3存储桶中，需注意完整数据集容量超过360 TB。用户可通过以下命令查看完整文件列表：
bash
aws s3 ls s3://nasa-surya-bench --no-sign-request

若需从AWS S3存储桶中下载单个文件，可参考该链接中的教程：[https://docs.aws.amazon.com/AmazonS3/latest/userguide/download-objects.html]。目前提供了多种从S3下载或同步数据的方案。

## 作者
Sujit Roy、Dinesha Vasanta Hegde、Johannes Schmude、Amy Lin、Vishal Gaur、Talwinder Singh、Rohit Lal
通讯邮箱：[e-mail]