SDO 2H Machine Learning Dataset

This dataset provides a compact Machine Learning ready dataset of SDO EUV and HMI medium-resolution  (1024x1024 pixels) images, for a total of 56,664 samples from May 14, 2010, to April 18, 2023, with a temporal cadence of 2 hours.

EUV images are provided at the following wavelength : 1600A, 304A, 211A, 193A, 171A and 94A.They are processed from the level 1.5 AIA-synoptic dataset (http://jsoc.stanford.edu/data/aia/synoptic/) and are successively:

corrected for instrument degradation

normalised by exposure time

log-transformed (x->log(1+x)), symetrically on positive and negative values

saturated to the 99.9 percentile maximum pixel value of the dataset, up to 2020*, for each channel

linearly scaled between 0 and 255, converted to 8bit integers and compressed as jpegs

The HMI's line-of-sight magnetograms (blos.zip) are retrieved from JSOC from the level 1.5 45-second line-of-sight serie and are successively :

downscaled to 1024x1024 pixels

standardized to a 2.4 arcec-to-pixel resolution (equal to the EUV images)

aligned with the EUV images

log-transformed (x->log(1+x))

saturated to the 99.9 percentile maximum pixel value of the dataset, up to 2020* 

linearly scaled between 0 and 255, with 127 representig original null values, 0 and 255 respecivelly the negative and positive saturation value (approximately 4644G before log-transformation)

converted to 8bit integers and compressed as jpegs

Downscaled and cropped images (224x448 pixels) used in Francisco et al., 2023 are aso provided in pcnn_images.zip

An outlier study is also provided in anomalies.zip, from which '{wavelength}_anomalies_grades.csv' files can be used to exclude the dates where abnormal samples of a given type ('anomalies_grade_scale.txt') are identified.

*The percentile values are computed on the pixels joint distribution using all sample from 2010 to 2019-12 included, so that the period starting from 2020-01 can be used as a completely independant test set.Original exposure and instrument degradation corrected values can be retrieved using the saturation values provided belows. Althought the JPEG encoding results in the loss of small scale information, the dataset processing preserve the physical intensity of the original inputs so that the provided compressed images can efficiently be used to estimate large and medium-scale Active Regions physical features.

HMI / BLOS
±4644 G

1600
9,360 DN

304
44,488 DN

171
29,599 DN

193
81,139 DN

211
8,179 DN

94
6,099 DN

本数据集提供了一套紧凑且可直接用于机器学习的太阳动力学天文台（SDO，Solar Dynamics Observatory）极紫外（EUV，Extreme Ultraviolet）与日震与磁像仪（HMI，Helioseismic and Magnetic Imager）中等分辨率（1024×1024像素）图像集，涵盖2010年5月14日至2023年4月18日期间的共56664个样本，时间采样间隔为2小时。

EUV图像包含以下波段：1600Å、304Å、211Å、193Å、171Å及94Å。这些图像源自1.5级AIA综合数据集（http://jsoc.stanford.edu/data/aia/synoptic/），并依次经过如下处理：
1. 仪器退化校正
2. 曝光时间归一化
3. 对正负像素值分别进行对称对数变换（x→log(1+x)）
4. 针对各通道，将像素值截断至截至2020年的数据集99.9%分位最大值*
5. 线性缩放至0~255区间，转换为8位整数后以JPEG格式压缩存储。

HMI的线视向磁图（blos.zip）源自JSOC的1.5级45秒线视向序列数据，依次经过如下处理：
1. 下采样至1024×1024像素
2. 标准化至2.4角秒/像素的分辨率（与EUV图像保持一致）
3. 与EUV图像配准对齐
4. 对数变换（x→log(1+x)）
5. 针对各通道，将像素值截断至截至2020年的数据集99.9%分位最大值*
6. 线性缩放至0~255区间，其中127对应原始零值，0和255分别对应负饱和与正饱和值（对数变换前约为±4644高斯）
7. 转换为8位整数后以JPEG格式压缩存储。

此外，本数据集还提供了Francisco等人2023年研究中使用的下采样裁剪图像（224×448像素），存储于pcnn_images.zip中。

数据集还附带了异常样本研究文件（anomalies.zip），其中"{wavelength}_anomalies_grades.csv"文件可用于剔除对应波段识别出异常样本的日期，异常类型可参考"anomalies_grade_scale.txt"。

* 百分位值基于2010年至2019年12月的全部样本联合像素分布计算得到，因此2020年1月起的数据可作为完全独立的测试集使用。可通过下文提供的饱和值恢复原始曝光校正与仪器校正后的数值。尽管JPEG编码会丢失部分小尺度信息，但本数据集的处理流程保留了原始输入的物理强度，因此所提供的压缩图像可有效用于估算大、中尺度活动区的物理特征。

HMI/BLOS：±4644 G
1600Å：9360 DN
304Å：44488 DN
171Å：29599 DN
193Å：81139 DN
211Å：8179 DN
94Å：6099 DN