Atmopheric Machine Learning Emulation Challenge, 1st Ed. (AMLEC-1)

Note: Unzip files to have access to the original data files (.h5, .csv, .xml) This dataset contains the train and test data used for the first edition of the Atmospheric Machine Learning Emulation Challenge (AMLEC-1), presented at ECMLPKDD 2025 (https://ecmlpkdd.org/2025/discovery-challenges/) and carried out within the EU ELIAS project (https://elias-ai.eu/opportunities/amlec/). The dataset contains a series of .h5 storing MODTRAN6 spectral simulations (transmittances, spherical albedo, path radiance), computed at a with various atmospheric and geometric conditions and two scenarios --atmospheric correction of hyperspectral data (A) and CO2 retrieval (B)-- each associated with its own spectral configuration. The training data (i.e., inputs and outputs of RTM simulations) is stored in HDF5 format with the following structure: Dimensions Name Description n_wl Number of wavelengths for which spectral data is provided n_funcs Number of atmospheric transfer functions n_comb Number of data points at which spectral data is provided n_param Dimensionality of the input variable space Data Components Name Description Dimensions Datatype LUTdata Atmospheric transfer functions (i.e. outputs) n_funcs*n_wvl x n_comb single LUTHeader Matrix of input variable values for each combination (i.e., inputs) n_param x n_comb double wvl Wavelength values associated with the atmospheric transfer functions (i.e., spectral grid) n_wvl double Note: Participants may choose to predict the spectral data either as a single vector of length n_funcs*n_wvl or as n_funcs separate vectors of lenght n_wvl. Testing input datasets (i.e., input for predictions) are stored in a tabulated .csv format with dimensions n_param x n_comb. During the challenge, participants only had access to this .csv data, while here we also provide the reference spectral simulations using for evaluation The training and testing dataset will be organized organized into scenario-specific folders: scenarioA (Atmospheric Correction), and scenarioB (CO2 Column Retrieval). Each folder will contain: A train with multiple .h5 files corresponding to different training sample sizes (e.g. train2000.h5contains 2000 samples). A reference subfolder containg two test files (refInterp and refExtrap) referring to the two aforementioned tracks (i.e., interpolation and extrapolation). Here is an example of how to load each dataset in python: import h5py import pandas as pd import numpy as np # Replace with the actual path to your training and testing data trainFile = 'train2000.h5' testFile = 'refInterp.csv' # Open the H5 file with h5py.File(file_path, 'r') as h5_file Ytrain = h5_file['LUTdata'][:] Xtrain = h5_file['LUTHeader'][:] wvl = h5_file['wvl'][:] # Read testing data df = pd.read_csv(testFile) Xtest = df.to_numpy() in Matlab: # Replace with the actual path to your training and testing data trainFile = 'train2000.h5'; testFile = 'refInterp.csv'; # Open the H5 file Ytrain = h5read(trainFile,'/LUTdata'); Xtrain = h5read(trainFile,'/LUTheader'); wvl = h5read(trainFile,'/wvl'); # Read testing data Xtest = importdata(testFile); and in R language: library(rhdf5) # Replace with the actual path to your training and testing data trainFile <- "train2000.h5" testFile <- "refInterp.csv" # Open the H5 file lut_data <- h5read(file_path, "LUTdata") lut_header <- h5read(file_path, "LUTHeader") wavelengths <- h5read(file_path, "wvl") # Read testing data Xtest <- as.matrix(read.table(file_path, sep = ",", header = TRUE))