2m Temperature Forecast by Deep Learning

This repository provides the preprocessed datasets, which are used in the study Temperature forecasting by deep learning methods by Gong et al. (2022). This allows the user to reproduce the presented results without running the preprocessing chain from the raw ERA5 data. Data description The datasets used to train, validate, and test the deep neural networks are based on the ERA5 reanalysis data provided by the European Centre for Medium-range Weather Forecast (ECMWF). Five different datasets have been created. All incorporate data between the years 2007 and 2019, but cover slightly varying domains over Central Europe and include different meteorological variables. The datasets are made available in compressed tar-archives (see Storage Location URL below). The file names thereby encapsulate some meta-information using the following naming convention: ERA5-Y[yyyy]-[yyyy]M[mm]to[mm]-[nx]x[ny]-[nn.nn]N[ee.ee]E-[var1]_[var2]_[var3] where - Y[yyyy]-[yyyy]M[mm]to[mm] denotes the years and the months describing the data period, -[nx]x[ny] is the number of grid points/pixels of the target domain in longitude and latitude direction, -[nn.nn]N[ee.ee]E stands for the geographical coordinates in degree of the target domain's south-west corner and -[var1]_[var2]_[var3] denote the short names of the variables according to ECMWF's parameter database In particular, the following datasets are provided: 1) era5-Y2007-2019M01to12-92x56-3840N0000E-2t_tcc_t850.tar.bz2: The target domain extends from 38.4°N to 54.9°N and 0.0°E to 27.3°E (92x56 grid points). The 2m-temperature (2t), the total cloud cover (tcc), and the 850 hPa temperature (t_850) are included as variables. This data corresponds to Datasets ID 1-3 in table A1 of the manuscript. 2) era5-Y2007-2019M01to12-80x48-3960N0180E-2t_tcc_t850.tar.bz2: The target domain extends from 39.6°N to 53.7°N and 1.8°E to 25.5°E (80x48 grid points). The 2t, tcc, and the t_850 are included as variables. This data corresponds to Dataset ID 4 in table A1 of the manuscript. 3) era5-Y2007-2019M01to12-72x44-4020N0300E-2t_tcc_t_850.tar.bz2: The target domain extends from 40.2°N to 53.1°N and 3.0°E to 24.3°E (72x44 grid points). The 2t, tcc, and t_850 are included as variables. This data corresponds to Dataset ID 5 in table A1 of the manuscript. 4) era5-Y2007-2019M01to12-80x48-3960N0180E-2t_t850.tar.bz2: The target domain extends from 39.6°N to 53.7°N and 1.8°E to 25.5°E (80x48 grid points). The 2t and the t_850 are the only variables included. This data set is actually a subset of No. 2. This data corresponds to Dataset ID 6 in table A1 of the manuscript. 5) era5-Y2007-2019M01to12-80x48-3960N0180E-2t.tar.bz2: The target domain extends from 39.6°N to 53.7°N and 1.8°E to 25.5°E (80x48 grid points). 2t is exclusively included. This data set is also a subset of No. 2. This data corresponds to Dataset ID 7 in table A1 of the manuscript. Data creation The original ERA5 data can be retrieved from the (MARS archive). Once access is granted, data can be downloaded by specifying a resolution of 0.3° in the retrieval script. The datasets provided in this repository are the processed ERA5 data after the extraction and the two preprocessing steps using the Atmospheric Machine learning Benchmarking System (AMBS) workflow tool (more details are provided in the README of the corresponding code repository). The data is available in TFRecords format that is used directly in the training step. Data access and decompression Data are stored in the archived and compressed format tar.bz2 and available via: https://datapub.fz-juelich.de/esde/esde-nfs/online_publication/2mT_by_DL/. After downloading, the compressed archives can be unpacked on Linux using tar xjf [filename].tar.bz2. On Windows, decompressing can be performed using WinZip. Dataset content After decompressing, the following subdirectory structure is created from each compressed tar-archive: - tfrecords_seq_len_[sequence_length]: This folder holds the TFRecords files that are streamed to the deep neural networks during training and postprocessing. Each TFRecord file contains 10 samples, where each sample comprises a sequence over [sequence_length] hours. - pickle: This folder contains the normalized hourly data saved in monthly pickle files (X_[month].pkl). The corresponding timestamps are included in T_[month].pkl. Furthermore, statistical information for each month is provided in the files stat_[month].json. - metadata.json: This file provides important meta information including the coordinates of the target domain, the included variables (e.g. 2t and t_850) and the origin of the processed data. - statsitic.json: This file includes the statistical information (maximum, minimum, and average values) used for normalizing the data. It also includes other information such as the total number of the timestamps (nfiles) and the list of JSON files (stat_[month].json) to compute the statistics. Data integrity and verification The tar-archives have been recursively checksummed with the md5 hash function. The generated file is uploaded to ensure the integrity of the files and no alteration to the dataset. To verify the integrity of the downloaded data, use the following snippet: find -type f -exec md5sum '{}' \; > md5sum.txt It will generate a single text file that should be identical to the file in this entry. License Original data by ECMWF Copyright "© 2022 European Centre for Medium-Range Weather Forecasts (ECMWF)". Source www.ecmwf.int. This data is published under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. https://creativecommons.org/licenses/by/4.0/ Contact Bing Gong ([e-mail])

本仓库提供了经预处理的数据集，用于Gong等人2022年发表的《基于深度学习方法的气温预报》研究。使用者无需从原始ERA5（ERA5）数据开始运行预处理流程，即可复现论文中展示的结果。

### 数据说明
用于训练、验证与测试深度神经网络的数据集，基于欧洲中期天气预报中心（European Centre for Medium-range Weather Forecasts, ECMWF）提供的ERA5再分析数据构建。本次共生成5组不同的数据集，所有数据集均覆盖2007年至2019年的时段，但针对中欧地区的研究区域略有差异，且包含的气象变量各不相同。

数据集以压缩tar归档文件的形式提供（详见下方存储位置链接），文件名遵循以下命名约定以封装部分元信息：
`ERA5-Y[yyyy]-[yyyy]M[mm]to[mm]-[nx]x[ny]-[nn.nn]N[ee.ee]E-[var1]_[var2]_[var3]`
其中：
- `Y[yyyy]-[yyyy]M[mm]to[mm]` 表示数据涵盖的年份与月份范围；
- `[nx]x[ny]` 代表目标区域在经度与纬度方向上的网格点数/像素数；
- `[nn.nn]N[ee.ee]E` 表示目标区域西南角的地理坐标（单位：度）；
- `[var1]_[var2]_[var3]` 为ECMWF参数数据库中对应的变量简称。

#### 具体提供的数据集
1. `era5-Y2007-2019M01to12-92x56-3840N0000E-2t_tcc_t850.tar.bz2`：目标区域范围为38.4°N至54.9°N、0.0°E至27.3°E，包含92×56个网格点。变量包含2米气温（2t）、总云量（tcc）以及850百帕气温（t_850）。该数据集对应论文附录表A1中的数据集ID 1至3。
2. `era5-Y2007-2019M01to12-80x48-3960N0180E-2t_tcc_t850.tar.bz2`：目标区域范围为39.6°N至53.7°N、1.8°E至25.5°E，包含80×48个网格点。变量包含2t、tcc以及t_850。该数据集对应论文附录表A1中的数据集ID 4。
3. `era5-Y2007-2019M01to12-72x44-4020N0300E-2t_tcc_t_850.tar.bz2`：目标区域范围为40.2°N至53.1°N、3.0°E至24.3°E，包含72×44个网格点。变量包含2t、tcc以及t_850。该数据集对应论文附录表A1中的数据集ID 5。
4. `era5-Y2007-2019M01to12-80x48-3960N0180E-2t_t850.tar.bz2`：目标区域范围为39.6°N至53.7°N、1.8°E至25.5°E，包含80×48个网格点。仅包含2t与t_850两个变量。该数据集实际为第2组数据集的子集，对应论文附录表A1中的数据集ID 6。
5. `era5-Y2007-2019M01to12-80x48-3960N0180E-2t.tar.bz2`：目标区域范围为39.6°N至53.7°N、1.8°E至25.5°E，包含80×48个网格点。仅包含2t变量。该数据集同样为第2组数据集的子集，对应论文附录表A1中的数据集ID 7。

### 数据创建
原始ERA5数据可从MARS存档获取。获得访问权限后，可在检索脚本中指定0.3°的分辨率进行数据下载。本仓库提供的数据集为经提取与两步预处理后的ERA5数据，预处理流程基于大气机器学习基准测试系统（Atmospheric Machine learning Benchmarking System, AMBS）工作流工具实现（更多细节可参见对应代码仓库的README文件）。数据以TFRecords（TFRecords）格式存储，可直接用于深度学习模型的训练。

### 数据获取与解压
数据以tar.bz2压缩归档格式存储，可通过以下链接获取：https://datapub.fz-juelich.de/esde/esde-nfs/online_publication/2mT_by_DL/
下载完成后，Linux系统可使用`tar xjf [filename].tar.bz2`命令解压归档文件；Windows系统可使用WinZip工具完成解压。

### 数据集内容
解压后，每个压缩归档将生成如下目录结构：
- `tfrecords_seq_len_[sequence_length]`：该文件夹存储训练与后处理阶段输入至深度神经网络的TFRecords文件。每个TFRecord文件包含10个样本，每个样本对应一段时长为`[sequence_length]`小时的序列数据。
- `pickle`：该文件夹存储按月份划分的归一化逐小时数据，保存为`X_[month].pkl`文件，对应的时间戳存储于`T_[month].pkl`文件中。此外，每个月份的统计信息存储于`stat_[month].json`文件中。
- `metadata.json`：该文件提供重要元信息，包括目标区域的坐标、包含的变量（如2t与t_850）以及预处理后数据的来源。
- `statistic.json`：该文件包含用于数据归一化的统计信息（最大值、最小值与平均值），同时包含总时间戳数量（nfiles）以及用于计算统计量的JSON文件列表（即`stat_[month].json`）等其他信息。

### 数据完整性与验证
所有tar归档文件均通过md5哈希函数进行递归校验，并生成校验文件以确保文件完整性与数据集未被篡改。验证下载数据完整性的命令如下：
`find -type f -exec md5sum '{}' ; > md5sum.txt`
该命令将生成一个文本文件，其内容应与本仓库中对应的校验文件完全一致。

### 许可证
原始ERA5数据的版权归"© 2022 European Centre for Medium-Range Weather Forecasts (ECMWF)"所有，来源网址为www.ecmwf.int。本数据集采用知识共享署名4.0国际许可协议（Creative Commons Attribution 4.0 International, CC BY 4.0）发布，许可协议链接为https://creativecommons.org/licenses/by/4.0/

### 联系方式
Bing Gong ([e-mail])