2na-97/FAKER-Air

--- license: mit task_categories: - time-series-forecasting tags: - climate - code pretty_name: FAKER-Air size_categories: - 10B<n<100B --- # FAKER-Air Dataset This repository contains the dataset used in **FAKER-Air**, consisting of ground-truth air quality observations interpolated onto a grid and CMAQ reanalysis data tailored for East Asia. - **Paper**: [Real-Time Long Horizon Air Quality Forecasting via Group-Relative Policy Optimization](https://www.arxiv.org/abs/2511.22169) - **Code**: [GitHub Repository](https://github.com/kaist-cvml/FAKER-Air) ## Dataset Structure The data is organized into two main directories inside `data/`: ### 1. Observations (`data/obs`) Ground-truth station data interpolated onto the CMAQ 27km grid. - **Format**: `.npz` (Compressed NumPy archives) - **Naming**: `YYYYMMDDHH_obs.npz` (e.g., `2016010100_obs.npz`) - **Content**: Contains arrays for pollutant concentrations (PM2.5, PM10, etc.) on the grid. - **Total Files**: ~74,000 files (Hourly data from 2016 to 2023+). ### 2. CMAQ Reanalysis (`data/cmaq`) Physics-based model outputs (Community Multiscale Air Quality). - **Format**: `.npy` and `.json` - **Structure**: `YYYY/MM/DD/NIER_27_01/` - **Files**: - `*_x_conc.npy`: Concentration fields. - `*_x_metcro2d.npy`: 2D Meteorological fields. - `*_x_metcro3d.npy`: 3D Meteorological fields. - `*_meta.json`: Metadata. ## How to Use You can download specific parts of the dataset using the `huggingface_hub` Python library. ### Prerequisites ```bash pip install huggingface_hub numpy ```` ### Download & Load Example ```python from huggingface_hub import snapshot_download import numpy as np import os # 1. Download the dataset (It will cache data locally) # To download only specific years or folders, use `allow_patterns`. local_dir = snapshot_download( repo_id="2na-97/FAKER-Air", repo_type="dataset", allow_patterns=[ "data/obs/2023*.npz", # Example: Only download OBS for 2023 "data/cmaq/2023/**" # Example: Only download CMAQ for 2023 ] ) print(f"Data downloaded to: {local_dir}") # 2. Load an OBS file obs_path = os.path.join(local_dir, "data/obs/2023010100_obs.npz") if os.path.exists(obs_path): data = np.load(obs_path) print("Keys in OBS:", data.files) # Example access: data['pm25'] # 3. Load a CMAQ file cmaq_path = os.path.join(local_dir, "data/cmaq/2023/01/01/NIER_27_01/20230101_x_conc.npy") if os.path.exists(cmaq_path): cmaq_data = np.load(cmaq_path) print("CMAQ Shape:", cmaq_data.shape) ``` ## Citation ```bibtex @misc{kang2026realtimelonghorizonair, title={Real-Time Long Horizon Air Quality Forecasting via Group-Relative Policy Optimization}, author={Inha Kang and Eunki Kim and Wonjeong Ryu and Jaeyo Shin and Seungjun Yu and Yoon-Hee Kang and Seongeun Jeong and Eunhye Kim and Soontae Kim and Hyunjung Shim}, year={2026}, eprint={2511.22169}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2511.22169}, } ```许可证：MIT许可证 任务类别：时间序列预测 标签：气候、代码 美观名称：FAKER-Air 数据规模分类：100亿 < 数据量 < 1000亿 # FAKER-Air 数据集 本仓库包含**FAKER-Air**所使用的数据集，该数据集由插值至规则网格的实测空气质量观测数据，以及专为东亚定制的Community Multiscale Air Quality（CMAQ）再分析数据组成。 - **论文**：[《基于组相对策略优化的实时长时序空气质量预测》](https://www.arxiv.org/abs/2511.22169) - **代码**：[GitHub 仓库](https://github.com/kaist-cvml/FAKER-Air) ## 数据集结构 数据在`data/`目录下分为两个主要子目录： ### 1. 观测数据（`data/obs`） 插值至CMAQ 27km网格的实测台站数据。 - **格式**：`.npz`（压缩NumPy存档） - **命名规则**：`YYYYMMDDHH_obs.npz`（例如：`2016010100_obs.npz`） - **内容**：包含网格上的污染物浓度数组（PM2.5、PM10等）。 - **总文件数**：约74000个文件（2016年至2023年及以后的逐小时数据）。 ### 2. CMAQ再分析数据（`data/cmaq`） 基于物理过程的模式输出结果（Community Multiscale Air Quality，CMAQ）。 - **格式**：`.npy` 与 `.json` - **目录结构**：`YYYY/MM/DD/NIER_27_01/` - **文件说明**： - `*_x_conc.npy`：浓度场数据。 - `*_x_metcro2d.npy`：二维气象场数据。 - `*_x_metcro3d.npy`：三维气象场数据。 - `*_meta.json`：元数据。 ## 使用方法 您可以通过Hugging Face Hub的Python库（`huggingface_hub`）下载数据集的特定部分。 ### 前置依赖 bash pip install huggingface_hub numpy ### 下载与加载示例 python from huggingface_hub import snapshot_download import numpy as np import os # 1. 下载数据集（数据将本地缓存） # 若仅需下载特定年份或文件夹，可使用`allow_patterns`参数。 local_dir = snapshot_download( repo_id="2na-97/FAKER-Air", repo_type="dataset", allow_patterns=[ "data/obs/2023*.npz", # 示例：仅下载2023年的观测数据 "data/cmaq/2023/**" # 示例：仅下载2023年的CMAQ数据 ] ) print(f"数据已下载至：{local_dir}") # 2. 加载观测数据文件 obs_path = os.path.join(local_dir, "data/obs/2023010100_obs.npz") if os.path.exists(obs_path): data = np.load(obs_path) print("观测数据文件包含的键：", data.files) # 示例访问方式：data['pm25'] # 3. 加载CMAQ数据文件 cmaq_path = os.path.join(local_dir, "data/cmaq/2023/01/01/NIER_27_01/20230101_x_conc.npy") if os.path.exists(cmaq_path): cmaq_data = np.load(cmaq_path) print("CMAQ数据形状：", cmaq_data.shape) ## 引用 bibtex @misc{kang2026realtimelonghorizonair, title={Real-Time Long Horizon Air Quality Forecasting via Group-Relative Policy Optimization}, author={Inha Kang and Eunki Kim and Wonjeong Ryu and Jaeyo Shin and Seungjun Yu and Yoon-Hee Kang and Seongeun Jeong and Eunhye Kim and Soontae Kim and Hyunjung Shim}, year={2026}, eprint={2511.22169}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2511.22169}, }