SimonSongHit/fengyun4A-cloud-detection-dataset

---
language: en
license: mit
tags:
- Cloud dection
- Fengyun-4A
pretty_name: Fengyun-4A cloud detection dataset
---

<!-- This file is uploaded as the Hugging Face dataset README; keep content user-facing (no internal maintainer or upload-only notes). -->

# Fengyun-4A cloud detection dataset

The cloud detection dataset for Fengyun-4A (FY-4A) geostationary satellite supporting both physical and spatio-temporal data-driven algorithms.

## Dataset description

Each folder contains cloud detection samples collected during the time period (UTC) indicated by the folder name, and includes the following three types of files:

- **`yyyymmddHHMM_yyyymmddHHMM.hdf5`** provides the index, latitude, longitude, land-surface type, 3×21×3×3 (time×feature×latitude×longitude) input matrices, and label (1 for cloudy and 0 for cloud-free) for each target FY-4A pixel.

- **`yyyymmddHHMM_yyyymmddHHMM_Atmos_i.dat`** provides vertical profiles of water vapor, temperature, and ozone; near-surface meteorological variables (2-m temperature, 2-m water vapor, 2-m pressure, and 10-m wind speed); surface properties (land-surface temperature, albedo, and emissivity); as well as elevation, latitude, longitude, satellite zenith, satellite azimuth, solar zenith, and solar azimuth angles for the i-th pixel (i=1,…,9) in each pixel block at the middle time instance.

- **`yyyymmddHHMM_yyyymmddHHMM_Aerosol_i.dat`** provides vertical profiles of nine aerosol species for the i-th pixel (i=1,…,9) in each pixel block at the middle time instance.

### File structure

Each folder is named `yyyymmddHHMM_yyyymmddHHMM` and contains 19 files.

```
{yyyymmddHHMM_yyyymmddHHMM}/
{yyyymmddHHMM_yyyymmddHHMM}.hdf5
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_1.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_2.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_3.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_4.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_5.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_6.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_7.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_8.dat
{yyyymmddHHMM_yyyymmddHHMM}_Atmos_9.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_1.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_2.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_3.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_4.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_5.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_6.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_7.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_8.dat
{yyyymmddHHMM_yyyymmddHHMM}_Aerosol_9.dat
```

Example:

- `201901010500_201901010514/`<br>
`201901010500_201901010514_Atmos_1.dat`<br>
`201901010500_201901010514_Atmos_2.dat`<br>
`201901010500_201901010514_Atmos_3.dat`<br>
`201901010500_201901010514_Atmos_4.dat`<br>
`201901010500_201901010514_Atmos_5.dat`<br>
`201901010500_201901010514_Atmos_6.dat`<br>
`201901010500_201901010514_Atmos_7.dat`<br>
`201901010500_201901010514_Atmos_8.dat`<br>
`201901010500_201901010514_Atmos_9.dat`<br>
`201901010500_201901010514_Aerosol_1.dat`<br>
`201901010500_201901010514_Aerosol_2.dat`<br>
`201901010500_201901010514_Aerosol_3.dat`<br>
`201901010500_201901010514_Aerosol_4.dat`<br>
`201901010500_201901010514_Aerosol_5.dat`<br>
`201901010500_201901010514_Aerosol_6.dat`<br>
`201901010500_201901010514_Aerosol_7.dat`<br>
`201901010500_201901010514_Aerosol_8.dat`<br>
`201901010500_201901010514_Aerosol_9.dat`<br>

## Variables
### Variables in `yyyymmddHHMM_yyyymmddHHMM.hdf5`

| Variable | Description | Type | Units |
|----------|-------------|------|-------|
| Index | Index of each target FY-4A pixel | Integer | — |
| Latitude | Latitude of each matched FY-4A pixel | Float | ° |
| Longitude | Longitude of each matched FY-4A pixel | Float | ° |
| Surface type | Land-surface type of each matched FY-4A pixel | String | — |
| Input | Satellite data for each pixel block (3×21×3×3) | Float | — |
| Label | Label of the target pixel for each pixel block | Integer | — |

### Features for each input matrix
| Feature | Description | Units |
|---------|-------------|-------|
| 1 | FY-4A AGRI Band 1 (0.45-0.49 μm) | — |
| 2 | FY-4A AGRI Band 2 (0.55-0.75 μm) | — |
| 3 | FY-4A AGRI Band 3 (0.75-0.90 μm) | — |
| 4 | FY-4A AGRI Band 4 (1.36-1.39 μm) | — |
| 5 | FY-4A AGRI Band 5 (1.58-1.64 μm) | — |
| 6 | FY-4A AGRI Band 6 (2.10-2.35 μm) | — |
| 7 | FY-4A AGRI Band 8 (3.50-4.00 μm) | K |
| 8 | FY-4A AGRI Band 9 (5.80-6.70 μm) | K |
| 9 | FY-4A AGRI Band 10 (6.90-7.30 μm) | K |
| 10 | FY-4A AGRI Band 11 (8.00-9.00 μm) | K |
| 11 | FY-4A AGRI Band 12 (10.3-11.3 μm) | K |
| 12 | FY-4A AGRI Band 13 (11.5-12.5 μm) | K |
| 13 | FY-4A AGRI Band 14 (13.2-13.8 μm) | K |
| 14 | Solar azimuth angle | ° |
| 15 | Solar zenith angle | ° |
| 16 | Solar glint angle | ° |
| 17 | Satellite azimuth angle | ° |
| 18 | Satellite zenith angle | ° |
| 19 | Surface elevation | m |
| 20 | Pixel longitude | ° |
| 21 | Pixel latitude | ° |

### Variables in `yyyymmddHHMM_yyyymmddHHMM_Atmos_i.dat`

| Variable | Description | Data type | Unit |
|----------|-------------|-----------|------|
| Index | Index of each tatget FY-4A pixel | Integer | -- |
| Pressure levels | Pressures of all layers for the i-th pixel in each pixel block | Float | hPa |
| Temperature profile | Temperature of all layers for the i-th pixel in each pixel block | Float | K |
| Water vapour profile | Water vapour concentration of all layers for the i-th pixel in each pixel block | Float | kg/kg |
| Ozone profile | Ozone concentration of all layers for the i-th pixel in each pixel block | Float | kg/kg |
| 2-m temperature | 2-m temperature of the i-th pixel in each pixel block | Float | K |
| 2-m water vapour | 2-m water vapour concentration of the i-th pixel in each pixel block | Float | kg/kg |
| 2-m pressure | 2-m pressure of the i-th pixel in each pixel block | Float | hPa |
| 10-m wind u | 10-m eastward wind speed of the i-th pixel in each pixel block | Float | m/s |
| 10-m wind v | 10-m northward wind speed of the i-th pixel in each pixel block | Float | m/s |
| Skin temperature | Land-surface temperature of the i-th pixel in each pixel block | Float | K |
| Elevation | Elevation of the i-th pixel in each pixel block | Float | km |
| Latitude | Latitude of the i-th pixel in each pixel block | Float | ° |
| Longitude | Longitude of the i-th pixel in each pixel block | Float | ° |
| Satellite zenith | Satellite zenith angle of the i-th pixel in each pixel block | Float | ° |
| Satellite azimuth | Satellite azimuth angle of the i-th pixel in each pixel block | Float | ° |
| Sun zenith | Sun zenith angle of the i-th pixel in each pixel block | Float | ° |
| Sun azimuth | Sun azimuth angle of the i-th pixel in each pixel block | Float | ° |
| Albedo1 | Land-surface albedo of the i-th pixel in each pixel block for FY-4A band 1 | Float | — |
| Albedo2 | Land-surface albedo of the i-th pixel in each pixel block for FY-4A band 2 | Float | — |
| Albedo3 | Land-surface albedo of the i-th pixel in each pixel block for FY-4A band 3 | Float | — |
| Albedo4 | Land-surface albedo of the i-th pixel in each pixel block for FY-4A band 4 | Float | — |
| Albedo5 | Land-surface albedo of the i-th pixel in each pixel block for FY-4A band 5 | Float | — |
| Albedo6 | Land-surface albedo of the i-th pixel in each pixel block for FY-4A band 6 | Float | — |
| Emissivity8 | Land-surface emissivity of the i-th pixel in each pixel block for FY-4A band 8 | Float | — |
| Emissivity11 | Land-surface emissivity of the i-th pixel in each pixel block for FY-4A band 11 | Float | — |
| Emissivity12 | Land-surface emissivity of the i-th pixel in each pixel block for FY-4A band 12 | Float | — |
| Emissivity13 | Land-surface emissivity of the i-th pixel in each pixel block for FY-4A band 13 | Float | — |

### Variables in `yyyymmddHHMM_yyyymmddHHMM_Aerosol_i.dat`

| Variable | Description | Data type | Unit |
|----------|-------------|-----------|------|
| Index | Index of each matched FY-4A pixel | Integer | kg/kg |
| BCAR | Black carbon aerosol mixing ratio | Float | kg/kg |
| DUS1 | Dust aerosol (0.03—0.55 μm) mixing ratio | Float | kg/kg |
| DUS2 | Dust aerosol (0.55—0.90 μm) mixing ratio | Float | kg/kg |
| DUS3 | Dust aerosol (0.90—20.0 μm) mixing ratio | Float | kg/kg |
| SULP | Sulphate aerosol mixing ratio | Float | kg/kg |
| SSA1 | Sea salt aerosol (0.03—0.50 μm) mixing ratio | Float | kg/kg |
| SSA2 | Sea salt aerosol (0.50—5.00 μm) mixing ratio | Float | kg/kg |
| SSA3 | Sea salt aerosol (5.00—20.0 μm) mixing ratio | Float | kg/kg |
| OMAT | Hydrophilic organic matter aerosol mixing ratio | Float | kg/kg |

## Usage

### Load from Hugging Face

```python
from huggingface_hub import snapshot_download

REPO_ID = "SimonSongHit/fengyun4A-cloud-detection-dataset"
cache_path = "G:/hf_cache" # Replace it with your cache path

print("Downloading dataset...")

root_dir = snapshot_download(
repo_id=REPO_ID,
repo_type="dataset",
cache_dir=cache_path
)

print("Data path:", root_dir)
```
### Use with TabPFN

A detailed usage example is provided in the notebook Dataset_download_and_usage_example.ipynb.

## Data Sources
This dataset integrates multiple satellite and reanalysis data sources:

- **FY-4A satellite data**: [National Satellite Meteorological Center (NSMC)](http://www.nsmc.org.cn/nsmc/cn/home/)
- **MODIS land-surface products (MCD12C1, MYD10C1, MCD43A3, MOD11B1)**: [NASA Earthdata Search](https://search.earthdata.nasa.gov/)
- **CALIPSO cloud-layer product**: [NASA Earthdata Search](https://search.earthdata.nasa.gov/)
- **ERA5 reanalysis data**: [Copernicus Climate Data Store](https://cds.climate.copernicus.eu/datasets/)
- **EAC4 reanalysis data**:[Copernicus Atmosphere Data Store](https://ads.atmosphere.copernicus.eu/datasets/)

## License
- MODIS and CALIPSO data are provided by :contentReference[oaicite:0]{index=0} and are subject to their [Data Usage Policy](https://www.earthdata.nasa.gov/engage/open-data-services-software/end-user-license-agreements).
- ERA5 data are provided by :contentReference[oaicite:1]{index=1} and are subject to their [Data Usage Policy](https://cds.climate.copernicus.eu/disclaimer-privacy).
- EAC4 data are provided by :contentReference[oaicite:2]{index=2} and are subject to their [Data Usage Policy](https://ads.atmosphere.copernicus.eu/disclaimer-privacy).