MultimodalBirds

MultimodalBirds: A Multimodal Image–Audio Dataset for Bird Species Classification. MultimodalBirds is a dataset for bird species classification and multimodal learning. It combines bird images from a curated subset of iNaturalist 2021 with bird vocalizations from the Xeno-Canto Bird Recordings Extended dataset by matching species information. The resulting dataset provides aligned image–audio pairs and supports research on multimodal fusion, modality imbalance, class imbalance, and model scalability. The dataset are provied in three versions for ease of use: unbalanced, regular and tiny. Citation If you use the MultimodalBirds dataset in your work, please cite it as: ```bibtex @misc{multimodalbirds2026,title={MultimodalBirds: A Multimodal Image–Audio Dataset for Bird Species Classification},author={Courtois, Jonathan and Pinguet, Jérémy and Meunier, Valentin and Chtioui, Fahd and Saulquin, Aurélie and Reyboz, Marina and Sassatelli, Gilles and Miramond, Benoît and Saighi, Sylvain and Boulet, Pierre and Castagnetti, Andrea},year={2026},url={https://zenodo.org/records/17633387}} ``` Dataset overview Data Sources Images: iNaturalist 2021 Birds Dataset Audio: Xeno-Canto Bird Recordings Extended (A-M) and Xeno-Canto Bird Recordings Extended (N-Z) All data are matched using the scientific species names. Dataset Versions The MultimodalBirds dataset contains 107,359 multimodal samples across 130 bird species. Each sample consists of a 128 × 128 × 3 RGB image paired with a 3-second audio recording of the same species. While the number of images per species is approximately 140 (from a fixed iNaturalist subset), the number of audio samples varies, reflecting natural imbalance in Xeno-Canto. Three dataset versions are provided (each is a subset of the previous): Unbalanced — 107,359 samples, 130 species; preserves real-world class and modality imbalance. Regular — 27,720 samples across 33 species; includes species with at least 140 images and 5 × 140 audio samples. Tiny — 2,800 samples across 10 species; 140 images and 140 audio samples per species. Dataset Processing The dataset was prepared with the following processing steps: Audio preprocessing: recordings were converted to mono and resampled to 16 kHz. Segmentation: audio recordings were split into 3-second clips with a 1.5-second hop (50% overlap). Validation: each clip was checked with BirdNET (birdnetlib v0.18.0); clips whose top prediction did not match the target species were discarded unless BirdNET confidence ≥ 0.5. Image preprocessing: images were standardized and limited to a maximum of 140 samples per species. Train/test split: 85% / 15% per species. Consecutive audio segments from the same recording are grouped and assigned entirely to either train or test to prevent data leakage. The code to construct this dataset from scratch is available at: https://bitbucket.org/edge-team-leat/multimodalbird/src/master/ Dataset Structure The MultimodalBirds dataset includes both **framework-agnostic** and **PyTorch-based loaders** to facilitate easy access and experiment setup. The dataset is organized in the following directory structure: ``` root │ ├── data/ Raw multimodal data │ │ │ └── <ebird_code>/ One folder per species │ │ │ ├── images/ Bird images (.jpg) │ │ ├── <image_id>.jpg │ │ └── ... │ │ │ └── sounds/ 3-second audio clips (.wav) │ ├── XC<recording_id>_<segment_id>.wav │ └── ... │ ├── metadata/ Dataset metadata and annotations │ │ │ ├── multimodal_bird_info.csv Main metadata file (all samples) │ └── label_mapping.csv Mapping between species names and labels │ ├── loaders/ Dataset loading utilities │ │ │ ├── data_loading.py Framework-agnostic dataset loader │ └── torch_loading.py PyTorch Dataset and DataLoader implementation │ ├── examples/ Usage examples │ │ │ ├── basic_example.py Example using framework-agnostic loader │ └── torch_example.py Example using PyTorch Dataset │ ├── pyproject.toml TOML file with dependencies │ └── README.md Dataset description file ``` Metadata description All the information of the dataset is stored in two CSV files. **label_mapping.csv** - **Total rows:** ~130 (one per species) - **Columns:** - `name`: Common name of the bird species - `ebird_code`: eBird species code (unique identifier) **multimodal_bird_info.csv** - **Total rows:** ~107,358 (one per audio subsample or image) - **Columns:** - `name`: Common name of the bird species (used for matching with the Images dataset) - `sci_name`: Scientific name of the bird species (used for matching with the Sounds dataset) - `rating`: Quality rating of the audio sample - `ebird_code`: eBird species code (unique identifier for ease of use) - `title`: Title of the audio sample or image sample - `channel`: Number of channel information (1 for sounds, 3 for RGB images) - `duration`: Duration of the audio subsample in seconds (3s) - `sampling_rate`: Sampling rate of the audio subsample (16000) - `time`: Timestamp of the origin audio sample (min:sec) - `bitrate_of_mp3`: Bitrate of the original audio file in bps - `filename`: Name of the audio file or image file - `file_type`: Type of the extension ('wav' for sound, 'jpg' for images) - `data_path`: Path to the sample - `base_filename`: Name of the origin audio file for this subsample - `BirdNet_class`: Classification result from BirdNet (used for matching with the Sounds dataset) - `BirdNet_score`: Confidence score from BirdNet (used for matching with the Sounds dataset) - `data_type`: Type of data ('sound' or 'image') - `index`: Index of the audio sample in its subgroup (used for the test/train leak avoidance) - `subgroup_id`: Identifier for the subgroup of the audio subsample within the original audio file (used for the test/train leak avoidance) - `for_test`: Indicator if the sample is for testing or training - `version`: Version of the dataset (1: unbalanced, 2: Regular, 3: Tiny) License This dataset is released under the CC BY 4.0 License. You are free to use, share, and adapt it as long as proper attribution is given.