MultimodalBirds

MultimodalBirds: A Multimodal Image–Audio Dataset for Bird Species Classification. MultimodalBirds is a dataset for bird species classification and multimodal learning. It combines bird images from a curated subset of iNaturalist 2021 with bird vocalizations from the Xeno-Canto Bird Recordings Extended dataset by matching species information. The resulting dataset provides aligned image–audio pairs and supports research on multimodal fusion, modality imbalance, class imbalance, and model scalability. The dataset are provied in three versions for ease of use: unbalanced, regular and tiny. Citation If you use the MultimodalBirds dataset in your work, please cite it as: ```bibtex @misc{multimodalbirds2026,title={MultimodalBirds: A Multimodal Image–Audio Dataset for Bird Species Classification},author={Courtois, Jonathan and Pinguet, Jérémy and Meunier, Valentin and Chtioui, Fahd and Saulquin, Aurélie and Reyboz, Marina and Sassatelli, Gilles and Miramond, Benoît and Saighi, Sylvain and Boulet, Pierre and Castagnetti, Andrea},year={2026},url={https://zenodo.org/records/17633387}} ``` Dataset overview Data Sources Images: iNaturalist 2021 Birds Dataset Audio: Xeno-Canto Bird Recordings Extended (A-M) and Xeno-Canto Bird Recordings Extended (N-Z) All data are matched using the scientific species names. Dataset Versions The MultimodalBirds dataset contains 107,359 multimodal samples across 130 bird species. Each sample consists of a 128 × 128 × 3 RGB image paired with a 3-second audio recording of the same species. While the number of images per species is approximately 140 (from a fixed iNaturalist subset), the number of audio samples varies, reflecting natural imbalance in Xeno-Canto. Three dataset versions are provided (each is a subset of the previous): Unbalanced — 107,359 samples, 130 species; preserves real-world class and modality imbalance. Regular — 27,720 samples across 33 species; includes species with at least 140 images and 5 × 140 audio samples. Tiny — 2,800 samples across 10 species; 140 images and 140 audio samples per species. Dataset Processing The dataset was prepared with the following processing steps: Audio preprocessing: recordings were converted to mono and resampled to 16 kHz. Segmentation: audio recordings were split into 3-second clips with a 1.5-second hop (50% overlap). Validation: each clip was checked with BirdNET (birdnetlib v0.18.0); clips whose top prediction did not match the target species were discarded unless BirdNET confidence ≥ 0.5. Image preprocessing: images were standardized and limited to a maximum of 140 samples per species. Train/test split: 85% / 15% per species. Consecutive audio segments from the same recording are grouped and assigned entirely to either train or test to prevent data leakage. The code to construct this dataset from scratch is available at: https://bitbucket.org/edge-team-leat/multimodalbird/src/master/ Dataset Structure The MultimodalBirds dataset includes both **framework-agnostic** and **PyTorch-based loaders** to facilitate easy access and experiment setup. The dataset is organized in the following directory structure: ``` root │ ├── data/ Raw multimodal data │ │ │ └── <ebird_code>/ One folder per species │ │ │ ├── images/ Bird images (.jpg) │ │ ├── <image_id>.jpg │ │ └── ... │ │ │ └── sounds/ 3-second audio clips (.wav) │ ├── XC<recording_id>_<segment_id>.wav │ └── ... │ ├── metadata/ Dataset metadata and annotations │ │ │ ├── multimodal_bird_info.csv Main metadata file (all samples) │ └── label_mapping.csv Mapping between species names and labels │ ├── loaders/ Dataset loading utilities │ │ │ ├── data_loading.py Framework-agnostic dataset loader │ └── torch_loading.py PyTorch Dataset and DataLoader implementation │ ├── examples/ Usage examples │ │ │ ├── basic_example.py Example using framework-agnostic loader │ └── torch_example.py Example using PyTorch Dataset │ ├── pyproject.toml TOML file with dependencies │ └── README.md Dataset description file ``` Metadata description All the information of the dataset is stored in two CSV files. **label_mapping.csv** - **Total rows:** ~130 (one per species) - **Columns:** - `name`: Common name of the bird species - `ebird_code`: eBird species code (unique identifier) **multimodal_bird_info.csv** - **Total rows:** ~107,358 (one per audio subsample or image) - **Columns:** - `name`: Common name of the bird species (used for matching with the Images dataset) - `sci_name`: Scientific name of the bird species (used for matching with the Sounds dataset) - `rating`: Quality rating of the audio sample - `ebird_code`: eBird species code (unique identifier for ease of use) - `title`: Title of the audio sample or image sample - `channel`: Number of channel information (1 for sounds, 3 for RGB images) - `duration`: Duration of the audio subsample in seconds (3s) - `sampling_rate`: Sampling rate of the audio subsample (16000) - `time`: Timestamp of the origin audio sample (min:sec) - `bitrate_of_mp3`: Bitrate of the original audio file in bps - `filename`: Name of the audio file or image file - `file_type`: Type of the extension ('wav' for sound, 'jpg' for images) - `data_path`: Path to the sample - `base_filename`: Name of the origin audio file for this subsample - `BirdNet_class`: Classification result from BirdNet (used for matching with the Sounds dataset) - `BirdNet_score`: Confidence score from BirdNet (used for matching with the Sounds dataset) - `data_type`: Type of data ('sound' or 'image') - `index`: Index of the audio sample in its subgroup (used for the test/train leak avoidance) - `subgroup_id`: Identifier for the subgroup of the audio subsample within the original audio file (used for the test/train leak avoidance) - `for_test`: Indicator if the sample is for testing or training - `version`: Version of the dataset (1: unbalanced, 2: Regular, 3: Tiny) License This dataset is released under the CC BY 4.0 License. You are free to use, share, and adapt it as long as proper attribution is given.MultimodalBirds：面向鸟类物种分类的多模态图像-音频数据集 MultimodalBirds是一款用于鸟类物种分类与多模态学习的数据集。它通过物种信息匹配，将精选自**iNaturalist 2021 (iNaturalist 2021)** 的鸟类图像子集，与**Xeno-Canto鸟类录音扩展数据集**的鸟类鸣叫声相结合。 该数据集提供对齐后的图像-音频配对样本，可支撑多模态融合、模态不平衡、类别不平衡以及模型可扩展性等方向的研究。为便于使用，数据集提供三个版本：不平衡版、标准版与微型版。 ## 引用 若您在研究工作中使用MultimodalBirds数据集，请按以下格式引用： bibtex @misc{multimodalbirds2026,title={MultimodalBirds: A Multimodal Image–Audio Dataset for Bird Species Classification},author={Courtois, Jonathan and Pinguet, Jérémy and Meunier, Valentin and Chtioui, Fahd and Saulquin, Aurélie and Reyboz, Marina and Sassatelli, Gilles and Miramond, Benoît and Saighi, Sylvain and Boulet, Pierre and Castagnetti, Andrea},year={2026},url={https://zenodo.org/records/17633387}} ## 数据集概览 ### 数据来源 - 图像数据：**iNaturalist 2021鸟类数据集** - 音频数据：**Xeno-Canto鸟类录音扩展数据集（A-M）** 与 **Xeno-Canto鸟类录音扩展数据集（N-Z）** 所有数据通过科学物种名称进行匹配对齐。 ### 数据集版本 MultimodalBirds数据集共包含107,359个多模态样本，涵盖130个鸟类物种。每个样本由一张128×128×3的RGB图像，以及对应物种的3秒音频录音组成。尽管每个物种的图像数量约为140（取自固定的iNaturalist子集），但音频样本数量存在差异，这反映了Xeno-Canto数据集中的天然不平衡特性。 本次提供的三个数据集版本均为前一版本的子集： 1. **不平衡版**：包含107,359个样本，覆盖130个物种，保留了真实场景下的类别与模态不平衡特性。 2. **标准版**：包含27,720个样本，覆盖33个物种，仅包含至少拥有140张图像与5×140个音频样本的物种。 3. **微型版**：包含2,800个样本，覆盖10个物种，每个物种均提供140张图像与140个音频样本。 ### 数据集处理流程 本数据集的构建经过以下处理步骤： 1. **音频预处理**：将原始录音转换为单声道，并重采样至16kHz。 2. **分段处理**：将音频录音切割为3秒的片段，步长为1.5秒（重叠率50%）。 3. **有效性验证**：使用**BirdNET (birdnetlib v0.18.0)** 对每个音频片段进行检测，若其Top预测结果与目标物种不匹配，则丢弃该片段，除非BirdNET的置信度≥0.5。 4. **图像预处理**：对图像进行标准化处理，并将每个物种的样本数量上限设置为140。 5. **训练/测试划分**：按每个物种85%训练集、15%测试集的比例进行划分。同一原始录音的连续音频片段将被分为一组，整体分配至训练集或测试集，以避免数据泄露。 可从以下地址获取从零构建该数据集的代码：https://bitbucket.org/edge-team-leat/multimodalbird/src/master/ ### 数据集组织结构 MultimodalBirds数据集同时提供**与框架无关的加载工具**与**基于PyTorch的加载器**，以简化数据访问与实验搭建流程。数据集的目录结构如下： root │ ├── data/ 原始多模态数据 │ │ │ └── <ebird_code>/ 每个物种对应一个文件夹 │ │ │ ├── images/ 鸟类图像（.jpg格式） │ │ ├── <image_id>.jpg │ │ └── ... │ │ │ └── sounds/ 3秒音频片段（.wav格式） │ ├── XC<recording_id>_<segment_id>.wav │ └── ... │ ├── metadata/ 数据集元数据与标注信息 │ │ │ ├── multimodal_bird_info.csv 主元数据文件（包含所有样本） │ └── label_mapping.csv 物种名称与标签的映射表 │ ├── loaders/ 数据集加载工具 │ │ │ ├── data_loading.py 与框架无关的数据集加载器 │ └── torch_loading.py PyTorch数据集与数据加载器实现 │ ├── examples/ 使用示例 │ │ │ ├── basic_example.py 使用框架无关加载器的示例代码 │ └── torch_example.py 使用PyTorch数据集的示例代码 │ ├── pyproject.toml 包含依赖项的TOML文件 │ └── README.md 数据集说明文件 ### 元数据说明 数据集的所有信息存储于两个CSV文件中： #### label_mapping.csv - **总行数**：约130行，每行对应一个鸟类物种 - **字段说明**： - `name`：鸟类物种的通用名称 - `ebird_code`：eBird物种代码（唯一标识符） #### multimodal_bird_info.csv - **总行数**：约107,358行，每行对应一个音频子样本或图像样本 - **字段说明**： - `name`：鸟类物种的通用名称（用于与图像数据集匹配） - `sci_name`：鸟类物种的科学名称（用于与音频数据集匹配） - `rating`：音频样本的质量评分 - `ebird_code`：eBird物种代码（便于使用的唯一标识符） - `title`：音频样本或图像样本的标题 - `channel`：通道数信息（音频为1，RGB图像为3） - `duration`：音频子样本的时长（单位：秒，固定为3s） - `sampling_rate`：音频子样本的采样率（固定为16000） - `time`：原始音频样本的时间戳（格式：分:秒） - `bitrate_of_mp3`：原始音频文件的比特率（单位：bps） - `filename`：音频文件或图像文件的名称 - `file_type`：文件扩展名类型（音频为`wav`，图像为`jpg`） - `data_path`：样本的存储路径 - `base_filename`：该音频子样本对应的原始音频文件名 - `BirdNet_class`：BirdNET的分类结果（用于与音频数据集匹配） - `BirdNet_score`：BirdNET输出的置信度得分（用于与音频数据集匹配） - `data_type`：数据类型（`sound`或`image`） - `index`：该音频样本在其子组内的索引（用于避免训练测试数据泄露） - `subgroup_id`：原始音频文件内音频子样本的子组标识符（用于避免训练测试数据泄露） - `for_test`：样本用于测试集还是训练集的标识 - `version`：数据集版本（1：不平衡版，2：标准版，3：微型版） ## 许可证 本数据集采用CC BY 4.0协议发布。只要注明正确的出处，您可自由使用、分享与改编本数据集。