MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps

MammAlps - A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps MammAlps is a multimodal and multi-view dataset of wildlife behavior monitoring. Nine camera-traps were placed at three monitoring sites in the Swiss National Park, from which we curated over 14 hours of video with audio, 2D segmentation maps and 8.5 hours of individual tracks densely labeled for species and behavior. Along with the data, we propose two benchmarks: Benchmark I - Multimodal species and behavior recognition: Based on 6`135 single clips centered on the animal, we propose a hierarchical and multimodal animal behavior recognition benchmark using audio, video and reference scene segmentation maps as inputs. Benchmark II - Multi-view Long-term event understanding: a second ecology-oriented benchmark aiming at identifying activities, species, number of individuals and meteorological conditions from 397 multi-view and long-term ecological events, including false positive triggers. General information Authors: Valentin Gabeff, Haozhe Qi, Brendan Flaherty, Gencer Sümbül, Alexander Mathis, Devis TuiaAffiliation: All authors were affiliated to the Ecole Polytechnique Fédérale de Lausanne (EPFL) at the time of the studyDate of collection: 08.2023 - 10.2023 (MM.YYYY - MM.YYYY)Geolocation data: Swiss National Park, Zernez, SwitzerlandAssociated publication URL: https://arxiv.org/abs/2503.18223Funding: This project was partially funded by EPFL's SV-ENAC I-PhD program (G.V.), Boehringer Ingelheim Fonds PhD stipend (H.Q.) and Swiss SNF grant (320030-227871) Dataset availability License: This dataset is released under the non-commercial CC BY-NC 4.0 license. Citation : When using the dataset, please use the following citation. The article is in press for CVPR, we will update the citation. @article{gabeff2025mammalps, title={MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps}, author={Valentin Gabeff and Haozhe Qi and Brendan Flaherty and Gencer Sumbül and Alexander Mathis and Devis Tuia}, year={2025}, journal={arXiv}, doi={10.48550/arXiv.2503.18223}, } Repository URL: https://zenodo.org/uploads/15040901Repository DOI: 10.5281/zenodo.15040901 Dataset version: v1 Data and files overview Data preparation: unzip mammalps_v1.zip. Repository structure: MammAlps ├── README.md └── mammalps_v1 └── benchmark_1 └── audios └── clips └── segmaps └── annotations └── benchmark_2 └── events_processed └── annotations └── raw_videos └── reference_scenes └── pictures └── annotations └── events2files.json └── raw_videos_mammalps_v1.csv └── file_list.txt benchmark_1: Data for multimodal species and behavior recognition tasks. It contains 3 modalities (video clips, audio and segmentation maps). clips: video clips are animal-centered videos with a 1:1 aspect ratio (380x380 pixels resolution) showing one animal individual expressing a single behavior. audios: audio files matching the associated clips segmaps: Video clip representing a segmentation map of the environment at the corresponding location/time as the animal clip. The animal is not included in the segmentation map as these are created from the reference scene pictures. annotations: Contains the train/val/test splits, along with the species and behavior labels. benchmark_2: Data for multi-view long-term event understanding tasks. It contains the feature extracted from the event by our offline token merging strategy. Raw videos can be found from the raw_video folder. Note that not all videos in raw_videos are used in this benchmark. events_processed: Pre-computed event features generated from concatenating the individual ToME video features and adding cosine positional encodings for the camera ID, elapsed time w.r.t event start, and the spatio-temporal source of the video tokens. Each file contains a 2D array where each row corresponds to one video token of fixed embedding size. annotations: Contains the train/test splits, along with the species, activity, meteorological and group size labels. raw_videos: raw_videos: Raw videos containing events for benchmark_1 and benchmark_2. Note that some videos are used only in one of the two benchmarks. annotations (TBA): Dense annotations for the raw videos containing individual tracks, species and behavioral information. reference_scenes: pictures: Reference picture for each camera trap. annotations: Segmentation maps of the reference pictures in different formats. events2files.json: Dictionary matching event ids to the corresponding list of file ids. raw_videos_mammalps_v1.csv: Raw videos metadata. file_list.txt: Relative path of all files in the current dataset. We refer the reader to the associated publication for details about data processing and tasks descriptions. Naming conventions: There are three cameras in each of the three sites. Sites are named as S1, S2 & S3; cameras are named as C1, C2 & C3. The raw video ids were preserved in the filenames and given as V<video_id>. Each video has a unique id named <file_id>. Events are a collection of videos that are not separated by more than 5 minutes of inactivity, and are named as E<event_id>. In each densely annotated video, individual animal tracks are named as ID<individual_id> and may be segmented if the individual leaves the frame momentarily, yielding multiple segments named as T<segment_id>*. Individual videos are divided into clips containing a single behavioral expression, named as c<clip_id>. * The 24 segments with an ID above 1 (animals re-entering the field-of-view) were unintentionally excluded from the dataset and are absent benchmark 1. We will add these in the future. Given these conventions, we named the files in MammAlps as follows: benchmark_1/clips: S<site_id>_C<camera_id>_E<event_id>_V<video_id>_ID<individual_id>_T<segment_id>/S<site_id>_C<camera_id>_E<event_id>_V<video_id>_ID<individual_id>_T<segment_id>_c<clip_id>.mp4 (e.g., S1_C2_E131_V0298_ID1_T1/S1_C2_E131_V0298_ID1_T1_c0.mp4) benchmark_1/audios: S<site_id>_C<camera_id>_E<event_id>_V<video_id>_ID<individual_id>_T<segment_id>/S<site_id>_C<camera_id>_E<event_id>_V<video_id>_ID<individual_id>_T<segment_id>_c<clip_id>.wav (e.g., S1_C2_E131_V0298_ID1_T1/S1_C2_E131_V0298_ID1_T1_c0.wav) benchmark_1/segmaps: S<site_id>_C<camera_id>_E<event_id>_V<video_id>_ID<individual_id>_T<segment_id>/S<site_id>_C<camera_id>_E<event_id>_V<video_id>_ID<individual_id>_T<segment_id>_c<clip_id>.npz (e.g., S1_C2_E131_V0298_ID1_T1/S1_C2_E131_V0298_ID1_T1_c0.npz) benchmark_2/events_processed: E<event_id>/E<event_id>_<PE>_<r_tome>.mp4 where:|PE indicates the type of positional encoding (dtci or dtci_source).|r_tome indicates the ToME reduction factor used to generate the video features.(e.g., E10/E10_dtci_r11.npy) raw_videos: S<site_id>/C<camera_id>/S<site_id>_C<camera_id>_E<event_id>_V<video_id>.mp4 (e.g., S1/C1/E10/S1_C1_E10_V0031.mp4) reference_scenes/pictures: S<site_id>_C<camera_id>_IMG_<img_id>.JPG (e.g., S1_C1_IMG_0881.JPG) reference_scenes/annotations: S<site_id>_C<camera_id>_IMG_<img_id>.npy or S<site_id>_C<camera_id>_IMG_<img_id>.png (e.g., S1_C1_IMG_0881.npy) Data files formats: audios: audios were extracted from the raw video files as raw waveforms (.wav) clips: clips are provided as compressed videos (.mp4) segmaps: segmentation map clips are provided in compressed numpy arrays (.npz) events_processed *: processed events are provided as numpy arrays (.npy). raw_videos: raw_videos are provided as compressed videos (.mp4) (raw output from the cameras) reference_scenes/pictures: reference scene pictures are given as images (.JPG) reference_scenes/annotations**: segmentation maps of the reference scenes are given either as images (.png) or as matrices (.npy) * The array is two-dimensional, where each row is a token of fixed dimension. The last two elements of each row contain the delta-time and camera id, which are used for positional encoding. Discard the last two elements from each row to conserve only the event features. ** When saved as .npy arrays, the array is two-dimensional corresponding to the dimensions of the reference image and the values correspond to the object class id. Annotation files format: Annotation files (Train/Val/Test) for benchmark I and II follow the epic-kitchen .csv format (space-separated). Benchmark I video_path start_s end_s activity actions species ---------------------------------------------------------------- <clip_path> 0 <clip_end_s> ActY <ActN;ActN> Spe Benchmark II event_path activities species met. cond group_size ---------------------------------------------------------------------- <processed_event_path> <ActY,...> <Spe,...> <Met> <Ind> reference_scenes/annotations/class_ids.jsonMapping between values in the annotation matrices of the reference scene segmentation maps (.npy) and the object classes. { "class": id, ... } events2files.json:Dictionary containing E<event_id> as keys and a list of <file_id> as values. { E<event_id>: [<file_id>, ], ... } raw_videos_mammalps_v1.csv fields: file_id: Unique video identifiersite_id: Site identifier (S1, S2 & S3)cam_id: Camera identifier for each site (C1, C2 & C3)video_id: Original video identifier (not unique)event_id: Event identifier to which the file is belonging toduration: raw video duration in secondsfps: raw video fpsresolution: raw video resolutiondate_time: video start date-time (YYYY-MM-DD hh:mm:ss)file_path: path to raw video (in raw_video folder)in_B1: True if the event is used in benchmark Iin_B2 True if the event is used in benchmark II Methodological informations We refer the reader to the associated publication for details about data processing and tasks descriptions. Benchmark evaluation code: The evaluation code for benchmark I and benchmark II is available at https://github.com/eceo-epfl/MammAlps. We will update this repository to include more demo notebooks. Data processing code (TBA): We are working on packaging the data processing code. Dense annotations: We are working on releasing the dense annotations for all raw_videos. Acknowledgements We thank members of the Mathis Group for Computational Neuroscience & AI (EPFL) and of the Environmental Computational Science and Earth Observation Laboratory (EPFL) for their feedback and fieldwork efforts. We also thank members of the Swiss National Park monitoring team for their support and feedback. The project was approved by the Research Commission of the National Park. Change log [02.04.2025]: Data release for Benchmark I and Benchmark II.