Molmo2-CapEval

# Molmo2-CapEval Molmo2-CapEval is a dataset of very long, detailed video captions from multiple annotators per video. It can be used to test the caption capability of vision-language models. Molmo2-Cap is part of the [Molmo2 dataset collection](https://huggingface.co/collections/allenai/molmo2-data) and was used to test the [Molmo2 family of models](https://huggingface.co/collections/allenai/molmo2). Quick links: - 📃 [Paper](https://allenai.org/papers/molmo2) - 🎥 [Blog with Videos](https://allenai.org/blog/molmo2) ## Evaluation code Please check out the `caption_eval.py` file for caption evaluation used in Molmo2 paper. ## Prepare videos Videos are from Vimeo, Ego4D, and BDD100K. Each row contains a video_id, source, video_start, video_end, and human annotations. Videos should be placed under MOLMO_DATA_DIR/video_datasets/video-caption-eval/ with the following structure: video-caption-eval/ ├── bdd100k/{video_id}.mov ├── ego4d/{video_id}.mp4 └── vimeo/{category}/vimeo_{video_id}.mp4 ### Vimeo (auto-download) Vimeo videos are bundled as vimeo_videos.zip in this repo and will be automatically downloaded and extracted when loading the dataset via the VixMoCaptionsEval class. No manual steps needed. ### BDD100K (manual download) Download BDD100K videos from the https://www.bdd100k.com/ and place them under video-caption-eval/bdd100k/. ### Ego4D (manual download + clipping) Download Ego4D videos from the https://ego4d-data.org/. Important: Some Ego4D videos need to be clipped to a specific time range. If a row in the dataset has video_start and video_end set (not null), you need to clip the original video and save it with the following naming convention: ffmpeg -ss {start} -to {end} -i {video_id}.mp4 -c copy {video_id}_bounded_decimal_2_{start}_{end}.mp4 For example, a video 0ae6293e-eda5-44f7-b56e-e8f27fcde953 clipped from 0.17s to 141.74s becomes: ego4d/0ae6293e-eda5-44f7-b56e-e8f27fcde953_bounded_decimal_2_0.17_141.74.mp4 Note: Trailing zeros in timestamps are stripped (e.g., 0.80 → 0.8, 3.00 → 3.0). Videos without video_start/video_end (null values) are used as-is — just place the full video as ego4d/{video_id}.mp4. ## License This dataset is licensed under ODC-BY. It is intended for research and educational use in accordance with Ai2’s [Responsible Use Guidelines](https://allenai.org/responsible-use). This dataset includes text captions generated from GPT-4.1 and GPT-5, which are subject to OpenAI’s [Terms of Use](https://openai.com/policies/row-terms-of-use/). Some contents of this dataset have been created based the use of third party datasets that are subject to academic and noncommercial research use only. Please refer to the Source Attribution file for more information about these sources.