Molmo2-VideoPointEval

# Molmo2-VideoPointEval `Molmo2-VideoPointEval` is a validation set of `Molmo2-VideoPoint` video pointing data collected from human annotators. It can be used to evaluate vision-language models' video pointing capability. Molmo2-VideoPointEval is part of the [Molmo2 dataset collection](https://huggingface.co/collections/allenai/molmo2-data) and was used to evaluate the [Molmo2 family of models](https://huggingface.co/collections/allenai/molmo2). Quick links: - 📃 [Paper](https://allenai.org/papers/molmo2) - 🎥 [Blog with Videos](https://allenai.org/blog/molmo2) ## Data Format - `video_source`: There are three video sources: `youtube`, `generated` and `MammalNet`. For YouTube videos, you need to download them by their `video_id`s. We provide a mapping from their IDs to the original YouTube URLs and public Google Cloud Storage URLs in `youtube_id_to_urls_mapping.json`. For videos from MammalNet, you can download them following the instructions in their Github repo [here](https://github.com/Vision-CAIR/MammalNet?tab=readme-ov-file#dataset-download). - `raw_timestamps` vs. `two_fps_timestamps`: We re-encoded all raw videos into 2FPS and annotated the 2FPS videos. You can find the `raw_frames` and `raw_timestamps` we extracted from the original videos, and the `two_fps_timestamps` we used in model evaluation. - `points` and `masks`: The `points` column represents a list of 2D absolute coordinates, where each `points[i]` corresponds to the groundtruth points annotated at `two_fps_timestamps[i]`. In this evaluation, we first perform a bipartite matching of predicted (time, x, y) points to groundtruth points with all triplets normalized to 0-100, and then we calculate the pointing recall, prediction, and F1 based on the `masks`, where each `masks[i]` contains a list of SAM2-generated masks in a 3-second window centered around the annotated gt `points[i]`. For more details, please refer to our paper. ## License This dataset is licensed under ODC-BY. A subset of videos from this dataset that are licensed as CC BY-4.0 may be downloaded from our Google Cloud Bucket via the URLs in `youtube_id_to_urls_mapping.json`. The dataset and videos are intended for research and educational use in accordance with Ai2’s [Responsible Use Guidelines](https://allenai.org/responsible-use). This dataset includes questions generated from GPT-4.1 and GPT-5, which are subject to OpenAI’s [Terms of Use](https://openai.com/policies/row-terms-of-use/).