wangyueqian/MMDuet2-data

---
language:
- en
license: apache-2.0
size_categories:
- 10K<n<100K
task_categories:
- video-text-to-text
---

<div align="center">
<h1 style="font-size: 32px; font-weight: bold;"> MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning </h1>
<br>

<p align="left">
📖 <a href="https://arxiv.org/abs/xxx" target="_blank">Paper</a> ·
⭐ <a href="https://github.com/yellow-binary-tree/MMDuet2" target="_blank">GitHub</a> ·
📊 <a href="https://huggingface.co/datasets/wangyueqian/MMDuet2-data" target="_blank">Dataset</a> ·
🤗 <a href="https://huggingface.co/wangyueqian/MMDuet2" target="_blank">Checkpoints</a>
</p>
</div>

This dataset repository contains the data for SFT, RL, and evaluate MMDuet2.

The textual annotations are in `sft/`, `rl/`, and `evaluate/` folders, respectively. Check the [Github codebase of MMDuet2](https://github.com/yellow-binary-tree/MMDuet2) for the usage of these files.

## Video Preparation
Videos in this dataset need to be downloaded separately.

- First, download the videos from their official websites respectively: [Live-WhisperX](https://huggingface.co/datasets/chenjoya/Live-WhisperX-526K/), [EgoExoLearn](https://github.com/OpenGVLab/EgoExoLearn), [Ego-Exo4D](https://ego-exo4d-data.org/), [LLaVA-Video](https://huggingface.co/datasets/lmms-lab/LLaVA-Video-178K), and [Tarsier2](https://huggingface.co/datasets/omni-research/Tarsier2-Recap-585K)

- Next, refer to the code in `video_to_frames-{dataset}.py` to extract frames from the video. Only videos used for proactive training (Live-WhisperX, EgoExoLearn, Ego-Exo4D) need to be extracted.

- To prevent the SFT stage from reading a large number of separate image files and putting pressure on the hard disk, you can choose to pack the extracted frames of the same video into an h5 file. This code is included in `code/frame_to_h5py-live_whisperx.py`.