PolyU-ChenLab/ETBench

--- license: cc-by-nc-sa-4.0 --- # E.T. Bench [arXiv](https://arxiv.org/abs/2409.18111) | [Project Page](https://polyu-chenlab.github.io/etbench) | [GitHub](https://github.com/PolyU-ChenLab/ETBench) E.T. Bench is a large-scale and high-quality benchmark for open-ended event-level video understanding. Categorized within a 3-level task taxonomy, it encompasses 7.3K samples under 12 tasks with 7K videos (251.4h total length) under 8 domains, providing comprehensive evaluations on 4 essential capabilities for time-sensitive video understanding. ## 📦 Data Preparation You may download the evaluation kit for E.T. Bench using the following command. ``` git lfs install git clone [e-mail]:datasets/PolyU-ChenLab/ETBench ``` Then, enter the directory and extract the files in the `videos` folder by running: ``` cd ETBench for path in videos/*.tar.gz; do tar -xvf $path -C videos; done ``` **[Optional]** You may also want to compress the videos (to lower FPS & resolution) for faster I/O. ``` python compress_videos.py --fps 3 --size 224 ``` <details> <summary><i>Arguments of <code>compress_videos.py</code></i></summary> - `--src_dir` Path to the videos folder (Default: `videos`) - `--tgt_dir` Path to the output folder (Default: `videos_compressed`) - `--fps` The target FPS for output (Default: `3`) - `--size` The length of the shortest side of output frames (Default: `224`) - `--workers` Number of workers to use (Default: `None` same as the number of CPUs) </details> This will compress all the videos to `3 FPS` and `224 pixels shortest side`. The audio will be removed as well. The output videos will be saved in `videos_compressed` folder with the same structure as `videos`. ## 🚀 Getting Started The folder for E.T. Bench is organized as follows. ``` ETBench ├─ annotations │ ├─ txt (annotations for sub-tasks, with timestamps as text) │ ├─ vid (annotations for sub-tasks, with timestamps as <vid> tokens) │ ├─ etbench_txt_v1.0.json (merged annotations in `txt` folder) │ └─ etbench_vid_v1.0.json (merged annotations in `vid` folder) ├─ evaluation │ ├─ compute_metrics.py (script for computing metrics) │ ├─ requirements.txt (requirements for the evaluation script) │ └─ subset.json (IDs of the subset for evaluating commercial models) ├─ videos (raw video files) ├─ videos_compressed (compressed video files) └─ compress_videos.py (script for compressing videos) ``` For full evaluation on 7,289 samples, you just need to use either of the following annotation file. - `etbench_txt_v1.0.json` - for models representing timestamps in pure text, e.g., '2.5 - 4.8 seconds' - `etbench_vid_v1.0.json` - for models using special tokens for timestamps, e.g., \<vid\> token in E.T. Chat Each JSON file contains a list of dicts with the following entries. ```python { "version": 1.0, # annotation version "idx": 0, # sample index "task": "tvg", # task "source": "qvhighlights", # source dataset "video": "qvhighlights/example.mp4", # path to video "duration": 35.0, # video duration (seconds) "src": [1.2, 15.0], # [optional] timestamps (seconds) in model inputs "tgt": [[15.0, 31.0], [31.4, 34.9]], # [optional] timestamps (seconds) in model outputs "p": 0, # [optional] index of correct answer (for RAR, ECA, RVQ, GVQ) "o": ["a", "b", "c", "d"], # [optional] answer candidates (for RAR, ECA, RVQ, GVQ) "g": ["a cat...", "it then..."], # [optional] ground truth captions (for DVC, SLC) "q": "...", # model input prompt "a": "..." # [to be added by the user] model response } ``` For each sample, you can simply load the corresponding video and send it together with the prompt in `q` to the model. In `vid` style annotations, all the timestamps in `q` have been replaced with `<vid>` and their original values can be found in `src`. After obtaining model outputs, you need to place raw text responses into the `a` entries of each sample and dump the entire list to a new JSON file. ***Please make sure the dumped file has exactly the same structure as the annotation file, except that each sample has a new `a` entry storing model outputs.*** Please refer to the [inference script](../etchat/eval/infer_etbench.py) of E.T. Chat as an example. ## 🔮 Compute Metrics Run the following command to install the requirements for the evaluation script. ``` pip install -r evaluation/requirements.txt ``` After that, compute the metrics by running ``` python evaluation/compute_metrics.py <path-to-the-dumped-json> # In case you want to evaluate on the subset with 470 samples (same as the commercial models in Table 1 of the paper) # python evaluation/compute_metrics.py <path-to-the-dumped-json> --subset ``` The evaluation log and computed metrics will be saved in `metrics.log` and `metrics.json`, respectively. ## 📖 Citation Please kindly cite our paper if you find this project helpful. ``` @inproceedings{liu2024etbench, title={E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding}, author={Liu, Ye and Ma, Zongyang and Qi, Zhongang and Wu, Yang and Chen, Chang Wen and Shan, Ying}, booktitle={Neural Information Processing Systems (NeurIPS)}, year={2024} } ```license: CC BY-NC-SA 4.0（知识共享署名-非商业性使用-相同方式共享4.0国际许可协议） # E.T. Bench [arXiv论文](https://arxiv.org/abs/2409.18111) | [项目主页](https://polyu-chenlab.github.io/etbench) | [GitHub仓库](https://github.com/PolyU-ChenLab/ETBench) E.T. Bench是一款面向开放式事件级视频理解的大规模高质量基准测试集。该数据集基于三级任务分类体系，涵盖12项任务下的7300余个样本、8个领域的7000段视频（总时长251.4小时），可针对时序敏感型视频理解的四项核心能力开展全面评估。 ## 📦 数据准备 您可通过以下命令下载E.T. Bench的评估工具包： git lfs install git clone [e-mail]:datasets/PolyU-ChenLab/ETBench 随后进入该目录并执行如下命令解压`videos`文件夹内的文件： cd ETBench for path in videos/*.tar.gz; do tar -xvf $path -C videos; done **【可选】** 您也可对视频进行压缩（降低帧率与分辨率）以提升输入/输出速度： python compress_videos.py --fps 3 --size 224 <details> <summary><i>参数说明：`compress_videos.py`</i></summary> - `--src_dir` 源视频文件夹路径（默认值：`videos`） - `--tgt_dir` 输出文件夹路径（默认值：`videos_compressed`） - `--fps` 目标输出帧率（默认值：3） - `--size` 输出帧最短边长度（默认值：224） - `--workers` 使用的工作进程数（默认值：`None`，与CPU核心数一致） </details> 该操作会将所有视频压缩至3帧率、最短边224像素的规格，同时移除音频。输出视频将保存至`videos_compressed`文件夹，且保持与`videos`文件夹一致的目录结构。 ## 🚀 快速上手 E.T. Bench的目录结构如下： ETBench ├─ annotations │ ├─ txt（子任务标注，时间戳以文本形式呈现） │ ├─ vid（子任务标注，时间戳以<vid>标记形式呈现） │ ├─ etbench_txt_v1.0.json（`txt`文件夹内的合并标注文件） │ └─ etbench_vid_v1.0.json（`vid`文件夹内的合并标注文件） ├─ evaluation │ ├─ compute_metrics.py（指标计算脚本） │ ├─ requirements.txt（评估脚本依赖文件） │ └─ subset.json（商用模型评估子集的ID列表） ├─ videos（原始视频文件） ├─ videos_compressed（压缩后的视频文件） └─ compress_videos.py（视频压缩脚本） 若需对全部7289个样本开展完整评估，可使用以下任一标注文件： - `etbench_txt_v1.0.json`：适用于以纯文本形式表示时间戳的模型（如‘2.5 - 4.8秒’） - `etbench_vid_v1.0.json`：适用于使用特殊标记表示时间戳的模型（如E.T. Chat中的<vid>标记） 每个JSON文件均包含一组字典结构，其字段说明如下： python { "version": 1.0, # 标注版本 "idx": 0, # 样本索引 "task": "tvg", # 任务类型 "source": "qvhighlights", # 来源数据集 "video": "qvhighlights/example.mp4", # 视频路径 "duration": 35.0, # 视频时长（单位：秒） "src": [1.2, 15.0], # 【可选】模型输入中的时间戳（单位：秒） "tgt": [[15.0, 31.0], [31.4, 34.9]], # 【可选】模型输出中的时间戳（单位：秒） "p": 0, # 【可选】正确答案索引（适用于RAR、ECA、RVQ、GVQ任务） "o": ["a", "b", "c", "d"], # 【可选】候选答案列表（适用于RAR、ECA、RVQ、GVQ任务） "g": ["a cat...", "it then..."], # 【可选】基准字幕（适用于DVC、SLC任务） "q": "...", # 模型输入提示词 "a": "..." # 【待用户补充】模型生成的响应 } 对于每个样本，您仅需加载对应视频，并将其与`q`字段中的提示词一同输入模型即可。在`vid`格式的标注文件中，`q`内的所有时间戳均已替换为`<vid>`标记，其原始值可在`src`字段中获取。 获取模型输出结果后，需将原始文本响应填入每个样本的`a`字段，并将完整列表导出为新的JSON文件。***请确保导出文件与原标注文件结构完全一致，仅新增`a`字段用于存储模型输出结果。*** 可参考E.T. Chat的推理脚本（`../etchat/eval/infer_etbench.py`）作为示例。 ## 🔮 指标计算 执行如下命令安装评估脚本的依赖包： pip install -r evaluation/requirements.txt 完成依赖安装后，执行以下命令计算指标： python evaluation/compute_metrics.py <path-to-the-dumped-json> # 若需对包含470个样本的子集进行评估（与论文表1中的商用模型评估设置一致） # python evaluation/compute_metrics.py <path-to-the-dumped-json> --subset 评估日志与计算得到的指标将分别保存至`metrics.log`与`metrics.json`文件。 ## 📖 引用声明 若本项目对您的研究有所帮助，请引用我们的论文： @inproceedings{liu2024etbench, title={E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding}, author={Liu, Ye and Ma, Zongyang and Qi, Zhongang and Wu, Yang and Chen, Chang Wen and Shan, Ying}, booktitle={Neural Information Processing Systems (NeurIPS)}, year={2024} }