VCGBench-Diverse

# 👁️ VCGBench-Diverse Benchmarks

---
## 📝 Description
Recognizing the limited diversity in existing video conversation benchmarks, we introduce VCGBench-Diverse to comprehensively evaluate the generalization ability of video LMMs. While VCG-Bench provides an extensive evaluation protocol, it is limited to videos from the ActivityNet200 dataset. Our benchmark comprises a total of 877 videos, 18 broad video categories and 4,354 QA pairs, ensuring a robust evaluation framework.

<p align="center">
<img src="vcgbench_diverse.png" alt="Contributions">
</p>

## Dataset Contents
1. `vcgbench_diverse_qa.json` - Contains VCGBench-Diverse question-answer pairs.
2. `videos.tar.gz` - Contains the videos corresponding to `vcgbench_diverse_qa.json`.
3. `human_annotated_video_descriptions` - Contains original human-annotated dense descriptions of the videos.
4. `gpt_evaluation_scripts` - Contains the GPT-3.5-Turbo evaluation scripts to evaluate a model's predictions.
5. `sample_predictions` - Contains the VideoGPT+ predictions on the VCGBench-Diverse. Compatible with `gpt_evaluation_scripts`.

In order to evaluate your model on `VCGBench-Diverse`, use question-answer pairs in `vcgbench_diverse_qa.json` to generate your model's predictions in format same as
`sample_predictions` and then use `gpt_evaluation_scripts` for the evalution.

## 💻 Download
To get started, follow these steps:
```
git lfs install
git clone https://huggingface.co/MBZUAI/VCGBench-Diverse
```

## 📚 Additional Resources
- **Paper:** [ArXiv](https://arxiv.org/abs/2406.09418).
- **GitHub Repository:** For training and updates: [GitHub](https://github.com/mbzuai-oryx/VideoGPT-plus).
- **HuggingFace Collection:** For downloading the pretrained checkpoints, VCGBench-Diverse Benchmarks and Training data, visit [HuggingFace Collection - VideoGPT+](https://huggingface.co/collections/MBZUAI/videogpt-665c8643221dda4987a67d8d).

## 📜 Citations and Acknowledgments

```bibtex
@article{Maaz2024VideoGPT+,
title={VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding},
author={Maaz, Muhammad and Rasheed, Hanoona and Khan, Salman and Khan, Fahad Shahbaz},
journal={arxiv},
year={2024},
url={https://arxiv.org/abs/2406.09418}
}

# 👁️ VCGBench-Diverse基准测试集

---
## 📝 数据集概述
针对现有视频对话基准测试集多样性不足的问题，我们提出VCGBench-Diverse，以全面评估视频大语言模型（Large Language Model, LLM）的泛化能力。尽管VCG-Bench已提供了较为完善的评估协议，但其仅局限于使用ActivityNet200数据集的视频。本基准测试集共计包含877段视频、18个宽泛的视频类别以及4354组问答对（Question-Answering pairs, QA对），可构建起一套稳健的评估框架。

<p align="center">
<img src="vcgbench_diverse.png" alt="贡献说明">
</p>

## 数据集内容
1. `vcgbench_diverse_qa.json` - 包含VCGBench-Diverse的问答对数据。
2. `videos.tar.gz` - 包含与`vcgbench_diverse_qa.json`对应的视频文件。
3. `human_annotated_video_descriptions` - 包含人工标注的视频详细描述文本。
4. `gpt_evaluation_scripts` - 包含用于评估模型预测结果的GPT-3.5-Turbo评估脚本。
5. `sample_predictions` - 包含VideoGPT+在VCGBench-Diverse上的模型预测结果，可与`gpt_evaluation_scripts`兼容使用。

若要在VCGBench-Diverse上评估你的模型，请使用`vcgbench_diverse_qa.json`中的问答对生成模型预测结果，预测格式需与`sample_predictions`保持一致，随后通过`gpt_evaluation_scripts`完成评估流程。

## 💻 下载方式
若要开始使用，请按照以下步骤操作：

git lfs install
git clone https://huggingface.co/MBZUAI/VCGBench-Diverse

## 📚 附加资源
- **论文**：[ArXiv](https://arxiv.org/abs/2406.09418)。
- **GitHub仓库**：用于获取训练相关内容与更新：[GitHub](https://github.com/mbzuai-oryx/VideoGPT-plus)。
- **HuggingFace合集**：若要下载预训练检查点（checkpoint）、VCGBench-Diverse基准测试集与训练数据，请访问 [HuggingFace合集 - VideoGPT+](https://huggingface.co/collections/MBZUAI/videogpt-665c8643221dda4987a67d8d)。

## 📜 引用与致谢
bibtex
@article{Maaz2024VideoGPT+,
title={VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding},
author={Maaz, Muhammad and Rasheed, Hanoona and Khan, Salman and Khan, Fahad Shahbaz},
journal={arxiv},
year={2024},
url={https://arxiv.org/abs/2406.09418}
}