Omni-Cloze

# Omni-Cloze Benchmark [**📖 Paper**](https://arxiv.org/abs/2510.12720) | [**🕵️ Omni-Detective Pipeline**](https://github.com/ddlBoJack/Omni-Captioner/blob/main/Omni-Detective/main.py) | [**🧑‍🏫 Omni-Cloze Benchmark**](https://huggingface.co/datasets/BoJack/Omni-Cloze) ## Guides **Omni-Cloze** frames detailed captioning evaluation as a **cloze-style multiple-choice proxy task**. Omni-Cloze is a unified benchmark for evaluating detailed captioning across audio-only, visual-only, and audio–visual settings. The dataset spans 9 main domains and 47 sub-categories covering diverse topics such as education, entertainment, sports, news, science, and lifestyle, with a total of 2k video clips with 70k fine-grained cloze blanks.  ## Quick Start ### 1. Prepare Video Data The video dataset is split into several tarball parts. Concatenate and extract them to the `videos/` directory: ```bash for f in videos.part*.tar; do tar -xvf "$f"; done ``` ### 2. Prepare Inference Results The core metadata file is `omni_cloze.jsonl`, which contains 2,320 audio-visual files and their corresponding cloze questions. To evaluate your model, you must first run inference and save the generated descriptions into a new field named **`predicted_caption`** within the JSONL file. Each line in your input file should follow this structure: ```json { "uuid": 1, "video_path": "./videos/0000001.mp4", "predicted_caption": "Your model's detailed description of the audio and visual content goes here...", "...": [...] } ``` ### 3. Run Evaluation The evaluation process uses an LLM to map your detailed captions to the specific cloze blanks. ```bash # 1. Set API environment variables export OPENAI_API_KEY="your-api-key-here" export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1" # 2. Specify input and output data paths, and the model predicted caption field name is "predicted_caption" in the input file input_file="your-input-file-here.jsonl" output_file="your-output-file-here.jsonl" # 3. Run the evaluation script python generate_prediction.py --input $input_file --output $output_file --workers 100 # 4. Run the statistics script python compute_acc.py --input $output_file --show-subcategory ``` ## Citation If you find **Omni-Cloze** useful for your research, please consider citing our work: ``` @article{omni-captioner, title={Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception}, author={Ma, Ziyang and Xu, Ruiyang and Xing, Zhenghao and Chu, Yunfei and Wang, Yuxuan and He, Jinzheng and Xu, Jin and Heng, Pheng-Ann and Yu, Kai and Lin, Junyang and others}, journal={Proc. ICLR}, year={2026} } ```