mteb/VidoreSyntheticDocQAEnergyRetrieval

--- annotations_creators: - derived language: - eng license: mit multilinguality: monolingual source_datasets: - vidore/syntheticDocQA_energy_test_beir task_categories: - visual-document-retrieval - image-to-text - text-to-image task_ids: [] dataset_info: - config_name: corpus features: - name: image dtype: image - name: id dtype: string - name: text dtype: 'null' - name: modality dtype: string splits: - name: test num_bytes: 313627553 num_examples: 977 download_size: 273000842 dataset_size: 313627553 - config_name: qrels features: - name: query-id dtype: string - name: corpus-id dtype: string - name: score dtype: int64 splits: - name: test num_bytes: 4274 num_examples: 100 download_size: 2838 dataset_size: 4274 - config_name: queries features: - name: id dtype: string - name: text dtype: string - name: image dtype: 'null' - name: modality dtype: string splits: - name: test num_bytes: 11260 num_examples: 100 download_size: 8470 dataset_size: 11260 configs: - config_name: corpus data_files: - split: test path: corpus/test-* - config_name: qrels data_files: - split: test path: qrels/test-* - config_name: queries data_files: - split: test path: queries/test-* tags: - mteb - text - image --- <!-- adapted from https://github.com/huggingface/huggingface_hub/blob/v0.30.2/src/huggingface_hub/templates/datasetcard_template.md --> <div align="center" style="padding: 40px 20px; background-color: white; border-radius: 12px; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.05); max-width: 600px; margin: 0 auto;"> <h1 style="font-size: 3.5rem; color: #1a1a1a; margin: 0 0 20px 0; letter-spacing: 2px; font-weight: 700;">VidoreSyntheticDocQAEnergyRetrieval</h1> <div style="font-size: 1.5rem; color: #4a4a4a; margin-bottom: 5px; font-weight: 300;">An <a href="https://github.com/embeddings-benchmark/mteb" style="color: #2c5282; font-weight: 600; text-decoration: none;" onmouseover="this.style.textDecoration='underline'" onmouseout="this.style.textDecoration='none'">MTEB</a> dataset</div> <div style="font-size: 0.9rem; color: #2c5282; margin-top: 10px;">Massive Text Embedding Benchmark</div> </div> Retrieve associated pages according to questions. | | | |---------------|---------------------------------------------| | Task category | t2i | | Domains | Academic | | Reference | https://arxiv.org/pdf/2407.01449 | Source datasets: - [vidore/syntheticDocQA_energy_test_beir](https://huggingface.co/datasets/vidore/syntheticDocQA_energy_test_beir) ## How to evaluate on this task You can evaluate an embedding model on this dataset using the following code: ```python import mteb task = mteb.get_task("VidoreSyntheticDocQAEnergyRetrieval") evaluator = mteb.MTEB([task]) model = mteb.get_model(YOUR_MODEL) evaluator.run(model) ``` <!-- Datasets want link to arxiv in readme to autolink dataset with paper --> To learn more about how to run models on `mteb` task check out the [GitHub repository](https://github.com/embeddings-benchmark/mteb). ## Citation If you use this dataset, please cite the dataset as well as [mteb](https://github.com/embeddings-benchmark/mteb), as this dataset likely includes additional processing as a part of the [MMTEB Contribution](https://github.com/embeddings-benchmark/mteb/tree/main/docs/mmteb). ```bibtex @article{faysse2024colpali, author = {Faysse, Manuel and Sibille, Hugues and Wu, Tony and Viaud, Gautier and Hudelot, C{\'e}line and Colombo, Pierre}, journal = {arXiv preprint arXiv:2407.01449}, title = {ColPali: Efficient Document Retrieval with Vision Language Models}, year = {2024}, } @article{enevoldsen2025mmtebmassivemultilingualtext, title={MMTEB: Massive Multilingual Text Embedding Benchmark}, author={Kenneth Enevoldsen and Isaac Chung and Imene Kerboua and Márton Kardos and Ashwin Mathur and David Stap and Jay Gala and Wissam Siblini and Dominik Krzemiński and Genta Indra Winata and Saba Sturua and Saiteja Utpala and Mathieu Ciancone and Marion Schaeffer and Gabriel Sequeira and Diganta Misra and Shreeya Dhakal and Jonathan Rystrøm and Roman Solomatin and Ömer Çağatan and Akash Kundu and Martin Bernstorff and Shitao Xiao and Akshita Sukhlecha and Bhavish Pahwa and Rafał Poświata and Kranthi Kiran GV and Shawon Ashraf and Daniel Auras and Björn Plüster and Jan Philipp Harries and Loïc Magne and Isabelle Mohr and Mariya Hendriksen and Dawei Zhu and Hippolyte Gisserot-Boukhlef and Tom Aarsen and Jan Kostkan and Konrad Wojtasik and Taemin Lee and Marek Šuppa and Crystina Zhang and Roberta Rocca and Mohammed Hamdy and Andrianos Michail and John Yang and Manuel Faysse and Aleksei Vatolin and Nandan Thakur and Manan Dey and Dipam Vasani and Pranjal Chitale and Simone Tedeschi and Nguyen Tai and Artem Snegirev and Michael Günther and Mengzhou Xia and Weijia Shi and Xing Han Lù and Jordan Clive and Gayatri Krishnakumar and Anna Maksimova and Silvan Wehrli and Maria Tikhonova and Henil Panchal and Aleksandr Abramov and Malte Ostendorff and Zheng Liu and Simon Clematide and Lester James Miranda and Alena Fenogenova and Guangyu Song and Ruqiya Bin Safi and Wen-Ding Li and Alessia Borghini and Federico Cassano and Hongjin Su and Jimmy Lin and Howard Yen and Lasse Hansen and Sara Hooker and Chenghao Xiao and Vaibhav Adlakha and Orion Weller and Siva Reddy and Niklas Muennighoff}, publisher = {arXiv}, journal={arXiv preprint arXiv:2502.13595}, year={2025}, url={https://arxiv.org/abs/2502.13595}, doi = {10.48550/arXiv.2502.13595}, } @article{muennighoff2022mteb, author = {Muennighoff, Niklas and Tazi, Nouamane and Magne, Loïc and Reimers, Nils}, title = {MTEB: Massive Text Embedding Benchmark}, publisher = {arXiv}, journal={arXiv preprint arXiv:2210.07316}, year = {2022} url = {https://arxiv.org/abs/2210.07316}, doi = {10.48550/ARXIV.2210.07316}, } ``` # Dataset Statistics <details> <summary> Dataset Statistics</summary> The following code contains the descriptive statistics from the task. These can also be obtained using: ```python import mteb task = mteb.get_task("VidoreSyntheticDocQAEnergyRetrieval") desc_stats = task.metadata.descriptive_stats ``` ```json { "test": { "num_samples": 1077, "number_of_characters": 8369, "documents_text_statistics": null, "documents_image_statistics": { "min_image_width": 1200, "average_image_width": 1763.8955987717502, "max_image_width": 3400, "min_image_height": 1166, "average_image_height": 2082.4831115660186, "max_image_height": 2340, "unique_images": 975 }, "queries_text_statistics": { "total_text_length": 8369, "min_text_length": 36, "average_text_length": 83.69, "max_text_length": 154, "unique_texts": 100 }, "queries_image_statistics": null, "relevant_docs_statistics": { "num_relevant_docs": 100, "min_relevant_docs_per_query": 1, "average_relevant_docs_per_query": 1.0, "max_relevant_docs_per_query": 1, "unique_relevant_docs": 77 }, "top_ranked_statistics": null } } ``` </details> --- *This dataset card was automatically generated using [MTEB](https://github.com/embeddings-benchmark/mteb)*VidoreSyntheticDocQAEnergyRetrieval is an English monolingual dataset for visual document retrieval tasks, containing both image and text information. Derived from the vidore/syntheticDocQA_energy_test_beir dataset, it includes a test set suitable for evaluating embedding models on visual document retrieval tasks.

搜集汇总

数据集介绍

构建方式

在学术文档检索领域，多模态嵌入模型的评估亟需高质量基准数据集。VidoreSyntheticDocQAEnergyRetrieval 数据集源自 vidore/syntheticDocQA_energy_test_beir，由 MTEB（大规模文本嵌入基准）项目衍生构建。其语料库包含 977 个图像文档，每个文档以图像形式呈现，无文本字段，旨在模拟纯视觉文档检索场景。查询集和相关性判断集各含 100 条样本，其中每个查询精确对应一个相关文档，形成一对一的检索映射。数据集的构建通过合成方式生成，确保查询与文档内容在能源领域主题上的高度一致性，为视觉文档检索任务提供了标准化的评估基础。

特点

该数据集的核心特点在于其多模态与领域专精的融合。作为视觉文档检索基准，它专注于能源领域的学术文献，语料库完全由图像构成，摒弃了传统文本表示，迫使模型依赖视觉语义进行理解与匹配。查询均为英文文本，长度在 36 至 154 字符之间，平均约 84 字符，兼具简洁性与信息密度。图像尺寸范围宽广（宽度 1200-3400 像素，高度 1166-2340 像素），平均宽度约 1764 像素，高度约 2082 像素，呈现多样化布局。此外，数据集仅有测试集划分，包含 1077 个样本，其中 77 个独特相关文档，凸显了检索任务的稀疏性与挑战性。

使用方法

使用该数据集进行模型评估需借助 MTEB 框架。用户可通过 mteb.get_task 方法加载任务，并利用 mteb.MTEB 类构建评估器，传入待评估的嵌入模型后调用 run 方法即可完成测试。模型需支持图像与文本的跨模态嵌入生成，以应对查询文本与图像文档的匹配。评估过程自动计算检索指标，无需手动处理数据格式。数据集以三个配置（corpus、queries、qrels）组织，分别存储文档图像、查询文本和相关性标签，便于按需加载。推荐参考 MTEB 官方仓库获取更详细的运行指南与模型适配方案。

背景与挑战

背景概述

在视觉语言模型与文档检索技术迅猛发展的背景下，如何高效地从包含丰富图文信息的学术文档中定位与能源领域相关的知识，成为一项亟待攻克的课题。mteb/VidoreSyntheticDocQAEnergyRetrieval数据集由Manuel Faysse等研究人员于2024年创建，其核心研究问题聚焦于评估多模态嵌入模型在文本到图像检索任务中的表现，尤其是在能源领域的学术文档场景下。该数据集源自vidore/syntheticDocQA_energy_test_beir，作为Massive Text Embedding Benchmark（MTEB）的一部分，为视觉文档检索提供了标准化的评测基准。它包含977个图文并茂的文档页面和100个精心设计的查询，每个查询对应一个唯一的相关文档，从而推动了多模态检索系统在专业学术领域中的性能评估与优化。

当前挑战

该数据集所面临的挑战主要体现在两个层面。首先，在领域问题层面，它致力于解决视觉文档检索中的跨模态对齐难题，即如何精准地将自然语言查询与包含图像、文本等多模态信息的文档页面进行匹配，这在能源等专业学术领域尤为复杂，因为术语专业且图文关系高度耦合。其次，在构建过程中，挑战在于合成查询与文档对的真实性与代表性平衡，确保生成的100个查询能有效覆盖能源领域的核心概念，同时保持每对仅一个相关文档的严格相关性标准，这要求对源文档的语义和视觉结构进行深度理解与人工校准，以避免噪声引入并保证评测的公平性。

常用场景

经典使用场景

VidoreSyntheticDocQAEnergyRetrieval 数据集专为视觉文档检索任务而设计，其核心应用在于评估模型根据自然语言查询从大量图文混杂的文档页面中精准定位相关页面的能力。该数据集依托于合成问答技术，构建了涵盖能源领域学术文献的检索场景，其中查询以文本形式呈现，而候选文档则以图像格式存储。这一设定使其成为检验跨模态检索模型，特别是视觉语言模型在文档级检索任务中性能的经典基准。研究者通常利用该数据集在标准的文本嵌入基准框架下，对模型的图文对齐与语义理解能力进行系统评估。

解决学术问题

在学术研究层面，该数据集主要解决了传统文本检索范式在处理富含图表、公式及排版信息的学术文档时所面临的语义鸿沟问题。现有基于纯文本的检索方法往往丢失了视觉布局中的关键信息，而 VidoreSyntheticDocQAEnergyRetrieval 通过引入图像模态，促使学界重新审视并量化模型对多模态文档内容的综合理解能力。它推动了视觉文档检索这一交叉领域的发展，为衡量模型是否能够跨越视觉与文本的表层差异、捕获深层语义关联提供了标准化的评测手段，从而引导研究聚焦于更贴近真实应用的多模态信息检索难题。

衍生相关工作

该数据集衍生了一系列密切相关的研究工作，最典型的是 ColPali 模型的提出，其论文《ColPali: Efficient Document Retrieval with Vision Language Models》直接引用了本数据集作为关键评测基准。此外，作为 MMTEB（大规模多语言文本嵌入基准）的一部分，该数据集还催生了关于多模态嵌入模型标准化评估方法的讨论，促进了跨模态检索领域评价体系的完善。后续研究在此基础上进一步探索了视觉语言模型在文档检索中的泛化能力，以及如何通过合成数据增强模型对复杂版面文档的适应性，推动了从纯文本检索向视觉文档检索的范式演进。