whybe-choi/en-vdr-hn

English Visual Document Retrieval (VDR) Hard Negatives 是一个多模态检索训练集，用于在英文文档页面上微调视觉-文档检索嵌入模型。查询是文本，文档是页面图像，每个挖掘行包含1个正例和7个挖掘的硬负例。硬负例是使用Qwen/Qwen3-VL-Embedding-8B模型挖掘的，挖掘过程在每个源数据集中进行，并排除了同一源数据集中具有相同查询的正例作为负例候选。数据集包含五个配置：corpus（去重后的图像存储，每个唯一页面图像一行）、naive（使用前七个有效检索到的负例）、shifted_by_n（通过跳过前N个候选来获取稍容易的负例，N=5）、marginpos（通过正例分数的绝对边际过滤负例，边际=0.05）和percpos（通过正例分数的百分比阈值过滤负例，阈值=95%）。数据集来源于五个英文VDR训练源，包括VisRAG-Ret-Train-In-domain-data、vdr-multilingual-train (en)、REAL-MM-RAG_FinTabTrainSet_rephrased、colpali_train_set和tatdqa_train。

English Visual Document Retrieval (VDR) Hard Negatives is a multimodal retrieval training dataset designed for fine-tuning visual-document retrieval embedding models on English document pages. Queries are text-based, while documents are represented as page images. Each mining entry contains 1 positive sample and 7 mined hard negatives. Hard negatives were mined using the Qwen/Qwen3-VL-Embedding-8B model, with the mining process executed per individual source dataset, and positive examples sharing the same query within the same source dataset were excluded from the negative candidate pool. The dataset offers five configurations:
- `corpus`: a deduplicated image repository where each unique page image occupies one row;
- `naive`: utilizes the top seven valid retrieved negative examples;
- `shifted_by_n`: generates slightly easier negative examples by skipping the first N candidates, with N=5;
- `marginpos`: filters negative examples using the absolute margin of positive example scores, with a margin value of 0.05;
- `percpos`: filters negative examples via a percentage threshold of positive example scores, with a threshold set to 95%.
The dataset is derived from five English VDR training sources, namely VisRAG-Ret-Train-In-domain-data, vdr-multilingual-train (en), REAL-MM-RAG_FinTabTrainSet_rephrased, colpali_train_set, and tatdqa_train.