patilli/HNC

HNC（Hard Negative Captions）是一个自动创建的数据集，包含用于图像-文本匹配（ITM）训练的困难负样本字幕，旨在实现视觉和语言（VL）领域的细粒度跨模态理解。此外，数据集还提供了一个具有挑战性的人工创建测试集，用于在具有不同组合复杂度的细粒度跨模态不匹配情况下对模型进行基准测试。结果表明，通过在HNC上进行训练，模型在诊断任务中检测不匹配的零样本能力得到了提升，并且在噪声视觉输入场景下表现稳健。此外，HNC模型在微调时提供了可比或更好的初始化。

The HNC dataset is an automatically generated set of hard negative captions designed to enhance fine-grained visual-linguistic comprehension capabilities in models. It includes automatically generated data for training and a manually created test set for evaluation. The dataset aims to address the issue of insufficient fine-grained understanding in existing image-text matching models when dealing with web-collected image-text pairs. Experiments demonstrate its effectiveness in improving zero-shot learning and mismatch detection tasks, as well as robustness under noisy visual input scenarios.