allenai/pixmo-point-explanations

PixMo-Point-Explanations是一个包含图像、问题和答案的数据集，答案中可能包含指向图像部分的点注释。该数据集用于训练视觉语言模型，使其能够通过文本和点的混合方式回答问题。数据集是PixMo数据集集合的一部分，并用于训练Molmo系列模型。数据集被认为是实验性的，因为模型在生成此类输出时可能会出现幻觉。因此，Molmo模型被训练为仅在输入问题前加上point_qa:时生成此类输出。数据集中图像以URL形式存储，包含解析后的响应、替代文本、内联文本和点列表等字段。图像哈希用于验证下载的图像与注释图像是否匹配。数据集遵循ODC-BY-1.0许可证，适用于研究和教育用途。

PixMo-Point-Explanations is a dataset of images, questions, and answers with explanations that can include in-line points that refer to parts of the image. It can be used to train vision language models to respond to questions through a mixture of text and points. The dataset is part of the PixMo dataset collection and was used to train the Molmo family of models. The dataset is considered experimental, as models can hallucinate more when generating outputs of this sort. Therefore, the Molmo models are trained to only generate outputs like this when specifically requested by prefixing input questions with point_qa:. Images are stored as URLs, and the dataset includes fields such as parsed responses, alt text, inline text, and lists of points. Image hashes are included to verify that the downloaded image matches the annotated image. The dataset is licensed under ODC-BY-1.0 and is intended for research and educational use.