multi-modal-vlm-visit-bench
收藏数据集卡片：multi-modal-vlm-visit-bench
数据集概述
该数据集由Argilla创建，包含多模态数据，适用于使用Argilla服务器进行探索和标注，或通过HuggingFace的datasets库直接加载。
数据集结构
数据集包含以下内容：
- 兼容HuggingFace
datasets格式的数据记录。 - 用于构建和整理数据集的标注指南（如果已在Argilla中定义）。
- 符合Argilla数据集格式的配置文件夹，位于
.argilla目录下。
数据集在Argilla中包含以下元素：字段、问题、建议、元数据、向量和指南。
字段
字段是数据记录的特征或文本，例如文本分类数据集的text列或指令跟随数据集的prompt列。
| 字段名称 | 标题 | 类型 | 必需 | Markdown |
|---|---|---|---|---|
| image | image | text | True | True |
| instruction | instruction | text | True | False |
| instruction-conditioned-caption | instruction-conditioned-caption | text | True | False |
问题
问题是向标注者提出的问题，可以是评分、文本、标签选择、多标签选择或排序类型。
| 问题名称 | 标题 | 类型 | 必需 | 描述 | 值/标签 |
|---|---|---|---|---|---|
| human-ratings-gpt4-correct | human-ratings-gpt4-correct | label_selection | True | 人类评分，指示GPT-4是否正确遵循了指令 | [true, false] |
| human-ratings-problem-in-caption | human-ratings-problem-in-caption | label_selection | True | 人类评分，指示标题中是否存在问题 | [true, false] |
| human-ratings-problem-in-gpt4 | human-ratings-problem-in-gpt4 | label_selection | True | 人类评分，指示GPT-4的响应中是否存在问题 | [true, false] |
| gpt4-prediction | gpt4-prediction | text | False | GPT-4对任务的预测 | N/A |
元数据
元数据是一个字典，用于提供关于数据记录的额外信息。
| 元数据名称 | 标题 | 类型 | 值 | 对标注者可见 |
|---|---|---|---|---|
| instruction-category | instruction-category | - | True |
向量
向量包含记录的向量表示，可用于搜索。
| 向量名称 | 标题 | 维度 |
|---|---|---|
| instruction-vector | instruction-vector | [1, 384] |
| instruction-conditioned-caption-vector | instruction-conditioned-caption-vector | [1, 384] |
数据实例
一个数据实例在Argilla中的示例如下：
json
{
"_server_id": "2bf0ce36-6faa-423b-a4c3-31189e03913d",
"fields": {
"image": "
",
"instruction": "What is this exercise called and how is it good for you?",
"instruction-conditioned-caption": "There is a pink foam mat with interlocking foam or rubber blue pieces on one half of it, sitting in the middle of a shady spot of grass behind a building and a sunnier spot. In the middle of the mat is a woman wearing grey pants that only come to her ankle and a pink halter-top style shirt. Shes putting all her weight on her thighs and hands, which are facing forward from her. Both of her legs are bent at the knees inward, so that the flats of her feet are touching her long black hair at the back of her head, and her hair dangles so it nearly touches her posterior, while her face is angled upwards towards the sky."
},
"id": "7b689a74-8583-4276-a9ef-9f80994be8c9",
"metadata": {
"instruction-category": "Exercise"
},
"responses": {},
"status": "pending",
"suggestions": {
"gpt4-prediction": {
"agent": null,
"score": null,
"value": "This exercise is called the "King Pigeon Pose" or "Eka Pada Rajakapotasana" in yoga. It is good for you as it stretches the thighs, groin, abdomen, chest, shoulders, and neck, while also stimulating the abdominal organs and improving posture and flexibility."
},
"human-ratings-gpt4-correct": {
"agent": null,
"score": null,
"value": "false"
},
"human-ratings-problem-in-caption": {
"agent": null,
"score": null,
"value": "false"
},
"human-ratings-problem-in-gpt4": {
"agent": null,
"score": null,
"value": "true"
}
},
"vectors": {}
}
在HuggingFace datasets中的相同记录示例如下：
json
{
"_server_id": "2bf0ce36-6faa-423b-a4c3-31189e03913d",
"gpt4-prediction.suggestion": "This exercise is called the "King Pigeon Pose" or "Eka Pada Rajakapotasana" in yoga. It is good for you as it stretches the thighs, groin, abdomen, chest, shoulders, and neck, while also stimulating the abdominal organs and improving posture and flexibility.",
"gpt4-prediction.suggestion.agent": null,
"gpt4-prediction.suggestion.score": null,
"human-ratings-gpt4-correct.suggestion": "false",
"human-ratings-gpt4-correct.suggestion.agent": null,
"human-ratings-gpt4-correct.suggestion.score": null,
"human-ratings-problem-in-caption.suggestion": "false",
"human-ratings-problem-in-caption.suggestion.agent": null,
"human-ratings-problem-in-caption.suggestion.score": null,
"human-ratings-problem-in-gpt4.suggestion": "true",
"human-ratings-problem-in-gpt4.suggestion.agent": null,
"human-ratings-problem-in-gpt4.suggestion.score": null,
"id": "7b689a74-8583-4276-a9ef-9f80994be8c9",
"image": "",
"instruction": "What is this exercise called and how is it good for you?",
"instruction-category": "Exercise",
"instruction-conditioned-caption": "There is a pink foam mat with interlocking foam or rubber blue pieces on one half of it, sitting in the middle of a shady spot of grass behind a building and a sunnier spot. In the middle of the mat is a woman wearing grey pants that only come to her ankle and a pink halter-top style shirt. Shes putting all her weight on her thighs and hands, which are facing forward from her. Both of her legs are bent at the knees inward, so that the flats of her feet are touching her long black hair at the back of her head, and her hair dangles so it nearly touches her posterior, while her face is angled upwards towards the sky.",
"instruction-conditioned-caption-vector": [
0.021473465487360954,
0.10754763334989548,
0.14798341691493988,
-0.14049002528190613,
0.010625330731272697,
-0.07629093527793884,
0.13141514360904694,
-0.05140950158238411,
-0.09660188853740692,
-0.2592792212963104,
-0.23375579714775085,
-0.08067195117473602,
0.12288053333759308,
-0.03611363098025322,
0.04131385684013367,
-0.028739627450704575,
-0.008648086339235306,
0.32250797748565674,
0.10550974309444427,
0.19984672963619232,
-0.03734481707215309,
-0.0022034691646695137,
0.07983627915382385,
-0.02013581618666649,
-0.1341937780380249,
-0.16509348154067993,
0.0715259537100792,
-0.09380444139242172,
-0.03984955698251724,
-0.025817451998591423,
0.5060305595397949,
0.12004397064447403,
0.07612147927284241,
-0.13307364284992218,
-0.032250773161649704,
-0.22835606336593628,
0.276922345161438,
0.0910184234380722,
-0.17201533913612366,
-0.11520933359861374,
0.13959485292434692,
0.17710253596305847,
0.14618510007858276,
-0.25805914402008057,
0.039814017713069916,
0.1329757571220398,
0.031686823815107346,
-0.030810443684458733,
0.25683125853538513,
-0.15260842442512512,
0.020481735467910767,
0.11013107001781464,
-0.032886043190956116,
0.015668530017137527,
0.03483792766928673,
-0.07092206180095673,
-0.1889929175376892,
0.01249205507338047,
0.23342226445674896,
-0.035175301134586334,
0.005187720060348511,
0.10122273862361908,
0.05438707768917084,
0.07043414562940598,
0.08355413377285004,
0.07310357689857483,
0.10765579342842102,
0.06553667038679123,
0.05527825653553009,
-0.08454061299562454,
-0.03585704043507576,
0.264997661113739,
-0.368277907371521,
-0.1793736219406128,
-0.12951549887657166,
-0.0031747817993164062,
0.0004681013524532318,
-0.11840999126434326,
0.2088143527507782,
0.04547523707151413,
-0.06620635837316513,
-0.018145756796002388,
-0.17441007494926453,
-0.1260131299495697,
-0.04789771884679794,
0.05233281850814819,
-0.0010442938655614853,
-0.05728473514318466,
0.05254557728767395,
-0.08983037620782852,
0.04343093931674957,
0.2849102020263672,
-0.06179475039243698,
0.19282130897045135,
0.02617977000772953,
-0.0691226124763
