wangclnlp/vision-feedback-mix-binarized

该数据集旨在提供大规模的视觉反馈数据，结合了多个高质量的视觉反馈数据集，包括LLaVA-Human-Preference-10K、VLFeedback、POVID_preference_data_for_VLLMs、RLHF-V-Dataset和RLAIF-V-Dataset。数据集格式包括样本ID、来源数据集、图像、图像路径和对话内容。标准化处理过程涉及对候选输出的排名和对话格式的统一。数据集可用于训练视觉奖励模型或进行DPO训练，以对齐视觉大语言模型。

The dataset Vision-Feedback-Mix-Binarized aims to provide large-scale vision feedback data. It combines several high-quality vision feedback datasets, including zhiqings/LLaVA-Human-Preference-10K, MMInstruction/VLFeedback, YiyangAiLab/POVID_preference_data_for_VLLMs, openbmb/RLHF-V-Dataset, and openbmb/RLAIF-V-Dataset. The dataset format includes sample ID, origin dataset source, image, image path, and conversation content. In the data format standardization process, the vlfeedback dataset was ranked based on average scores and the output with the highest average score was selected as the preferred output, with a randomly selected remaining output as the dispreferred one. For the LLaVA-RLHF dataset, historical conversations were converted into a single-round dialogue format and dialogue roles were unified. The dataset can be used to train a vision reward model or conduct DPO training to align a vision large language model. The limitations of the dataset are that it was initially mixed without filtering, and filtering is currently being conducted to reduce the sample size and improve the overall quality of the dataset.

原始信息汇总

数据集卡片：Vision-Feedback-Mix-Binarized

简介

该数据集旨在提供大规模的视觉反馈数据。它结合了以下高质量的视觉反馈数据集：

• zhiqings/LLaVA-Human-Preference-10K: 9,422个样本
• MMInstruction/VLFeedback: 80,258个样本
• YiyangAiLab/POVID_preference_data_for_VLLMs: 17,184个样本
• openbmb/RLHF-V-Dataset: 5,733个样本
• openbmb/RLAIF-V-Dataset: 83,132个样本

此外，我们还提供了一个清洗版本的数据集：wangclnlp/vision-feedback-mix-binarized-cleaned。

混合反馈数据格式描述

bash { "id": int, # 样本ID "origin_pre_dataset": string, # 反馈数据集的来源 "image": Image, # 图像 "image_path": string, # 图像路径 "conversations": string # 对话 # [{"from": "human","value": "a prompt or historical conversations"},{"from": "gpt","preference_values": ["prefered output", "dispreferred output"]}] }

标准化数据格式的步骤

1. 在vlfeedback数据集中，我们根据帮助性、伦理考虑和视觉忠实度的平均分数对所有候选输出进行排序。此外，为了创建vlfeedback二值化数据集，我们选择平均分数最高的输出作为首选输出，并随机选择其余输出中的一个作为非首选输出。
2. 在LLaVA-RLHF数据集中，为了统一单轮对话格式，我们将历史对话转换为一个“提示字符串”，通过合并对话角色（USER: 和 ASSISTANT:）。

使用方法

您可以下载此数据集来训练视觉奖励模型或进行DPO训练，以对齐视觉大语言模型。我们推荐使用Vision-LLM-Alignment，它提供了专门为训练视觉LLM设计的SFT、奖励模型训练和RLHF/DPO代码。

局限性

我们最初混合了这些视觉反馈数据而未进行过滤。我们正在过滤这些大规模的视觉反馈数据，以减少样本数量并提高数据集的整体质量。过滤后的版本将很快发布。