PairUG
收藏资源简介：
PairUG数据集是字节跳动公司创建的一个高质量的统一视觉-语言模型训练数据集，包含16K对理解-生成（UG）样本，用于强化学习微调。该数据集通过将数据重组为理解-生成对，并据此对数据进行对齐优化，旨在解决统一视觉-语言模型在强化学习中存在的理解与生成任务不平衡的问题。
PairUG Dataset is a high-quality unified vision-language model training dataset created by ByteDance. It contains 16K pairs of Understanding-Generation (UG) samples for reinforcement learning fine-tuning. This dataset reorganizes data into Understanding-Generation pairs and conducts alignment optimization based on these pairs, aiming to resolve the imbalance issue between understanding and generation tasks of unified vision-language models during reinforcement learning.
PairUni数据集概述
数据集基本信息
- 数据集名称：PairUni
- 核心功能：统一多模态训练框架
- 主要用途：统一视觉语言模型的理解和生成任务训练
方法概述
- 核心创新：将数据重组为理解-生成对
- 数据增强：使用GPT-o3为理解样本生成描述，为生成样本生成问答对
- 配对结构：
- 对齐对：来自相同实例
- 检索对：语义相关的不同数据点
- 训练框架：Pair-GRPO（基于组相对策略优化的配对感知变体）
数据集详情
- 数据集名称：PairUG
- 数据规模：16K理解-生成对
- 数据格式：JSONL格式
- 评估模型：Janus-Pro UVLMs
数据格式规范
json { "similarity": 0.88, "generate_ann": { "image_path": "data/images/geneval_train_e52c9d7d6c674fd8b2c8b5d2ec43efac.png", "prompt": "a photo of a towel and a zebra", "question": "Which statement best describes the contrast between the material draped on the animal and the animal’s own surface pattern? A. The fabric is smooth and plain, whereas the coat shows bold stripes. B. Both the fabric and the coat display identical striping. C. The fabric is covered with polka dots, while the coat is entirely plain. D. The fabric appears coarse and burlap-like, while the coat looks scaly.
Answer with the options letter from the given choices directly.", "answer": "A", "tag": "geneval_train" }, "understand_ann": { "image_path": "data/images/detection_f2436089737d4f0181f246926c8a2558.png", "prompt": "In open savanna grassland, a small cluster of five plains zebras stands closely together, black-and-white striped bodies angling different directions amid tall yellowish grass under daylight, with erect manes and ears.", "question": "What type of pattern dominates the animals’ coats? A. Stripes B. Polka dots C. Solid gray D. Checkered
Answer with the options letter from the given choices directly.", "answer": "A", "tag": "detection" } }
项目结构
PairUni/ ├── janus/ # Janus模型实现 ├── open_r1/ # PairGRPO训练框架 ├── rewards/ # 奖励模型 ├── configs/ # 训练配置 └── train.sh # 训练启动脚本
引用信息
bibtex @article{pairuni2024, title={PairUni: Unified Multimodal Training with GRPO}, author={Your Name}, journal={arXiv preprint arXiv:XXXX.XXXXX}, year={2024} }
许可证
- Apache License 2.0
