TIGER-Lab/RationalRewards_DiffusionNFT_TrainData

--- pretty_name: DiffusionRL Train Data license: mit task_categories: - text-to-image - image-to-image language: - en size_categories: - 10K<n<100K configs: - config_name: default data_files: - split: train path: t2i/train.jsonl tags: - text-to-image, - image-to-image, - image-generation --- **TLDR:** this is the diffusion RL training dataset for text-to-image generation and image editing, from the following paper. --- # RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time <div align="left"> <strong>Haozhe Wang</strong>¹ &nbsp; <strong>Cong Wei</strong>² &nbsp; <strong>Weiming Ren</strong>² &nbsp; <strong>Jiaming Liu</strong>³ &nbsp; <strong>Fangzhen Lin</strong>¹ &nbsp; <strong>Wenhu Chen</strong>²<br> ¹ HKUST &nbsp; ² University of Waterloo &nbsp; ³ Alibaba </div> ----- <div align="center"> <a href="https://arxiv.org/abs/2604.11626"> <img src="https://img.shields.io/badge/arXiv-2604.11626-B31B1B?style=for-the-badge&logo=arxiv&logoColor=white" alt="Paper (arXiv)"> </a> <a href="https://tiger-ai-lab.github.io/RationalRewards/"> <img src="https://img.shields.io/badge/Project%20Page-0A66C2?style=for-the-badge&logo=googlechrome&logoColor=white" alt="Project Page"> </a> <a href="https://github.com/TIGER-AI-Lab/RationalRewards"> <img src="https://img.shields.io/badge/Github-181717?style=for-the-badge&logo=github&logoColor=white" alt="GitHub"> </a> <a href="https://huggingface.co/TIGER-Lab/RationalRewards-8B-T2I"> <img src="https://img.shields.io/badge/Model%20(T2I)-FFD966?style=for-the-badge&logo=huggingface&logoColor=black" alt="Model (T2I)"> </a> <a href="https://huggingface.co/TIGER-Lab/RationalRewards-8B-Edit"> <img src="https://img.shields.io/badge/Model%20(Edit)-FFD966?style=for-the-badge&logo=huggingface&logoColor=black" alt="Model (Edit)"> </a> <a href="https://huggingface.co/collections/TIGER-Lab/rationalrewards"> <img src="https://img.shields.io/badge/HF%20Collection-RationalRewards-FFD966?style=for-the-badge&logo=huggingface&logoColor=black" alt="HF Collection"> </a> <br> <a href="https://huggingface.co/datasets/TIGER-Lab/RationalRewards-SFTData"> <img src="https://img.shields.io/badge/SFT%20Dataset-FFB7B2?style=for-the-badge&logo=huggingface&logoColor=black" alt="SFT Dataset"> </a> <a href="https://huggingface.co/datasets/TIGER-Lab/RationalRewards-EvalData-GenAIBench-MMRB2-ERBench"> <img src="https://img.shields.io/badge/Eval%20Dataset-FFB7B2?style=for-the-badge&logo=huggingface&logoColor=black" alt="Eval Dataset"> </a> <a href="https://huggingface.co/datasets/TIGER-Lab/RationalRewards_DiffusionNFT_TrainData"> <img src="https://img.shields.io/badge/Diffusion%20RL%20Training%20Dataset-FFB7B2?style=for-the-badge&logo=huggingface&logoColor=black" alt="Diffusion RL Training Dataset"> </a> </div> **RationalRewards** is a reasoning-based reward model and toolkit for visual generation. Instead of reducing preference into one opaque scalar, it generates explicit multi-dimensional critiques before scoring, turning reward models from passive evaluators into active optimization interfaces. **About the name:** "Rational" means being reasonable, sensible, in Chinese, 理性的 **RationalRewards supports optimization in complementary spaces**: - **train-time optimization** through RL with structured, interpretable reward signals, and - **test-time optimization** through a Generate-Critique-Refine loop without parameter updates. ## Key Results Instantiated via PARROT on a Qwen3-VL-Instruct-8B backbone, RationalRewards achieves state-of-the-art preference prediction among open-source reward models and remains competitive with Gemini-2.5-Pro. As an RL reward, it consistently improves generators beyond scalar baselines across both text-to-image and image-editing tasks. Most interestingly, RationalRewards' test-time prompt tuning, requiring no parameter updates, matches or exceeds RL-based fine-tuning on several benchmarks.  *Train-time RL and test-time prompt tuning with RationalRewards across visual generation benchmarks.* ## Why Reasoning Rewards? Most reward models collapse instruction following, visual quality, composition, and plausibility into one scalar. This removes the structure of human judgment and often leads to brittle optimization. RationalRewards keeps those dimensions explicit so generators receive semantically grounded feedback about what to fix and why. ### Why do reasoning rewards resist reward hacking? Scalar rewards are vulnerable to reward hacking because they collapse rich judgment into one number that can rise even when outputs do not truly improve. RationalRewards introduces an implicit regularization: before giving scores, it must produce coherent, multi-dimensional critiques tied to concrete evaluation axes. This constrains optimization to evidence-backed reasoning and improves the monotonic relationship between reward and observed quality during RL. ### Why are preference-trained rewards more stable than generic VLM judges? Generic VLM judges can be strong analysts, but as reward functions they often show high-variance pointwise scoring across semantically similar samples. That variance becomes optimization noise in RL. PARROT trains RationalRewards directly for preference discrimination, yielding lower-variance, preference-aligned scores. The practical outcome is more stable optimization steps and better reward reliability, even with a smaller model footprint. ### Why do reasoning rewards enable test-time scaling? Reasoning feedback can be reused after generation, not only during training. In a Generate-Critique-Refine loop, RationalRewards critiques the produced image, identifies concrete deficiencies, and proposes targeted prompt updates. Unlike pre-hoc prompt enhancement that rewrites blindly, this is post-hoc and reactive to actual failures. That makes test-time compute more effective at eliciting latent generator capability, often approaching or surpassing RL fine-tuning gains without parameter updates.  *RationalRewards supports optimization in both parameter space (RL) and prompt space (test-time refinement).* ## Method: Preference-Anchored Rationalization (PARROT) Human rationale annotation is expensive. PARROT recovers high-quality rationale supervision from preference-only data in three phases: 1. **Anchored generation:** a teacher VLM proposes rationale candidates consistent with known labels. 2. **Consistency filtering:** hallucinated or non-predictive rationales are removed. 3. **Distillation:** a student model learns to critique-before-score without seeing labels. This gives a practical path from abundant preference datasets to scalable reasoning supervision.  *PARROT pipeline: anchored rationale generation, consistency filtering, and distillation.* ## Empirical Evidence RationalRewards strengthens both alignment quality and downstream optimization.  *State-of-the-art preference prediction among open-source reward models.*  *Structured critique channels reduce shortcut exploitation compared with scalar-only rewards.* To better show optimization behavior, we also include diffusion RL training evolution results. The figure below visualizes how RationalRewards-guided training improves over time, illustrating that benefits are not only visible at the final checkpoint but emerge consistently throughout training. This constrast sharply with scalar rewards suffering reward hacking, as we demonstrate in Figure 12 in the paper.  *Evolution of diffusion RL performance under RationalRewards-guided optimization.*  *Generate-Critique-Refine at test time can match or exceed RL fine-tuning on several benchmarks.*  *Additional qualitative use cases enabled by explicit reasoning feedback.* ## Citation ```bibtex @article{rationalrewards2026, title = {RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time}, author = {Haozhe Wang and Cong Wei and Weiming Ren and Jiaming Liu and Fangzhen Lin and Wenhu Chen}, journal = {arXiv preprint}, year = {2026} } ```--- pretty_name: DiffusionRL训练数据集 license: MIT许可证 task_categories: - 文生图（text-to-image） - 图生图（image-to-image） language: - 英语 size_categories: - 10K < 样本数 < 100K configs: - config_name: 默认配置 data_files: - split: 训练集 path: t2i/train.jsonl tags: - 文生图 - 图生图 - 图像生成 --- **摘要概览：** 本数据集为适用于文生图生成与图像编辑任务的扩散强化学习训练数据集，源自下述论文。 --- # 《RationalRewards：基于推理的奖励机制在训练与测试阶段赋能视觉生成》 <div align="left"> <strong>王浩哲</strong>¹ &nbsp; <strong>魏聪</strong>² &nbsp; <strong>任伟明</strong>² &nbsp; <strong>刘佳明</strong>³ &nbsp; <strong>林芳桢</strong>¹ &nbsp; <strong>陈文虎</strong>²<br> ¹ 香港科技大学（HKUST）&nbsp; ² 滑铁卢大学（University of Waterloo）&nbsp; ³ 阿里巴巴（Alibaba） </div> ----- <div align="center"> <a href="https://arxiv.org/abs/2604.11626"> <img src="https://img.shields.io/badge/arXiv-2604.11626-B31B1B?style=for-the-badge&logo=arxiv&logoColor=white" alt="论文（arXiv）"> </a> <a href="https://tiger-ai-lab.github.io/RationalRewards/"> <img src="https://img.shields.io/badge/Project%20Page-0A66C2?style=for-the-badge&logo=googlechrome&logoColor=white" alt="项目主页"> </a> <a href="https://github.com/TIGER-AI-Lab/RationalRewards"> <img src="https://img.shields.io/badge/Github-181717?style=for-the-badge&logo=github&logoColor=white" alt="GitHub"> </a> <a href="https://huggingface.co/TIGER-Lab/RationalRewards-8B-T2I"> <img src="https://img.shields.io/badge/Model%20(T2I)-FFD966?style=for-the-badge&logo=huggingface&logoColor=black" alt="文生图模型（T2I）"> </a> <a href="https://huggingface.co/TIGER-Lab/RationalRewards-8B-Edit"> <img src="https://img.shields.io/badge/Model%20(Edit)-FFD966?style=for-the-badge&logo=huggingface&logoColor=black" alt="图像编辑模型（Edit）"> </a> <a href="https://huggingface.co/collections/TIGER-Lab/rationalrewards"> <img src="https://img.shields.io/badge/HF%20Collection-RationalRewards-FFD966?style=for-the-badge&logo=huggingface&logoColor=black" alt="Hugging Face合集"> </a> <br> <a href="https://huggingface.co/datasets/TIGER-Lab/RationalRewards-SFTData"> <img src="https://img.shields.io/badge/SFT%20Dataset-FFB7B2?style=for-the-badge&logo=huggingface&logoColor=black" alt="监督微调数据集（SFT Dataset）"> </a> <a href="https://huggingface.co/datasets/TIGER-Lab/RationalRewards-EvalData-GenAIBench-MMRB2-ERBench"> <img src="https://img.shields.io/badge/Eval%20Dataset-FFB7B2?style=for-the-badge&logo=huggingface&logoColor=black" alt="评测数据集"> </a> <a href="https://huggingface.co/datasets/TIGER-Lab/RationalRewards_DiffusionNFT_TrainData"> <img src="https://img.shields.io/badge/Diffusion%20RL%20Training%20Dataset-FFB7B2?style=for-the-badge&logo=huggingface&logoColor=black" alt="扩散强化学习训练数据集"> </a> </div> **RationalRewards** 是一款面向视觉生成任务、基于推理机制的奖励模型与工具包。与传统将偏好压缩为单一模糊标量的方式不同，它会在给出评分前生成明确的多维度评价反馈，将奖励模型从被动的评估者转变为主动的优化接口。 **名称释义：** "Rational" 意为合乎理性、通情达理，对应中文的「理性的」。 **RationalRewards 支持在互补空间中开展优化：** - **训练阶段优化：** 通过强化学习（RL）结合结构化、可解释的奖励信号实现； - **测试阶段优化：** 通过「生成-评价-细化」循环完成，无需更新模型参数。 ## 核心实验结果 以 Qwen3-VL-Instruct-8B 作为骨干模型并通过 PARROT（Preference-Anchored Rationalization，偏好锚定合理化框架）实例化后，RationalRewards 在开源奖励模型中实现了当前最优的偏好预测性能，且可与 Gemini-2.5-Pro 比肩。作为强化学习奖励信号，它在文生图与图像编辑两类任务中均能持续提升生成器性能，优于单一标量基准方法。尤为值得关注的是，无需参数更新的RationalRewards测试阶段提示词调优策略，在多个基准测试中可达到甚至超越基于强化学习的微调效果。  *基于RationalRewards的训练阶段强化学习与测试阶段提示词调优在视觉生成基准测试中的表现。* ## 为何选择推理式奖励？ 多数奖励模型会将指令遵循、视觉质量、构图合理性与真实性压缩为单一标量评分，这丢弃了人类判断的结构化逻辑，往往导致优化过程不稳定。RationalRewards 保留了这些评价维度的显式性，使生成器能够获得语义锚定的反馈，明确需要修正的内容及原因。 ### 推理式奖励为何能够抵御奖励黑客攻击？ 单一标量奖励极易受到奖励黑客攻击，因为它将丰富的人类判断压缩为单一数值，即便生成结果并未真正提升，该数值仍可能走高。RationalRewards 引入了隐式正则化机制：在给出评分前，必须生成与具体评价维度绑定的连贯多维度评价反馈。这将优化过程约束于有证据支撑的推理逻辑，同时强化了强化学习阶段奖励得分与实际生成质量之间的单调相关性。 ### 基于偏好训练的奖励为何比通用视觉语言模型（VLM）评判器更稳定？ 通用视觉语言模型（VLM）评判器可以成为优秀的分析者，但作为奖励函数时，它们在语义相似的样本上往往会出现高方差的逐点评分。这种方差会成为强化学习中的优化噪声。PARROT 直接针对偏好区分任务训练 RationalRewards，从而生成低方差、与偏好对齐的评分。实际效果表现为优化过程更稳定，奖励可靠性更强，即便模型参数量更小。 ### 推理式奖励为何能够支持测试阶段扩展？ 推理反馈不仅可在训练阶段使用，还可在生成完成后复用。在「生成-评价-细化」循环中，RationalRewards 会对生成的图像进行评价，识别具体缺陷，并提出针对性的提示词更新方案。与预先盲目改写的提示词增强不同，该策略属于事后响应式优化，针对实际生成失败进行调整。这使得测试阶段的计算能够更有效地激发生成器的潜在能力，在无需更新参数的情况下，往往能达到甚至超越强化学习微调的效果提升。  *RationalRewards 支持在参数空间（强化学习）与提示词空间（测试阶段细化）中开展优化。* ## 方法：偏好锚定合理化框架（PARROT, Preference-Anchored Rationalization） 人工标注推理反馈成本高昂。PARROT 可通过三个阶段从仅包含偏好标注的数据中恢复高质量的推理监督信号： 1. **锚定生成：** 教师视觉语言模型生成与已知标签一致的推理候选结果。 2. **一致性过滤：** 移除幻觉生成或无预测价值的推理内容。 3. **知识蒸馏：** 学生模型在未接触标签数据的情况下，学习「先评价、后评分」的范式。 该方法为从海量偏好数据集到可扩展的推理监督信号提供了切实可行的路径。  *PARROT 流程：锚定推理生成、一致性过滤与知识蒸馏。* ## 实验验证 RationalRewards 同时提升了对齐质量与下游优化效果。  *在开源奖励模型中实现当前最优的偏好预测性能。*  *与仅使用标量奖励的方法相比，结构化评价通道减少了捷径利用现象。* 为更清晰地展示优化行为，我们还提供了扩散强化学习训练的演化结果。下图可视化了RationalRewards引导的训练随时间推移的性能提升过程，表明其收益不仅体现在最终的模型 checkpoint 中，而是在整个训练过程中持续显现。这与单一标量奖励易受奖励黑客攻击的现象形成鲜明对比，具体可参见论文中的图12。  *基于RationalRewards优化的扩散强化学习性能演化曲线。*  *测试阶段的「生成-评价-细化」策略在多个基准测试中可达到甚至超越强化学习微调效果。*  *由显式推理反馈支持的更多定性应用案例。* ## 引用 bibtex @article{rationalrewards2026, title = {RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time}, author = {Haozhe Wang and Cong Wei and Weiming Ren and Jiaming Liu and Fangzhen Lin and Wenhu Chen}, journal = {arXiv preprint}, year = {2026} }