zlab-princeton/UEval

Name: zlab-princeton/UEval
Creator: zlab-princeton
Published: 2026-06-14T03:59:13+08:00

Hugging Face2026-02-02 更新2026-06-14 收录

多模态数据评估

统一模型基准测试

数据链接：

https://hf-mirror.com/datasets/zlab-princeton/UEval 数据链接链接失效反馈

官方服务：

资源简介：

--- dataset_info: features: - name: id dtype: string - name: task dtype: string - name: prompt dtype: string - name: image_rubrics sequence: string - name: text_rubrics sequence: string - name: image_ref sequence: image - name: text_ref dtype: string splits: - name: test num_bytes: 301995195.0 num_examples: 1000 download_size: 292190640 dataset_size: 301995195.0 configs: - config_name: default data_files: - split: test path: data/test-* --- # UEval: A Benchmark for Unified Multimodal Generation > [**UEval: A Benchmark for Unified Multimodal Generation**](https://arxiv.org/abs/2502.12150) > *[Bo Li](https://primerl.github.io/), [Yida Yin](https://davidyyd.github.io), [Wenhao Chai](https://wenhaochai.com/), [Xingyu Fu](https://zeyofu.github.io/)\*, [Zhuang Liu](https://liuzhuang13.github.io)\* (* indicates co-advising) > Princeton University > [[Paper]](https://arxiv.org/abs/2601.22155) [[Project page]](https://zlab-princeton.github.io/UEval/) [[Code]](https://github.com/zlab-princeton/UEval) --- <img src="https://github.com/user-attachments/assets/f66379a8-c571-4cba-8e9b-78d158ecd26c" width=100% height=100% class="center"> We introduce **UEval**, a benchmark to evaluate unified models, i.e., models capable of generating both images and text. UEval comprises 1,000 expert-curated prompts that require both images and text in the model outputs, sourced from 8 diverse real-world domains. ## Results We evaluate recent unified models on all 8 tasks in our benchmark. Overall, frontier models consistently outperform open-source ones across all tasks: GPT-5-Thinking achieves the highest average score of 66.4, while the best open-source model obtains only 49.1. The gap between proprietary and open-source models is very large: the strongest frontier model (e.g., GPT-5-Thinking) outperforms the best open-source model (e.g., Emu 3.5) by over 17 points on average. | Model | Space | Textbook | Diagram | Paper | Art | Life | Tech | Exercise | Avg | | ------------------------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | | *Reference* | 96.2 | 94.4 | 93.1 | 96.2 | 90.6 | 87.7 | 90.6 | 89.2 | 92.2 | | Janus-Pro | 21.0 | 31.0 | 37.4 | 15.2 | 26.4 | 23.0 | 17.6 | 11.5 | 22.9 | | Show-o2 | 25.4 | 33.1 | 33.2 | 17.4 | 25.6 | 15.6 | 17.4 | 13.1 | 22.6 | | MMaDA | 10.8 | 20.0 | 14.2 | 13.3 | 15.7 | 15.8 | 12.4 | 12.6 | 14.4 | | BAGEL | 29.8 | 42.5 | 37.2 | 20.0 | 39.0 | 33.6 | 24.8 | 21.4 | 31.0 | | Emu3.5 | **59.1** | **57.4** | **41.1** | **31.6** | **59.3** | **62.0** | **37.0** | **45.4** | **49.1** | | Gemini-2.0-Flash | 65.2 | 55.2 | 47.6 | 45.8 | **70.4** | 58.0 | 50.2 | 48.0 | 55.1 | | Gemini-2.5-Flash | 78.0 | 74.0 | 66.4 | **71.6** | 66.6 | 63.0 | **58.2** | 50.0 | 66.0 | | GPT-5-Instant | 77.3 | 77.9 | 62.3 | 55.1 | 71.2 | **69.7** | 50.7 | 57.6 | 65.2 | | GPT-5-Thinking | **84.0** | **78.0** | **67.8** | 51.9 | 67.8 | 63.8 | 57.0 | **61.4** | **66.4** | <img width="1368" height="1417" alt="Image" src="https://github.com/user-attachments/assets/ede55cbf-fffd-4278-8fed-34d8c95596a8" /> <img width="1398" height="1542" alt="Image" src="https://github.com/user-attachments/assets/add410df-b267-4693-8a3d-b9c8e45108fb" /> ## Citation If you find this repository helpful, please consider citing: ```bibtex @article{li2026ueval, title = {UEval: A Benchmark for Unified Multimodal Generation}, author = {Li, Bo and Yin, Yida and Chai, Wenhao and Fu, Xingyu and Liu, Zhuang}, journal = {arXiv preprint arXiv:2601.22155}, year = {2026} } ```

--- 数据集信息：特征： - 名称：id 数据类型：字符串 - 名称：task 数据类型：字符串 - 名称：prompt 数据类型：字符串 - 名称：image_rubrics 数据类型：字符串序列 - 名称：text_rubrics 数据类型：字符串序列 - 名称：image_ref 数据类型：图像序列 - 名称：text_ref 数据类型：字符串划分： - 名称：测试集字节数：301995195.0 样本数量：1000 下载大小：292190640 数据集大小：301995195.0 配置项： - 配置名称：default 数据文件： - 划分：测试集路径：data/test-* --- # UEval：统一多模态生成基准 > [**UEval：统一多模态生成基准**](https://arxiv.org/abs/2502.12150) > *[Bo Li](https://primerl.github.io/)、[Yida Yin](https://davidyyd.github.io)、[Wenhao Chai](https://wenhaochai.com/)、[Xingyu Fu](https://zeyofu.github.io/)*、[Zhuang Liu](https://liuzhuang13.github.io)*（* 表示共同指导） > 普林斯顿大学 > [[论文]](https://arxiv.org/abs/2601.22155) [[项目主页]](https://zlab-princeton.github.io/UEval/) [[代码]](https://github.com/zlab-princeton/UEval) --- <img src="https://github.com/user-attachments/assets/f66379a8-c571-4cba-8e9b-78d158ecd26c" width=100% height=100% class="center"> 我们提出**UEval**，一款用于评估统一模型的基准测试集——这类模型可同时生成图像与文本。UEval包含1000条由专家精心甄选的提示词，这些提示词要求模型输出同时涵盖图像与文本内容，其来源覆盖8个多样化的真实世界领域。 ## 实验结果我们在该基准测试的全部8项任务上评估了近期推出的统一模型。总体而言，前沿闭源模型在所有任务上的表现始终优于开源模型：GPT-5-Thinking取得最高平均得分66.4，而表现最佳的开源模型仅获得49.1。闭源与开源模型之间的性能差距悬殊：表现最优的前沿模型（如GPT-5-Thinking）在平均得分上比表现最佳的开源模型（如Emu 3.5）高出17分以上。 | 模型名称 | 场景 | 教科书 | 示意图 | 学术论文 | 艺术创作 | 日常生活 | 技术场景 | 习题任务 | 平均得分 | | ------------------------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | | *参考基准* | 96.2 | 94.4 | 93.1 | 96.2 | 90.6 | 87.7 | 90.6 | 89.2 | 92.2 | | Janus-Pro | 21.0 | 31.0 | 37.4 | 15.2 | 26.4 | 23.0 | 17.6 | 11.5 | 22.9 | | Show-o2 | 25.4 | 33.1 | 33.2 | 17.4 | 25.6 | 15.6 | 17.4 | 13.1 | 22.6 | | MMaDA | 10.8 | 20.0 | 14.2 | 13.3 | 15.7 | 15.8 | 12.4 | 12.6 | 14.4 | | BAGEL | 29.8 | 42.5 | 37.2 | 20.0 | 39.0 | 33.6 | 24.8 | 21.4 | 31.0 | | Emu3.5 | **59.1** | **57.4** | **41.1** | **31.6** | **59.3** | **62.0** | **37.0** | **45.4** | **49.1** | | Gemini-2.0-Flash | 65.2 | 55.2 | 47.6 | 45.8 | **70.4** | 58.0 | 50.2 | 48.0 | 55.1 | | Gemini-2.5-Flash | 78.0 | 74.0 | 66.4 | **71.6** | 66.6 | 63.0 | **58.2** | 50.0 | 66.0 | | GPT-5-Instant | 77.3 | 77.9 | 62.3 | 55.1 | 71.2 | **69.7** | 50.7 | 57.6 | 65.2 | | GPT-5-Thinking | **84.0** | **78.0** | **67.8** | 51.9 | 67.8 | 63.8 | 57.0 | **61.4** | **66.4** | <img width="1368" height="1417" alt="Image" src="https://github.com/user-attachments/assets/ede55cbf-fffd-4278-8fed-34d8c95596a8" /> <img width="1398" height="1542" alt="Image" src="https://github.com/user-attachments/assets/add410df-b267-4693-8a3d-b9c8e45108fb" /> ## 引用说明若您认为本仓库对您的研究有所帮助，请引用如下文献： bibtex @article{li2026ueval, title = {UEval: A Benchmark for Unified Multimodal Generation}, author = {Li, Bo and Yin, Yida and Chai, Wenhao and Fu, Xingyu and Liu, Zhuang}, journal = {arXiv preprint arXiv:2601.22155}, year = {2026} }

提供机构：

zlab-princeton

zlab-princeton/UEval

资源简介：

相关数据集