有没有相关的论文或文献参考？
这个数据集是基于什么背景创建的？
数据集的作者是谁？
能帮我联系到这个数据集的作者吗？
这个数据集如何下载？
UI-Genie-RM-517k, UI-Genie-Agent-16k|图形用户界面代理数据集|机器学习数据集
收藏arXiv2025-05-28 更新2025-05-29 收录
下载链接：
https://github.com/Euphoria16/UI-Genie
下载链接
链接失效反馈资源简介：
UI-Genie是一个自我改进框架，旨在解决GUI代理中的两个关键挑战：轨迹结果的验证具有挑战性，高质量的训练数据不可扩展。该框架通过奖励模型和自我改进流程分别解决这两个挑战。奖励模型UI-Genie-RM具有图像-文本交错架构，可以有效地处理历史上下文并统一动作级别和任务级别的奖励。为了支持UI-Genie-RM的训练，我们开发了精心设计的生成数据策略，包括基于规则的验证、受控轨迹破坏和硬负挖掘。为了解决第二个挑战，自我改进流程通过在动态环境中通过奖励引导的探索和结果验证来增强代理和奖励模型，逐步扩展可解决的复杂GUI任务。为了训练模型，我们生成了UI-Genie-RM-517k和UI-Genie-Agent-16k，建立了第一个专为GUI代理设计的奖励特定数据集，同时展示了无需手动注释即可生成高质量合成轨迹的能力。实验结果表明，UI-Genie在多个GUI代理基准测试中实现了最先进的性能，具有三代数据-模型自我改进。我们开源了我们的完整框架实现和生成的数据集，以促进GUI代理的进一步研究。
提供机构：
香港中文大学多媒体实验室 & vivo人工智能实验室
创建时间：
2025-05-28
原始信息汇总
UI-Genie数据集概述
数据集基本信息
- 名称: UI-Genie
- 类型: 移动GUI代理训练数据集
- 开发框架: 自改进框架(MLLM-based)
- 核心创新: 通过迭代代理-奖励模型协同进化消除人工标注需求
关键特性
- 无人工标注的合成轨迹生成
- 代理与奖励模型协同进化
- 自动化高质量数据集生成
- 跨多个基准测试达到SOTA性能
数据集组成
| 数据集名称 | 规模 | 描述 | 状态 |
|---|---|---|---|
| UI-Genie-RM-517k | 517K | 首个GUI代理奖励数据集 | 即将发布 |
| UI-Genie-Agent-16k | 16K | 高质量合成轨迹数据集 | 已发布(含5K样本) |
模型性能
代理模型
| 模型 | 参数量 | AndroidControl-Low(SR) | AndroidControl-High(SR) | AndroidLab(SR) | Android Arena(SR) |
|---|---|---|---|---|---|
| UI-Genie-Agent | 3B | 93.8 | 72.9 | 28.8 | - |
| UI-Genie-Agent | 7B | 94.3 | 74.2 | 38.7 | 20.4 |
| UI-Genie-Agent | 72B | 94.8 | 77.0 | 41.2 | - |
奖励模型
| 模型 | 参数量 | Step-Level F1 | Outcome-Level F1 |
|---|---|---|---|
| UI-Genie-RM | 7B | 79.6 | 82.1 |
数据来源
- AndroidControl训练集
- AMEX训练集
- AndroidLab训练集
- UI-Genie-Agent-16k合成数据集
相关资源
- 论文: https://arxiv.org/abs/2505.21496
- 模型: https://huggingface.co/HanXiao1999/UI-Genie-Agent-7B
- 数据集: https://huggingface.co/datasets/HanXiao1999/UI-Genie-Agent-5k
AI搜集汇总
数据集介绍
构建方式
UI-Genie数据集的构建采用了创新的自增强框架，通过奖励模型（UI-Genie-RM）和自增强流程协同工作。具体而言，奖励模型采用图像-文本交错架构，有效处理历史上下文并统一动作级和任务级奖励。数据生成策略包括基于规则的验证、受控轨迹破坏和硬负样本挖掘，确保了数据的高质量和多样性。此外，通过奖励引导的探索和动态环境中的结果验证，逐步扩展可解决的复杂GUI任务，生成了UI-Genie-RM-517k和UI-Genie-Agent-16k两个数据集，无需人工标注。
特点
UI-Genie数据集具有以下显著特点：首先，它是首个专门为GUI代理设计的奖励数据集，包含517k个奖励数据样本，涵盖动作级和任务级奖励。其次，数据集通过合成轨迹生成策略确保了高质量的数据，避免了人工标注的繁琐和成本。此外，数据集具有高度的多样性和复杂性，覆盖了多种GUI任务和交互场景，适用于训练和评估多模态大语言模型（MLLM）在GUI代理任务中的表现。最后，数据集的构建过程具有可扩展性，能够通过自增强框架不断优化和扩展。
使用方法
UI-Genie数据集的使用方法主要包括以下几个方面：首先，奖励数据集（UI-Genie-RM-517k）可用于训练和评估GUI代理的奖励模型，通过动作级和任务级奖励的联合优化提升代理的交互能力。其次，代理数据集（UI-Genie-Agent-16k）可用于训练GUI代理模型，通过合成轨迹数据增强代理的泛化能力和任务完成率。此外，数据集还可用于评估GUI代理在静态和动态环境中的表现，支持多种评估指标（如成功率、动作类型准确率等）。最后，数据集的开源框架和实现细节为后续研究提供了便利，支持进一步的模型优化和应用扩展。
背景与挑战
背景概述
UI-Genie-RM-517k和UI-Genie-Agent-16k是由CUHK MMLab和vivo AI Lab于2025年联合推出的多模态大型语言模型（MLLM）数据集，旨在解决移动图形用户界面（GUI）智能体训练中的关键挑战。该数据集通过创新的自改进框架生成高质量合成轨迹，无需人工标注，显著提升了GUI智能体的轨迹验证和训练数据扩展能力。其核心研究问题聚焦于如何通过奖励模型和自改进流程，有效解决GUI智能体领域中的轨迹验证难题和高质量训练数据的可扩展性问题。这一突破性工作为移动GUI智能体的发展提供了首个专用奖励数据集，并在多个基准测试中达到了最先进的性能水平。
当前挑战
UI-Genie数据集面临的主要挑战体现在两个维度：领域问题挑战方面，GUI智能体的轨迹验证具有独特复杂性，其完成状态高度依赖历史上下文，现有评估框架难以提供准确的中间动作验证；构建过程挑战方面，缺乏可靠的验证方法导致当前训练仍依赖人工标注的操作轨迹，这些数据存在耗时、昂贵且难以扩展的固有缺陷。具体而言，构建过程中需克服三大技术难题：基于规则的验证需要精确匹配动作类型、坐标精度和语义一致性；受控轨迹破坏需通过早期截断、跨任务替换和冗余延续等机制生成负样本；硬负样本挖掘则需识别初始奖励模型误分类的样本以增强模型鲁棒性。
常用场景
经典使用场景
UI-Genie-RM-517k和UI-Genie-Agent-16k数据集在移动GUI代理领域具有广泛的应用场景。这些数据集主要用于训练和评估基于多模态大语言模型（MLLM）的GUI代理，特别是在移动设备上的自动化任务执行。经典使用场景包括移动应用的自动化测试、用户界面交互的模拟、以及复杂多步骤任务的自动化执行。通过提供高质量的合成轨迹数据和专门的奖励模型数据，这些数据集为研究者提供了强大的工具来开发和优化GUI代理。
实际应用
在实际应用中，UI-Genie-RM-517k和UI-Genie-Agent-16k数据集可以用于开发智能助手、自动化测试工具和辅助技术。例如，这些数据集可以用于训练代理来自动化常见的移动应用操作，如日历管理、电子邮件处理和社交媒体互动。此外，它们还可以用于开发辅助技术，帮助残障人士更轻松地使用移动设备。这些应用场景展示了数据集在现实世界中的广泛潜力和实用价值。
衍生相关工作
UI-Genie-RM-517k和UI-Genie-Agent-16k数据集衍生了一系列相关的研究工作。例如，基于这些数据集的研究可以探索如何进一步优化奖励模型的架构，以提高轨迹评估的准确性。此外，这些数据集还可以用于研究如何将自改进管道应用于其他领域的代理开发，如桌面应用或网页自动化。这些衍生工作不仅扩展了数据集的应用范围，还为GUI代理研究的未来发展提供了新的方向。
以上内容由AI搜集并总结生成
数据主题
5,000+
优质数据集
54 个
任务类型
进入经典数据集
热门数据集
中国区域交通网络数据集
该数据集包含中国各区域的交通网络信息，包括道路、铁路、航空和水路等多种交通方式的网络结构和连接关系。数据集详细记录了各交通节点的位置、交通线路的类型、长度、容量以及相关的交通流量信息。
data.stats.gov.cn 收录
UniProt
UniProt（Universal Protein Resource）是全球公认的蛋白质序列与功能信息权威数据库，由欧洲生物信息学研究所（EBI）、瑞士生物信息学研究所（SIB）和美国蛋白质信息资源中心（PIR）联合运营。该数据库以其广度和深度兼备的蛋白质信息资源闻名，整合了实验验证的高质量数据与大规模预测的自动注释内容，涵盖从分子序列、结构到功能的全面信息。UniProt核心包括注释详尽的UniProtKB知识库（分为人工校验的Swiss-Prot和自动生成的TrEMBL），以及支持高效序列聚类分析的UniRef和全局蛋白质序列归档的UniParc。其卓越的数据质量和多样化的检索工具，为基础研究和药物研发提供了无可替代的支持，成为生物学研究中不可或缺的资源。
www.uniprot.org 收录
Canadian Census
**Overview** The data package provides demographics for Canadian population groups according to multiple location categories: Forward Sortation Areas (FSAs), Census Metropolitan Areas (CMAs) and Census Agglomerations (CAs), Federal Electoral Districts (FEDs), Health Regions (HRs) and provinces. **Description** The data are available through the Canadian Census and the National Household Survey (NHS), separated or combined. The main demographic indicators provided for the population groups, stratified not only by location but also for the majority by demographical and socioeconomic characteristics, are population number, females and males, usual residents and private dwellings. The primary use of the data at the Health Region level is for health surveillance and population health research. Federal and provincial departments of health and human resources, social service agencies, and other types of government agencies use the information to monitor, plan, implement and evaluate programs to improve the health of Canadians and the efficiency of health services. Researchers from various fields use the information to conduct research to improve health. Non-profit health organizations and the media use the health region data to raise awareness about health, an issue of concern to all Canadians. The Census population counts for a particular geographic area representing the number of Canadians whose usual place of residence is in that area, regardless of where they happened to be on Census Day. Also included are any Canadians who were staying in that area on Census Day and who had no usual place of residence elsewhere in Canada, as well as those considered to be 'non-permanent residents'. National Household Survey (NHS) provides demographic data for various levels of geography, including provinces and territories, census metropolitan areas/census agglomerations, census divisions, census subdivisions, census tracts, federal electoral districts and health regions. In order to provide a comprehensive overview of an area, this product presents data from both the NHS and the Census. NHS data topics include immigration and ethnocultural diversity; aboriginal peoples; education and labor; mobility and migration; language of work; income and housing. 2011 Census data topics include population and dwelling counts; age and sex; families, households and marital status; structural type of dwelling and collectives; and language. The data are collected for private dwellings occupied by usual residents. A private dwelling is a dwelling in which a person or a group of persons permanently reside. Information for the National Household Survey does not include information for collective dwellings. Collective dwellings are dwellings used for commercial, institutional or communal purposes, such as a hotel, a hospital or a work camp. **Benefits** - Useful for canada public health stakeholders, for public health specialist or specialized public and other interested parties. for health surveillance and population health research. for monitoring, planning, implementation and evaluation of health-related programs. media agencies may use the health regions data to raise awareness about health, an issue of concern to all canadians. giving the addition of longitude and latitude in some of the datasets the data can be useful to transpose the values into geographical representations. the fields descriptions along with the dataset description are useful for the user to quickly understand the data and the dataset. **License Information** The use of John Snow Labs datasets is free for personal and research purposes. For commercial use please subscribe to the [Data Library](https://www.johnsnowlabs.com/marketplace/) on John Snow Labs website. The subscription will allow you to use all John Snow Labs datasets and data packages for commercial purposes. **Included Datasets** - [Canadian Population and Dwelling by FSA 2011](https://www.johnsnowlabs.com/marketplace/canadian-population-and-dwelling-by-fsa-2011) - This Canadian Census dataset covers data on population, total private dwellings and private dwellings occupied by usual residents by forward sortation area (FSA). It is enriched with the percentage of the population or dwellings versus the total amount as well as the geographical area, province, and latitude and longitude. The whole Canada's population is marked as 100, referring to 100% for the percentages. - [Detailed Canadian Population Statistics by CMAs and CAs 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-cmas-and-cas-2011) - This dataset covers the population statistics of Canada by Census Metropolitan Areas (CMAs) and Census Agglomerations (CAs). It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by FED 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-fed-2011) - This dataset covers the population statistics of Canada from 2011 by Federal Electoral District of 2013 Representation Order. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by Health Region 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-health-region-2011) - This dataset covers the population statistics of Canada by health region. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. - [Detailed Canadian Population Statistics by Province 2011](https://www.johnsnowlabs.com/marketplace/detailed-canadian-population-statistics-by-province-2011) - This dataset covers the population statistics of Canada by provinces and territories. It is categorized also by citizen/immigration status, ethnic origin, religion, mobility, education, language, work, housing, income etc. There is detailed characteristics categorization within these stated categories that are in 5 layers. **Data Engineering Overview** **We deliver high-quality data** - Each dataset goes through 3 levels of quality review - 2 Manual reviews are done by domain experts - Then, an automated set of 60+ validations enforces every datum matches metadata & defined constraints - Data is normalized into one unified type system - All dates, unites, codes, currencies look the same - All null values are normalized to the same value - All dataset and field names are SQL and Hive compliant - Data and Metadata - Data is available in both CSV and Apache Parquet format, optimized for high read performance on distributed Hadoop, Spark & MPP clusters - Metadata is provided in the open Frictionless Data standard, and its every field is normalized & validated - Data Updates - Data updates support replace-on-update: outdated foreign keys are deprecated, not deleted **Our data is curated and enriched by domain experts** Each dataset is manually curated by our team of doctors, pharmacists, public health & medical billing experts: - Field names, descriptions, and normalized values are chosen by people who actually understand their meaning - Healthcare & life science experts add categories, search keywords, descriptions and more to each dataset - Both manual and automated data enrichment supported for clinical codes, providers, drugs, and geo-locations - The data is always kept up to date – even when the source requires manual effort to get updates - Support for data subscribers is provided directly by the domain experts who curated the data sets - Every data source’s license is manually verified to allow for royalty-free commercial use and redistribution. **Need Help?** If you have questions about our products, contact us at [info@johnsnowlabs.com](mailto:info@johnsnowlabs.com).
Databricks 收录
中国近海台风路径集合数据集(1945-2024)
1945-2024年度，中国近海台风路径数据集，包含每个台风的真实路径信息、台风强度、气压、中心风速、移动速度、移动方向。 数据源为获取温州台风网(http://www.wztf121.com/)的真实观测路径数据，经过处理整合后形成文件，如使用csv文件需使用文本编辑器打开浏览，否则会出现乱码，如要使用excel查看数据，请使用xlsx的格式。
国家海洋科学数据中心 收录
MedDialog
MedDialog数据集（中文）包含了医生和患者之间的对话（中文）。它有110万个对话和400万个话语。数据还在不断增长，会有更多的对话加入。原始对话来自好大夫网。
github 收录