OpenDFM/MobA-MobBench

--- license: cc-by-nc-sa-4.0 language: - en - zh tags: - GUI - Agent size_categories: - n<1K configs: - config_name: default data_files: - split: MobBench path: "MobBench_tasks_release_v1.0.json" --- <div align="center"> <img src="./title.png"></img> <img src="./overview.png" width="500em" ></img> **🎮 MobA manipulates mobile phones just like how you would.** 🌐 [Website](https://github.com/OpenDFM/MobA) | 📃 [Paper](https://arxiv.org/abs/2410.13757/) | 🤗 [MobBench](https://huggingface.co/datasets/OpenDFM/MobA-MobBench) | 🗃️ [Code](https://github.com/OpenDFM/MobA) [简体中文](./README_zh.md) | English </div> ## 🔥 News - **[2024.10.18]** We open-source MobA on [GitHub](https://github.com/OpenDFM/MobA), and our paper is available on [arXiv](https://arxiv.org/abs/2410.13757). ## 📖 Introduction Current mobile assistants are limited by dependence on system APIs or struggle with complex user instructions and diverse interfaces due to restricted comprehension and decision-making abilities. To address these challenges, we propose MobA, a novel Mobile phone Agent powered by multimodal large language models that enhances comprehension and planning capabilities through a sophisticated two-level agent architecture. The high-level Global Agent (GA) is responsible for understanding user commands, tracking history memories, and planning tasks. The low-level Local Agent (LA) predicts detailed actions in the form of function calls, guided by sub-tasks and memory from the GA. Integrating a Reflection Module allows for efficient task completion and enables the system to handle previously unseen complex tasks. MobA demonstrates significant improvements in task execution efficiency and completion rate in real-life evaluations, underscoring the potential of MLLM-empowered mobile assistants. ## 🔧 Deployment > MobA is still under development, and we are keeping updating the code. Please stay tuned! ### System Requirements Make sure you have installed [Android Debug Bridge (ADB)](https://developer.android.google.cn/tools/adb), and you have connected your Android device to your computer. You should be able to see your devides with command `adb devices`. ### Environment Setup ```shell conda create -n moba python=3.12 conda activate moba pip install numpy opencv-python openai generativeai pillow colorama ``` You may also use `requirements.txt` to install the required packages (However it is not recommended since there are many unused packages). ### Run MobA You need to specify the configuration file in `config.yaml` before running MobA. You can find the configuration file in the `moba` folder. ```bash vim ./moba/config.yaml cd ./moba/agent python executor.py ``` You should be able to run MobA smoothly on Windows now. You can find MobBench, the fifty tasks we tested in the paper, on [huggingface](https://huggingface.co/datasets/OpenDFM/MobA-MobBench). ## 📑 Citation If you find our work useful, please cite us! ```bib @misc{zhu2024moba, title={MobA: A Two-Level Agent System for Efficient Mobile Task Automation}, author={Zichen Zhu and Hao Tang and Yansi Li and Kunyao Lan and Yixuan Jiang and Hao Zhou and Yixiao Wang and Situo Zhang and Liangtai Sun and Lu Chen and Kai Yu}, year={2024}, eprint={2410.13757}, archivePrefix={arXiv}, primaryClass={cs.MA}, url={https://arxiv.org/abs/2410.13757}, } ``` ## 📧 Contact Us If you have any questions, please feel free to contact me via email `[e-mail]`.--- license: 知识共享署名-非商业性使用-相同方式共享4.0（CC BY-NC-SA 4.0） language: - 英语 - 汉语 tags: - 图形用户界面（GUI） - 智能体（Agent） size_categories: - 样本数少于1000 configs: - config_name: 默认配置 data_files: - split: MobBench path: "MobBench_tasks_release_v1.0.json" --- <div align="center"> <img src="./title.png"></img> <img src="./overview.png" width="500em"></img> **🎮 MobA 如同人类一般操作手机。** 🌐 [官网](https://github.com/OpenDFM/MobA) | 📃 [论文](https://arxiv.org/abs/2410.13757/) | 🤗 [MobBench数据集](https://huggingface.co/datasets/OpenDFM/MobA-MobBench) | 🗃️ [代码](https://github.com/OpenDFM/MobA) [简体中文](./README_zh.md) | English </div> ## 🔥 最新动态 - **[2024.10.18]** 我们已于 [GitHub](https://github.com/OpenDFM/MobA) 开源MobA项目，相关研究论文已上线 [arXiv](https://arxiv.org/abs/2410.13757) 平台。 ## 📖 研究背景与方案 当前移动智能助手普遍存在两大局限：一是过度依赖系统API，二是由于理解与决策能力受限，难以处理复杂用户指令与多样的界面场景。为解决上述痛点，我们提出MobA——一种基于多模态大语言模型（Multimodal Large Language Model, MLLM）的新型手机智能体，通过精妙的双层智能体架构强化理解与规划能力。其中，高层全局智能体（Global Agent, GA）负责理解用户指令、追踪历史记忆并规划整体任务；低层本地智能体（Local Agent, LA）则在全局智能体的子任务规划与历史记忆引导下，以函数调用的形式预测具体执行动作。此外，系统集成了反思模块，可高效完成任务并支持处理此前未见过的复杂任务。在真实场景评估中，MobA在任务执行效率与完成率上均展现出显著提升，充分彰显了基于多模态大语言模型的移动助手的应用潜力。 ## 🔧 部署指南 > MobA仍处于开发阶段，我们将持续更新代码，敬请期待！ ### 系统要求 请确保已安装 [Android调试桥（Android Debug Bridge, ADB）](https://developer.android.google.cn/tools/adb)，并将Android设备连接至计算机。执行命令`adb devices`后，应可在终端中看到已连接的设备列表。 ### 环境搭建 shell conda create -n moba python=3.12 conda activate moba pip install numpy opencv-python openai generativeai pillow colorama 你也可以通过`requirements.txt`安装所需依赖包（但不推荐，因其中包含大量未使用的冗余组件）。 ### 运行MobA 在运行MobA前，需先在`config.yaml`配置文件中指定相关参数，该配置文件可在`moba`文件夹中找到。 bash vim ./moba/config.yaml cd ./moba/agent python executor.py 目前你已可在Windows系统上流畅运行MobA。论文中用于测试的50个任务组成的MobBench数据集，可在 [Hugging Face](https://huggingface.co/datasets/OpenDFM/MobA-MobBench) 平台获取。 ## 📑 引用说明 若您的研究工作受益于本项目，请引用如下成果： bib @misc{zhu2024moba, title={MobA: A Two-Level Agent System for Efficient Mobile Task Automation}, author={Zichen Zhu and Hao Tang and Yansi Li and Kunyao Lan and Yixuan Jiang and Hao Zhou and Yixiao Wang and Situo Zhang and Liangtai Sun and Lu Chen and Kai Yu}, year={2024}, eprint={2410.13757}, archivePrefix={arXiv}, primaryClass={cs.MA}, url={https://arxiv.org/abs/2410.13757}, } ## 📧 联系方式 若您有任何疑问或建议，欢迎通过邮箱`[e-mail]`与我们取得联系。