Echoverse

# Echoverse <div align="center"> <a href="https://aka.ms/echoverse-techreport"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b" alt="Paper"></a> <a href="https://aka.ms/echoverse-techreport"><img src="https://img.shields.io/badge/Blog-Post-6f42c1" alt="Blog"></a> <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-blue" alt="License"></a> </div> Synthetic **computer-use / web-agent** evaluation environments. Each *environment* is a self-contained world that drills specific UI skills; an agent is given a natural-language `goal` and must drive the environment's UI to the target state, graded against a `reference_state_change` or `reference_answer`. This dataset bundles, for every environment: - **`<env>/test_tasks.parquet`** — the shipped test tasks (previewable in the dataset viewer, with per-column statistics and distribution graphs). - **`<env>/<grounding>.db`** — the grounding **SQLite** database each task runs against (LFS binary asset; not shown in the viewer). Use the **config dropdown** at the top of the viewer to switch between environments. ## Environments | Config | Focus | Tasks | Grounding DB | |---|---|---|---| | `datepickers` | Date-picker widgets across ~10 everyday contexts | 109 | `datepicker_grounded.db` | | `datepickers_ood` | Out-of-distribution date-picker scenarios | 150 | `datepicker_ood_grounded.db` | | `echoforge` | Project/entity management over a large app DB | 101 | `echoforge.db` | | `echostay` | Lodging/stay management workflows | 117 | `echostay.db` | | `nested_filter` | Nested filter-widget selection | 100 | `nested_filter.db` | | `nested_filter_ood` | Out-of-distribution nested-filter scenarios | 145 | `nested_filter_ood.db` | ## Columns Scalar fields are stored as typed columns (so the viewer shows value distributions); nested objects are stored as **JSON strings** to preserve their structure without breaking the stats widget. Common columns: `id`, `goal`, `task_type` (`read`/`write`), `difficulty`, `category`, `reference_answer`, `reference_state_change`, `db_file`. Environment-specific columns include: `env_name`, `datepicker_type`, `style`, `canonical_answer` (datepickers); `estimated_actions`, `verification_query`, `metadata` (echoforge / echostay); `widget_family`, `widget_id`, `widget_category`, `canonical_filter_state` (nested_filter). JSON-string columns: `extra_website_args`, `metadata`, `canonical_filter_state`. ## Loading ```python from datasets import load_dataset # one environment ds = load_dataset("microsoft/Echoverse", "echoforge", split="test") print(ds[0]) ``` ## Grounding databases Each environment's SQLite grounding DB is stored alongside its tasks and can be downloaded with the `hf` CLI (requires access to this private dataset): ```bash # a single environment's grounding DB hf download microsoft/Echoverse echoforge/echoforge.db \ --repo-type dataset --local-dir ./echoverse_data # datepickers hf download microsoft/Echoverse datepickers/datepicker_grounded.db \ --repo-type dataset --local-dir ./echoverse_data ``` Each task's `db_file` field names the grounding DB it expects. ## Responsible AI These environments and tasks are released for **research purposes** — they are fictional, synthetic worlds for research on and evaluation of computer-use agents, and are not affiliated with or endorsed by any third party. Before using them, please read the [Transparency Note](https://github.com/microsoft/Echoverse/blob/main/TRANSPARENCY.md), which describes intended uses, out-of-scope uses, limitations, and best practices. ## License Released under the [MIT License](https://opensource.org/licenses/MIT).