cua-verse/Aguvis

# cua-verse/Aguvis

cua-lite preprocessed version of Aguvis (xlangai/aguvis-stage1 + aguvis-stage2) merged into one repo. Stage-1 contributes grounding:action sub-datasets (OmniAct, RICO, UI-RefExp, GUIEnv, SeeClick, WebUI); stage-2 contributes trajectory data (AITW, Android-Control, CoAT, GUIDE, MiniWoB). Both stages share the unified cua-lite SFT schema; the original stage boundary is preserved in metadata.others.

## Origin

- [https://huggingface.co/datasets/xlangai/aguvis-stage1](https://huggingface.co/datasets/xlangai/aguvis-stage1)
- [https://huggingface.co/datasets/xlangai/aguvis-stage2](https://huggingface.co/datasets/xlangai/aguvis-stage2)

## Schema

Each row has these columns:

| column | type | notes |
|---|---|---|
| `image_ids` | list[string] | content-addressed ids (`<sha256>.<ext>`), used for cross-parquet dedup |
| `images` | list[Image] | bytes embedded at HF push time; matches `image_ids` index-for-index |
| `messages` | list[struct] | OpenAI-style turns with `role` + structured `content` |
| `metadata` | struct | `{platform, task_type, split, others{...}}` |

Coordinate values in `messages` are normalized to `[0, 1000]` integers.

## Layout

```
<platform>/<task_type>/<split>/<variant>.parquet
```

- `platform` ∈ {desktop, mobile, web}
- `task_type` ∈ {understanding, grounding:action, grounding:point, grounding:bbox, trajectory}
- `split` ∈ {train, eval} — `eval` is an in-distribution held-out slice (not used in training)

## Stats

| platform | task_type | variant | train | eval |
|---|---|---|---:|---:|
| desktop | grounding:action | omniact | 5,392 | 99 |
| mobile | grounding:action | ricoig16k | 15,774 | 359 |
| mobile | grounding:action | ricosca | 171,212 | 2,000 |
| mobile | grounding:action | ui_refexp | 15,268 | 356 |
| mobile | grounding:action | widget_cap | 99,485 | 1,940 |
| mobile | trajectory | aitw | 1,698 | 30 |
| mobile | trajectory | android_control | 12,318 | 263 |
| mobile | trajectory | coat | 1,306 | 23 |
| mobile | trajectory | guide | 595 | 12 |
| web | grounding:action | guienv | 325,972 | 2,000 |
| web | grounding:action | seeclick | 269,121 | 2,000 |
| web | grounding:action | seeclick_mi | 269,119 | 2,000 |
| web | grounding:action | webui | 56,302 | 1,087 |
| web | trajectory | miniwob | 1,775 | 31 |

## Image store

Images are content-addressed by SHA-256 and deduplicated within this repo.
The mapping `image_id → bytes` is embedded in every parquet via
[`datasets.Image`](https://huggingface.co/docs/datasets/image_load). The
`reverse.py` script in [cua-lite/cua-lite](https://github.com/cua-lite/cua-lite/tree/main/scripts/hf_upload)
reconstructs a local `image_store/` + reference-only parquets, letting you
re-dedup across datasets on your own machine.

- Total images: **516,962**
- Store size: **209.74 GB**

## Notes

Sub-datasets (variants) may have heterogeneous source licenses. See metadata.others.source for provenance. Many Aguvis sub-datasets overlap with standalone cua-lite datasets (Mind2Web, AMEX, etc.); deduplicate before mixing for training.

## License & Citation

See original datasets (xlangai/aguvis-stage1 and aguvis-stage2)

See https://aguvis-project.github.io/