omnilingual-gaia2

# OmnilingualGAIA2 OmnilingualGAIA2 is the multilingual extension of the [GAIA2](https://huggingface.co/datasets/meta-agents-research-environments/gaia2-cli) agentic benchmark: `6400` scenarios covering 10 target languages, each machine-translated from its English reference by a pipeline that enforces a cross-surface preservation contract. For the methodology, per-language quality analysis, and the evaluation of frontier agents on this benchmark, see [the paper](https://arxiv.org/abs/2608.08775). ## Layout One config per (language, subset) pair, a single `test` split each, with the same columns as [`meta-agents-research-environments/gaia2-cli`](https://huggingface.co/datasets/meta-agents-research-environments/gaia2-cli): | column | description | |---|---| | `scenario_id` | Stable GAIA2 scenario identifier, shared across languages | | `scenario` | Full scenario definition as a JSON string (see below) | | `language` | ISO 639-3 code | | `script` | ISO 15924 code | | `subset` | GAIA2 capability subset | ```python import json from datasets import load_dataset ds = load_dataset("facebook/omnilingual-gaia2", "spa_Latn_execution", split="test") scenario = json.loads(ds[0]["scenario"]) ``` `scenario` is the standard GAIA2 scenario object, with top-level keys `metadata` (scenario id, duration, start time, hints, tags), `apps` (the serialized app universe the agent acts on), `events` (the oracle event DAG), `version`, and `augmentation`. Scenario IDs are aligned across languages, so any two configs sharing a subset can be compared row for row. ## Coverage | language | execution | search | ambiguity | adaptability | total | |---|---|---|---|---|---| | `cmn_Hans` | 160 | 160 | 160 | 160 | 640 | | `deu_Latn` | 160 | 160 | 160 | 160 | 640 | | `fra_Latn` | 160 | 160 | 160 | 160 | 640 | | `hin_Deva` | 160 | 160 | 160 | 160 | 640 | | `ind_Latn` | 160 | 160 | 160 | 160 | 640 | | `ita_Latn` | 160 | 160 | 160 | 160 | 640 | | `jpn_Jpan` | 160 | 160 | 160 | 160 | 640 | | `por_Latn` | 160 | 160 | 160 | 160 | 640 | | `spa_Latn` | 160 | 160 | 160 | 160 | 640 | | `tur_Latn` | 160 | 160 | 160 | 160 | 640 | ## Running the benchmark Materialize a config into a `dataset_root` of scenario JSON files, which the [gaia2-cli runner](https://github.com/facebookresearch/meta-agents-research-environments) consumes directly: ```python import json, pathlib from datasets import load_dataset config = "spa_Latn_execution" dest = pathlib.Path.home() / "datasets" / "omnilingual-gaia2" / config dest.mkdir(parents=True, exist_ok=True) for row in load_dataset("facebook/omnilingual-gaia2", config, split="test"): (dest / f"{row['scenario_id']}.json").write_text(row["scenario"]) ``` ```bash gaia2-runner run-dataset --dataset ~/datasets/omnilingual-gaia2/spa_Latn_execution ... ``` **Note on the verifier**: Scoring non-English trajectories with the stock English verifier is **not** a fair comparison. We are shipping the multilingual verifier used for the paper's numbers in a PR on top of [Meta Agents Research Environments](https://github.com/facebookresearch/meta-agents-research-environments); until it lands, treat cross-language comparisons produced with the default judge as indicative only. ## Provenance and limitations The source dataset is [meta-agents-research-environments/gaia2-cli](https://huggingface.co/datasets/meta-agents-research-environments/gaia2-cli). Translations are machine-generated with a translation pipeline involving [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) and have **not** been fully human-verified. ## Citation ```bibtex @misc{caciolai2026omnilingualgaia2evaluatingmultilingualgap, title={OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents}, author={Andrea Caciolai and Pere-Lluís Huguet Cabot and Chierh Cheng and Albert Ventayol-Boada and Gabriel Mejia Gonzalez and Christophe Ropers and Lucas Bandarkar and Sebastian Ruder and Darlene Sakakihara and Elliot Yun and Pierre Andrews and Grégoire Mialon and Romain Froger and Marta R. Costa-jussà}, year={2026}, eprint={2608.08775}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2608.08775}, } ```