omnilingual-gaia2-results

# Omnilingual-GAIA2 Results The graded rollouts behind the [Omnilingual-GAIA2 leaderboard](https://huggingface.co/spaces/facebook/omnilingual-gaia2). ## Access **This dataset is currently private.** Everything below — the clone, the `load_dataset` call, the `hf://` query — works for a reader who has been granted access and is authenticated with an `HF_TOKEN` that carries read permission on this repository, either in the environment or via `hf auth login`. For anyone else the same commands fail with a 401 or a 404, and there is no public mirror. It is written this way rather than hedged so that the failure is expected instead of confusing; if the dataset is later made public the snippets are unchanged and the token stops being needed. The leaderboard itself does not depend on this. The five derived tables are committed in the public Space repository under `data/tables/`, so every published number is readable without access here. What needs access is the rollout grain: re-deriving those tables, reading a trace, or querying below the published aggregation. The parquet files are git-lfs objects (see `.gitattributes`), so a `git clone` needs git-lfs installed; `snapshot_download` and `load_dataset` handle it themselves. ## Layout ``` data/<provider>__<model>__<harness>__<lang>.parquet ``` One file per (submission, language), which is the unit the ingester writes, so a new language adds a shard and rewrites none. Seven submissions across eleven languages is 77 files. `submission` and `lang` are also columns, so this glob is the whole archive: ``` data/*.parquet ``` Languages are FLORES codes, with `eng_Latn` the comparator every cell is scored against: ``` eng_Latn cmn_Hans deu_Latn fra_Latn hin_Deva ind_Latn ita_Latn jpn_Jpan por_Latn spa_Latn tur_Latn ``` Capabilities: ``` execution search ambiguity adaptability ``` ## Schema 22 columns, 147,840 rows, 77 shards, 279 MB. Published numbers come from these twelve: ``` submission scenario_id capability lang run success num_agent_events failure_reason trace_published harness judge_model judge_prompt_version ``` Three describe the shape of the rollout rather than its verdict: ``` num_oracle_events n_turns n_events ``` Four more are what the runs recorded and nothing published reads: ``` agent_response model started_at pass_at ``` And three are JSON strings — heterogeneous shapes, one per tool, per checker and per provider, so parse with `json.loads` on the rows you want rather than expecting a struct: ``` judgments per-turn judge details events the tool-call trace, each entry `{app, fn, args, ret, write, turn, is_agent}` reasoning chain-of-thought, a list of `{seq, turn, reasoning}` ``` Those three plus `agent_response` are 98% of the bytes; `reasoning` alone is 62%. A scan that projects the other eighteen columns never decompresses any of them. ### `num_oracle_events` How many events the scenario's oracle trace contains — the length of the reference solution, and so a rough measure of how much the scenario asks for. It is a property of the **scenario**, not of the rollout: every rollout of one scenario carries the same value, in every language and from every agent. The runner never computed it — it writes a literal `0` on the `judged_in_container` path — so this column was backfilled from the GAIA2 scenario definitions, as the count of `events[]` entries whose `class_name` is `OracleEvent`. It is now authoritative and non-null on all 147,840 rows. Range: min 1, median 7, max 41. One caveat if you derive a rate from it. `search` is uniformly 1 oracle event across all 160 of its scenarios, so any "fraction of oracle events completed" metric degenerates to pass/fail there and is not the same quantity it is on the other three capabilities. ### `n_turns`, `n_events` and truncation `n_turns` is how many turns the user took with the agent, counted off `events.jsonl` as a running count of `AgentUserInterface.send_message_to_agent`. It is non-null on every row. For the six `openclaw` submissions it is lossless. For `moonshotai__kimi-2.6__opencode` it is **partial**: that submission published no trace, so the count is reconstructed from `daemon_judgments.agent_events`, which holds write actions only — roughly 31% of the tool calls, with no return values. It undercounts, and 145 of its rollouts read 0 where the true answer is almost certainly 1. It is published rather than nulled because it is the only turn evidence those rollouts have; treat Kimi's turn counts as a lower bound. `n_events` is the **true, uncapped** event count. The `events` blob itself is capped at 150 entries, so for 3,703 rollouts it is a truncated prefix and `len(events) < n_events`. This column exists so that the truncation is visible instead of silent: counting tool calls with `len(events)` is quietly wrong on exactly the rollouts that did the most work. It is null for Kimi, where an event count off a third of the events would be wrong rather than partial. ### `reasoning` Full chain-of-thought, with no length cap: 1,069,891,434 characters over 1,024,523 records in 63,333 rollouts, the longest single blob 222,665 characters. Present for three submissions only — `google__gemma-4-31b-it__openclaw`, `qwen__qwen-3.6-35b-a3b__openclaw` and `qwen__qwen-3.6-27b-dense__openclaw`, the open-weight families served locally. Null for the other four, and by construction rather than by omission: * `anthropic__claude-4.7-opus__openclaw` — `thinking` is stripped at launch for Bedrock, so nothing reached the trace to record; * `openai__gpt-5.4__openclaw` and `google__gemini-3.1-pro__openclaw` — the providers return no reasoning text. Gemini returns an opaque `thought_signature`, which is a handle and not a trace, and is not stored as though it were; * `moonshotai__kimi-2.6__opencode` — emits reasoning, but its traces were not retained and are not recoverable. A further 27 rollouts among the three that have it had a trace that was read and found to contain no reasoning. They are null too, following the ingester's convention of writing null rather than `[]` for an extraction that came back empty. ### Nulls are meaningful ``` success null, not false, where ungradable (195 rollouts) harness, judge_model, judge_prompt_version null where the submission recorded nothing judgments, events null where nothing was published n_events null for kimi-2.6; see above reasoning null for four of seven submissions; see above ``` `moonshotai__kimi-2.6__opencode` published no traces at all, so its `events` is null on all 21,120 rows. ### Fields that were here and are gone `stratum`, `thinking`, `cohort` and `overflow` were columns of this archive and are not any more. None of them was a measurement. `stratum` was the string `all` on every row, hardcoded by the ingester for a subsampling regime that was never run. `thinking` was a config knob, null on every row, and misleading where it was not: the recorded value is stripped at launch for Bedrock. `cohort` and `overflow` were one fact stored twice — internal bookkeeping about which context-window experiment a row came out of, which is not something a published archive owes its readers. ## Reading it Both snippets need read access and an `HF_TOKEN` while the dataset is private; see [Access](#access). ```python from datasets import load_dataset ds = load_dataset("facebook/omnilingual-gaia2-results", split="train") ``` Or query without pulling the traces down: ```python import duckdb duckdb.sql(""" SELECT submission, lang, avg(success::int) AS pass_rate FROM 'hf://datasets/facebook/omnilingual-gaia2-results/data/*.parquet' WHERE success IS NOT NULL GROUP BY 1, 2 ORDER BY 3 DESC """) ``` ## Verifying the published tables The five leaderboard tables live in the Space repo under `data/tables/`. They are a build product of these rows and of the `omnigaia/aggregate.py` that reduces them, so the check is that the committed tables still follow from both: ```bash git clone https://huggingface.co/spaces/facebook/omnilingual-gaia2 cd omnilingual-gaia2 && uv sync uv run python scripts/rederive.py --archive /path/to/this/dataset --check ``` `uv sync` installs the dev group by default, so no extra flag is needed. The comparison is byte-for-byte and exits non-zero on any disagreement. Run it by hand. There is no CI on either repository and nothing runs this automatically; a maintainer types it before merging, and so should you. ## Grading Every row published so far records `judge_model` `gpt-oss-120b` and `judge_prompt_version` `omnilingual-gaia2` — the rubric localised per language. That is not the judge GAIA2's own published results use, so these scores are comparable to each other and not to a GAIA2 number from elsewhere. The two columns are per-row rather than assumed precisely so that a future submission graded otherwise says so. The leaderboard's About tab carries the same disclosure. ## Licence CC-BY-NC-4.0. That is the licence the data carries; it is not a statement about who can fetch it, which is the repository's private visibility — see [Access](#access). Model-generated agent traces; scores are as graded by the judge recorded in each row, and are not endorsements of any model.