gaia2-cli

# GAIA2 CLI Benchmark dataset for [gaia2-cli](https://github.com/meta-agents-research-environments/gaia2-cli), the CLI-based agent evaluation harness. ## Schema Each row has two columns: | Column | Type | Description | |--------|------|-------------| | `scenario_id` | string | Unique scenario identifier (e.g. `scenario_universe_21_1qgjj6`) | | `scenario` | string | Complete scenario as a JSON string | ## Usage ```python from datasets import load_dataset import json # Load a specific config (160 scenarios) ds = load_dataset("meta-agents-research-environments/gaia2-cli", "adaptability", split="test") # Access a scenario scenario = json.loads(ds[0]["scenario"]) print(scenario.keys()) # dict_keys(['metadata', 'apps', 'events', 'version', 'augmentation']) # Load all configs (800 scenarios) ds = load_dataset("meta-agents-research-environments/gaia2-cli", split="test") ``` Available configs: `adaptability`, `ambiguity`, `execution`, `search`, `time`. ## With the runner The `gaia2-runner` downloads and caches this dataset automatically: ```bash gaia2-runner run-dataset \ --dataset meta-agents-research-environments/gaia2-cli \ --splits adaptability \ --image localhost/gaia2-oc:latest \ --provider anthropic --model claude-opus-4-6 \ --judge-provider anthropic --judge-model claude-opus-4-6 ``` Or in a TOML config: ```toml [target] dataset = "meta-agents-research-environments/gaia2-cli" splits = "all" ``` ## Export to JSON To export scenarios as individual JSON files: ```bash python scripts/export_hf_to_json.py --splits all --dest ~/gaia2_datasets/gaia2-cli ```