meta-agents-research-environments/gaia2-cli

---
pretty_name: GAIA2 CLI
configs:
- config_name: default
data_files:
- split: test
path: data/*
- config_name: adaptability
data_files:
- split: test
path: data/adaptability-*
- config_name: ambiguity
data_files:
- split: test
path: data/ambiguity-*
- config_name: execution
data_files:
- split: test
path: data/execution-*
- config_name: search
data_files:
- split: test
path: data/search-*
- config_name: time
data_files:
- split: test
path: data/time-*
default_config_name: default
---

# GAIA2 CLI

Benchmark dataset for [gaia2-cli](https://github.com/meta-agents-research-environments/gaia2-cli), the CLI-based agent evaluation harness.

## Schema

Each row has two columns:

| Column | Type | Description |
|--------|------|-------------|
| `scenario_id` | string | Unique scenario identifier (e.g. `scenario_universe_21_1qgjj6`) |
| `scenario` | string | Complete scenario as a JSON string |

## Usage

```python
from datasets import load_dataset
import json

# Load a specific config (160 scenarios)
ds = load_dataset("meta-agents-research-environments/gaia2-cli", "adaptability", split="test")

# Access a scenario
scenario = json.loads(ds[0]["scenario"])
print(scenario.keys()) # dict_keys(['metadata', 'apps', 'events', 'version', 'augmentation'])

# Load all configs (800 scenarios)
ds = load_dataset("meta-agents-research-environments/gaia2-cli", split="test")
```

Available configs: `adaptability`, `ambiguity`, `execution`, `search`, `time`.

## With the runner

The `gaia2-runner` downloads and caches this dataset automatically:

```bash
gaia2-runner run-dataset \
--dataset meta-agents-research-environments/gaia2-cli \
--splits adaptability \
--image localhost/gaia2-oc:latest \
--provider anthropic --model claude-opus-4-6 \
--judge-provider anthropic --judge-model claude-opus-4-6
```

Or in a TOML config:

```toml
[target]
dataset = "meta-agents-research-environments/gaia2-cli"
splits = "all"
```

## Export to JSON

To export scenarios as individual JSON files:

```bash
python scripts/export_hf_to_json.py --splits all --dest ~/gaia2_datasets/gaia2-cli
```