ClawBenchV2Trace

# ClawBench V2 Traces **Full execution traces for every V2 model run scored on [ClawBench](https://huggingface.co/datasets/TIGER-Lab/ClawBench).** |[**🏆 Leaderboard**](https://claw-bench.com/leaderboard) | [**📊 Benchmark**](https://huggingface.co/datasets/TIGER-Lab/ClawBench) | [**📖 Paper**](https://arxiv.org/abs/2604.08523) | [**💻 Code**](https://github.com/TIGER-AI-Lab/ClawBench) | [**🎬 V1 Traces**](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace) | Companion to [`TIGER-Lab/ClawBench`](https://huggingface.co/datasets/TIGER-Lab/ClawBench) (task definitions) and [`NAIL-Group/ClawBenchV1Trace`](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace) (V1 traces). This dataset publishes the raw execution data for every **V2** model run we've evaluated — one directory per (task × model × attempt), each with the screen recording, network capture, browser actions, agent reasoning, and the final intercepted request. Anyone can re-grade, debug, or build new evaluators on top of these traces without re-running the agent. The corpus this scores is the [V2 130-task set](https://huggingface.co/datasets/TIGER-Lab/ClawBench) — newer, expanded coverage relative to the original V1 153-task set. ## Layout One directory per run, named `hermes-v2-<task_id>-<task_slug>-<model>-<timestamp>/`. Each run directory contains: ``` <run-dir>/ └── data/ ├── recording.mp4 # ~10 MB Full session recording (visual layer) ├── requests.jsonl # ~1–10 MB Network — every HTTP request/response ├── actions.jsonl # ~50 KB Browser action stream ├── agent-messages.jsonl # ~500 KB Agent LLM reasoning trace ├── interception.json # ~1 KB Final HTTP request that was intercepted (or `intercepted=false`) ├── judge.json # ~1 KB LLM judge verdict on the intercepted payload ├── agent-stdout.log └── agent-stderr.log ``` A run directory is fully self-contained: re-grading or replaying a single run only needs that one directory. ## Models covered (rolling — new models added as evaluated) V2 results land as new model runs complete. As of **2026-05-12** (806 total V2 runs): | Rank | Model | Harness | Intercepted | Reward | Runs | |------|-------|---------|-------------|--------|------| | 1 | `claude-opus-4-7` *(partial)* | hermes | **54.7%** | 13.3% | 75 | | 2 | `glm-5.1` | hermes | **48.5%** | 18.5% | 130 | | 3 | `gpt-5.5` *(partial)* | hermes | **48.1%** | 11.1% | 81 | | 4 | `deepseek-v4-pro` | hermes | **43.8%** | 10.0% | 130 | | 5 | `openrouter/owl-alpha` | hermes | **14.6%** | 4.6% | 130 | | 6 | `deepseek-v4-flash` | hermes | **3.1%** | 1.5% | 130 | Ranked by `intercepted / 130` (Stage 1, corpus-normalized) with Reward as tiebreak; partial batches are penalized for unattempted tasks. Fresh data + V1 lineup on the [live leaderboard Space](https://huggingface.co/spaces/TIGER-Lab/ClawBench) and the project [website](https://claw-bench.com/). For V1 traces (the original 153-task set with Claude Opus 4.6, Sonnet 4.6, Haiku 4.5, GPT-5.4, Kimi K2.5, etc.), see [`NAIL-Group/ClawBenchV1Trace`](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace). ## Download The full dataset is large — each run averages ~10–15 MB. Most users want a subset. ```bash # All V2 runs for one model hf download --repo-type dataset NAIL-Group/ClawBenchV2Trace \ --include "*-claude-opus-4-7-*" # All V2 runs for one task across all models hf download --repo-type dataset NAIL-Group/ClawBenchV2Trace \ --include "hermes-v2-1010-*" # Everything (large) hf download --repo-type dataset NAIL-Group/ClawBenchV2Trace ``` ## Scoring See [`eval/scoring.md`](https://github.com/TIGER-AI-Lab/ClawBench/blob/main/eval/scoring.md) for the two-stage rubric: 1. **Interception** — final HTTP request matched the per-task URL/method `eval_schema`. 2. **Judge** — LLM judge (default `deepseek/deepseek-v4-pro` via OpenRouter) verifies the payload fulfills the natural-language instruction. `final_pass = intercepted AND judge_match`. Every `data/judge.json` in this dataset records the judge verdict, reason, and the exact judge model used. Reproduce any number on the leaderboard: ```bash pip install clawbench-eval export OPENROUTER_API_KEY=sk-or-v1-... hf download --repo-type dataset NAIL-Group/ClawBenchV2Trace \ --include "*-<your-model>-*" --local-dir ./v2-traces python scripts/clawbench_rescore.py --judge-model deepseek/deepseek-v4-pro \ --only-batch ./v2-traces --force ``` ## License & Citation Apache 2.0 — see [LICENSE](https://github.com/TIGER-AI-Lab/ClawBench/blob/main/LICENSE). ```bibtex @article{zhang2026clawbench, title={ClawBench: Can AI Agents Complete Everyday Online Tasks?}, author={Yuxuan Zhang and Yubo Wang and Yipeng Zhu and Penghui Du and Junwen Miao and Xuan Lu and Wendong Xu and Yunzhuo Hao and Songcheng Cai and Xiaochen Wang and Huaisong Zhang and Xian Wu and Yi Lu and Minyi Lei and Kai Zou and Huifeng Yin and Ping Nie and Liang Chen and Dongfu Jiang and Wenhu Chen and Kelsey R. Allen}, journal={arXiv preprint arXiv:2604.08523}, year={2026} } ```