WildClawBench-Harbor

<h1 align="center">WildClawBench-Harbor</h1> <div align="center"> [](https://internlm.github.io/WildClawBench/) [](https://github.com/internlm/WildClawBench) [](https://arxiv.org/abs/2605.10912) <br> [](https://huggingface.co/datasets/internlm/WildClawBench) [](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories) </div> This repository is the [WildClawBench](https://huggingface.co/datasets/internlm/WildClawBench) benchmark converted to the [Harbor](https://github.com/harbor-framework/harbor) task format, so that all 60 tasks can be run directly with `harbor run` against any Harbor-supported agent (Claude Code, OpenHands, Codex CLI, custom agents, ...). WildClawBench is a benchmark for real-world, long-horizon agent evaluation: 60 hand-built tasks that require multi-step tool use (10–60+ calls), multimodal processing (video, images, documents), coding on real repositories, and robustness against prompt injection. See the [original dataset card](https://huggingface.co/datasets/internlm/WildClawBench) and paper for the full benchmark description. ## The WildClawBench Family | Repository | What's inside | |---|---| | **[WildClawBench](https://huggingface.co/datasets/internlm/WildClawBench)** | The benchmark itself: task data and Docker images for all four harnesses, run via the [official pipeline](https://github.com/internlm/WildClawBench) | | **[WildClawBench-Harbor](https://huggingface.co/datasets/internlm/WildClawBench-Harbor)** (this repo) | All 60 tasks in [Harbor](https://github.com/harbor-framework/harbor) format — evaluate any Harbor-supported agent with a single `harbor run` | | **[WildClawBench-Trajectories](https://huggingface.co/datasets/internlm/WildClawBench-Trajectories)** | Complete agent trajectories from our frontier-model evaluations, browsable in the HF Agent Trace Viewer | ## Task Categories | Category | Tasks | Focus | |---|---|---| | 01 Productivity Flow | 10 | Information synthesis, document processing, scheduling | | 02 Code Intelligence | 12 | Codebase comprehension, visual reasoning, script generation | | 03 Social Interaction | 6 | Multi-turn communication, routing, thread consolidation | | 04 Search & Retrieval | 11 | Constraint search, conflict resolution, data extraction | | 05 Creative Synthesis | 11 | Video/audio processing, cross-modal generation | | 06 Safety Alignment | 10 | Prompt-injection resistance, credential awareness, risky-operation refusal | Tasks with a `_zh` suffix use Chinese-language instructions; the rest are in English. ## Repository Structure Each task is a self-contained Harbor task directory at the repository root, named `<category>_task_<n>_<slug>`: ``` 01_Productivity_Flow_task_1_arxiv_digest/ ├── task.toml # Harbor task config (schema 1.4): environment, timeouts, metadata ├── instruction.md # The task prompt given to the agent ├── environment/ │ └── .wildclaw/ # Workspace assets copied into the container │ ├── run-warmup.sh # Environment warmup, run as a healthcheck before the agent starts │ └── skills/ # Task-available skills (e.g. agent-browser) └── tests/ ├── test.sh # Verifier entry point; writes the reward to /logs/verifier/ ├── grader.py # Grading logic ├── checks.py └── transcript_loader.py ``` Key `task.toml` settings shared across tasks: - **Environment**: a prebuilt Docker image (`wildclawbench-ubuntu:v1.3`) with `workdir = /tmp_workspace` and `network_mode = "public"` (tasks involve live web access). - **Timeouts**: 20 min for the agent, 10 min for the verifier (per task; a few tasks differ). - **Verifier**: runs in the shared container (`environment_mode = "shared"`) and is injected only after the agent finishes, so graders are never visible to the agent during execution. - **Artifacts**: task outputs under `/tmp_workspace/results` are collected after each run. ## Quick Start Install Harbor: ```bash uv tool install harbor # or: pip install harbor ``` Download this dataset: ```bash hf download internlm/WildClawBench-Harbor --repo-type dataset --local-dir ./WildClawBench-Harbor ``` Run a single task: ```bash harbor run -p ./WildClawBench-Harbor/01_Productivity_Flow_task_1_arxiv_digest \ -a claude-code -m anthropic/claude-opus-4-1 ``` Run the full benchmark: ```bash harbor run -p ./WildClawBench-Harbor -a claude-code -m anthropic/claude-opus-4-1 --n-concurrent 4 ``` ### Prerequisites - **Docker** must be available; tasks run in the `wildclawbench-ubuntu:v1.3` image, shipped as a tarball in the original WildClawBench repo. Load it once before running: ```bash hf download internlm/WildClawBench Images/wildclawbench-ubuntu_v1.3.tar --repo-type dataset --local-dir . docker load -i Images/wildclawbench-ubuntu_v1.3.tar ``` - **`BRAVE_API_KEY`** (optional): search-oriented tasks pass this through to the container for the Brave Search API. Export it before running if your agent relies on it. - Tasks require **outbound network access** (they browse the live web); results for search/retrieval tasks may vary slightly over time as web content changes. ## Scoring Each task's verifier writes a reward (`0`/`1`, or structured JSON metrics for partially-scored tasks) to `/logs/verifier/`. Harbor aggregates rewards across tasks; the benchmark score is the mean reward. Note that for the Safety Alignment category, "success" means the agent *refused or safely handled* the adversarial request. ## Differences from the Original Release - The original [WildClawBench](https://huggingface.co/datasets/internlm/WildClawBench) ships tasks as markdown specs plus an OpenClaw-based runner; this repository repackages each task into the standard Harbor layout (`task.toml` / `instruction.md` / `environment/` / `tests/`) with no changes to task content or grading logic. - Task naming is flattened to `<category>_task_<n>_<slug>` directories at the repository root. ## Citation If you use this benchmark, please cite the original WildClawBench paper: ```bibtex @article{ding2026wildclawbench, title={Wildclawbench: A benchmark for real-world, long-horizon agent evaluation}, author={Ding, Shuangrui and Dai, Xuanlang and Xing, Long and Ding, Shengyuan and Liu, Ziyu and JingYi, Yang and Yang, Penghui and Zhang, Zhixiong and Wei, Xilin and Fang, Xinyu and others}, journal={arXiv preprint arXiv:2605.10912}, year={2026} } ```