openthoughts-tblite

# NousResearch/openthoughts-tblite This dataset is a reformatted version of [OpenThoughts-TBLite](https://huggingface.co/datasets/open-thoughts/OpenThoughts-TBLite) for use with the [Hermes Agent](https://github.com/NousResearch/hermes-agent) Terminal-Bench evaluation framework. ## Source OpenThoughts-TBLite was created by the [OpenThoughts](https://www.openthoughts.ai/) Agent team in collaboration with [Snorkel AI](https://snorkel.ai/) and [Bespoke Labs](https://bespokelabs.ai/). It is a difficulty-calibrated subset of [Terminal-Bench 2.0](https://www.tbench.ai/leaderboard/terminal-bench/2.0) designed for faster iteration when developing terminal agents. - **Original dataset:** [open-thoughts/OpenThoughts-TBLite](https://huggingface.co/datasets/open-thoughts/OpenThoughts-TBLite) - **GitHub:** [open-thoughts/OpenThoughts-TBLite](https://github.com/open-thoughts/OpenThoughts-TBLite) - **Blog post:** [openthoughts.ai/blog/openthoughts-tblite](https://www.openthoughts.ai/blog/openthoughts-tblite) ## What we changed The original TBLite dataset stores tasks as individual directories with Dockerfiles, instruction files, and test scripts. We converted it to a flat tabular format matching the [NousResearch/terminal-bench-2](https://huggingface.co/datasets/NousResearch/terminal-bench-2) schema so it can be used directly with our evaluation harness: 1. **Built and pushed Docker images** for all 100 tasks to Docker Hub as `nousresearch/tblite-<task-name>:latest` 2. **Packed test suites** (`tests/` directories) as base64-encoded tar.gz archives in `tests_tar` 3. **Packed build contexts** (`environment/` directories) as base64-encoded tar.gz archives in `environment_tar` 4. **Extracted metadata** (category, difficulty, tags, timeouts) from each task's `task.toml` No task content was modified -- the instructions, Dockerfiles, and test scripts are identical to the originals. ## Schema | Column | Type | Description | |--------|------|-------------| | `task_name` | string | Unique task identifier (directory name from TBLite) | | `instruction` | string | Natural language task prompt shown to the agent | | `docker_image` | string | Pre-built Docker Hub image (`nousresearch/tblite-<name>:latest`) | | `category` | string | Task category (e.g., `security`, `machine-learning`) | | `difficulty` | string | Difficulty level (`easy`, `medium`, `hard`, `very-hard`, `expert`) | | `tags` | string | JSON-encoded list of tags | | `agent_timeout_sec` | float | Recommended agent timeout from task.toml | | `test_timeout_sec` | float | Recommended test/verifier timeout from task.toml | | `environment_tar` | string | Base64 tar.gz of the Docker build context (fallback if image unavailable) | | `tests_tar` | string | Base64 tar.gz of the test suite (pytest scripts uploaded to sandbox) | | `test_sh` | string | Content of `tests/test.sh` (the test runner script) | ## Difficulty Distribution Calibrated using Claude Haiku 4.5 as a reference model: | Difficulty | Pass Rate Range | Tasks | |------------|----------------|-------| | Easy | >= 70% | 24 | | Medium | 40-69% | 43 | | Hard | 10-39% | 28 | | Very Hard | < 10% | 1 | | Expert | < 10% | 2 | ## Usage ### With Hermes Agent ```bash python environments/benchmarks/tblite/tblite_env.py evaluate ``` ### With Python ```python from datasets import load_dataset ds = load_dataset("NousResearch/openthoughts-tblite", split="train") print(f"{len(ds)} tasks") print(ds[0]["task_name"], ds[0]["instruction"][:100]) ``` ## Citation ```bibtex @software{OpenThoughts-TBLite, author = {OpenThoughts-Agent team, Snorkel AI, Bespoke Labs}, month = Feb, title = {{OpenThoughts-TBLite: A High-Signal Benchmark for Iterating on Terminal Agents}}, howpublished = {https://www.openthoughts.ai/blog/openthoughts-tblite}, year = {2026} } ```