TaskTrove

# TaskTrove **TaskTrove** is an open-source collection of agentic task datasets, released by the [OpenThoughts-Agent](https://www.open-thoughts.ai/blog/agent) team. It is the task complement to **[AgentTrove](https://huggingface.co/datasets/open-thoughts/AgentTrove)** — the agent traces in AgentTrove were generated by running models against these task datasets using the [Harbor](https://github.com/open-thoughts/OpenThoughts-Agent) framework. > **v3.2 (current)** — replaced the old swegym task dataset (`laion__swegym-tasks-patched-validated-v2`, 989 tasks) with `laion/swegym-tasks-patched-validated-v5` (2,438 tasks, patched + validated). No other dataset changed. > > **v3.1** — verifier fix for `DCAgent__code-contests-noblock` (8,728 tasks): the v1/v2/v3 `tests/test.sh` ran pytest under `set -euo pipefail`, so a failing solution aborted the script before `reward.txt` was written — failed solutions raised `RewardFileNotFoundError` and were dropped instead of recorded as `reward=0`, silently biasing the reward distribution toward `1.0`. The verifier now wraps pytest (`set +e` / capture `PYTEST_EXIT` / `set -e`) so a reward is always recorded. Only this one dataset's `test.sh` changed; all other task content is byte-identical to v3. > > **v3** — additive expansion of v2: all 96 v2 datasets are retained, plus **20 new Nemotron-Gym RLVR task datasets** converted from the [nvidia/Nemotron-Post-Training-v3](https://huggingface.co/collections/nvidia/nemotron-post-training-v3) collection (instruction-following, math, science, knowledge, reasoning, multi-turn chat, and single-step agentic pivots). Prior releases remain resolvable at the [`v1`](https://huggingface.co/datasets/open-thoughts/TaskTrove/tree/v1) and [`v2`](https://huggingface.co/datasets/open-thoughts/TaskTrove/tree/v2) tags. --- ## Repository Structure Each source dataset is stored as a subdirectory named `org__name/`, where the original HuggingFace repo `org/name` has its `/` replaced with `__`. Each contains the task binaries as `tasks.parquet` (columns `path`: str, `task_binary`: gzip tar). --- ## Task Format All tasks are valid [Harbor](https://github.com/open-thoughts/OpenThoughts-Agent) task binaries. Tasks fall into two categories: - **With verifiers** (RL training / eval): a `verifier` scores trajectories at episode end. - **Without verifiers** (SFT / datagen): define the environment + instruction, rely on external grading. --- ## New in v3: Nemotron-Gym RLVR conversions (20) Converted with the OpenThoughts-Agent `data.nemotron_gym` framework. Grading is self-contained where the source carries a deterministic gold (string/regex/numeric/JSON-schema/grid), and LLM-judge where grading is inherently subjective (equivalence, rubric, hybrid IFEval+judge). Each task instruction explicitly directs the agent to write its answer to the grader's file path. - Instruction-following: citation-formatting, free-form-formatting, structured-outputs (json/yaml/xml/toml/csv), multi-turn chat, adversarial inverse-IFEval, CFBench, SysBench - Math / science / knowledge: math (sympy/latex boxed, with external-pointer hydration), litmus-bench (chemistry), science (Physics/Bio/Chem equivalence judge), QA-abstention, ReasoningGym, ARC-AGI (transductive + python-inductive), Multichallenge (advanced + vanilla) - Single-step agentic pivots: function-calling, conversational-tool-use, SWE, indirect-prompt-injection (graded on the agent's next action) --- ## Usage ```bash python -m scripts.datagen.extract_tasks_from_parquet \ --parquet open-thoughts/TaskTrove \ --output_dir $SCRATCH/tasks/tasktrove --on_exist overwrite ``` See the [OpenThoughts-Agent repository](https://github.com/open-thoughts/OpenThoughts-Agent) for the datagen/RL pipeline. --- ## Resolving prior versions ```python from huggingface_hub import snapshot_download snapshot_download(repo_id="open-thoughts/TaskTrove", repo_type="dataset", revision="v2", local_dir="./tasktrove_v2") # or "v1" ``` --- ## Citation ```bibtex @misc{openthoughts-agent, author = {Team, OpenThoughts-Agent}, month = Dec, title = {{OpenThoughts-Agent}}, howpublished = {https://www.open-thoughts.ai/blog/agent}, year = {2025} } ```