deepseek-v4-pro-0813-agentic

# DeepSeek-V4-Pro 0813 Agentic (DS4) A standalone, verifiable-first agentic training corpus: **19,072 training traces** plus **2,135 held-out evaluation rows** (validation 1,070 / test 1,065), generated by **DeepSeek-V4-Pro 0813** (`deepseek-v4-pro-0813`, official API, thinking mode) across **13 verifiable task families**, each row admitted only after passing a deterministic programmatic verifier. The corpus is designed to be directly usable for SFT, GRPO/RLVR, and **NeMo Gym / NeMo RL** (verified against the NeMo Gym JSONL contract — see "NeMo-RL readiness" below). - Teacher: `deepseek-v4-pro-0813` (DeepSeek official API, the 2026-08-13 release; visible reasoning captured per turn) - 19,072 training rows · 1,070 validation · 1,065 test (seed-grouped, zero leakage; 21,207 rows across all splits) - 13 families · 46.95M input / 16.78M output / 12.15M reasoning tokens - 100% of rows carry visible reasoning; 27,541 real tool-call steps in 5,997 tool rows; 1,438 error-recovery trajectories; 4,921 rows with parallel tool-call turns - Every row re-verified by replay (21,207/21,207) and every verifier covered by positive+negative smoke tests (108/108) ## Purpose The corpus fills the domains that benefit agentic performance, following the NVIDIA Nemotron 3 Ultra "teacher" recipe (verifiable instruction-following, structured outputs, programmatic constraint checkers, RLVR reward contracts) and grounded in a full-device analysis of 2,499 real Hermes agent sessions (tool chain shapes, parallel-call rates, recovery shares, reasoning-length bands). See `MANIFEST.json` for campaign provenance and `docs` references inside the companion repository. ## Families | Family | Rows | What the verifier checks | |---|---|---| | tool_call_schema_adherence | 5,541 | native function calls against frozen tool schemas; REAL sandbox execution (fs/sqlite/json tools); final-state equality; recovery + parallel-call lanes; hygiene (thrash detection) | | verified_constraint_instruction | 4,210 | per-row programmatic constraint checkers: keyword count/position, start/end words, paragraph/sentence/word counts, letter bans, punctuation bans, phrase anchors, list items, headers, multi-constraint + negated variants | | structured_outputs | 1,951 | strict parsing (JSON/YAML/XML/TOML/CSV) + schema validation (types/enums/patterns/required, extra-field rejection) + EXACT ground-truth value equality for extraction lanes | | stateful_dialogue | 1,856 | state recall + state mutation across 4 turns (booking/inventory/schedule/budget) with value verifiers | | verifiable_math | 1,812 | exact recomputation (arithmetic/algebra/percentages/fractions/probability/geometry/series) with numeric or Fraction equality | | science_verifiable | 1,748 | independent recomputation (physics kinematics/circuits/KE, chemistry moles, biology Punnett) within tolerance | | multilingual_multi_turn | 1,649 | language script checks (zh/ja/ar/es/fr/de) + bullet-count + 2-sentence constraints across turns | | planning_decomposition | 761 | numbered plan artifact (3-5 steps) + full execution of the plan in the deliverable | | grounded_long_context | 551 | 16K-32K synthetic corpora: aggregation, multi-hop, contradiction tasks with required document-ID citations and exact values | | delegation_orchestration | 397 | 3-way sub-task decomposition with per-task outputs + merged summary coverage | | grounded_citations | 361 | multi-hop claims over a synthetic corpus with REQUIRED citation formats ([1]/[2] bracket or (doc-0001) inline) and fact checking | | memory_context_management | 194 | summary-based compression and discard-all context management with exact retention semantics | | code_execution | 176 unique solutions | sandboxed Python with HIDDEN tests (never shown to the model) run on the final code; try-fix trajectories | ## Splits Seed-grouped deterministic splits (SHA-256 of `family:seed` mod 100): **train 90% / validation 5% / test 5%** — 19,072 / 1,070 / 1,065. Prompt-level grouping guarantees no train/test leakage across seeds. ## Repository layout Four loadable parquet configs (partitioned into `train` / `validation` / `test` shards), plus lossless raw JSONL under `raw/` for auditing. Configs are selected by name; raw files are fetched with `huggingface_hub` (not via `load_dataset`). | Config | train | validation | test | Contents | |---|---|---|---|---| | `canonical` | 19,072 | 1,070 | 1,065 | full audit view (all columns below) | | `sft_openai` | 19,072 | 1,070 | 1,065 | SFT-ready view | | `responses_api` | 19,072 | 1,070 | 1,065 | NeMo Gym native view | | `rl_tool_prompts` | 15,916 | 891 | 895 | GRPO prompts + reward contracts | 21,207 rows across all splits in canonical/sft_openai/responses_api; rl_tool_prompts covers the 17,702 verifiable rows. Parquet files are `dataset-format` sharded (`<split>-00000-of-00001.parquet`), generated from the raw JSONL via `datasets` 4.3.0; SHAs are recorded in `MANIFEST.json` under `parquet_configs`. ## File formats ### Config `canonical` — full audit view (one JSON object per row) | Column | Type | Description | |---|---|---| | `id` | string | stable row id, `ds4-…` prefixed | | `split` | string | `train` / `validation` / `test` | | `messages` | list[dict] | full conversation in OpenAI shape (see below) | | `tools` | list | tool definitions (empty for non-tool families) | | `task_type` | string | `ds4-<family>` | | `source` | string | `ds4-<bucket>` | | `domain` | string | family name | | `subdomain` | string | bucket (task variant) | | `teacher_model` / `teacher_provider` | string | `deepseek-v4-pro-0813`¹ / `deepseek-official-api` | | `trace_kind` | string | `single_turn` (11,967) / `multi_turn` (3,699) / `multi_turn_tool` (5,541) | | `disposition` | string | `traj_success` (19,769) / `traj_recovery` (1,438 — contains at least one tool error followed by recovery) | | `ground_truth_json` | string | verifier contract / expected values | | `verifier_passed` | bool | always `true` (acceptance gate) | | `rubric` | list | NVIDIA-style `[{question, pass_criteria}]` | | `n_assistant_turns` | int | 1-12 (median 1, p90 4) | | `n_tool_calls` | int | 0-17 (p90 6; 5,997 rows have >=1) | | `reasoning_present` | bool | 100% true (visible CoT captured) | | `seed` / `attempts` | int | deterministic generator seed; attempts until acceptance | | `schema_version` | string | `5.0-ds4-production` | | `source_repository` / `source_license` | string | `local:supplement_ds4` / `other; synthetic research corpus` | | `usage` | dict | prompt/completion/reasoning token counts | | `run_id` | string | campaign run provenance | ¹ Rows record the API model id as sent during generation (`deepseek-v4-pro`), which resolved to the 0813 release (`deepseek-v4-pro-0813`, released 2026-08-13) for every generation run (2026-08-15/16). **Message object shape** (variable keys by role; every assistant turn carries `reasoning_content` when reasoning was produced): ```json {"role": "system", "content": "..."} {"role": "user", "content": "..."} {"role": "assistant", "content": "...", "reasoning_content": "...", "tool_calls": [{"id": "call_x", "type": "function", "function": {"name": "fs_read", "arguments": "{\"path\": \"data/a.txt\"}"}}]} {"role": "tool", "tool_call_id": "call_x", "name": "fs_read", "content": "{\"content\": \"...\"}"} ``` ### Config `sft_openai` — SFT-ready view `{id, split, messages, tools, sampling_weight (1.0), source, domain}` — no chat template applied (template at train time, per-model), `arguments` as JSON STRING per OpenAI format. ### Config `responses_api` — NeMo Gym schema view (trace + verifier contracts) `{id, split, responses_create_params: {input: [messages...]}, verifier_metadata: {task_type, bucket, expected, rubric}, agent_ref: {type: "responses_api_agents", name: "ds4_<family>_simple_agent"}}` — matches the NeMo Gym JSONL contract (`responses_create_params` per the OpenAI Responses API schema; `agent_ref` routes each row to its agent server; the `expected` block is the full verifier contract for building the resources-server `verify()`). NOTE: `responses_create_params.input` carries the FULL verified reference trajectory (system/user/assistant/tool turns) — use it to build and test verifiers and to replay traces, NOT as a rollout seed. Rollout prompts (prompt-only) live in `rl_tool_prompts` (verified: zero assistant turns in `prompt_messages_json`). In the parquet view, `verifier_metadata.expected` is a JSON STRING (its schema varies by family, which parquet cannot unify); `json.loads()` it before use. The raw JSONL (`raw/responses_api.jsonl`) keeps the original nested object. ### Config `rl_tool_prompts` — GRPO prompts + reward contracts (17,702 rows; rollout-ready) `{id, family, bucket, split, prompt_messages_json, tools_json, reward_contract_json, sampling_weight}` covering 11 verifiable families — prompt-only messages (verified: zero assistant turns), tool definitions, and `reward_contract_json` = the exact programmatic contract usable as a reward function. This is the RL rollout view; use it as the NeMo Gym training dataset (prompt + tools + reward contract per row). ## Token statistics - Input 46,952,693 · output 16,779,172 · reasoning 12,150,867 - Token-level blend (share of input+output mass): tool 50.0% · long-context 19.5% · constraint-instruction 9.3% · planning 5.4% · stateful 4.1% · delegation 2.4% · structured-outputs 2.4% · citations 2.2% · multilingual 2.1% · science 0.8% · math 0.7% · memory 0.6% · code 0.5% - Rows are ordered easy→hard within each family (by attempts-until-acceptance) in `canonical.jsonl`. ## Generation & quality methodology - Teacher: `deepseek-v4-pro-0813` (DeepSeek-V4-Pro, 2026-08-13 release) via the official DeepSeek API; thinking mode; generation parameters: temperature 0.7, top_p 0.95, max_tokens 8,192 (16,384 for long-context). - Fleet: up to 400 tmux-isolated worker processes over a SQLite-WAL ledger (pending → leased → raw_saved → verified → accepted), crash-safe with orphan-lease recovery, per-family circuit breakers, and a hard budget stop. - Acceptance requires: structural validity + privacy + dedup + ONE registered objective verifier per family (the table above). Failed rows are retried up to 3 attempts, then retained as rejected evidence — never relabeled. - Tool rows execute REAL sandbox tools (workspace fs, sqlite, json transforms, python subprocess with timeout); no simulated tool results. - Every accepted row re-verified by deterministic replay (21,207/21,207). - Dedup: exact-prompt (max 3 independent completions per unique prompt; tool rows deduped on fixture identity) + 5-gram SimHash (hamming <= 3) on >=300-char non-tool finals; code/planning deduped on prompt+solution identity (only identical solutions collapse). - Reference bands (from the 2,499-session Hermes tool-use analysis) used by the periodic quality monitor; observed: recovery share 24.0% of tool rows, parallel-turn presence 82.1% of tool rows, reasoning-present 100%. - PII: synthetic prompts and fixtures only; no mined or user data. ## Provenance & licensing Fully synthetic research corpus. Prompts are deterministic template-generated; every assistant response is a DeepSeek-V4-Pro 0813 (`deepseek-v4-pro-0813`) output. License: `other` — synthetic research corpus. Verify DeepSeek's API terms for your intended training use before commercial application. This repo is additive to (not a subset of) `r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation`: fresh seed namespaces, disjoint templates, and a shingle-blocking cross-dedup gate documented in the manifest. ## Usage recipes SFT (responses-only masking, template at train time): ```python from datasets import load_dataset ds = load_dataset("r0b0tlab/deepseek-v4-pro-0813-agentic", "sft_openai") # messages: list[{role, content, reasoning_content, tool_calls}] # train assistant turns only; apply your model's chat template at train time. ``` GRPO/RLVR (NeMo Gym or TRL): ```python # "responses_api" for NeMo Gym; "rl_tool_prompts" for TRL: ds = load_dataset("r0b0tlab/deepseek-v4-pro-0813-agentic", "rl_tool_prompts") # reward_contract_json per row = programmatic verifier contract ``` Full audit view (all columns): ```python ds = load_dataset("r0b0tlab/deepseek-v4-pro-0813-agentic", "canonical") ``` Lossless raw JSONL (byte-identical to generation output, not a datasets config): ```python from huggingface_hub import hf_hub_download p = hf_hub_download("r0b0tlab/deepseek-v4-pro-0813-agentic", "raw/canonical.jsonl", repo_type="dataset") ``` ## NeMo-RL readiness Verified 2026-08-16 against the NeMo Gym data-prep contract (docs.nvidia.com/nemo/gym/data) and NVIDIA's released RL datasets (e.g. `nvidia/Nemotron-RL-Instruction-Following-Structured-Outputs-v2`, `nvidia/Nemotron-RL-instruction_following`): | Check | Result | |---|---| | `responses_create_params` present per row (OpenAI Responses API schema) | PASS — 21,207/21,207 rows | | `agent_ref.type == "responses_api_agents"`, `<family>_simple_agent` naming | PASS — matches NVIDIA's own convention | | Prompt-only rollout input (NVIDIA ships prompt-only `input`) | `rl_tool_prompts` PASS (0 assistant turns) · `responses_api` carries the full reference trajectory — do not use as rollout seed; project to the pre-assistant prefix (system+user) or use `rl_tool_prompts` | | Programmatic reward contracts | PASS — all 11 RL families carry self-contained contracts (code_exec, math variants, science variants, sqlite/fs tool chains, planning, citations, long-context, compression, delegation, structured-output schema+ground_truth, constraint id+params list mirroring NVIDIA's `instruction_id_list`/`kwargs` pattern) | | Built-in agent servers | N/A by design — the 13 `ds4_<family>_simple_agent` names require registering custom NeMo Gym agent servers; `verifier_metadata.expected` / `reward_contract_json` provide the `verify()` contracts, and reference checker implementations ship with the campaign code (scripts/supplement_ds4) | | Conversational families (stateful_dialogue, multilingual_multi_turn; 3,505 rows) | NOT in `rl_tool_prompts` — their RL rollout requires an environment that scripts continuation user turns (judge-style, as NVIDIA's MultiTurnChat) | Consumption path: download the repo → place the raw JSONL views under a resources-server config → run `gym dataset collate --mode train_preparation` to validate and generate rollout metrics. `agent_ref` will be matched against your registered agent servers during collation. ## Known limitations - grounded_long_context is 2.6% of rows (16-32K aggregation is hard for the teacher; ~69% yield) — below a 4% inventory floor; supplement for long-context-heavy students. - memory_context_management yield 48.5% (compression with exact retention). - code_execution spans 10 task templates; 176 rows, deduped on prompt+solution identity (identical solutions collapse). - Generation ran during the 2026-08-15/16 DeepSeek pricing change; cost accounting in the campaign reports. ## Citation ```bibtex @misc{r0b0tlab_ds4_0813_2026, title = {DeepSeek-V4-Pro 0813 Agentic (DS4): a verifiable-first agentic training corpus}, author = {r0b0tlab}, year = {2026}, url = {https://huggingface.co/datasets/r0b0tlab/deepseek-v4-pro-0813-agentic} } ``` Teacher model: DeepSeek-V4-Pro 0813 (`deepseek-v4-pro-0813`) (DeepSeek, 2026). Methodology informed by the NVIDIA Nemotron 3 Ultra technical report (NVIDIA, 2026).