qwen3.8-max-glm5.2-kimi-k3-distillation

# Multi-Teacher Distillation Dataset (57,937 traces) A quality-filtered, deduplicated, multi-teacher SFT corpus combining traces from three frontier models across math, code, reasoning, instruction-following, tool-use, science, long-context, multilingual, and creative dialogue domains. ## Teachers | Teacher | Provider | Traces | |---------|----------|-------:| | Qwen3.8-Max-Preview | Alibaba Cloud Model Studio | 48,283 | | GLM-5.2 | Z.AI Coding Plan | 5,307 | | Kimi Code K3 | Moonshot AI (Kimi) | 4,347 | ## Quick Start ### General SFT (recommended entrypoint) ```python from datasets import load_dataset # Default config — rebalanced sampling weights ds = load_dataset("r0b0tlab/qwen3.8-max-glm5.2-distillation-51389", "sft_balanced") for batch in ds["train"]: messages = batch["messages"] # list of {role, content, reasoning_content, tool_calls, trainable} tools = batch["tools"] # list of tool definitions weight = batch["sampling_weight"] # float — use in WeightedRandomSampler # ... feed to your trainer with responses-only loss masking ``` Each message in the `messages` list has this structure: ```python { "role": "assistant", # system | user | assistant | tool "content": "The answer is 30.", # final text output "reasoning_content": "Buzz and...",# chain-of-thought (empty for non-reasoning turns) "tool_calls": [], # [{id, type, function: {name, arguments}}] "tool_call_id": "", # for tool-role messages "name": "", # optional tool name "trainable": True # True only on assistant turns } ``` ### Native GLM-4.7-Flash training (pre-tokenized) Skip the chat template entirely — `input_ids` and `labels` are already computed: ```python import pyarrow.dataset as ds table = ds.dataset(".../data/glm47_native", format="parquet") # Each row: # input_ids: list[int] — exact token IDs from GLM-4.7-Flash tokenizer # labels: list[int] — same length, -100 on non-assistant tokens # total_tokens: int # assistant_tokens: int — count of labels != -100 # mask_method: str — "prefix-differential" or "structural-redaction" # renderer_revision: str — pinned tokenizer version # Train with standard causal LM loss: # loss = CrossEntropyLoss(model(input_ids), labels, ignore_index=-100) ``` ### Tool-use / agent training ```python ds = load_dataset("r0b0tlab/qwen3.8-max-glm5.2-distillation-51389", "sft_tools") # 5,909 rows with real tool-call trajectories (not JSON-in-content) # Then RL on deterministic tool tasks: rl = load_dataset("r0b0tlab/qwen3.8-max-glm5.2-distillation-51389", "rl_tool_prompts") # prompt_messages_json + tools_json + reward_contract_json ``` ## Dataset Summary | Metric | Value | |--------|------:| | Total traces | 57,937 | | Train / Validation / Test | 52,205 / 2,872 / 2,860 | | Teachers | 3 (Qwen3.8-Max, GLM-5.2, Kimi K3) | | Domains | 9 normalized + 16 raw | | Configs (Parquet views) | 24 | | Exact retokenization | PASS · 0 mask failures · 3 tokenizers | | Max tokens (GLM) | 56,428 | | Splits | Template-cluster-grouped · deterministic · zero leakage | ### Build pipeline ``` Base dataset (52,083) ── freeze + inventory ├── K3 supplement: 4,354 accepted → 694 already in base → 3,660 new ├── Qwen V4 production: 2,131 accepted + 70 canary └── raw merge upper bound: 57,944 ↓ normalize → quality filter → deduplicate ↓ 57,937 canonical rows (7 near-dups removed) ↓ capability taxonomy → mixture analysis → sampling policies ↓ exact retokenization (GLM/Qwen/Llama) → 24 training configs ``` ## Domain Distribution ### Normalized domains | Domain | Rows | % | |--------|-----:|---:| | Math | 15,072 | 26.0% | | Code | 14,719 | 25.4% | | Reasoning | 10,463 | 18.1% | | Instruction | 8,524 | 14.7% | | Agent Tools | 5,952 | 10.3% | | Long Context | 1,263 | 2.2% | | Science | 844 | 1.5% | | Dialogue | 664 | 1.1% | | Multilingual | 436 | 0.8% | ### Source breakdown (top 20) | Source | Rows | |--------|-----:| | Evol-Code | 8,098 | | MetaMathQA | 6,267 | | CodeAlpaca | 5,586 | | tulu-3 | 5,395 | | glm5.2-agent-tool-synthetic | 5,307 | | SciQ | 3,947 | | NuminaMath-CoT | 2,870 | | CommonssenseQA | 2,816 | | OrcaMath | 2,319 | | Qwen3.8-Max V4 Supplement | 2,201 | | Dolly | 1,710 | | QASC | 1,682 | | K3 strict_instruction | 1,026 | | K3 grounded_long_context | 978 | | ARC-Easy | 971 | | K3 science_logic_data | 844 | | K3 math_formal | 720 | | GSM8K | 640 | | MATH/algebra | 578 | | OpenBookQA | 574 | ## Capability Mixture | Capability | % of rows | SOTA floor | Status | |-----------|--------:|--------|--------| | Reasoning-bearing | 84.6% | >=30% | PASS (over-indexed for generalist) | | Tool-use (real trajectories) | 10.3% | >=13% | Below floor — mitigated by `sft_balanced` weights | | Long-context (exact GLM >=4096 tokens) | 2.9% | >=4% | Below floor | | Science | 19.5% | >=4% | PASS | | Multilingual | 0.8% | >=5% | Below floor | | Single-turn | 86.7% | <=65% | Above ceiling — mitigated by `sft_balanced` weights | Under `sft_balanced` sampling weights, the effective mixture at training time shifts: | Capability | Weighted % under `sft_balanced` | |-----------|------:| | Tools | 37.5% | | Single-turn | 55.9% (under ceiling) | | Long-context | 5.0% (clears floor) | | Reasoning | 90.4% | | Science | 19.9% | ## Configs (24 Parquet views) ### Training configs | Config | Rows | Use case | |--------|-----:|----------| | `sft_balanced` | 57,937 | **Default generalist SFT** — rebalanced sampling weights | | `sft` | 57,937 | Uniform weights (no rebalancing) | | `sft_reasoning` | 48,986 | Reasoning-only (chain-of-thought traces) | | `sft_reasoning_specialist` | 48,986 | Same rows, specialist weighting | | `sft_final` | 57,937 | Final-content only (reasoning stripped) | | `sft_instruction_final` | 8,524 | Instruction-following, final-only targets | | `sft_tools` | 5,909 | Real tool-call trajectories | | `sft_agent` | 5,909 | Multi-turn agent traces | | `rl_tool_prompts` | 5,909 | GRPO/RLVR tool-use prompts + reward contracts | | `sft_math` | 15,072 | Math specialist | | `sft_code` | 14,719 | Code specialist | | `sft_science` | 11,307 | Science specialist | | `sft_long_context` | 1,275 | Long-context (thin — see gaps) | | `sft_multilingual` | 436 | Multilingual (thin — see gaps) | | `sft_dialogue` | 664 | Multi-turn creative/dialogue | ### Ablation configs | Config | Rows | Isolates | |--------|-----:|----------| | `sft_k3_only` | 4,347 | Kimi K3 teacher only | | `sft_qwen_v4_supplement` | 2,201 | Qwen V4 supplement only | | `sft_glm_agent` | 5,307 | GLM-5.2 agent tools only | ### Infrastructure configs | Config | Rows | Content | |--------|-----:|---------| | `canonical` | 57,937 | Full audit columns (messages, tools, metadata, verifier flags, provenance) | | `openai_messages` | 57,937 | OpenAI-compatible message format | | `prompt_completion_text` | 52,028 | Text-only prompt/completion (no tool rows) | | `glm47_native` | 57,937 | Pre-tokenized: input_ids + labels (GLM-4.7-Flash) | | `token_stats` | 57,937 | Exact multi-tokenizer stats per row | | `smoke` | 256 | Coverage slice for CI/testing | ## Column Schemas ### `sft_balanced` / `sft` / specialist `sft_*` configs | Column | Type | Description | |--------|------|-------------| | `parent_id` | string | Source lineage ID | | `id` | string | Stable release row ID | | `split` | string | `train` / `validation` / `test` | | `messages` | list[struct] | Conversation turns (see Quick Start) | | `tools` | list[struct] | Tool definitions (empty if no tools) | | `sampling_weight` | float | Weighted sampling multiplier | | `source` | string | Source label (e.g., `MetaMathQA`, `k3_math_formal`) | | `domain` | string | Domain label (e.g., `math`, `code`, `agent_tool`) | ### `glm47_native` (pre-tokenized) | Column | Type | Description | |--------|------|-------------| | `parent_id` | string | Source lineage ID | | `id` | string | Stable release row ID | | `split` | string | Split label | | `input_ids` | list[int32] | Exact token IDs from GLM-4.7-Flash | | `labels` | list[int32] | Same length as input_ids; -100 on non-assistant tokens | | `total_tokens` | int32 | len(input_ids) | | `assistant_tokens` | int32 | count of labels != -100 | | `mask_method` | string | `prefix-differential` or `structural-redaction` | | `sampling_weight` | float | Sampling weight | | `source` | string | Source label | | `domain` | string | Domain label | | `renderer_revision` | string | Pinned tokenizer: `zai-org/GLM-4.7-Flash@7dd20894...` | ### `token_stats` (exact multi-tokenizer) Contains total/assistant token counts for GLM-4.7-Flash, Qwen3-0.6B, and Llama-3.1-8B per row. Key columns: `glm47_total_tokens`, `glm47_assistant_tokens`, `glm47_loss_ratio`, `glm47_context_bucket`, `qwen3_*`, `llama31_final_*`, `max_validated_tokens`. ### `canonical` (full audit view) All columns from the schemas above plus: `teacher_model`, `teacher_provider`, `trace_kind`, `disposition`, `ground_truth_json`, `verifier_passed`, `family_oracle_passed`, `reference_agreement`, `format_score_raw`, `quality_flags`, `source_record_hash`, `prompt_cluster_id`, `template_cluster_id`, `dedup_cluster_id`, `schema_version`, `source_repository`, `source_revision`, `source_license`, `source_split`, `source_item_id`. ### `rl_tool_prompts` | Column | Description | |--------|-------------| | `family` | Tool family (e.g., `file_ops`, `code_debug`) | | `prompt_messages_json` | System + user messages (JSON) | | `tools_json` | Tool definitions (JSON) | | `expected_terminal_oracle_json` | Ground-truth answer for reward | | `reward_contract_json` | Reward specification | | `sampling_weight` | Sampling weight | ## Token Statistics All 57,937 rows were retokenized with three pinned tokenizers: | Tokenizer | Repo | Revision | |-----------|------|----------| | GLM-4.7-Flash | `zai-org/GLM-4.7-Flash` | `7dd20894a642a0aa287e9827cb1a1f7f91386b67` | | Qwen3-0.6B | `Qwen/Qwen3-0.6B` | `c1899de289a04d12100db370d81485cdf75e47ca` | | Llama-3.1-8B-Instruct | `NousResearch/Meta-Llama-3.1-8B-Instruct` | `d10aef7999a2b5ba950ab3974312feeedbfe0b77` | GLM-4.7-Flash context bucket distribution: | Bucket | Rows | |--------|-----:| | <= 4096 tokens | 56,243 (97.1%) | | 4097-8192 | 1,484 (2.6%) | | 8193-16384 | 127 (0.2%) | | 16385-32768 | 69 (0.1%) | | 32769-65536 | 14 (<0.1%) | Loss mask methods: `prefix-differential` (most rows — prefix-preserving chat template) and `structural-redaction` (multi-turn tool rows — diff-based mask). Zero mask failures. ## Loss Masking Two methods, automatically selected per row: 1. **Prefix-differential**: The chat template is prefix-preserving (adding a new message doesn't change earlier tokens). Labels are assigned by diffing successive template renders — tokens from assistant messages get their real IDs, all others get -100. 2. **Structural-redaction**: For complex multi-turn tool conversations where the template isn't prefix-preserving. Each assistant payload is redacted in a copy of the conversation, and the diff between full and redacted renders identifies assistant tokens. Both methods guarantee: no user/system/tool content in the loss, no fabricated labels, at least one proven assistant span per row. ## Deduplication Three layers applied: 1. **Exact identity hash** — canonical hash of normalized (system, user turns, tools, fixture) 2. **Semantic near-dup** — 5-gram Jaccard >= 0.9 using blake2b shingles 3. **Prompt cluster overlap** — rows sharing a prompt cluster with base dataset are deduped Result: 7 near-duplicates removed from 5,861 eligible supplement rows; zero exact duplicates in final set. ## Quality Filtering | Gate | Method | |------|--------| | Structural | Role order, non-empty assistant turns, tool-call/result linkage | | Teacher identity | K3 rows verified `kimi-code/k3`; Qwen rows verified `qwen3.8-max-preview` | | Tool trajectories | Real tool-role results required (no pseudo-tool JSON in content) | | Reasoning presence | K3 reasoning_content folded onto assistant messages | | ID uniqueness | Zero duplicate IDs across base + supplements | K3 double-count guard: 694 K3 rows already in base dataset were identified by ID intersection and excluded from supplements (only 3,660 new K3 rows admitted). ## Curriculum Recommendations The `curriculum_stages.json` in the hub package defines a multi-stage schedule: | Stage | Config | Method | Notes | |-------|--------|--------|-------| | A | `sft_balanced` | SFT, 1 epoch | Default generalist entrypoint | | B | `sft_tools` + `sft_agent` | SFT, 0.5 epoch | Tool/agent specialization | | C (optional) | `sft_reasoning_specialist` | SFT, 0.5 epoch, low LR | Dataset is already reasoning-heavy | | D (optional) | `rl_tool_prompts` | GRPO/RLVR | Verifiable tool tasks only | ## Provenance and License This dataset is a synthetic research corpus built from teacher model outputs. Each row carries full provenance in the `canonical` config: `source_repository`, `source_revision`, `source_license`, `source_item_id`, `teacher_model`, `teacher_provider`. Base dataset rows (52,083) originate from public sources including MetaMathQA (MIT), GSM8K (MIT), SciQ (CC0), ARC (CC-BY-SA), OpenBookQA (CC-BY-SA 4.0), CommonsenseQA (CC-BY-SA), QASC (Apache-2.0), CodeAlpaca, Evol-Code, tulu-3, Dolly, MATH, NuminaMath-CoT, OrcaMath. Supplement rows from K3 (kimi-code/k3) and Qwen V4 (qwen3.8-max-preview) are labeled `other; synthetic research corpus`. The repo name retains the historical `51389` from the original v1 release; the actual row count is 57,937. ## Known Limitations - **Tools (10.3%)** and **long-context (2.9%)** are below SOTA floors by inventory. The `sft_balanced` config uses sampling weights to compensate at train time, but genuine coverage gaps remain until additional generation completes. - **Multilingual (0.8%)** is critically thin; domain-only classification without a language detector means some multilingual content in base instruction rows may not be tagged. - **Reasoning is over-indexed (84.6%)** — appropriate for reasoning-forward students, but generalist trainings should use `sft_balanced` weights or supplement with non-reasoning data. - **Single-turn dominates (86.7%)** — `sft_balanced` down-weights single-turn to ~56% effective share. - Full exact tokenization covers GLM-4.7-Flash, Qwen3, and Llama-3.1. Other tokenizers require applying your own chat template to the `sft` configs. - This package is a local pre-release (rc3). The qualitative review of the compile is in progress. ## Citation If you use this dataset, cite the teacher models and upstream sources that made it possible: ```bibtex @misc{qwen38, title={Qwen3.8-Max-Preview}, author={Alibaba Cloud}, year={2026} } @misc{glm52, title={GLM-5.2}, author={Z.AI}, year={2026} } @misc{kimi-k3, title={Kimi Code K3}, author={Moonshot AI}, year={2026} } @misc{metamathqa, title={MetaMathQA: Bridging Visual and Textual Reasoning}, author={Yu, Shuo and others}, year={2024}, url={https://github.com/meta-math/MetaMath} } ```