Beam-Reasoning-Traces

# Beam Reasoning Traces Verified math and science reasoning traces from Reflection's [Beam-1.0-501B-A23B](https://huggingface.co/reflection/Beam-1.0-501B-A23B), mostly at `reasoning_effort=medium`. A further set covers the same prompts at `low`, `high` and `xhigh`. They were built to distill Beam into a small model, and were used to train [ProCreations/Dory-Beam-4B](https://huggingface.co/ProCreations/Dory-Beam-4B), a looped NVIDIA Dory-4B with Beam's reasoning-effort setting that thinks 1.9–4× less than the base model. Made for the Build with Beam challenge. | Config | Rows | What it is | |---|---|---| | `sft` (default) | 53,306 | One verified-correct trace per prompt (the shortest correct one), in chat `messages` form | | `all` | 124,945 | Every generation over 62,417 prompts, correct or not, with the grader's verdict | | `effort_sft` | 57,764 | The reasoning-effort training set: low, medium and high traces for 20k prompts, tagged as Dory-Beam's chat template renders them | | `effort_traces` | 60,000 | Every Beam generation at `low`, `high` and `xhigh` for those 20k prompts, correct or not | ## Evaluation outputs The `eval` configs hold every graded answer behind the [Dory-Beam-4B results](https://huggingface.co/ProCreations/Dory-Beam-4B#results), so the tables can be audited without re-running anything: | Config | Rows | What it is | |---|---|---| | `eval` | 42,228 | 23 model × loop-count × effort configurations, 1,836 answers each (GPQA-Diamond ×2 samples, MATH-500 ×1, AIME 2024/2025 ×8, AMC 2023 ×4, 300 held-out ×1) | | `eval_ifeval` | 3,246 | IFEval at 2 loops for base Dory, Dory-Beam-4B and its v1 (541 prompts × 2 samples), with strict and loose per-instruction results | | `eval_problems` | 900 | Prompt and reference answer for every non-GPQA problem | The models: - `dory1.0-4b`: base Dory, also in its `efficient` mode. - `dory-beam-4b`: v2. `reasoning_mode` is `effort-low`, `effort-medium` or `effort-high`, or `normal` for the untagged default. - `dory-beam-4b-v1`: the first release. - `dory-beam-4b-accurate`: the 70/30 base-to-fine-tune blend. - Ablations: `run-a-sft` (the fine-tune before blending) with its `step400`/`step800` checkpoints, `run-a2-random-trace` (trained on random correct traces), and `blend-70-30`. Each row has `model`, `loops`, `reasoning_mode`, `bench`, `problem_id`, `sample`, `correct`, `extracted_answer`, `reference_answer`, `output_tokens`, `finish_reason`, `max_output_tokens`, `resumed_tokens`, `thinking_closed` and the full `text` (reasoning, then `</think>` and the answer). Base Dory and Dory-Beam-4B in normal mode use a 64k-token limit: answers that hit the original 32k limit (16k for IFEval) were resumed from where they stopped, and `resumed_tokens` counts the tokens added. The other configurations stay at 32k, where they were rarely cut off. GPQA rows omit the question and `text`, because GPQA's authors ask that its questions not appear online in plain text. Grades, extracted letters and token counts are kept. [`audit_eval.py`](https://huggingface.co/ProCreations/Dory-Beam-4B/blob/main/code/scripts/audit_eval.py) recomputes every table and the paired bootstrap confidence intervals straight from these files: `python audit_eval.py dory-beam-4b:2 dory1.0-4b:2`, or `dory-beam-4b:2-effort-low` for one effort level. ## How it was made **Prompts.** These are decontaminated, auto-verifiable problems from two sources: - [zwhe99/DeepMath-103K](https://huggingface.co/datasets/zwhe99/DeepMath-103K) (MIT): difficulty 5–9, closed-form answers - [nvidia/OpenScienceReasoning-2](https://huggingface.co/datasets/nvidia/OpenScienceReasoning-2) (CC-BY-4.0): multiple choice, single-letter answers Any prompt sharing a 13-gram with MATH-500, AIME 2024/2025, AMC 2023 or GPQA was dropped. A separate check against the 198 GPQA-Diamond questions found zero 13-gram overlap. 300 further pool prompts were held out for evaluation and are not in this dataset. **Generation.** Every trace comes from Beam-1.0-501B-A23B with `reasoning_effort=medium`, apart from 30 pilot rows at `high`. Most were generated by self-hosting the open weights with SGLang (`lmsysorg/sglang:beam`) at temperature 0.7 and top-p 0.99, usually with 2 samples per prompt. The rest came from the Reflection API with its default sampling: | `teacher` | Checkpoint | `sft` rows | Math accuracy | Science accuracy | Median tokens | |---|---|---|---|---|---| | `selfhost-nvfp4` | Beam NVFP4 on 2× B300 | 48,826 | 85.8% | 76.5% | 1,248 | | `selfhost-fp8` | Beam FP8 on 4× B300 | 2,640 | 86.3% | 78.1% | 1,152 | | `api` | Reflection API | 1,840 | 83.5% | 78.7% | 987 | Self-hosted Beam matched the API's accuracy but wrote about 30–37% longer reasoning on token-identical prompts. This held for both NVFP4 and FP8, so quantization is unlikely to be the cause; sampling or serving differences on Reflection's side are more likely. Keeping the shortest correct of two samples narrows the gap. **Verification.** Math answers are taken from the last `\boxed{}` and compared with [math-verify](https://github.com/huggingface/Math-Verify). Science answers must match the gold letter. Truncated generations always count as incorrect. **Reasoning effort.** For the `effort_*` configs, self-hosted Beam (NVFP4, the same teacher as most `sft` rows) answered 20,000 randomly chosen `sft` prompts once each at `low`, `high` and `xhigh`. That was 60,000 generations, all graded the same way. - **Accuracy:** 93.1% at low, 91.8% at high and 92.3% at xhigh. - **Mean length:** 1,074, 2,233 and 2,420 tokens, against 1,648 for these prompts' medium traces. Because xhigh was barely longer than high, the training level `high` takes the longer correct answer of the two. `effort_sft` has 18,616 `low` and 19,148 `high` examples. It also has 20,000 medium examples (a random correct trace per prompt), half tagged `medium` and half left untagged. Tagged prompts end with `\n\n{reasoning effort: <level>}`. ## Fields `sft`: `id`, `domain` (`math`/`science`), `source`, `difficulty` (DeepMath only), `teacher`, `reasoning_effort`, `messages` (user prompt, then an assistant turn with `reasoning_content` and `content`), `gold`, `n_samples` and `n_correct` (Beam's generations and correct ones for this prompt). `all`: `id`, `domain`, `source`, `topic`, `difficulty`, `teacher`, `reasoning_effort`, `prompt`, `reasoning`, `content`, `gold`, `answer_type`, `pred` (the extracted answer), `correct`, `finish_reason`, `completion_tokens`. Self-hosted SGLang requests reported only total usage across samples, so their per-sample token counts are approximate. `effort_sft`: `id` (`<prompt id>#<level>`), `prompt_id`, `reasoning_effort` (the level trained), `beam_effort` (the Beam setting that produced it), `tagged`, `domain`, `teacher`, `messages`, `gold`. `effort_traces`: the `all` fields, plus `trained_as` (the level a trace was used for in `effort_sft`, if any). `run_a_ids.txt` lists the 52,660 prompts in the snapshot that Dory-Beam-4B was trained on. Later rows come from nightly API top-ups. ## Use ```python from datasets import load_dataset ds = load_dataset("ProCreations/Beam-Reasoning-Traces", "sft", split="train") ``` The `messages` format works with any chat template that renders `reasoning_content`, including Dory's and Qwen3's. To train on another model, the code in the [model repo](https://huggingface.co/ProCreations/Dory-Beam-4B/tree/main/code) rebuilds everything: `build_pool.py`, `generate.py`, `verify.py` and `build_dataset.py`. ## License CC-BY-4.0. Prompts come from DeepMath-103K (MIT) and OpenScienceReasoning-2 (CC-BY-4.0, please credit NVIDIA). Responses were generated by Beam-1.0-501B-A23B, which is released under Apache-2.0.