Data and models — Verifiable Rewards for Calibrated Probabilistic Forecasting

This archive accompanies the paper *Verifiable Rewards for Calibrated Probabilistic Forecasting* andits code repository. It contains the prepared training and evaluation data, the win-rate teachertable, the per-play predictions needed to reproduce every figure and table, and the trained LoRAadapters. The code that consumes these files is in the companion repository (`data/`, `eval/`, `train/`,`paper/build_assets.py`). Game data is derived from the public[nflverse](https://github.com/nflverse) play-by-play release. ## Contents ```grpo/ train.jsonl GRPO training prompts (2015–2022), one play per line eval.jsonl selection split (2023) test.jsonl test split (2024)winrate_buckets.json the empirical-rate teacher p̂(x): bucket -> shrunk win rateqa/ qa_eval.jsonl held-out 2023 game states with ground-truth features qa_test.jsonl held-out 2024 game states with ground-truth featurespreds/ preds_test_direct.json per-play predictions, direct RLVR model (test 2024) preds_test_masked.json per-play predictions, masked-CoT RLVR model preds_test_deepseek.json per-play predictions, DeepSeek-V4 zero-shot reasoning_judged_eval.jsonl blinded-judge labels for the reasoning-faithfulness analysis vegas_eval.json betting-market reference probabilitiesadapters/ direct/ LoRA adapter for the direct model (base: Qwen2.5-7B-Instruct) masked/ LoRA adapter for the masked-CoT model``` ## Schemas **`grpo/*.jsonl`** — one play per line: | field | description ||---|---|| `prompt` | the game-state question shown to the model (contains the public pregame spread only) || `actual_outcome` | 1 if the possession team won the game, else 0 (the verifiable label) || `target` | `p̂(x)`, the state-conditioned empirical win rate used as the reward target || `vegas_wp` | the live market win probability — **evaluation only; never shown in `prompt`** || `game_id`, `season`, `difficulty`, `meta` | provenance and bookkeeping | The reward is `r = 1 − (p − target)²`, where `p` is the probability parsed from the model's answer. **`qa/*.jsonl`** — held-out game states with ground-truth `features` (`posteam`, `defteam`, `qtr`,`time`, `game_seconds_remaining`, `down`, `ydstogo`, `yardline_100`, `score_differential`,`vegas_wp`, `wp`, `epa`, `spread_line`). Used by the blinded reasoning audit. **`winrate_buckets.json`** — the teacher table. Buckets are score margin × time remaining × pregamespread; each value is the bucket's training win fraction with hierarchical empirical-Bayes shrinkage(pseudocount 25) toward coarser buckets. Built from training-season outcomes only. **`preds/preds_test_*.json`** — per-play records `{game_id, prob, outcome, vegas, parsed, truncated}`,where `prob` is the model's probability and `vegas` is the market reference. These reproduce allheld-out metrics, the paired bootstraps, and the reliability and triangulation figures. ## Loading the adapters ```pythonfrom peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct")tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")model = PeftModel.from_pretrained(base, "adapters/direct") # or adapters/masked``` The tokenizer is unchanged by LoRA; load it from the base model. ### Adapter provenance | Adapter | Run | Checkpoint | Test Brier / ECE | LoRA ||---|---|---|---|---|| `direct/` | WS1, lr 2e-5 + grad-accum 16 | 200 (eval-selected) | 0.1442 / 0.0292 | r16, α32 || `masked/` | WS2, lr 3e-5, β=0, answer-span mask | 200 (eval-selected) | 0.1522 / 0.0293 | r16, α32 | Both adapt `Qwen/Qwen2.5-7B-Instruct`. SHA-256 of `adapter_model.safetensors`: ```direct 7d52fff15d423f96af9ace4dc982f1d88432f117395537118392566004ba9cebmasked 103ffc2cb6b4a871d6c47779d63c448991fe286a8b5e1ab5a4e913591de80979``` ## Provenance and license - Play-by-play data is derived from the nflverse project and is licensed **CC-BY-4.0**.- The prepared files in this archive (`grpo/`, `qa/`, `winrate_buckets.json`, `preds/`) are released under **CC-BY-4.0**.- The LoRA adapters are derived from **Qwen2.5-7B-Instruct** and are subject to that model's **Apache-2.0** license. No personal data is included. The data describes publicly recorded NFL game states and outcomes.