casino-benchmark

# LLM Casino Benchmark Evaluates LLM strategic reasoning in casino games (Blackjack and European Roulette) under provably fair conditions with shared cryptographic seeds. Each model plays 100 simulations (50 per game, 25 rounds each) starting with $1,000. | | | |-----------|-------| | **Total Rows** | 59,483 | | **Decision Rows** | 35,000 | | **Models Evaluated** | 14 | | **Simulations** | 1,400 (100 per model) | | **Games** | Blackjack, European Roulette | | **Rounds per Simulation** | 25 | | **Starting Balance** | $1,000 per simulation | | **File Format** | Parquet (Zstandard compression, 35 MB) | ## Leaderboard | Model ID | Display Name | Total Profit | Win Rate | Median Latency | |----------|-------------|-------------|----------|----------------| | `gemini-3-flash` | Gemini 3 Flash | +$3,396.0 | 60.0% | 5,664ms | | `grok-4.1-reasoning` | Grok 4.1 Fast Reasoning | +$2,099.0 | 54.1% | 7,788ms | | `kimi-k2-thinking` | Moonshot Kimi K2 Thinking | +$1,773.0 | 60.0% | 35,823ms | | `gemini-3-pro` | Gemini 3 Pro Preview | +$558.5 | 52.2% | 16,078ms | | `deepseek-reasoner` | DeepSeek Reasoner | -$37.3 | 54.3% | 74,342ms | | `gpt-5.2` | GPT 5.2 | -$380.5 | 56.9% | 6,751ms | | `mimo-flash` | Xiaomi Mimo V2 Flash | -$1,673.9 | 52.2% | 32,887ms | | `minimax-m2.1` | MiniMax M2.1 | -$2,059.4 | 58.2% | 12,543ms | | `claude-opus-4-5` | Claude 4.5 Opus | -$2,368.0 | 53.5% | 13,105ms | | `glm-4.7` | GLM 4.7 | -$3,072.5 | 53.2% | 23,559ms | | `claude-sonnet-4-5` | Claude 4.5 Sonnet | -$3,588.5 | 57.9% | 16,149ms | | `qwen3-max` | Qwen3 Max | -$5,765.2 | 61.6% | 5,870ms | | `mistral-large-3` | Mistral Large 3 | -$3,915.0 | 50.3% | 4,862ms | | `claude-haiku-4-5` | Claude 4.5 Haiku | -$7,788.5 | 61.7% | 9,496ms | Each model played 50 blackjack simulations and 50 roulette simulations (2,500 decision rows per model). ## Tasks This benchmark defines two tasks in [`eval.yaml`](eval.yaml): | Task ID | Game | Mode | Description | |---------|------|------|-------------| | `blackjack` | Blackjack | `heads_up` | 6-deck shoe, dealer stands on all 17s (S17), blackjack pays 3:2, double after split allowed, up to 3 splits, late surrender. Cards dealt via SHA-256 deterministic shuffling. | | `roulette` | European Roulette | `classic` | Single-zero wheel (37 numbers, 2.7% house edge), all standard bet types, multiple simultaneous bets per spin. Spin results via `SHA256(seed + spin_number) mod 37`. | ## Dataset Structure ### Data Fields | Field | Type | Description | |-------|------|-------------| | `sim_id` | int64 | Unique simulation identifier | | `game_type` | string | `"blackjack"` or `"roulette"` | | `mode` | string | `"heads_up"` (blackjack) or `"classic"` (roulette) | | `row_type` | string | Row classification (always `"decision"`) | | `step` | int64 | Step number within the simulation | | `game_num` | int64 | Round/spin number within simulation (1-25) | | `participant_id` | binary | UUID of the participant | | `model_id` | string | Model identifier slug | | `model_name` | string | Human-readable model name | | `decision_id` | binary | UUID for each individual decision | | `decision_type` | string | `"bet"`, `"action"`, or `"insurance"` | | `action` | string | Specific action taken (see Actions below) | | `latency_ms` | float64 | LLM response time in milliseconds | | `bet_amount` | float64 | Amount wagered on this decision | | `bet_type` | string | Roulette bet type (empty for blackjack) | | `bet_value` | string | Roulette bet target value (empty for blackjack) | | `balance_before` | float64 | Player balance before the decision | | `hand_cards` | string | Cards in hand, e.g. `"5♣ 10♣"` (blackjack only) | | `hand_value` | int64 | Numeric hand value (blackjack only) | | `is_soft` | bool | Whether the hand contains a usable ace | | `dealer_upcard` | string | Dealer's visible card, e.g. `"10♣"` (blackjack only) | | `dealer_upcard_value` | int64 | Numeric value of dealer upcard | | `valid_actions` | string | JSON array of valid actions for this decision | | `outcome` | string | Result: `"win"`, `"lose"`, `"push"`, `"blackjack"`, `"mixed"`, `"pass"`, or empty for intermediate steps | | `bet_total` | float64 | Total bet for the hand (may differ from `bet_amount` after doubles/splits) | | `payout` | float64 | Amount paid out | | `profit` | float64 | Net profit/loss (`payout - bet_total`) | | `balance_after` | float64 | Player balance after resolution | | `spin_result` | int64 | Roulette winning number (roulette only) | | `spin_color` | string | Roulette result color (roulette only) | | `dealer_final_value` | int64 | Dealer's final hand value (blackjack only) | | `dealer_bust` | bool | Whether the dealer busted (blackjack only) | | `hands_detail` | string | JSON array of per-hand detail for split hands | | `prompt` | string | Full prompt sent to the LLM | | `response` | string | Raw JSON response from the LLM | | `thinking_content` | string | Extended thinking / chain-of-thought content | ### Actions | Action | Count | Context | |--------|-------|---------| | `bet` | 34,930 | Wagering phase (both games) | | `stand` | 10,521 | Blackjack gameplay | | `hit` | 9,097 | Blackjack gameplay | | `double` | 1,660 | Blackjack gameplay | | `no_insurance` | 1,511 | Blackjack insurance decision | | `surrender` | 1,122 | Blackjack gameplay | | `split` | 571 | Blackjack gameplay | | `pass` | 70 | Roulette (skipping a spin) | | `insurance` | 1 | Blackjack insurance decision | ### Outcomes | Outcome | Count | Description | |---------|-------|-------------| | `win` | 17,160 | Player won the round | | `lose` | 14,245 | Player lost the round | | `mixed` | 1,261 | Split hands with mixed results | | `push` | 1,214 | Tie with dealer | | `blackjack` | 1,050 | Natural blackjack (21 on first two cards) | | `pass` | 70 | Player chose not to bet | | *(empty)* | 24,483 | Intermediate actions (hit, stand, etc.) without final outcome | ### Row Types The dataset contains two categories of rows: - **Outcome rows** (35,000): Rows with a non-empty `outcome` field representing resolved decisions with final results. - **Intermediate rows** (24,483): Rows with an empty `outcome` field representing mid-hand actions in blackjack (hit, stand, etc.) that precede the final resolution of a hand. ### Data Splits All rows are in a single split. No validation or test split exists. ### Example Data Point ```json { "sim_id": 1550, "game_type": "blackjack", "mode": "heads_up", "row_type": "decision", "step": 5, "game_num": 3, "model_id": "claude-opus-4-5", "model_name": "Claude 4.5 Opus", "decision_type": "action", "action": "stand", "latency_ms": 4521.0, "bet_amount": 50.0, "balance_before": 1050.0, "hand_cards": "K♠ 9♦", "hand_value": 19, "is_soft": false, "dealer_upcard": "7♣", "dealer_upcard_value": 7, "outcome": "win", "bet_total": 50.0, "payout": 100.0, "profit": 50.0, "balance_after": 1100.0, "dealer_final_value": 17, "dealer_bust": false, "prompt": "...", "response": "{\"action\": \"stand\", \"reasoning\": \"With 19 against a 7, standing is optimal.\"}", "thinking_content": "" } ``` ## Dataset Creation ### Pipeline Overview 1. **Seed Generation**: 50 cryptographic seeds generated via `secrets.token_hex(32)`. Each seed is shared across all 14 models to ensure identical game scenarios. 2. **Simulation Execution**: Each model plays 50 blackjack and 50 roulette simulations (25 rounds each) against deterministic game engines. 3. **Per-Simulation Storage**: Individual Parquet files saved per simulation in `benchmark_data/sim_{id}_{game_type}/`. 4. **Merging**: All simulation Parquets concatenated into a single file. 5. **Normalization**: Filtered to shared seeds only, deduplicated, JSON response formatting cleaned. ### Provably Fair Randomness Game outcomes are deterministic given a seed. For blackjack, each card is dealt using `SHA256(seed + round_number + "deal" + position)`. For roulette, each spin resolves via `SHA256(seed + spin_number) mod 37`. The same seed produces the same dealer cards and wheel results regardless of which model is playing. While random outcomes are identical per seed, game states can diverge because models make different decisions. A model that hits in blackjack advances the card position counter differently than one that stands, producing different subsequent cards. ### LLM Prompting Models receive structured prompts containing: - Current game state (hand, dealer upcard, balance, valid actions) - Recent history (last 5-10 rounds) - Bet constraints and payout tables - Instructions to respond in JSON with an `action` and `reasoning` field Standard models use temperature 0.7; reasoning models use 0.6. Max response tokens are 500 for standard models. Invalid or unparseable responses fall back to safe defaults (stand in blackjack, pass in roulette). ## Considerations for Using the Data ### Latency Variability Response latencies range from ~1 second to ~62 minutes. Models with extended thinking capabilities (DeepSeek Reasoner, Kimi K2 Thinking) show substantially higher latencies. Latency values reflect real API response times and are affected by provider load, network conditions, and model-specific processing. ### Statistical Power Each model has 2,500 outcome decisions across 100 simulations. This provides reasonable statistical power for comparing aggregate performance, but per-game-type comparisons (1,250 decisions per model per game) have wider confidence intervals. ### Licensing Model outputs (the `response` and `thinking_content` fields) are generated by third-party LLM providers and is subject to their respective terms of service.