eval-Hermes-4-14B-reasoning-old

# h4-e3-overlong-masked-30k-rerun Evaluation Results

## Summary

| Benchmark | Score | Metric | Samples | Overlong rate |
|-----------|-------|--------|---------|---------------|
| aime24 | 0.527 | math_pass@1:64_samples | 64 | 6.6% |
| aime25 | 0.414 | math_pass@1:64_samples | 64 | 8.1% |
| arenahard | 0.782 | eval/overall_winrate | 500 | 0.0% |
| bbh_generative | 0.844 | extractive_match | 1 | 5.8% |
| creative-writing-v3 | 0.617 | creative_writing_score | 96 | 0.0% |
| drop_generative_nous | 0.827 | drop_acc | 1 | 2.4% |
| eqbench3 | 0.805 | eqbench_score | 135 | 0.0% |
| gpqa_diamond | 0.556 | gpqa_pass@1:8_samples | 8 | 8.5% |
| ifeval | 0.501 | inst_level_loose_acc | 1 | 6.7% |
| lcb-v6-aug2024+ | 0.452 | eval/pass_1 | 1 | 12.7% |
| math_500 | 0.926 | math_pass@1:4_samples | 4 | 0.9% |
| mmlu_generative | 0.838 | extractive_match | 1 | 2.3% |
| mmlu_pro | 0.733 | pass@1:1_samples | 1 | 6.6% |
| musr_generative | 0.591 | extractive_match | 1 | 2.9% |
| obqa_generative | 0.934 | extractive_match | 1 | 2.2% |
| rewardbench | 0.270 | eval/ties_error_rate | 1 | 2.5% |
| simpleqa_nous | 0.054 | fuzzy_match | 1 | 9.6% |

Overlong rate: 3,554 / 64,523 samples (5.5%) missing closing `</think>` tag

## Detailed Results

### aime24

| Metric | Score | Std Error |
|--------|-------|----------|
| math_pass@1:1_samples | 0.500 | 0.093 |
| math_pass@1:4_samples | 0.533 | 0.071 |
| math_pass@1:8_samples | 0.517 | 0.071 |
| math_pass@1:16_samples | 0.512 | 0.068 |
| math_pass@1:32_samples | 0.526 | 0.066 |
| math_pass@1:64_samples | 0.527 | 0.064 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:54:12
**Temperature:** 0.6
**Overlong samples:** 6.6% (126 / 1920)

### aime25

| Metric | Score | Std Error |
|--------|-------|----------|
| math_pass@1:1_samples | 0.467 | 0.093 |
| math_pass@1:4_samples | 0.433 | 0.067 |
| math_pass@1:8_samples | 0.450 | 0.069 |
| math_pass@1:16_samples | 0.435 | 0.068 |
| math_pass@1:32_samples | 0.421 | 0.068 |
| math_pass@1:64_samples | 0.414 | 0.067 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 01:00:56
**Temperature:** 0.6
**Overlong samples:** 8.1% (156 / 1920)

### arenahard

| Metric | Score | Std Error |
|--------|-------|----------|
| eval/overall_winrate | 0.782 | 0.000 |
| eval/total_samples | 500.000 | 0.000 |
| eval/win_count | 363.000 | 0.000 |
| eval/tie_count | 57.000 | 0.000 |
| eval/loss_count | 80.000 | 0.000 |
| eval/win_rate | 0.726 | 0.000 |
| eval/tie_rate | 0.114 | 0.000 |
| eval/loss_rate | 0.160 | 0.000 |
| eval/winrate_arena-hard-v0.1 | 0.782 | 0.000 |

**Model:** h4-14b-think
**Evaluation Time (hh:mm:ss):** 00:05:56
**Temperature:** 0.6
**Overlong samples:** 0.0% (0 / 500)

### bbh_generative

| Metric | Score | Std Error |
|--------|-------|----------|
| extractive_match | 0.844 | 0.019 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:39:25
**Temperature:** 0.6
**Overlong samples:** 5.8% (318 / 5511)

### creative-writing-v3

| Metric | Score | Std Error |
|--------|-------|----------|
| creative_writing_score | 0.617 | 0.000 |
| num_samples | 96.000 | 0.000 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** N/A
**Temperature:** N/A
**Overlong samples:** 0.0% (0 / 96)

### drop_generative_nous

| Metric | Score | Std Error |
|--------|-------|----------|
| drop_acc | 0.827 | 0.004 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:46:08
**Temperature:** 0.6
**Overlong samples:** 2.4% (227 / 9536)

### eqbench3

| Metric | Score | Std Error |
|--------|-------|----------|
| eqbench_score | 0.805 | 0.000 |
| num_samples | 135.000 | 0.000 |

**Model:** h4-14b-think
**Evaluation Time (hh:mm:ss):** N/A
**Temperature:** N/A
**Overlong samples:** 0.0% (0 / 135)

### gpqa_diamond

| Metric | Score | Std Error |
|--------|-------|----------|
| gpqa_pass@1:1_samples | 0.561 | 0.035 |
| gpqa_pass@1:4_samples | 0.557 | 0.029 |
| gpqa_pass@1:8_samples | 0.556 | 0.027 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:31:23
**Temperature:** 0.6
**Overlong samples:** 8.5% (135 / 1584)

### ifeval

| Metric | Score | Std Error |
|--------|-------|----------|
| prompt_level_strict_acc | 0.320 | 0.020 |
| inst_level_strict_acc | 0.457 | 0.001 |
| prompt_level_loose_acc | 0.368 | 0.021 |
| inst_level_loose_acc | 0.501 | 0.001 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:12:34
**Temperature:** 0.6
**Overlong samples:** 6.7% (36 / 541)

### lcb-v6-aug2024+

| Metric | Score | Std Error |
|--------|-------|----------|
| eval/pass_1 | 0.452 | 0.000 |
| eval/easy_pass_1 | 0.930 | 0.000 |
| eval/medium_pass_1 | 0.538 | 0.000 |
| eval/hard_pass_1 | 0.133 | 0.000 |
| eval/completion_length | 56586.253 | 0.000 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 04:18:48
**Temperature:** N/A
**Overlong samples:** 12.7% (923 / 7264)

### math_500

| Metric | Score | Std Error |
|--------|-------|----------|
| math_pass@1:1_samples | 0.934 | 0.011 |
| math_pass@1:4_samples | 0.926 | 0.009 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:13:14
**Temperature:** 0.6
**Overlong samples:** 0.9% (17 / 2000)

### mmlu_generative

| Metric | Score | Std Error |
|--------|-------|----------|
| extractive_match | 0.838 | 0.003 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 01:04:09
**Temperature:** 0.6
**Overlong samples:** 2.3% (327 / 14042)

### mmlu_pro

| Metric | Score | Std Error |
|--------|-------|----------|
| pass@1:1_samples | 0.733 | 0.004 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 02:08:06
**Temperature:** 0.6
**Overlong samples:** 6.6% (793 / 12032)

### musr_generative

| Metric | Score | Std Error |
|--------|-------|----------|
| extractive_match | 0.591 | 0.030 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:10:38
**Temperature:** 0.6
**Overlong samples:** 2.9% (22 / 756)

### obqa_generative

| Metric | Score | Std Error |
|--------|-------|----------|
| extractive_match | 0.934 | 0.011 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:08:17
**Temperature:** 0.6
**Overlong samples:** 2.2% (11 / 500)

### rewardbench

| Metric | Score | Std Error |
|--------|-------|----------|
| eval/percent_correct | 0.048 | 0.000 |
| eval/total_samples | 1865.000 | 0.000 |
| eval/correct_samples | 89.000 | 0.000 |
| eval/format_compliance_rate | 0.055 | 0.000 |
| eval/avg_response_length | 7469.826 | 0.000 |
| eval/response_length_std | 9635.779 | 0.000 |
| eval/judgment_entropy | 0.000 | 0.000 |
| eval/most_common_judgment_freq | 1.000 | 0.000 |
| eval/format_error_rate | 1.000 | 0.000 |
| eval/avg_ties_rating | 3.495 | 0.000 |
| eval/ties_error_rate | 0.270 | 0.000 |
| eval/percent_correct_Factuality | 0.000 | 0.000 |
| eval/percent_correct_Precise IF | 0.000 | 0.000 |
| eval/percent_correct_Math | 0.000 | 0.000 |
| eval/percent_correct_Safety | 0.000 | 0.000 |
| eval/percent_correct_Focus | 0.000 | 0.000 |
| eval/percent_correct_Ties | 0.873 | 0.000 |
| eval/choice_samples | 1763.000 | 0.000 |
| eval/ties_samples | 102.000 | 0.000 |
| eval/choice_format_compliance_rate | 0.000 | 0.000 |
| eval/ties_format_compliance_rate | 1.000 | 0.000 |
| eval/wrong_answer_a_bias_rate | 0.000 | 0.000 |
| eval/wrong_answer_total_count | 1763.000 | 0.000 |
| eval/wrong_answer_a_count | 0.000 | 0.000 |

**Model:** h4-14b-think
**Evaluation Time (hh:mm:ss):** 00:11:23
**Temperature:** 0.6
**Overlong samples:** 2.5% (47 / 1865)

### simpleqa_nous

| Metric | Score | Std Error |
|--------|-------|----------|
| exact_match | 0.038 | 0.003 |
| fuzzy_match | 0.054 | 0.003 |

**Model:** h4-e3-overlong-masked-30k-rerun
**Evaluation Time (hh:mm:ss):** 00:39:19
**Temperature:** 0.6
**Overlong samples:** 9.6% (416 / 4321)