eval-Hermes-4.3-36B

# 36bpsychev2 Evaluation Results ## Summary | Benchmark | Score | Metric | Samples | Overlong rate | |-----------|-------|--------|---------|---------------| | aime24 | 0.719 | math_pass@1:64_samples | 64 | 17.6% | | aime25 | 0.693 | math_pass@1:64_samples | 64 | 18.8% | | bbh_generative | 0.864 | extractive_match | 1 | 4.8% | | drop_generative_nous | 0.835 | drop_acc | 1 | 2.7% | | gpqa_diamond | 0.655 | gpqa_pass@1:8_samples | 8 | 2.2% | | ifeval | 0.779 | inst_level_loose_acc | 1 | 7.8% | | math_500 | 0.938 | math_pass@1:4_samples | 4 | 1.8% | | mmlu_generative | 0.877 | extractive_match | 1 | 0.1% | | mmlu_pro | 0.807 | pass@1:1_samples | 1 | 1.1% | | musr_generative | 0.697 | extractive_match | 1 | 0.4% | | obqa_generative | 0.966 | extractive_match | 1 | 0.2% | | simpleqa_nous | 0.060 | fuzzy_match | 1 | 8.6% | Overlong rate: 1,846 / 54,663 samples (3.4%) missing closing `</think>` tag ## Detailed Results ### aime24 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.767 | 0.079 | | math_pass@1:4_samples | 0.700 | 0.068 | | math_pass@1:8_samples | 0.713 | 0.064 | | math_pass@1:16_samples | 0.719 | 0.061 | | math_pass@1:32_samples | 0.731 | 0.061 | | math_pass@1:64_samples | 0.719 | 0.062 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 02:57:47 **Temperature:** 0.6 **Overlong samples:** 17.6% (337 / 1920) ### aime25 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.633 | 0.089 | | math_pass@1:4_samples | 0.692 | 0.062 | | math_pass@1:8_samples | 0.662 | 0.063 | | math_pass@1:16_samples | 0.694 | 0.061 | | math_pass@1:32_samples | 0.693 | 0.062 | | math_pass@1:64_samples | 0.693 | 0.063 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 03:06:24 **Temperature:** 0.6 **Overlong samples:** 18.8% (360 / 1920) ### bbh_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.864 | 0.017 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 01:27:42 **Temperature:** 0.6 **Overlong samples:** 4.8% (262 / 5511) ### drop_generative_nous | Metric | Score | Std Error | |--------|-------|----------| | drop_acc | 0.835 | 0.004 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 01:22:41 **Temperature:** 0.6 **Overlong samples:** 2.7% (255 / 9536) ### gpqa_diamond | Metric | Score | Std Error | |--------|-------|----------| | gpqa_pass@1:1_samples | 0.641 | 0.034 | | gpqa_pass@1:4_samples | 0.663 | 0.027 | | gpqa_pass@1:8_samples | 0.655 | 0.026 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 01:18:07 **Temperature:** 0.6 **Overlong samples:** 2.2% (35 / 1584) ### ifeval | Metric | Score | Std Error | |--------|-------|----------| | prompt_level_strict_acc | 0.654 | 0.020 | | inst_level_strict_acc | 0.740 | 0.000 | | prompt_level_loose_acc | 0.702 | 0.020 | | inst_level_loose_acc | 0.779 | 0.000 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 00:25:46 **Temperature:** 0.6 **Overlong samples:** 7.8% (42 / 541) ### math_500 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.942 | 0.010 | | math_pass@1:4_samples | 0.938 | 0.008 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 00:33:46 **Temperature:** 0.6 **Overlong samples:** 1.8% (35 / 2000) ### mmlu_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.877 | 0.003 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 01:46:37 **Temperature:** 0.6 **Overlong samples:** 0.1% (16 / 14042) ### mmlu_pro | Metric | Score | Std Error | |--------|-------|----------| | pass@1:1_samples | 0.807 | 0.004 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 04:43:51 **Temperature:** 0.6 **Overlong samples:** 1.1% (128 / 12032) ### musr_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.697 | 0.028 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 00:23:45 **Temperature:** 0.6 **Overlong samples:** 0.4% (3 / 756) ### obqa_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.966 | 0.008 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 00:17:21 **Temperature:** 0.6 **Overlong samples:** 0.2% (1 / 500) ### simpleqa_nous | Metric | Score | Std Error | |--------|-------|----------| | exact_match | 0.039 | 0.003 | | fuzzy_match | 0.060 | 0.004 | **Model:** 36bpsychev2 **Evaluation Time (hh:mm:ss):** 01:27:06 **Temperature:** 0.6 **Overlong samples:** 8.6% (372 / 4321)