eval-Hermes-4.3-36B-centralized

# 36btorchtitan Evaluation Results ## Summary | Benchmark | Score | Metric | Samples | Overlong rate | |-----------|-------|--------|---------|---------------| | aime24 | 0.706 | math_pass@1:64_samples | 64 | 24.6% | | aime25 | 0.669 | math_pass@1:64_samples | 64 | 26.8% | | bbh_generative | 0.847 | extractive_match | 1 | 9.3% | | drop_generative_nous | 0.817 | drop_acc | 1 | 6.9% | | gpqa_diamond | 0.649 | gpqa_pass@1:8_samples | 8 | 4.2% | | ifeval | 0.740 | inst_level_loose_acc | 1 | 13.1% | | math_500 | 0.923 | math_pass@1:4_samples | 4 | 3.1% | | mmlu_generative | 0.866 | extractive_match | 1 | 1.4% | | mmlu_pro | 0.797 | pass@1:1_samples | 1 | 2.9% | | musr_generative | 0.647 | extractive_match | 1 | 17.2% | | obqa_generative | 0.918 | extractive_match | 1 | 5.0% | | simpleqa_nous | 0.056 | fuzzy_match | 1 | 22.8% | Overlong rate: 4,041 / 54,663 samples (7.4%) missing closing `</think>` tag ## Detailed Results ### aime24 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.767 | 0.079 | | math_pass@1:4_samples | 0.733 | 0.070 | | math_pass@1:8_samples | 0.700 | 0.064 | | math_pass@1:16_samples | 0.690 | 0.064 | | math_pass@1:32_samples | 0.702 | 0.066 | | math_pass@1:64_samples | 0.706 | 0.066 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 03:10:32 **Temperature:** 0.6 **Overlong samples:** 24.6% (473 / 1920) ### aime25 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.633 | 0.089 | | math_pass@1:4_samples | 0.683 | 0.077 | | math_pass@1:8_samples | 0.646 | 0.072 | | math_pass@1:16_samples | 0.650 | 0.069 | | math_pass@1:32_samples | 0.662 | 0.068 | | math_pass@1:64_samples | 0.669 | 0.067 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 03:16:06 **Temperature:** 0.6 **Overlong samples:** 26.8% (515 / 1920) ### bbh_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.847 | 0.019 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 02:05:45 **Temperature:** 0.6 **Overlong samples:** 9.3% (513 / 5511) ### drop_generative_nous | Metric | Score | Std Error | |--------|-------|----------| | drop_acc | 0.817 | 0.004 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 02:08:58 **Temperature:** 0.6 **Overlong samples:** 6.9% (656 / 9536) ### gpqa_diamond | Metric | Score | Std Error | |--------|-------|----------| | gpqa_pass@1:1_samples | 0.677 | 0.033 | | gpqa_pass@1:4_samples | 0.657 | 0.028 | | gpqa_pass@1:8_samples | 0.649 | 0.027 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 01:27:00 **Temperature:** 0.6 **Overlong samples:** 4.2% (67 / 1584) ### ifeval | Metric | Score | Std Error | |--------|-------|----------| | prompt_level_strict_acc | 0.595 | 0.021 | | inst_level_strict_acc | 0.697 | 0.001 | | prompt_level_loose_acc | 0.654 | 0.020 | | inst_level_loose_acc | 0.740 | 0.001 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 00:32:45 **Temperature:** 0.6 **Overlong samples:** 13.1% (71 / 541) ### math_500 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.924 | 0.012 | | math_pass@1:4_samples | 0.923 | 0.008 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 00:40:12 **Temperature:** 0.6 **Overlong samples:** 3.1% (63 / 2000) ### mmlu_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.866 | 0.003 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 02:32:05 **Temperature:** 0.6 **Overlong samples:** 1.4% (200 / 14042) ### mmlu_pro | Metric | Score | Std Error | |--------|-------|----------| | pass@1:1_samples | 0.797 | 0.004 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 05:52:54 **Temperature:** 0.6 **Overlong samples:** 2.9% (343 / 12032) ### musr_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.647 | 0.029 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 00:41:25 **Temperature:** 0.6 **Overlong samples:** 17.2% (130 / 756) ### obqa_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.918 | 0.012 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 00:23:38 **Temperature:** 0.6 **Overlong samples:** 5.0% (25 / 500) ### simpleqa_nous | Metric | Score | Std Error | |--------|-------|----------| | exact_match | 0.038 | 0.003 | | fuzzy_match | 0.056 | 0.003 | **Model:** 36btorchtitan **Evaluation Time (hh:mm:ss):** 02:52:46 **Temperature:** 0.6 **Overlong samples:** 22.8% (985 / 4321)