eval-Hermes-4-70B-nonreasoning

# hermes-70b-nonreasoning Evaluation Results ## Summary | Benchmark | Score | Metric | Samples | Overlong rate | |-----------|-------|--------|---------|---------------| | aime24 | 0.095 | math_pass@1:64_samples | 64 | 99.4% | | aime25 | 0.073 | math_pass@1:64_samples | 64 | 98.2% | | arenahard | 0.568 | eval/overall_winrate | 500 | 0.0% | | bbh_generative | 0.805 | extractive_match | 1 | 100.0% | | creative-writing-v3 | 0.491 | creative_writing_score | 96 | 0.0% | | drop_generative_nous | 0.784 | drop_acc | 1 | 100.0% | | eqbench3 | 0.739 | eqbench_score | 135 | 0.0% | | gpqa_diamond | 0.333 | gpqa_pass@1:8_samples | 8 | 100.0% | | ifeval | 0.823 | inst_level_loose_acc | 1 | 99.8% | | lcb-v6-aug2024+ | 0.255 | eval/pass_1 | 1 | 99.4% | | math_500 | 0.710 | math_pass@1:4_samples | 4 | 100.0% | | mmlu_generative | 0.767 | extractive_match | 1 | 100.0% | | mmlu_pro | 0.549 | pass@1:1_samples | 1 | 100.0% | | musr_generative | 0.563 | extractive_match | 1 | 100.0% | | obqa_generative | 0.900 | extractive_match | 1 | 100.0% | | rewardbench | 0.448 | eval/percent_correct | 1 | 94.3% | | simpleqa_nous | 0.133 | fuzzy_match | 1 | 100.0% | Overlong rate: 63,597 / 64,523 samples (98.6%) missing closing `</think>` tag ## Detailed Results ### aime24 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.100 | 0.056 | | math_pass@1:4_samples | 0.117 | 0.046 | | math_pass@1:8_samples | 0.104 | 0.043 | | math_pass@1:16_samples | 0.087 | 0.040 | | math_pass@1:32_samples | 0.093 | 0.042 | | math_pass@1:64_samples | 0.095 | 0.039 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:18:04 **Temperature:** 0.6 **Overlong samples:** 99.4% (1908 / 1920) ### aime25 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.100 | 0.056 | | math_pass@1:4_samples | 0.058 | 0.029 | | math_pass@1:8_samples | 0.067 | 0.030 | | math_pass@1:16_samples | 0.069 | 0.031 | | math_pass@1:32_samples | 0.068 | 0.029 | | math_pass@1:64_samples | 0.073 | 0.029 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:23:21 **Temperature:** 0.6 **Overlong samples:** 98.2% (1885 / 1920) ### arenahard | Metric | Score | Std Error | |--------|-------|----------| | eval/overall_winrate | 0.568 | 0.000 | | eval/total_samples | 500.000 | 0.000 | | eval/win_count | 233.000 | 0.000 | | eval/tie_count | 103.000 | 0.000 | | eval/loss_count | 164.000 | 0.000 | | eval/win_rate | 0.466 | 0.000 | | eval/tie_rate | 0.206 | 0.000 | | eval/loss_rate | 0.328 | 0.000 | | eval/winrate_arena-hard-v0.1 | 0.568 | 0.000 | **Model:** h4-70b-nothink-arena **Evaluation Time (hh:mm:ss):** 00:04:40 **Temperature:** 0.6 **Overlong samples:** 0.0% (0 / 500) ### bbh_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.805 | 0.023 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:29:33 **Temperature:** 0.6 **Overlong samples:** 100.0% (5511 / 5511) ### creative-writing-v3 | Metric | Score | Std Error | |--------|-------|----------| | creative_writing_score | 0.491 | 0.000 | | num_samples | 96.000 | 0.000 | **Model:** h4-70b-nonreasoning-cwlf **Evaluation Time (hh:mm:ss):** N/A **Temperature:** N/A **Overlong samples:** 0.0% (0 / 96) ### drop_generative_nous | Metric | Score | Std Error | |--------|-------|----------| | drop_acc | 0.784 | 0.004 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:37:28 **Temperature:** 0.6 **Overlong samples:** 100.0% (9536 / 9536) ### eqbench3 | Metric | Score | Std Error | |--------|-------|----------| | eqbench_score | 0.739 | 0.000 | | num_samples | 135.000 | 0.000 | **Model:** h4-70b-nothink-arena **Evaluation Time (hh:mm:ss):** N/A **Temperature:** N/A **Overlong samples:** 0.0% (0 / 135) ### gpqa_diamond | Metric | Score | Std Error | |--------|-------|----------| | gpqa_pass@1:1_samples | 0.359 | 0.034 | | gpqa_pass@1:4_samples | 0.340 | 0.025 | | gpqa_pass@1:8_samples | 0.333 | 0.022 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:16:42 **Temperature:** 0.6 **Overlong samples:** 100.0% (1584 / 1584) ### ifeval | Metric | Score | Std Error | |--------|-------|----------| | prompt_level_strict_acc | 0.701 | 0.020 | | inst_level_strict_acc | 0.789 | 0.000 | | prompt_level_loose_acc | 0.749 | 0.019 | | inst_level_loose_acc | 0.823 | 0.000 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:17:28 **Temperature:** 0.6 **Overlong samples:** 99.8% (540 / 541) ### lcb-v6-aug2024+ | Metric | Score | Std Error | |--------|-------|----------| | eval/pass_1 | 0.255 | 0.000 | | eval/easy_pass_1 | 0.743 | 0.000 | | eval/medium_pass_1 | 0.210 | 0.000 | | eval/hard_pass_1 | 0.022 | 0.000 | | eval/completion_length | 2398.123 | 0.000 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:56:48 **Temperature:** N/A **Overlong samples:** 99.4% (7224 / 7264) ### math_500 | Metric | Score | Std Error | |--------|-------|----------| | math_pass@1:1_samples | 0.706 | 0.020 | | math_pass@1:4_samples | 0.710 | 0.017 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:14:30 **Temperature:** 0.6 **Overlong samples:** 100.0% (1999 / 2000) ### mmlu_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.767 | 0.004 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:55:09 **Temperature:** 0.6 **Overlong samples:** 100.0% (14042 / 14042) ### mmlu_pro | Metric | Score | Std Error | |--------|-------|----------| | pass@1:1_samples | 0.549 | 0.005 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 01:06:59 **Temperature:** 0.6 **Overlong samples:** 100.0% (12032 / 12032) ### musr_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.563 | 0.031 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:03:02 **Temperature:** 0.6 **Overlong samples:** 100.0% (756 / 756) ### obqa_generative | Metric | Score | Std Error | |--------|-------|----------| | extractive_match | 0.900 | 0.013 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:02:00 **Temperature:** 0.6 **Overlong samples:** 100.0% (500 / 500) ### rewardbench | Metric | Score | Std Error | |--------|-------|----------| | eval/percent_correct | 0.448 | 0.000 | | eval/total_samples | 1865.000 | 0.000 | | eval/correct_samples | 835.000 | 0.000 | | eval/format_compliance_rate | 0.823 | 0.000 | | eval/avg_response_length | 2605.972 | 0.000 | | eval/response_length_std | 9095.798 | 0.000 | | eval/judgment_entropy | 1.529 | 0.000 | | eval/most_common_judgment_freq | 0.358 | 0.000 | | eval/format_error_rate | 0.187 | 0.000 | | eval/avg_ties_rating | 4.184 | 0.000 | | eval/ties_error_rate | 0.301 | 0.000 | | eval/percent_correct_Factuality | 0.358 | 0.000 | | eval/percent_correct_Precise IF | 0.269 | 0.000 | | eval/percent_correct_Math | 0.437 | 0.000 | | eval/percent_correct_Safety | 0.409 | 0.000 | | eval/percent_correct_Focus | 0.560 | 0.000 | | eval/percent_correct_Ties | 0.794 | 0.000 | | eval/choice_samples | 1763.000 | 0.000 | | eval/ties_samples | 102.000 | 0.000 | | eval/choice_format_compliance_rate | 0.813 | 0.000 | | eval/ties_format_compliance_rate | 1.000 | 0.000 | | eval/wrong_answer_a_bias_rate | 0.354 | 0.000 | | eval/wrong_answer_total_count | 1009.000 | 0.000 | | eval/wrong_answer_a_count | 357.000 | 0.000 | **Model:** h4-70b-nothink-arena **Evaluation Time (hh:mm:ss):** 00:13:01 **Temperature:** 0.6 **Overlong samples:** 94.3% (1759 / 1865) ### simpleqa_nous | Metric | Score | Std Error | |--------|-------|----------| | exact_match | 0.089 | 0.004 | | fuzzy_match | 0.133 | 0.005 | **Model:** hermes-70b-nonreasoning **Evaluation Time (hh:mm:ss):** 00:16:58 **Temperature:** 0.6 **Overlong samples:** 100.0% (4321 / 4321)