Qwen2.5-32B-Instruct_eval_636d

# mlfoundations-dev/Qwen2.5-32B-Instruct_eval_636d Precomputed model outputs for evaluation. ## Evaluation Results ### Summary | Metric | AIME24 | AMC23 | MATH500 | MMLUPro | JEEBench | GPQADiamond | LiveCodeBench | CodeElo | CodeForces | |--------|------|-----|-------|-------|--------|-----------|-------------|-------|----------| | Accuracy | 16.7 | 65.0 | 78.4 | 30.6 | 49.3 | 42.6 | 49.1 | 13.8 | 19.6 | ### AIME24 - **Average Accuracy**: 16.67% ± 1.41% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 20.00% | 6 | 30 | | 2 | 26.67% | 8 | 30 | | 3 | 16.67% | 5 | 30 | | 4 | 20.00% | 6 | 30 | | 5 | 13.33% | 4 | 30 | | 6 | 13.33% | 4 | 30 | | 7 | 10.00% | 3 | 30 | | 8 | 16.67% | 5 | 30 | | 9 | 16.67% | 5 | 30 | | 10 | 13.33% | 4 | 30 | ### AMC23 - **Average Accuracy**: 65.00% ± 1.06% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 62.50% | 25 | 40 | | 2 | 60.00% | 24 | 40 | | 3 | 70.00% | 28 | 40 | | 4 | 70.00% | 28 | 40 | | 5 | 67.50% | 27 | 40 | | 6 | 62.50% | 25 | 40 | | 7 | 62.50% | 25 | 40 | | 8 | 65.00% | 26 | 40 | | 9 | 67.50% | 27 | 40 | | 10 | 62.50% | 25 | 40 | ### MATH500 - **Accuracy**: 78.40% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 78.40% | 392 | 500 | ### MMLUPro - **Average Accuracy**: 30.60% ± 0.00% - **Number of Runs**: 1 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 30.60% | 153 | 500 | ### JEEBench - **Average Accuracy**: 49.30% ± 0.58% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 47.91% | 246.75 | 515 | | 2 | 49.76% | 256.25 | 515 | | 3 | 50.24% | 258.75 | 515 | ### GPQADiamond - **Average Accuracy**: 42.59% ± 1.31% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 39.90% | 79 | 198 | | 2 | 42.42% | 84 | 198 | | 3 | 45.45% | 90 | 198 | ### LiveCodeBench - **Average Accuracy**: 49.12% ± 0.97% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 50.68% | 259 | 511 | | 2 | 47.36% | 242 | 511 | | 3 | 49.32% | 252 | 511 | ### CodeElo - **Average Accuracy**: 13.81% ± 0.00% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 13.81% | 54 | 391 | | 2 | 13.81% | 54 | 391 | | 3 | 13.81% | 54 | 391 | ### CodeForces - **Average Accuracy**: 19.65% ± 0.34% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 19.43% | 88 | 453 | | 2 | 19.21% | 87 | 453 | | 3 | 20.31% | 92 | 453 |