DeepSeek-R1-Distill-Qwen-1.5B_eval_5554

# mlfoundations-dev/DeepSeek-R1-Distill-Qwen-1.5B_eval_5554 Precomputed model outputs for evaluation. ## Evaluation Results ### Summary | Metric | AIME24 | AMC23 | MATH500 | MMLUPro | JEEBench | GPQADiamond | LiveCodeBench | CodeElo | CodeForces | HLE | HMMT | AIME25 | LiveCodeBenchv5 | |--------|------|-----|-------|-------|--------|-----------|-------------|-------|----------|---|----|------|---------------| | Accuracy | 32.7 | 71.8 | 80.8 | 31.1 | 32.5 | 31.1 | 27.2 | 8.8 | 8.5 | 15.0 | 15.3 | 23.7 | 15.4 | ### AIME24 - **Average Accuracy**: 32.67% ± 2.39% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 40.00% | 12 | 30 | | 2 | 16.67% | 5 | 30 | | 3 | 33.33% | 10 | 30 | | 4 | 33.33% | 10 | 30 | | 5 | 33.33% | 10 | 30 | | 6 | 40.00% | 12 | 30 | | 7 | 20.00% | 6 | 30 | | 8 | 36.67% | 11 | 30 | | 9 | 36.67% | 11 | 30 | | 10 | 36.67% | 11 | 30 | ### AMC23 - **Average Accuracy**: 71.75% ± 1.23% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 67.50% | 27 | 40 | | 2 | 72.50% | 29 | 40 | | 3 | 75.00% | 30 | 40 | | 4 | 77.50% | 31 | 40 | | 5 | 70.00% | 28 | 40 | | 6 | 70.00% | 28 | 40 | | 7 | 70.00% | 28 | 40 | | 8 | 77.50% | 31 | 40 | | 9 | 65.00% | 26 | 40 | | 10 | 72.50% | 29 | 40 | ### MATH500 - **Accuracy**: 80.80% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 80.80% | 404 | 500 | ### MMLUPro - **Accuracy**: 31.09% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 31.09% | N/A | N/A | ### JEEBench - **Average Accuracy**: 32.48% ± 0.78% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 32.67% | 168.25 | 515 | | 2 | 30.73% | 158.25 | 515 | | 3 | 34.03% | 175.25 | 515 | ### GPQADiamond - **Average Accuracy**: 31.14% ± 2.31% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 32.32% | 64 | 198 | | 2 | 25.76% | 51 | 198 | | 3 | 35.35% | 70 | 198 | ### LiveCodeBench - **Average Accuracy**: 27.20% ± 1.48% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 28.38% | 145 | 511 | | 2 | 24.27% | 124 | 511 | | 3 | 28.96% | 148 | 511 | ### CodeElo - **Average Accuracy**: 8.78% ± 0.37% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 8.70% | 34 | 391 | | 2 | 8.18% | 32 | 391 | | 3 | 9.46% | 37 | 391 | ### CodeForces - **Average Accuracy**: 8.46% ± 0.53% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 8.17% | 37 | 453 | | 2 | 7.73% | 35 | 453 | | 3 | 9.49% | 43 | 453 | ### HLE - **Average Accuracy**: 15.01% ± 0.32% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 14.62% | 75 | 513 | | 2 | 15.79% | 81 | 513 | | 3 | 14.62% | 75 | 513 | ### HMMT - **Average Accuracy**: 15.33% ± 1.43% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 16.67% | 5 | 30 | | 2 | 6.67% | 2 | 30 | | 3 | 13.33% | 4 | 30 | | 4 | 16.67% | 5 | 30 | | 5 | 10.00% | 3 | 30 | | 6 | 16.67% | 5 | 30 | | 7 | 13.33% | 4 | 30 | | 8 | 16.67% | 5 | 30 | | 9 | 20.00% | 6 | 30 | | 10 | 23.33% | 7 | 30 | ### AIME25 - **Average Accuracy**: 23.67% ± 1.20% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 26.67% | 8 | 30 | | 2 | 26.67% | 8 | 30 | | 3 | 23.33% | 7 | 30 | | 4 | 20.00% | 6 | 30 | | 5 | 23.33% | 7 | 30 | | 6 | 26.67% | 8 | 30 | | 7 | 20.00% | 6 | 30 | | 8 | 16.67% | 5 | 30 | | 9 | 30.00% | 9 | 30 | | 10 | 23.33% | 7 | 30 | ### LiveCodeBenchv5 - **Average Accuracy**: 15.45% ± 0.27% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 15.72% | 58 | 369 | | 2 | 14.91% | 55 | 369 | | 3 | 15.72% | 58 | 369 |