DeepSeek-R1-Distill-Qwen-1.5B_OpenThoughts3_eval_5554

# mlfoundations-dev/DeepSeek-R1-Distill-Qwen-1.5B_OpenThoughts3_eval_5554 Precomputed model outputs for evaluation. ## Evaluation Results ### Summary | Metric | AIME24 | AMC23 | MATH500 | MMLUPro | JEEBench | GPQADiamond | LiveCodeBench | CodeElo | CodeForces | AIME25 | HLE | LiveCodeBenchv5 | HMMT | |--------|------|-----|-------|-------|--------|-----------|-------------|-------|----------|------|---|---------------|----| | Accuracy | 0.0 | 2.2 | 1.6 | 7.0 | 16.2 | 10.3 | 10.0 | 0.9 | 1.3 | 0.7 | 5.6 | 12.8 | 0.0 | ### AIME24 - **Average Accuracy**: 0.00% ± 0.00% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 0.00% | 0 | 30 | | 2 | 0.00% | 0 | 30 | | 3 | 0.00% | 0 | 30 | | 4 | 0.00% | 0 | 30 | | 5 | 0.00% | 0 | 30 | | 6 | 0.00% | 0 | 30 | | 7 | 0.00% | 0 | 30 | | 8 | 0.00% | 0 | 30 | | 9 | 0.00% | 0 | 30 | | 10 | 0.00% | 0 | 30 | ### AMC23 - **Average Accuracy**: 2.25% ± 0.55% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 2.50% | 1 | 40 | | 2 | 2.50% | 1 | 40 | | 3 | 2.50% | 1 | 40 | | 4 | 5.00% | 2 | 40 | | 5 | 5.00% | 2 | 40 | | 6 | 0.00% | 0 | 40 | | 7 | 0.00% | 0 | 40 | | 8 | 2.50% | 1 | 40 | | 9 | 2.50% | 1 | 40 | | 10 | 0.00% | 0 | 40 | ### MATH500 - **Accuracy**: 1.60% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 1.60% | 8 | 500 | ### MMLUPro - **Accuracy**: 7.03% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 7.03% | N/A | N/A | ### JEEBench - **Average Accuracy**: 16.18% ± 0.74% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 17.77% | 91.5 | 515 | | 2 | 16.17% | 83.25 | 515 | | 3 | 14.61% | 75.25 | 515 | ### GPQADiamond - **Average Accuracy**: 10.27% ± 1.55% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 10.10% | 20 | 198 | | 2 | 7.07% | 14 | 198 | | 3 | 13.64% | 27 | 198 | ### LiveCodeBench - **Average Accuracy**: 10.05% ± 4.50% - **Number of Runs**: 6 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 20.55% | 105 | 511 | | 2 | 20.55% | 105 | 511 | | 3 | 19.18% | 98 | 511 | | 4 | 0.00% | 0 | 511 | | 5 | 0.00% | 0 | 511 | | 6 | 0.00% | 0 | 511 | ### CodeElo - **Average Accuracy**: 0.85% ± 0.09% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 0.77% | 3 | 391 | | 2 | 0.77% | 3 | 391 | | 3 | 1.02% | 4 | 391 | ### CodeForces - **Average Accuracy**: 1.25% ± 0.39% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 0.66% | 3 | 453 | | 2 | 1.99% | 9 | 453 | | 3 | 1.10% | 5 | 453 | ### AIME25 - **Average Accuracy**: 0.67% ± 0.42% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 0.00% | 0 | 30 | | 2 | 0.00% | 0 | 30 | | 3 | 0.00% | 0 | 30 | | 4 | 0.00% | 0 | 30 | | 5 | 3.33% | 1 | 30 | | 6 | 0.00% | 0 | 30 | | 7 | 0.00% | 0 | 30 | | 8 | 0.00% | 0 | 30 | | 9 | 0.00% | 0 | 30 | | 10 | 3.33% | 1 | 30 | ### HLE - **Average Accuracy**: 5.59% ± 0.35% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 5.07% | 26 | 513 | | 2 | 5.26% | 27 | 513 | | 3 | 6.43% | 33 | 513 | ### LiveCodeBenchv5 - **Average Accuracy**: 12.83% ± 0.45% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 13.28% | 49 | 369 | | 2 | 11.92% | 44 | 369 | | 3 | 13.28% | 49 | 369 | ### HMMT - **Average Accuracy**: 0.00% ± 0.00% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 0.00% | 0 | 30 | | 2 | 0.00% | 0 | 30 | | 3 | 0.00% | 0 | 30 | | 4 | 0.00% | 0 | 30 | | 5 | 0.00% | 0 | 30 | | 6 | 0.00% | 0 | 30 | | 7 | 0.00% | 0 | 30 | | 8 | 0.00% | 0 | 30 | | 9 | 0.00% | 0 | 30 | | 10 | 0.00% | 0 | 30 |