gemini_gemini-2.5-pro-preview-05-06_eval_68b3

# gemini__gemini-2.5-pro-preview-05-06 Evaluation Results Precomputed model outputs for evaluation. ## Evaluation Results ### Summary | Metric | MATH500 | CodeForces | CodeElo | MMLUPro | JEEBench | LiveCodeBenchv5 | LiveCodeBench | AIME25 | AIME24 | AMC23 | HLE | HMMT | |--------| -------|----------|-------|-------|--------|---------------|-------------|------|------|-----|---|---- | | Accuracy | 93.8 | 57.5 | 59.5 | 13.1 | 44.6 | 62.3 | 73.7 | 76.3 | 92.3 | 100.0 | 15.8 | 76.7 | ### MATH500 - **Accuracy**: 93.8% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 93.8% | 469 | 500 | ### CodeForces - **Average Accuracy**: 57.5% ± 0.6% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 57.0% | 258 | 453 | | 2 | 56.7% | 257 | 453 | | 3 | 58.7% | 266 | 453 | ### CodeElo - **Average Accuracy**: 59.5% ± 1.1% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 57.8% | 226 | 391 | | 2 | 61.6% | 241 | 391 | | 3 | 59.1% | 231 | 391 | ### MMLUPro - **Accuracy**: 13.1% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 13.1% | 1573 | 12032 | ### JEEBench - **Average Accuracy**: 44.6% ± 0.1% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 44.9% | 230.5 | 513 | | 2 | 44.3% | 227.5 | 513 | | 3 | 44.4% | 228.0 | 513 | ### LiveCodeBenchv5 - **Average Accuracy**: 62.3% ± 0.5% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 61.2% | 226 | 369 | | 2 | 62.9% | 232 | 369 | | 3 | 62.9% | 232 | 369 | ### LiveCodeBench - **Average Accuracy**: 73.7% ± 0.8% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 72.6% | 371 | 511 | | 2 | 75.1% | 384 | 511 | | 3 | 73.4% | 375 | 511 | ### AIME25 - **Average Accuracy**: 76.3% ± 1.0% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 76.7% | 23 | 30 | | 2 | 73.3% | 22 | 30 | | 3 | 80.0% | 24 | 30 | | 4 | 80.0% | 24 | 30 | | 5 | 80.0% | 24 | 30 | | 6 | 76.7% | 23 | 30 | | 7 | 70.0% | 21 | 30 | | 8 | 73.3% | 22 | 30 | | 9 | 76.7% | 23 | 30 | | 10 | 76.7% | 23 | 30 | ### AIME24 - **Average Accuracy**: 92.3% ± 1.3% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 86.7% | 26 | 30 | | 2 | 93.3% | 28 | 30 | | 3 | 93.3% | 28 | 30 | | 4 | 86.7% | 26 | 30 | | 5 | 96.7% | 29 | 30 | | 6 | 96.7% | 29 | 30 | | 7 | 93.3% | 28 | 30 | | 8 | 96.7% | 29 | 30 | | 9 | 86.7% | 26 | 30 | | 10 | 93.3% | 28 | 30 | ### AMC23 - **Average Accuracy**: 100.0% ± 0.0% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 100.0% | 40 | 40 | | 2 | 100.0% | 40 | 40 | | 3 | 100.0% | 40 | 40 | | 4 | 100.0% | 40 | 40 | | 5 | 100.0% | 40 | 40 | | 6 | 100.0% | 40 | 40 | | 7 | 100.0% | 40 | 40 | | 8 | 100.0% | 40 | 40 | | 9 | 100.0% | 40 | 40 | | 10 | 100.0% | 40 | 40 | ### HLE - **Average Accuracy**: 15.8% ± 0.1% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 15.7% | 80 | 510 | | 2 | 15.7% | 80 | 510 | | 3 | 15.9% | 81 | 510 | ### HMMT - **Average Accuracy**: 76.7% ± 1.4% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 76.7% | 23 | 30 | | 2 | 80.0% | 24 | 30 | | 3 | 76.7% | 23 | 30 | | 4 | 83.3% | 25 | 30 | | 5 | 73.3% | 22 | 30 | | 6 | 76.7% | 23 | 30 | | 7 | 80.0% | 24 | 30 | | 8 | 80.0% | 24 | 30 | | 9 | 66.7% | 20 | 30 | | 10 | 73.3% | 22 | 30 |