b2_code_fasttext_pos_all_neg_sql_1k_eval_636d

# mlfoundations-dev/b2_code_fasttext_pos_all_neg_sql_1k_eval_636d Precomputed model outputs for evaluation. ## Evaluation Results ### Summary | Metric | AIME24 | AMC23 | MATH500 | MMLUPro | JEEBench | GPQADiamond | LiveCodeBench | CodeElo | CodeForces | |--------|------|-----|-------|-------|--------|-----------|-------------|-------|----------| | Accuracy | 17.0 | 53.0 | 72.2 | 27.4 | 40.7 | 35.7 | 30.5 | 7.1 | 9.3 | ### AIME24 - **Average Accuracy**: 17.00% ± 1.60% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 20.00% | 6 | 30 | | 2 | 26.67% | 8 | 30 | | 3 | 10.00% | 3 | 30 | | 4 | 20.00% | 6 | 30 | | 5 | 20.00% | 6 | 30 | | 6 | 10.00% | 3 | 30 | | 7 | 13.33% | 4 | 30 | | 8 | 13.33% | 4 | 30 | | 9 | 20.00% | 6 | 30 | | 10 | 16.67% | 5 | 30 | ### AMC23 - **Average Accuracy**: 53.00% ± 1.83% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 50.00% | 20 | 40 | | 2 | 62.50% | 25 | 40 | | 3 | 52.50% | 21 | 40 | | 4 | 42.50% | 17 | 40 | | 5 | 55.00% | 22 | 40 | | 6 | 47.50% | 19 | 40 | | 7 | 47.50% | 19 | 40 | | 8 | 57.50% | 23 | 40 | | 9 | 57.50% | 23 | 40 | | 10 | 57.50% | 23 | 40 | ### MATH500 - **Accuracy**: 72.20% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 72.20% | 361 | 500 | ### MMLUPro - **Average Accuracy**: 27.40% ± 0.00% - **Number of Runs**: 1 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 27.40% | 137 | 500 | ### JEEBench - **Average Accuracy**: 40.74% ± 0.77% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 39.22% | 202.0 | 515 | | 2 | 40.53% | 208.75 | 515 | | 3 | 42.48% | 218.75 | 515 | ### GPQADiamond - **Average Accuracy**: 35.69% ± 0.14% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 35.35% | 70 | 198 | | 2 | 35.86% | 71 | 198 | | 3 | 35.86% | 71 | 198 | ### LiveCodeBench - **Average Accuracy**: 30.53% ± 0.85% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 31.90% | 163 | 511 | | 2 | 28.96% | 148 | 511 | | 3 | 30.72% | 157 | 511 | ### CodeElo - **Average Accuracy**: 7.08% ± 0.23% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 7.16% | 28 | 391 | | 2 | 6.65% | 26 | 391 | | 3 | 7.42% | 29 | 391 | ### CodeForces - **Average Accuracy**: 9.35% ± 0.39% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 9.93% | 45 | 453 | | 2 | 8.61% | 39 | 453 | | 3 | 9.49% | 43 | 453 |