b2_code_fasttext_pos_all_neg_sql_0.3k_eval_636d

# mlfoundations-dev/b2_code_fasttext_pos_all_neg_sql_0.3k_eval_636d Precomputed model outputs for evaluation. ## Evaluation Results ### Summary | Metric | AIME24 | AMC23 | MATH500 | MMLUPro | JEEBench | GPQADiamond | LiveCodeBench | CodeElo | CodeForces | |--------|------|-----|-------|-------|--------|-----------|-------------|-------|----------| | Accuracy | 16.3 | 55.5 | 72.4 | 26.6 | 42.7 | 44.4 | 30.3 | 7.3 | 10.1 | ### AIME24 - **Average Accuracy**: 16.33% ± 0.99% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 13.33% | 4 | 30 | | 2 | 20.00% | 6 | 30 | | 3 | 16.67% | 5 | 30 | | 4 | 13.33% | 4 | 30 | | 5 | 16.67% | 5 | 30 | | 6 | 16.67% | 5 | 30 | | 7 | 13.33% | 4 | 30 | | 8 | 13.33% | 4 | 30 | | 9 | 16.67% | 5 | 30 | | 10 | 23.33% | 7 | 30 | ### AMC23 - **Average Accuracy**: 55.50% ± 0.92% - **Number of Runs**: 10 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 60.00% | 24 | 40 | | 2 | 55.00% | 22 | 40 | | 3 | 50.00% | 20 | 40 | | 4 | 60.00% | 24 | 40 | | 5 | 52.50% | 21 | 40 | | 6 | 55.00% | 22 | 40 | | 7 | 57.50% | 23 | 40 | | 8 | 55.00% | 22 | 40 | | 9 | 55.00% | 22 | 40 | | 10 | 55.00% | 22 | 40 | ### MATH500 - **Accuracy**: 72.40% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 72.40% | 362 | 500 | ### MMLUPro - **Average Accuracy**: 26.60% ± 0.00% - **Number of Runs**: 1 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 26.60% | 133 | 500 | ### JEEBench - **Average Accuracy**: 42.67% ± 0.59% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 43.06% | 221.75 | 515 | | 2 | 41.26% | 212.5 | 515 | | 3 | 43.69% | 225.0 | 515 | ### GPQADiamond - **Average Accuracy**: 44.44% ± 0.63% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 45.96% | 91 | 198 | | 2 | 43.94% | 87 | 198 | | 3 | 43.43% | 86 | 198 | ### LiveCodeBench - **Average Accuracy**: 30.27% ± 0.68% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 30.14% | 154 | 511 | | 2 | 31.51% | 161 | 511 | | 3 | 29.16% | 149 | 511 | ### CodeElo - **Average Accuracy**: 7.33% ± 0.76% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 7.67% | 30 | 391 | | 2 | 8.44% | 33 | 391 | | 3 | 5.88% | 23 | 391 | ### CodeForces - **Average Accuracy**: 10.08% ± 0.29% - **Number of Runs**: 3 | Run | Accuracy | Questions Solved | Total Questions | |-----|----------|-----------------|----------------| | 1 | 10.38% | 47 | 453 | | 2 | 9.49% | 43 | 453 | | 3 | 10.38% | 47 | 453 |