mlfoundations-dev/a1_code_primeintellect_real_world_swe_eval_636d

mlfoundations-dev/a1codeprimeintellectrealworldsweeval_636d

Precomputed model outputs for evaluation.

Evaluation Results

Summary

AIME24

• —Average Accuracy: 12.33% ± 1.42%
• —Number of Runs: 10

AMC23

• —Average Accuracy: 52.25% ± 2.22%
• —Number of Runs: 10

MATH500

• —Accuracy: 72.40% | Accuracy | Questions Solved | Total Questions | |----------|-----------------|----------------| | 72.40% | 362 | 500 |

MMLUPro

• —Average Accuracy: 24.40% ± 0.00%
• —Number of Runs: 1

JEEBench

• —Average Accuracy: 36.31% ± 1.42%
• —Number of Runs: 3

GPQADiamond

• —Average Accuracy: 38.55% ± 0.14%
• —Number of Runs: 3

LiveCodeBench

• —Average Accuracy: 15.92% ± 0.56%
• —Number of Runs: 3

CodeElo

• —Average Accuracy: 7.84% ± 0.37%
• —Number of Runs: 3

CodeForces

• —Average Accuracy: 8.39% ± 0.34%
• —Number of Runs: 3