m-a-p/AetherCode
AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions Introduction Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap… See the full description on the dataset page: https://huggingface.co/datasets/m-a-p/AetherCode.
<div align="center"> <h1>AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions</h1> </div>
<div align="center" style="line-height: 1;"> <a href="https://arxiv.org/abs/2508.16402" target="blank" style="margin: 2px;"> <img alt="2508.16402" src="https://img.shields.io/badge/arXiv-2508.16402-red?logo=arxiv&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/papers/2508.16402" target="blank" style="margin: 2px;"> <img alt="Paper" src="https://img.shields.io/badge/Paper-HF-blue?logo=huggingface&logoColor=white" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/datasets/m-a-p/AetherCode" target="blank" style="margin: 2px;"> <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-m--a--p/AetherCode-536af5" style="display: inline-block; vertical-align: middle;"/> </a> <a href="https://huggingface.co/datasets/m-a-p/AetherCode/blob/main/LICENSE" style="margin: 2px;"> <img alt="Dataset License" src="https://img.shields.io/badge/DatasetLicense-CC--BY--4.0-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/> </a> </div>
Introduction
Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.
Highlights
Problem Curation from Top-Tier Competitions: AetherCode is among the first benchmark to systematically collect problems from premier programming competitions worldwide, including the Olympiad in Informatics (OI) and the International Collegiate Programming Contest (ICPC). Our process involved a comprehensive collection, meticulous cleaning, and format conversion of problems from PDF to a Markdown+LaTeX structure. Each problem statement was manually proofread for correctness, and a team of competitive programming experts mannotated each problem with classification tags.
High-Quality Test Cases: We developed a hybrid methodology, combining automated generation with expert annotation, to create high-quality test cases for every problem. We evaluated the correctness and comprehensiveness of our test cases by validating them against a large corpus of collected solutions, enforcing a standard of zero false positives and zero false negatives.
Quickstart
from datasets import load_dataset
# Load AetherCode v1 2024 (Jan 2024 - Dec 2024)
ds = load_dataset("m-a-p/AetherCode", "v1_2024")
# Load AetherCode v1 2025 (Jan 2025 - May 2025)
ds = load_dataset("m-a-p/AetherCode", "v1_2025")
Evaluation Results
Contest List and Release Schedule
In future updates, we will maintain both an open set and a private set simultaneously. The open set contains complete test cases, while the private set does not include test cases to ensure a fair evaluation. In subsequent updates, new problems will be added to the private set, and the test cases of problems in the private set will be gradually made public. In AetherCode v1, the open set includes all problems from 2024 (in subset v1_2024), and the private set includes problems from January to May 2025 (in subset v1_2025). If you need the evaluation results for the private set, please use the evaluation script we provide and send the LLM's responses to zh.wang@bytedance.com.
AetherCode v1 (Jan 2024 - May 2025) include problems from the following contests.
License
This project is licensed under CC-BY-4.0. See the LICENSE file for details.
Citation
@misc{aethercode,
title={AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions},
author={Zihan Wang and Jiaze Chen and Zhicheng Liu and Markus Mak and Yidi Du and Geonsik Moon and Luoqi Xu and Aaron Tua and Kunshuo Peng and Jiayi Lu and Mingfei Xia and Boqian Zou and Chenyang Ran and Guang Tian and Shoutai Zhu and Yeheng Duan and Zhenghui Kang and Zhenxing Lin and Shangshu Li and Qiang Luo and Qingshen Long and Zhiyong Chen and Yihan Xiao and Yurong Wu and Daoguang Zan and Yuyi Fu and Mingxuan Wang and Ming Ding},
year={2025},
eprint={2508.16402},
archivePrefix={arXiv},
primaryClass={cs.SE},
url={https://arxiv.org/abs/2508.16402},
}