vcoder17/codeforces

1---2dataset_info:3- config_name: default4  features:5  - name: id6    dtype: string7  - name: aliases8    sequence: string9  - name: contest_id10    dtype: string11  - name: contest_name12    dtype: string13  - name: contest_type14    dtype: string15  - name: contest_start16    dtype: int6417  - name: contest_start_year18    dtype: int6419  - name: index20    dtype: string21  - name: time_limit22    dtype: float6423  - name: memory_limit24    dtype: float6425  - name: title26    dtype: string27  - name: description28    dtype: string29  - name: input_format30    dtype: string31  - name: output_format32    dtype: string33  - name: interaction_format34    dtype: string35  - name: note36    dtype: string37  - name: examples38    list:39    - name: input40      dtype: string41    - name: output42      dtype: string43  - name: editorial44    dtype: string45  - name: rating46    dtype: int6447  - name: tags48    sequence: string49  - name: testset_size50    dtype: int6451  - name: official_tests52    list:53    - name: input54      dtype: string55    - name: output56      dtype: string57  - name: official_tests_complete58    dtype: bool59  - name: input_mode60    dtype: string61  - name: generated_checker62    dtype: string63  - name: executable64    dtype: bool65  - name: generated_tests66    dtype: int6467  splits:68  - name: train69    num_bytes: 536120607170    num_examples: 955671  - name: test72    num_bytes: 10843117573    num_examples: 46874  download_size: 275529513475  dataset_size: 546963724676- config_name: verifiable77  features:78  - name: id79    dtype: string80  - name: aliases81    sequence: string82  - name: contest_id83    dtype: string84  - name: contest_name85    dtype: string86  - name: contest_type87    dtype: string88  - name: contest_start89    dtype: int6490  - name: contest_start_year91    dtype: int6492  - name: index93    dtype: string94  - name: time_limit95    dtype: float6496  - name: memory_limit97    dtype: float6498  - name: title99    dtype: string100  - name: description101    dtype: string102  - name: input_format103    dtype: string104  - name: output_format105    dtype: string106  - name: interaction_format107    dtype: string108  - name: note109    dtype: string110  - name: examples111    list:112    - name: input113      dtype: string114    - name: output115      dtype: string116  - name: editorial117    dtype: string118  - name: rating119    dtype: int64120  - name: tags121    sequence: string122  - name: testset_size123    dtype: int64124  - name: official_tests125    list:126    - name: input127      dtype: string128    - name: output129      dtype: string130  - name: official_tests_complete131    dtype: bool132  - name: input_mode133    dtype: string134  - name: generated_checker135    dtype: string136  - name: executable137    dtype: bool138  - name: generated_tests139    dtype: int64140  splits:141  - name: train142    num_bytes: 4677870975.757639143    num_examples: 8338144  - name: test145    num_bytes: 97773409.93589744146    num_examples: 422147  download_size: 2461572745148  dataset_size: 4775644385.693537149- config_name: verifiable-prompts150  features:151  - name: id152    dtype: string153  - name: aliases154    sequence: string155  - name: contest_id156    dtype: string157  - name: contest_name158    dtype: string159  - name: contest_type160    dtype: string161  - name: contest_start162    dtype: int64163  - name: contest_start_year164    dtype: int64165  - name: index166    dtype: string167  - name: time_limit168    dtype: float64169  - name: memory_limit170    dtype: float64171  - name: title172    dtype: string173  - name: description174    dtype: string175  - name: input_format176    dtype: string177  - name: output_format178    dtype: string179  - name: interaction_format180    dtype: string181  - name: note182    dtype: string183  - name: examples184    list:185    - name: input186      dtype: string187    - name: output188      dtype: string189  - name: editorial190    dtype: string191  - name: rating192    dtype: int64193  - name: tags194    sequence: string195  - name: testset_size196    dtype: int64197  - name: official_tests198    list:199    - name: input200      dtype: string201    - name: output202      dtype: string203  - name: official_tests_complete204    dtype: bool205  - name: input_mode206    dtype: string207  - name: generated_checker208    dtype: string209  - name: executable210    dtype: bool211  - name: generated_tests212    dtype: int64213  - name: prompt214    dtype: string215  - name: language216    dtype: string217  splits:218  - name: train219    num_bytes: 9392758048220    num_examples: 16676221  - name: test222    num_bytes: 190953774223    num_examples: 844224  download_size: 4936052469225  dataset_size: 9583711822226configs:227- config_name: default228  data_files:229  - split: train230    path: data/train-*231  - split: test232    path: data/test-*233- config_name: verifiable234  data_files:235  - split: train236    path: verifiable/train-*237  - split: test238    path: verifiable/test-*239- config_name: verifiable-prompts240  data_files:241  - split: train242    path: verifiable-prompts/train-*243  - split: test244    path: verifiable-prompts/test-*245license: cc-by-4.0246---247# Dataset Card for CodeForces248 249## Dataset description250[CodeForces](https://codeforces.com/) is one of the most popular websites among competitive programmers, hosting regular contests where participants must solve challenging algorithmic optimization problems. The challenging nature of these problems makes them an interesting dataset to improve and test models’ code reasoning capabilities. This dataset includes more than **10k unique problems** covering the very first contests all the way to 2025. Additionally, we generated and extensively validated custom checker code for problems with multiple solutions, and generated additional challenging test cases.251 252## Why most datasets are actually not "verifiable"253 254Competitive programming problems require solutions to solve any valid test case within the input constraints within certain time and memory limits. Usually, the most challenging test cases requiring truly optimized (and correct) solutions consist of **very large test inputs**.255 256While contest platforms like CodeForces let users see test cases, they **truncate** them to at most \~400 characters. This means that the problems on "verifiable" datasets often only contain the most simple, trivial test cases that can be solved with an easy brute force solution.257Additionally, many problems allow for multiple correct answers, and therefore require a special program, a "checker" to validate if a user submitted answer is correct (~30% of problems).258Fully verifable problems thus require two things: being able to properly validate test cases (checkers), and having challenging test cases that ensure solutions are correct.259 260## What sets this dataset apart261 262Besides covering more problems than previous efforts, this dataset:263 264- **editorials**: includes *editorials*, an explanation, written by the contest organizers, explaining the correct solution (60% of problems)265- **latex images**: has proper text versions of latex rendered equations (images), OCR'd with `Qwen/Qwen2.5-VL-7B-Instruct`266- **checkers**: to make sure we could evaluate problems with multiple correct possible answers, we:267  1. Used real human contestant solutions, we made sure problems were executable268  2. With those solutions, we determined which problems might require a custom checker (when some correct solutions produce a result that does not match the "correct" output)269  3. Generated 20-100 custom checkers using DeepSeek-R1 (depending on the problem)270  4. Tested them until one properly validated the real human correct solutions (judged their outputs to be correct)271- **additional test cases**: to ensure proper verifiability, we generated additional test cases for problems whose public test cases had been truncated:272  1. Had DeepSeek-R1 create test case generators covering tricky edge cases and making full use of the input limits. Coming up with test cases is considerably easier than fully solving the problem273  2. Used one of the correct human contestant solutions to obtain the "correct output" of each test case274  3. Run multiple correct human solutions through the test case input+output, and removed test cases where the correct solutions did not agree on the result275  4. Using some incorrect solutions (when available) that passed multiple public tests (and sometimes even all public test cases), we ordered the generated test cases by how hard they are (e.g., if all incorrect solutions solve it, then it's quite easy) and also by the size of the input, under the assumption that larger inputs correspond to harder test cases276  5. Selected the top hardest (according to our heuristic) test cases, as well as some other randomly sampled test cases of differing difficulties, to ensure intermediate rewards as model solutions improve277 278## Subsets279We have split the dataset into two subsets:280- `default`: all problems (~10k problems)281- `verifiable`: only problems that are `executable` AND that either have `official_tests_complete` or some `generated_tests` available (8760 problems)282- `verifiable-prompts`: same as `verifiable` but with 2 prompts for generation per problem (one in python and one in cplusplus). See columns `language` and `prompt`283 284 285## Downloading generated tests286Due to their large size (~110GB), you need to download the generated test cases separately:287```bash288pip install -U huggingface_hub[cli,hf_xet]289# change PATH_TO_SAVE_TESTCASES. Increase --max-workers according to your machine's capacity290huggingface-cli download open-r1/codeforces --repo-type=dataset --include='generated_tests/*.parquet' --max-workers=8 --local-dir PATH_TO_SAVE_TESTCASES 291```292Test cases are split per contest and named `test_cases_XXXX.parquet` where X is the contest ID. Each parquet file has 4 columns: `problem_id` (to be matched with `id` in the main dataset), `input` (str), `output` (str) and `test_case_i` (int).293 294## Splits295We provide a `test` split with problems from late 2024 and early 2025. Please avoid training on these.296 297## Data fields298### General problem/contest fields299- `id` (str): unique submission ID300- `aliases` (list[str]): list of other problem IDs that are copies of this one (quite common between Div 1. and Div 2. contests, where some problems are in both)301- `index` (str): usually a letter, indicating the index of this problem in the contest302- `contest_id` (str): the ID of the contest this problem belongs to303- `contest_name` (str): name of the contest this problem belongs to304- `contest_type` (str): 'ICPC', 'CF' or 'IOI'305- `contest_start` (int): Contest start time in unix format306- `contest_start_year` (int): year of the contest307### Problem statement fields308- `time_limit` (float): execution time limit for each test case, in seconds309- `memory_limit` (float): execution memory limit for each test case, in megabytes310- `title` (str): problem title311- `description` (str): main problem description312- `input_format` (str): explanation of how test case inputs will be structured313- `output_format` (str): explanation of how test case outputs should be structured314- `interaction_format` (str): (for interactive problems only, which our execution currently does not support): describes how the program should interact with the grader (which we do not have)315- `note` (str): short explanation of how the examples are solved316- `examples` (list[{input: str, output: str}]): example test cases that are shown in the problem statement317### Problem metadata318- `editorial` (str): explanation of the solution from the original problem authors (when available)319- `rating` (str): problem rating (difficulty).320- `tags` (list[str]): problem tags321### Test and execution data322- `testset_size` (int): number of tests in the full testset. This is obtained from the `passedTestCount` value of correct solutions to this problem323- `official_tests` (list[{input: str, output: str}]): all test cases from CodeForces that are not **truncated** (< 400 chars)324- `official_tests_complete` (bool): whether we have all the official test cases for this problem (no test case was truncated)325- `input_mode` (str): 'stdio' or 'file'. How the input and output should be sent to the program. stdio=standard input. file=read from 'input.txt' and write to 'output.txt'326- `generated_checker` (str): python program (checker.py) that should be run with `python checker.py input.txt correct_output.txt solution_output.txt` to validate the solution, when there are multiple possible answers to this problem (prints 0-1 or 0-100 to stdout)327- `executable` (bool): whether we have been able to run at least 3 human created solutions to this problem and had them pass the `official_tests` (submissions are in `open-r1/codeforces-submissions`)328- `generated_tests` (int): number of generated and validated additional tests created for improved verifiability. See "Downloading generated tests" above.329### RL/generation related fields330For the `verifiable-prompts` subset only:331 332- `language`: `python` or `cpp`. Each problem has a row with `python` and another with `cpp`333- `prompt`: fully formatted prompt asking the model to generate a solution that solves the problem within the given time and memory constraints, in language `language`. Ready to use for RL334 335## Loading the dataset336 337```python338from datasets import load_dataset339 340ds = load_dataset("open-r1/codeforces-submissions", split="train", name='default')341OR342ds = load_dataset("open-r1/codeforces-submissions", split="train", name='verifiable')343```344See other CodeForces related datasets in [this collection](https://huggingface.co/collections/open-r1/codeforces-68234ed24aa9d65720663bd2).345 346## Verifying problems347We recommend using our [**compile**](https://github.com/guipenedo/piston/blob/master/packages/codeforces/1.0.0/compile) and [**run**](https://github.com/guipenedo/piston/blob/master/packages/codeforces/1.0.0/run) scripts developed specifically for this dataset.348Here's an example of the payload to evaluate a problem using [piston](https://github.com/huggingface/open-r1/blob/main/slurm/piston/README.md), which runs these two scripts under the hood:349```python350source_code = "..." # source_code is the model generated code351endpoint = "http://piston_endpoint:piston_port"352extension, piston_language = "cpp", "cf_c++17"353# problem_data is a row from this dataset354test_case = problem_data['official_tests'][0]  # if this problem also has generated_tests, you should run those too355payload = {356    "language": piston_language,357    "version": "*", 358    "files": [359        {360            "name": f"main.{extension}",361            "content": source_code362        },363        {364            "name": "input.txt",365            "content": test_case['input']366        },367        {368            "name": "correct_output.txt", 369            "content": test_case['output']370        },371        *([{"name": "checker.py", "content": problem_data['generated_checker']}] if problem_data['generated_checker'] else []),372        {373            "name": "grader_config",374            "content": "\n".join(375                f"{key}={value}" for key, value in {376                    "TIME_LIMIT": problem_data['time_limit'],377                    "MEMORY_LIMIT": problem_data['memory_limit'],378                    "INPUT_MODE": problem_data['input_mode']379                }.items()380            )381        }382    ]383}384result = requests.post(f"{endpoint}/execute", json=payload, headers={"Content-Type": "application/json"})385# example correct result:386# {387#   "compile": {388#     "stdout": "",389#     "stderr": "",390#     "code": 0,391#     "signal": null,392#     "output": ""393#   },394#   "run": {395#     "stdout": "1\n",  <- this is the actual score. 0=wrong/TLE/MLE; 1=correct396#     "stderr": "Output is correct (diff)\n",397#     "code": 0,398#     "signal": null,399#     "output": "1\nOutput is correct (diff)\n"400#   },401#   "language": "c++",402#   "version": "1.0.0"403# }404 405# example incorrect solution:406# {407#   "compile": {408#     "stdout": "Skipping compile - python3\n",409#     "stderr": "",410#     "code": 0,411#     "signal": null,412#     "output": "Skipping compile - python3\n"413#   },414#   "run": {415#     "stdout": "0\n",416#     "stderr": "Output isn't correct (checker)\n",417#     "code": 0,418#     "signal": null,419#     "output": "0\nOutput isn't correct (checker)\n"420#   },421#   "language": "python3",422#   "version": "1.0.0"423# }424result_is_correct = result and 'compile' in result and result['compile']['code'] == 0 and result['run']['code'] == 0 and result['run']['stdout'].split() == '1'425```426 427## License428The dataset is licensed under the Open Data Commons Attribution License (ODC-By) 4.0 license.429 430## Citation431 432If you find CodeForces useful in your work, please consider citing it as:433 434```435@misc{penedo2025codeforces,436      title={CodeForces}, 437      author={Guilherme Penedo and Anton Lozhkov and Hynek Kydlíček and Loubna Ben Allal and Edward Beeching and Agustín Piqueres Lajarín and Quentin Gallouédec and Nathan Habib and Lewis Tunstall and Leandro von Werra},438      year={2025},439      publisher = {Hugging Face},440      journal = {Hugging Face repository},441      howpublished = {\url{https://huggingface.co/datasets/open-r1/codeforces}}442}443```