OpenCoder-LLM/opc-fineweb-code-corpus

OpenCoder Dataset

The OpenCoder dataset is composed of the following datasets:

• —opc-sft-stage1: the sft data used for opencoder sft-stage1
• —opc-sft-stage2: the sft data used for opencoder sft-stage2
• —opc-annealing-corpus: the synthetic data & algorithmic corpus used for opencoder annealing
• —opc-fineweb-code-corpus: the code-related page recalled from fineweb <-- you are here
• —opc-fineweb-math-corpus: the math-related page recalled from fineweb
• —refineCode-code-corpus-meta: the meta-data of RefineCode

Detailed information about the data can be found in our paper.

opc-fineweb-code-corpus

This code-related data from Fineweb was specifically used in OpenCoder pre-training. We employ fastText in three iterative rounds to recall a final dataset of 55B code and math-related data. You can find math-related data at OpenCoder-LLM/fineweb-math-corpus.

This work belongs to [INF](https://www.infly.cn/).

Citation Information

Please consider citing our paper if you find this dataset useful:

@inproceedings{Huang2024OpenCoderTO,
  title = {OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models},
  author = {Siming Huang and Tianhao Cheng and Jason Klein Liu and Jiaran Hao and Liuyihan Song and Yang Xu and J. Yang and J. H. Liu and Chenchen Zhang and Linzheng Chai and Ruifeng Yuan and Zhaoxiang Zhang and Jie Fu and Qian Liu and Ge Zhang and Zili Wang and Yuan Qi and Yinghui Xu and Wei Chu},
  year = {2024},
  url = {https://arxiv.org/pdf/2411.04905}
}