Aniket200325/coder-pretrain-60gb
Coding LLM Pretraining Corpus Built with build_pretrain_dataset.ipynb + finalize_local.py (DuckDB). Composition Source Collected rows Collected size Final rows Final size code 8,415,197 30.00 GB 8,413,850 29.88 GB web 3,492,492 15.00 GB 3,390,372 14.50 GB math 659,720 3.00 GB 658,089 2.95 GB wiki 519,584 2.00 GB 519,535 1.99 GB docs 4,504,069 10.00 GB 4,289,545 9.74 GB Total final corpus: 59.06 GB of raw text (17,271,391 documents) across… See the full description on the dataset page: https://huggingface.co/datasets/Aniket200325/coder-pretrain-60gb.
0146
../
cpp-00000.parquetdownload
cpp-00001.parquetdownload
cpp-00002.parquetdownload
cpp-00003.parquetdownload
cpp-00004.parquetdownload
cpp-00005.parquetdownload
cpp-00006.parquetdownload
cpp-00007.parquetdownload
cpp-00008.parquetdownload
cpp-00009.parquetdownload
cpp-00010.parquetdownload
cpp-00011.parquetdownload
css-00000.parquetdownload
css-00001.parquetdownload
css-00002.parquetdownload
css-00003.parquetdownload
go-00000.parquetdownload
go-00001.parquetdownload
go-00002.parquetdownload
go-00003.parquetdownload
go-00004.parquetdownload
go-00005.parquetdownload
go-00006.parquetdownload
go-00007.parquetdownload
html-00000.parquetdownload
html-00001.parquetdownload
html-00002.parquetdownload
html-00003.parquetdownload
java-00000.parquetdownload
java-00001.parquetdownload
java-00002.parquetdownload
java-00003.parquetdownload
java-00004.parquetdownload
java-00005.parquetdownload
java-00006.parquetdownload
java-00007.parquetdownload
java-00008.parquetdownload
java-00009.parquetdownload
java-00010.parquetdownload
java-00011.parquetdownload
javascript-00000.parquetdownload
javascript-00001.parquetdownload
javascript-00002.parquetdownload
javascript-00003.parquetdownload
javascript-00004.parquetdownload
javascript-00005.parquetdownload
javascript-00006.parquetdownload
javascript-00007.parquetdownload
javascript-00008.parquetdownload
javascript-00009.parquetdownload
javascript-00010.parquetdownload
javascript-00011.parquetdownload
javascript-00012.parquetdownload
javascript-00013.parquetdownload
javascript-00014.parquetdownload
javascript-00015.parquetdownload
python-00000.parquetdownload
python-00001.parquetdownload
python-00002.parquetdownload
python-00003.parquetdownload
python-00004.parquetdownload
python-00005.parquetdownload
python-00006.parquetdownload
python-00007.parquetdownload
python-00008.parquetdownload
python-00009.parquetdownload
python-00010.parquetdownload
python-00011.parquetdownload
python-00012.parquetdownload
python-00013.parquetdownload
python-00014.parquetdownload
python-00015.parquetdownload
python-00016.parquetdownload
python-00017.parquetdownload
python-00018.parquetdownload
python-00019.parquetdownload
python-00020.parquetdownload
python-00021.parquetdownload
python-00022.parquetdownload
python-00023.parquetdownload
python-00024.parquetdownload
python-00025.parquetdownload
python-00026.parquetdownload
python-00027.parquetdownload
python-00028.parquetdownload
python-00029.parquetdownload
python-00030.parquetdownload
python-00031.parquetdownload
rust-00000.parquetdownload
rust-00001.parquetdownload
rust-00002.parquetdownload
rust-00003.parquetdownload
rust-00004.parquetdownload
rust-00005.parquetdownload
rust-00006.parquetdownload
rust-00007.parquetdownload
shell-00000.parquetdownload
shell-00001.parquetdownload
shell-00002.parquetdownload
shell-00003.parquetdownload
sql-00000.parquetdownload
sql-00001.parquetdownload
sql-00002.parquetdownload
sql-00003.parquetdownload
typescript-00000.parquetdownload
typescript-00001.parquetdownload
typescript-00002.parquetdownload
typescript-00003.parquetdownload
typescript-00004.parquetdownload
typescript-00005.parquetdownload
typescript-00006.parquetdownload
typescript-00007.parquetdownload
typescript-00008.parquetdownload
typescript-00009.parquetdownload
typescript-00010.parquetdownload
typescript-00011.parquetdownload
typescript-00012.parquetdownload
typescript-00013.parquetdownload
typescript-00014.parquetdownload
typescript-00015.parquetdownload