CodedotAI/code_clippy
This dataset was generated by selecting GitHub repositories from a large collection of repositories. These repositories were collected from https://seart-ghs.si.usi.ch/ and Github portion of [The Pile](https://github.com/EleutherAI/github-downloader) (performed on July 7th, 2021). The goal of this dataset is to provide a training set for pretraining large language models on code data for helping software engineering researchers better understand their impacts on software related tasks such as autocompletion of code. The dataset is split into train, validation, and test splits. There is a version containing duplicates (209GBs compressed) and ones where exact duplicates (132GBs compressed) are removed. Contains mostly JavaScript and Python code, but other programming languages are included as well to various degrees.
1*.7z filter=lfs diff=lfs merge=lfs -text2*.arrow filter=lfs diff=lfs merge=lfs -text3*.bin filter=lfs diff=lfs merge=lfs -text4*.bin.* filter=lfs diff=lfs merge=lfs -text5*.bz2 filter=lfs diff=lfs merge=lfs -text6*.ftz filter=lfs diff=lfs merge=lfs -text7*.gz filter=lfs diff=lfs merge=lfs -text8*.h5 filter=lfs diff=lfs merge=lfs -text9*.joblib filter=lfs diff=lfs merge=lfs -text10*.lfs.* filter=lfs diff=lfs merge=lfs -text11*.model filter=lfs diff=lfs merge=lfs -text12*.msgpack filter=lfs diff=lfs merge=lfs -text13*.onnx filter=lfs diff=lfs merge=lfs -text14*.ot filter=lfs diff=lfs merge=lfs -text15*.parquet filter=lfs diff=lfs merge=lfs -text16*.pb filter=lfs diff=lfs merge=lfs -text17*.pt filter=lfs diff=lfs merge=lfs -text18*.pth filter=lfs diff=lfs merge=lfs -text19*.rar filter=lfs diff=lfs merge=lfs -text20saved_model/**/* filter=lfs diff=lfs merge=lfs -text21*.tar.* filter=lfs diff=lfs merge=lfs -text22*.tflite filter=lfs diff=lfs merge=lfs -text23*.tgz filter=lfs diff=lfs merge=lfs -text24*.xz filter=lfs diff=lfs merge=lfs -text25*.zip filter=lfs diff=lfs merge=lfs -text26*.zstandard filter=lfs diff=lfs merge=lfs -text27*tfevents* filter=lfs diff=lfs merge=lfs -text28