23ws-LLMcoder/LLMcoder-GitHub-Python-Mix-Direct

1---2# For reference on dataset card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=13# Doc / guide: https://huggingface.co/docs/hub/datasets-cards4{}5---6 7# Dataset Card for LLMcoder-GitHub-Python-Mix-Direct8 9Python target autocomplete suggestions in the format of conversations for OpenAI's fine-tuning.10 11## Dataset Details12 13### Dataset Description14 15<!-- Provide a longer summary of what this dataset is. -->16 17 18 19- **Curated by:** [More Information Needed]20- **Funded by [optional]:** [More Information Needed]21- **Shared by [optional]:** [More Information Needed]22- **Language(s) (NLP):** [More Information Needed]23- **License:** [More Information Needed]24 25### Dataset Sources [optional]26 27The data has been scraped from the following public GitHub repositories on 2023-11-15:28 29```30https://github.com/numpy/numpy31https://github.com/pandas-dev/pandas32https://github.com/matplotlib/matplotlib33https://github.com/scikit-learn/scikit-learn34https://github.com/python-pillow/Pillow35https://github.com/psaegert/pmtrendviz36https://github.com/psaegert/nli-nec37https://github.com/graphdeco-inria/gaussian-splatting38https://github.com/lllyasviel/ControlNet39https://github.com/maltfield/awesome-lemmy-instances40https://github.com/Aleph-Alpha/aleph-alpha-client41https://github.com/MaartenGr/BERTopic42https://github.com/MilesCranmer/PySR43https://github.com/AUTOMATIC1111/stable-diffusion-webui44https://github.com/microsoft/Codex-CLI45https://github.com/dropbox/hydra46https://github.com/HLearning/unet_keras47https://github.com/hmason/ml_class48https://github.com/django/django49https://github.com/encode/django-rest-framework50https://github.com/pallets/flask51https://github.com/postmanlabs/httpbin52https://github.com/jakevdp/PythonDataScienceHandbook53https://github.com/donnemartin/data-science-ipython-notebooks54https://github.com/tensorflow/tensorflow55```56 57## Uses58 59This dataset is intended to be used for fine-tuning of GPT-3.5-Turbo via OpenAI's fine-tuning API.60 61## Dataset Structure62 63`train_completions.jsonl` includes a list of 100 target conversations. Each conversation is structured as follows:64 65```json66[67    {"role": "system", "content": <system prompt>},68    {"role": "user", "content": <first half of the code>},69    {"role": "assistant", "content": <small target completion from ground truth>}70]71```72 73## Dataset Creation74 75### Curation Rationale76 77This dataset was created to fine-tune GPT-3.5-Turbo to provide more reliably formatted autocomplete suggestions for python code.78 79#### Data Collection and Processing80 81We scraped a mix of 25 GitHub repositories related to python and sampled 4 random python files weighted by their length.82The files were split in 2 parts at a uniformly random point in each file.83Next, if the input was longer than 10k tokens, we truncated the input file to 250 to 10000 tokens from the beginning.84The output was manually truncated to reasonably short code completions.85 86#### Who are the source data producers?87 88Paul Saegert89 90[More Information Needed]91 92#### Personal and Sensitive Information93 94This dataset may contain personal or sentitive information.95 96## Bias, Risks, and Limitations97 98The dataset contains only python code from the most popular, trending, or personal projects. It may be biased towards a particular style of code.99 100## Citation [optional]101 102<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->103 104**BibTeX:**105 106[More Information Needed]