datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
github-repos-pythonThe Github repository retrieval source for [code-rag-bench], containing all Python files from the entire GitHub dump (in github-repos)
Python-Code-23k-ShareGPTThis dataset is in Vicuna/ShareGPT format. There are 23000+ set of conversations. Each set having 2 conversations.
Along with the Python code detailed explanation is provided.
This dataset was generated using GPT-3.5, GPT-4 etc.
Python-Code-LargePython-Code-Large
Python-Code-Large is a large-scale corpus of Python source code comprising more than 2 million rows of Python code. The dataset is designed to support research in large language model (LLM) pretraining, code intelligence, software engineering automation, and program analysis for the Python ecosystem.
By providing a high-volume, language-specific corpus, Python-Code-Large enables systematic experimentation in Python-focused model training, domain adaptation, and downstream… See the full description on the dataset page: https://huggingface.co/datasets/Lovett01/Python-Code-Large.Python-Code-LargePython-Code-Large
Python-Code-Large is a large-scale corpus of Python source code comprising more than 2 million rows of Python code. The dataset is designed to support research in large language model (LLM) pretraining, code intelligence, software engineering automation, and program analysis for the Python ecosystem.
By providing a high-volume, language-specific corpus, Python-Code-Large enables systematic experimentation in Python-focused model training, domain adaptation, and downstream… See the full description on the dataset page: https://huggingface.co/datasets/ajibawa-2023/Python-Code-Large.python-code-instructions-85k
Python Code Instructions - 85K
Instruction-tuning dataset of Python functions paired with short natural-language instructions derived from repository docstrings.
What changed in this release
This release keeps the original public rows and format, but makes the dataset easier to use responsibly:
exact duplicate rows were removed again using normalized instruction + output hashing
deterministic train, validation, and test splits were added
the dataset card now documents… See the full description on the dataset page: https://huggingface.co/datasets/NickIBrody/python-code-instructions-85k.code-text-python
Dataset is imported from CodeXGLUE and pre-processed using their script.
Where to find in Semeru:
The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/code-to-text/python in Semeru
CodeXGLUE -- Code-To-Text
Task Definition
The task is to generate natural language comments for a code, and evaluted by smoothed bleu-4 score.
Dataset
The dataset we use comes from CodeSearchNet and we filter the dataset as the following:
Remove… See the full description on the dataset page: https://huggingface.co/datasets/semeru/code-text-python.python_codeStable-Code-Python-SFT
Stable Code Python SFT
The Stable Code Python SFT dataset is a high-quality synthetic dataset derived from the
stabilityai/stable-code-instruct-3b model for the purpose of supervised fine-tuning (SFT). Please refer to the
Versioning section for dataset versions.
Note: If you would like to contribute to this repository,
please read the CONTRIBUTING first.
TableofContents
Features
File Structure
Metadata
Usage
Versioning
License
TeamContact
Reference
Citation… See the full description on the dataset page: https://huggingface.co/datasets/bunyaminergen/Stable-Code-Python-SFT.python_code_docstring_ast_corpus
Overview
This dataset contains 34,000+ rows of code-docstring-ast data along with additional metadata. Data was gathered from various Python libraries and frameworks and their
publicly available GitHub repos. This dataset was created for the purpose of training the CodeT5+ transformer on AST-enhanced code-to-doc tasks.
Sources
The dataset was gathered from various GitHub repos sampled from this repo by Vinta.
The 26 repos are:
matplotlib
pytorch
cryptography
django… See the full description on the dataset page: https://huggingface.co/datasets/Mir-2002/python_code_docstring_ast_corpus.python_documentation_codeDataset created using https://github.com/jeffmeloy/py2dataset using the Python code from the following:
https://github.com/ansible/ansible
https://github.com/apache/airflow
https://github.com/arogozhnikov/einops
https://github.com/arviz-devs/arviz
https://github.com/astropy/astropy
https://github.com/biopython/biopython
https://github.com/bjodah/chempy
https://github.com/bokeh/bokehhttps://github.com/CalebBell/thermo
https://github.com/camDavidsonPilon/lifelines
https://github.com/coin-or/pulp… See the full description on the dataset page: https://huggingface.co/datasets/jeffmeloy/python_documentation_code.ReForm-Python2Dafny-Dataset
Re:Form Datasets
This repository contains the datasets associated with the paper "Re:Form -- Reducing Human Priors in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny".
The paper introduces a framework that leverages Reinforcement Learning (RL) within Large Language Models (LLMs) to reduce reliance on human-annotated priors for formal software verification. The datasets provided are integral for training and evaluating models in formal language… See the full description on the dataset page: https://huggingface.co/datasets/Veri-Code/ReForm-Python2Dafny-Dataset.MultiRoundConvos-Code-JS-HTML-CSS-Pythoncode.evol.instruct.wiz.oss_python.jsonpython_codehttps://huggingface.co/datasets/ajibawa-2023/Python-Code-23k-ShareGPT
features: coding, single-turn, task
length: 22.6k
code_search_net_python_tokenizedcode-judge-ast-python-15k-it1
code-judge-ast-python-15k-it1
15k examples of Python code + yes/no structural property questions with deterministic AST ground truth. 3 points per example, balanced to 40-60% per property.
Source: The Stack v1 (deduplicated Python).
SO-Python_QA-filtered-2023-no_code-tanh_scoreSO dataset of pythontag data
Question filters:
images
links
code blocks
Q_Score > 0
Answer_count > 0
Answers filters:
images
links
code blocks
Scores are tanh applied to scaled with AbsMaxScaler to IQR range of Original SO Answers' scores
openai-finetune-python-code-instructionspython-text-to-codepython-mal-code-dataset
Python Code Dataset
This dataset contains extracted Python code from various repositories for fine-tuning code-generation models.
python-code-simplified
What This Is
This is a dataset of "simplified" Python Code from https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca.
Our simplification attempts to remove all comments, and reduce these names/strings to a single letter without impacting its structure/logic.
You should use the "minimized" field. The "original" field is the original output from iamtarun/python_code_instructions_18k_alpaca.
Why It Exists
We used this dataset to generate… See the full description on the dataset page: https://huggingface.co/datasets/neuronpedia-org/python-code-simplified.python_code_version2python_code_instructions_510_alpacacode_instruction_alpaca_kkmcb_qa_python_codePython-Code-Instructions-7kpython-code-trainingPython_Codepython_code_instructions_1k_alpacapython_code_instructions_18k_chatML_chinese2