Shuu12121/python-treesitter-dedupe-filtered-datasetsV2

1---2license: apache-2.03language:4- en5tags:6- python7- code-search8- text-to-code9- code-to-text10- source-code11configs:12- config_name: default13  data_files:14  - split: train15    path: data/train-*16  - split: validation17    path: data/validation-*18  - split: test19    path: data/test-*20dataset_info:21  features:22  - name: code23    dtype: string24  - name: docstring25    dtype: string26  - name: func_name27    dtype: string28  - name: language29    dtype: string30  - name: repo31    dtype: string32  - name: path33    dtype: string34  - name: url35    dtype: string36  - name: license37    dtype: string38  splits:39  - name: train40    num_bytes: 153290582041    num_examples: 77690042  - name: validation43    num_bytes: 2772345144    num_examples: 1559945  - name: test46    num_bytes: 2970736847    num_examples: 1592648  download_size: 46498924649  dataset_size: 159033663950---51 52# Python CodeSearch Dataset (Shuu12121/python-treesitter-dedupe-filtered-datasetsV2)53 54## Dataset Description55This dataset contains Python functions paired with their documentation strings (docstrings), extracted from open-source Python repositories on GitHub.56It is formatted similarly to the CodeSearchNet challenge dataset.57 58Each entry includes:59- `code`: The source code of a python function or method.60- `docstring`: The docstring or Javadoc associated with the function/method.61- `func_name`: The name of the function/method.62- `language`: The programming language (always "python").63- `repo`: The GitHub repository from which the code was sourced (e.g., "owner/repo").64- `path`: The file path within the repository where the function/method is located.65- `url`: A direct URL to the function/method's source file on GitHub (approximated to master/main branch).66- `license`: The SPDX identifier of the license governing the source repository (e.g., "MIT", "Apache-2.0").67Additional metrics if available (from Lizard tool):68- `ccn`: Cyclomatic Complexity Number.69- `params`: Number of parameters of the function/method.70- `nloc`: Non-commenting lines of code.71- `token_count`: Number of tokens in the function/method.72 73## Dataset Structure74The dataset is divided into the following splits:75 76- `train`: 776,900 examples77- `validation`: 15,599 examples78- `test`: 15,926 examples79 80## Data Collection81The data was collected by:821. Identifying popular and relevant Python repositories on GitHub.832. Cloning these repositories.843. Parsing Python files (`.py`) using tree-sitter to extract functions/methods and their docstrings/Javadoc.854. Filtering functions/methods based on code length and presence of a non-empty docstring/Javadoc.865. Using the `lizard` tool to calculate code metrics (CCN, NLOC, params).876. Storing the extracted data in JSONL format, including repository and license information.887. Splitting the data by repository to ensure no data leakage between train, validation, and test sets.89 90## Intended Use91This dataset can be used for tasks such as:92- Training and evaluating models for code search (natural language to code).93- Code summarization / docstring generation (code to natural language).94- Studies on Python code practices and documentation habits.95 96## Licensing97The code examples within this dataset are sourced from repositories with permissive licenses (typically MIT, Apache-2.0, BSD).98Each sample includes its original license information in the `license` field.99The dataset compilation itself is provided under a permissive license (e.g., MIT or CC-BY-SA-4.0),100but users should respect the original licenses of the underlying code.101 102## Example Usage103```python104from datasets import load_dataset105 106# Load the dataset107dataset = load_dataset("Shuu12121/python-treesitter-dedupe-filtered-datasetsV2")108 109# Access a split (e.g., train)110train_data = dataset["train"]111 112# Print the first example113print(train_data[0])114```115