Shuu12121/javascript-treesitter-filtered-datasetsV2

1---2license: apache-2.03language:4- en5tags:6- javascript7- js8- code-search9- text-to-code10- code-to-text11- source-code12- frontend13- backend14- web-development15dataset_info:16  features:17  - name: code18    dtype: string19  - name: docstring20    dtype: string21  - name: func_name22    dtype: string23  - name: language24    dtype: string25  - name: repo26    dtype: string27  - name: path28    dtype: string29  - name: url30    dtype: string31  - name: license32    dtype: string33  splits:34  - name: train35    num_bytes: 92538816036    num_examples: 70335437  - name: validation38    num_bytes: 4600208339    num_examples: 4189940  - name: test41    num_bytes: 1821722742    num_examples: 1713843  download_size: 19636737044  dataset_size: 98960747045configs:46- config_name: default47  data_files:48  - split: train49    path: data/train-*50  - split: validation51    path: data/validation-*52  - split: test53    path: data/test-*54---55 56# Javascript CodeSearch Dataset (Shuu12121/javascript-treesitter-filtered-datasetsV2)57 58## Dataset Description59This dataset contains JavaScript functions and methods paired with their JSDoc comments, extracted from open-source JavaScript repositories on GitHub.60It is formatted similarly to the CodeSearchNet challenge dataset.61 62Each entry includes:63- `code`: The source code of a javascript function or method.64- `docstring`: The docstring or Javadoc associated with the function/method.65- `func_name`: The name of the function/method.66- `language`: The programming language (always "javascript").67- `repo`: The GitHub repository from which the code was sourced (e.g., "owner/repo").68- `path`: The file path within the repository where the function/method is located.69- `url`: A direct URL to the function/method's source file on GitHub (approximated to master/main branch).70- `license`: The SPDX identifier of the license governing the source repository (e.g., "MIT", "Apache-2.0").71Additional metrics if available (from Lizard tool):72- `ccn`: Cyclomatic Complexity Number.73- `params`: Number of parameters of the function/method.74- `nloc`: Non-commenting lines of code.75- `token_count`: Number of tokens in the function/method.76 77## Dataset Structure78The dataset is divided into the following splits:79 80- `train`: 703,354 examples81- `validation`: 41,899 examples82- `test`: 17,138 examples83 84## Data Collection85The data was collected by:861. Identifying popular and relevant Javascript repositories on GitHub.872. Cloning these repositories.883. Parsing Javascript files (`.js`) using tree-sitter to extract functions/methods and their docstrings/Javadoc.894. Filtering functions/methods based on code length and presence of a non-empty docstring/Javadoc.905. Using the `lizard` tool to calculate code metrics (CCN, NLOC, params).916. Storing the extracted data in JSONL format, including repository and license information.927. Splitting the data by repository to ensure no data leakage between train, validation, and test sets.93 94## Intended Use95This dataset can be used for tasks such as:96- Training and evaluating models for code search (natural language to code).97- Code summarization / docstring generation (code to natural language).98- Studies on Javascript code practices and documentation habits.99 100## Licensing101The code examples within this dataset are sourced from repositories with permissive licenses (typically MIT, Apache-2.0, BSD).102Each sample includes its original license information in the `license` field.103The dataset compilation itself is provided under a permissive license (e.g., MIT or CC-BY-SA-4.0),104but users should respect the original licenses of the underlying code.105 106## Example Usage107```python108from datasets import load_dataset109 110# Load the dataset111dataset = load_dataset("Shuu12121/javascript-treesitter-filtered-datasetsV2")112 113# Access a split (e.g., train)114train_data = dataset["train"]115 116# Print the first example117print(train_data[0])118```119