datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
code_search_net
Dataset Card for CodeSearchNet corpus
Dataset Summary
CodeSearchNet corpus is a dataset of 2 milllion (comment, code) pairs from opensource libraries hosted on GitHub. It contains code and documentation for several programming languages.
CodeSearchNet corpus was gathered to support the CodeSearchNet challenge, to explore the problem of code retrieval using natural language.
Supported Tasks and Leaderboards
language-modeling: The dataset can be used to… See the full description on the dataset page: https://huggingface.co/datasets/code-search-net/code_search_net.code-search-net-python
Dataset Card for "code-search-net-python"
Dataset Description
Homepage: None
Repository: https://huggingface.co/datasets/Nan-Do/code-search-net-python
Paper: None
Leaderboard: None
Point of Contact: @Nan-Do
Dataset Summary
This dataset is the Python portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-python.CodeSearchNetEmploying the MTEB evaluation framework's dataset version, utilize the code below for assessment:
import mteb
import logging
from sentence_transformers import SentenceTransformer
from mteb import MTEB
logger = logging.getLogger(__name__)
model_name = 'intfloat/e5-base-v2'
model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(
tasks=[
"AppsRetrieval",
"CodeFeedbackMT",
"CodeFeedbackST",
"CodeTransOceanContest",
"CodeTransOceanDL"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet.CodeSearchNet-ccrEmploying the MTEB evaluation framework's dataset version, utilize the code below for assessment:
import mteb
import logging
from sentence_transformers import SentenceTransformer
from mteb import MTEB
logger = logging.getLogger(__name__)
model_name = 'intfloat/e5-base-v2'
model = SentenceTransformer(model_name)
tasks = mteb.get_tasks(
tasks=[
"AppsRetrieval",
"CodeFeedbackMT",
"CodeFeedbackST",
"CodeTransOceanContest",
"CodeTransOceanDL"… See the full description on the dataset page: https://huggingface.co/datasets/CoIR-Retrieval/CodeSearchNet-ccr.CodeSearchNetRetrieval
CodeSearchNetRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
The dataset is a collection of code snippets and their corresponding natural language queries. The task is to retrieve the most relevant code snippet for a given query.
Task category
t2t
Domains
Programming, Written
Reference
https://huggingface.co/datasets/code_search_net/
Source datasets:
code-search-net/code_search_net
How to evaluate on this task
You can evaluate an embedding… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CodeSearchNetRetrieval.SWE-bench_Verified-code-searchinstructional_code-search-net-python
Dataset Card for "instructional_code-search-net-python"
Dataset Summary
This is an instructional dataset for Python.
The dataset contains two different kind of tasks:
Given a piece of code generate a description of what it does.
Given a description generate a piece of code that fulfils the description.
Languages
The dataset is in English.
Data Splits
There are no splits.
Dataset Creation
May of 2023
Curation Rationale
This… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/instructional_code-search-net-python.CodeSearchNet-ccrSWE-rebench-code-searchcode_search_net_clean
Dataset Card for "code_search_net_clean"
More Information needed
code_search_net
CodeSearchNet
This is an unofficial reupload of the code_search_net dataset in the parquet format. I have also removed the columns func_code_tokens, func_documentation_tokens, and split_name as they are not relevant. The original repository relies on a Python module that is downloaded and executed to unpack the dataset, which is a potential security risk but importantly raises an annoying warning. As a plus, parquets load faster.
Original model card:
Dataset Card for… See the full description on the dataset page: https://huggingface.co/datasets/claudios/code_search_net.codesearchnet
Dataset Card for CodeSearchNet
This dataset is a collection of comment-code pairs of various programming languages. See code_search_net for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "comment", "code"
Column types: str, str
Examples:{
'comment': 'Computes the new parent id for the node being moved.\n\n@return int',
'code': "protected function… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/codesearchnet.SWE-bench_Lite-code-searchcode_search_net_python_10000_examplescode_x_glue_tc_nl_code_search_adv
Dataset Card for "code_x_glue_tc_nl_code_search_adv"
Dataset Summary
CodeXGLUE NL-code-search-Adv dataset, available at https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code/NL-code-search-Adv
The dataset we use comes from CodeSearchNet and we filter the dataset as the following:
Remove examples that codes cannot be parsed into an abstract syntax tree.
Remove examples that #tokens of documents is < 3 or >256
Remove examples that documents contain special tokens… See the full description on the dataset page: https://huggingface.co/datasets/google/code_x_glue_tc_nl_code_search_adv.code-search-net-java
Dataset Card for "code-search-net-java"
Dataset Summary
This dataset is the Java portion of the CodeSarchNet annotated with a summary column.The code-search-net dataset includes open source functions that include comments found at GitHub.The summary is a short description of what the function does.
Languages
The dataset's comments are in English and the functions are coded in Java
Data Splits
Train, test, validation labels are included in the… See the full description on the dataset page: https://huggingface.co/datasets/Nan-Do/code-search-net-java.owl_code_search_hard_negative_datasets-Pre_kd
Owl Code Search Hard Negative Datasets
Knowledge Distillation (KD) ベースのハードネガティブ付きコード検索データセットです。コード検索モデルShuu12121/CodeSearch-ModernBERT-Crow-v3-large-len1024-Plusを教師モデルとして、各コメントと説明コメントのペアのデータセットから各クエリに対する関数の類似度スコアを計算し、ハードネガティブ（正解に類似しているが不正解の文書）を付与しています。
概要
目的: コード検索モデルの Contrastive Learning / Knowledge Distillation ファインチューニング
言語: Go, Java, JavaScript, PHP, Python, Ruby, Rust, TypeScript（8言語）
総サンプル数: 4,787,740
データサイズ: 8.73 GB（展開後） / 3.37 GB（ダウンロード時）
フォーマット:… See the full description on the dataset page: https://huggingface.co/datasets/Shuu12121/owl_code_search_hard_negative_datasets-Pre_kd.SWE-rebench-code-searchSWE-smith-py-code-searchSWE-smith-py-code-searchcode_search_le512codesearchnet-codegen
Dataset Card for CodeSearchNet for CodeGen
This is a processed version of the CodeSearchNet dataset. Namely, I separated the doc (documentation/docstring), sign (function signature), and output (function body) into separate fields; doc and sign are concatenated (according to the correct order of the programming language) into the problem field, making it suitable for the code generation task.
Dataset Details
Dataset Description
Curated by: [More… See the full description on the dataset page: https://huggingface.co/datasets/pengyunie/codesearchnet-codegen.multilingual-codesearch-hard-negativesV2
multilingual-codesearch-hard-negativesV2
このデータセットは、多言語コード検索タスク向けに設計された、ハードネガティブペアを提供します。複数のプログラミング言語から収集した、フィルタ済みのコード／ドックストリング対を基に構成されています。
データセット概要
各サンプルには以下の情報が含まれます。
query_docstring: 関数やメソッドの自然言語による説明
positive_code: 対応する正しいコード実装
hard_negative_code: 類似しているが意味が異なるコード（FAISSによる近傍検索で選定）
similarity_score: クエリとハードネガティブ間のFAISS類似度スコア（内積／コサイン類似度）
付加情報: リポジトリ、ファイルパス、ライセンス情報、言語
対応言語
Python
Java
JavaScript
PHP
Ruby
Go
Rust
データセット作成方法… See the full description on the dataset page: https://huggingface.co/datasets/Shuu12121/multilingual-codesearch-hard-negativesV2.CodeSearchNet-Pythoncode_search_netcode-search-net/code_search_net , already loaded and converted to paraquet so you dont have to enable remote execution to use it .
SWE-smith-code-search
SWE-smith Code Search
SWE-smith Code Search is a code-localization training dataset derived from
SWE-bench/SWE-smith-py.
It converts each eligible SWE-smith mutation patch into structured file,
class/module, and function/method targets. The release contains 39,291 examples
from 131 Python repositories and is intended for training and evaluating coding
agents that must locate relevant code before editing it.
中文简介：本数据集从 SWE-smith Python 原始任务中提取文件、类/模块、函数/方法三级
代码定位标签，可直接用于 Code… See the full description on the dataset page: https://huggingface.co/datasets/LeeXugar/SWE-smith-code-search.CodeSearchNetCCRetrieval
CodeSearchNetCCRetrieval
An MTEB dataset
Massive Text Embedding Benchmark
The dataset is a collection of code snippets. The task is to retrieve the most relevant code snippet for a given code snippet.
Task category
t2t
Domains
Programming, Written
Reference
https://arxiv.org/abs/2407.02883
Source datasets:
CoIR-Retrieval/CodeSearchNet-ccr
How to evaluate on this task
You can evaluate an embedding model on this dataset using the following code:
import… See the full description on the dataset page: https://huggingface.co/datasets/mteb/CodeSearchNetCCRetrieval.codesearchnet-python-raw-457kann-codesearch-4m
Cleaning
Unlike the original dataset, the func_code_string column was updated to remove any comments and keep just the code.
The original version can still be found in the whole_func_string.
import re
def remove_comments_docstrings(code, language):
if language == 'python':
# Remove docstrings
code = re.sub(r'"""(.*?)"""', '', code, flags=re.DOTALL)
code = re.sub(r"'''(.*?)'''", '', code, flags=re.DOTALL)
# Remove comments
code =… See the full description on the dataset page: https://huggingface.co/datasets/unum-cloud/ann-codesearch-4m.SWE-Gym-code-search