code-embedding
embeddings-fine-tuning-filtered-code
Overview
This dataset is composed of high quality code retrieval data sources with mined hard negatives annotated with bi-encoder and cross-encoder scores. It can be used to train a strong code retrieval model by itself but is better used after a large-scale contrastive pre-training, for example using the CoRNStack dataset.
The negatives were mined following the NV-Retriever setup: the closest documents to each query are mined as negatives, and false negatives are filtered out… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-fine-tuning-filtered-code.embeddings-fine-tuning-filtered-code-edit
Overview
This dataset is composed of high quality code-edit retrieval data with mined hard negatives annotated with bi-encoder and cross-encoder scores. It can be used to train a strong code retrieval model by itself but is better used after a large-scale contrastive pre-training, for example using the CoRNStack dataset.
The negatives were mined following the NV-Retriever setup: the closest documents to each query are mined as negatives, and false negatives are filtered out if… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-fine-tuning-filtered-code-edit.transformers_code_embeddings_v3transformers_code_embeddings_v2_pooltransformers_code_embeddings
Transformers Code Embeddings
Compact index of function/class definitions from src/transformers/models/**/modeling_*.py for cross-model similarity. Built to help surface reusable code when modularizing models.
Contents
embeddings.safetensors — float32, L2-normalized embeddings shaped [N, D].
code_index_map.json — {int_id: "relative/path/to/modeling_*.py:SymbolName"}.
code_index_tokens.json — {identifier: [sorted_unique_tokens]} for Jaccard.
How these were built… See the full description on the dataset page: https://huggingface.co/datasets/hf-internal-testing/transformers_code_embeddings.code-embedding-dataset
Code-to-Doc Embedding Dataset
AI-generated code documentation pairs for training code embedding / retrieval models.
Dataset Description
Each record contains a code anchor (real production code) paired with:
positive: A rich natural-language documentation of what the code does
queries: 4 natural-language search queries a developer might use to find this code
label: A short semantic label (3-8 words)
This dataset is designed for training bi-encoder embedding models (e.g.… See the full description on the dataset page: https://huggingface.co/datasets/archit11/code-embedding-dataset.