code-embeddings
embeddings-fine-tuning-filtered-code
Overview
This dataset is composed of high quality code retrieval data sources with mined hard negatives annotated with bi-encoder and cross-encoder scores. It can be used to train a strong code retrieval model by itself but is better used after a large-scale contrastive pre-training, for example using the CoRNStack dataset.
The negatives were mined following the NV-Retriever setup: the closest documents to each query are mined as negatives, and false negatives are filtered out… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-fine-tuning-filtered-code.embeddings-fine-tuning-filtered-code-edit
Overview
This dataset is composed of high quality code-edit retrieval data with mined hard negatives annotated with bi-encoder and cross-encoder scores. It can be used to train a strong code retrieval model by itself but is better used after a large-scale contrastive pre-training, for example using the CoRNStack dataset.
The negatives were mined following the NV-Retriever setup: the closest documents to each query are mined as negatives, and false negatives are filtered out if… See the full description on the dataset page: https://huggingface.co/datasets/lightonai/embeddings-fine-tuning-filtered-code-edit.transformers_code_embeddings_v3transformers_code_embeddings
Transformers Code Embeddings
Compact index of function/class definitions from src/transformers/models/**/modeling_*.py for cross-model similarity. Built to help surface reusable code when modularizing models.
Contents
embeddings.safetensors — float32, L2-normalized embeddings shaped [N, D].
code_index_map.json — {int_id: "relative/path/to/modeling_*.py:SymbolName"}.
code_index_tokens.json — {identifier: [sorted_unique_tokens]} for Jaccard.
How these were built… See the full description on the dataset page: https://huggingface.co/datasets/hf-internal-testing/transformers_code_embeddings.transformers_code_embeddings_v2_poolb2_calc_negative_embeddings_code_stackexchange_codereview