aysinghal/ide-code-retrieval-qwen3-0.6b

ide-code-retrieval-qwen3-0.6b

A SentenceTransformer model fine-tuned from Qwen/Qwen3-Embedding-0.6B for IDE code retrieval -- mapping natural-language commit queries to relevant source code documents via dense vector similarity.

Note: This is an intermediate checkpoint at step 9,000 / 9,150 (98.4% through 3 epochs). Training loss is still decreasing, so a later checkpoint may perform better.

Model Description

This model encodes both short natural-language queries (commit messages, search queries) and longer code documents into a shared embedding space. Retrieval is performed by computing cosine similarity between the query embedding and candidate code embeddings.

• —Base model: Qwen/Qwen3-Embedding-0.6B (0.6B parameters)
• —Max sequence length: 1024 tokens
• —Output dimensionality: 1024 (normalized)
• —Similarity function: Cosine similarity

Training Details

Dataset

• —Source: aysinghal/code-retrieval-training-dataset
• —Total pairs: 2,465,694
• —Train split: 2,342,409 pairs (95%)
• —Eval split: 123,285 pairs (5%)
• —Text strategy: truncate (max 4096 chars)
• —Negatives: Explicit hard negatives from the dataset
• —Pre-tokenized: Yes (token IDs stored on disk for zero-overhead data loading)

Loss Function

MultipleNegativesRankingLoss (InfoNCE) with explicit hard negatives. Each training example consists of an anchor (query), a positive (relevant code), and a hard negative (similar but irrelevant code). In-batch negatives provide additional contrast.

Hyperparameters

Hardware

• —GPUs: 2x NVIDIA L40S
• —Total training steps: 9,150 (3 epochs)

Training Progress (at checkpoint step 9,000)

• —Training loss: 2.8684 (step 50) → 0.4907 (step 9000)
• —Best eval loss: 0.1070 (step 7,320)
• —Progress: 9,000 / 9,150 steps (98.4%)

Evaluation Results

<details> <summary>Full training loss history (click to expand)</summary>

</details>

Usage

Loading the Model

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("aysinghal/ide-code-retrieval-qwen3-0.6b")

Computing Embeddings

queries = [
    "fix null pointer exception in user authentication",
    "add retry logic to API client",
]
code_docs = [
    "def authenticate(user):\n    if user is None:\n        raise ValueError...",
    "class APIClient:\n    def request(self, url, retries=3):\n        ...",
]

query_embeddings = model.encode(queries)
code_embeddings = model.encode(code_docs)

# Compute cosine similarities
from sentence_transformers.util import cos_sim
similarities = cos_sim(query_embeddings, code_embeddings)
print(similarities)

Intended Use

• —Primary use case: Retrieving relevant code files/functions given a natural-language query (commit message, bug description, feature request)
• —Search pipeline: Encode a corpus of code documents offline, then at query time encode the query and find nearest neighbors via cosine similarity

Limitations

• —This is an early checkpoint (98.4% through training). The loss curve is still decreasing, so later checkpoints will likely perform better.
• —Trained on a specific code retrieval dataset; may not generalize to all programming languages or query styles without further fine-tuning.
• —Max context is 1024 tokens -- very long files are truncated.

Citation

If you use this model, please cite the base model:

@article{qwen3embedding,
  title={Qwen3-Embedding},
  author={Qwen Team},
  year={2025}
}