Modelpublic

rileyseaburg/concept-first-codegen

sourceHugging Faceapache-2.0updated 8mo agoView on Hugging Face

0likes8downloads

Model Card

library_name: transformers tags:

—code-generation
—concept-embedding
—jepa
—pytorch
—gguf license: apache-2.0 ---

Concept-First Code Generation

Inspired by VL-JEPA: Predict concept embeddings first, then generate code conditioned on them.

The Idea

Traditional autoregressive models predict tokens one at a time, which can lead to losing coherence or hallucinating APIs. The Concept-First approach solves this by:

1.Concept Encoder: Encoding code snippets into semantic embeddings.
2.Concept Predictor: Predicting what the code embedding should look like given a query.
3.Concept-Conditioned Generation: Retrieving similar concepts to guide the LLM.

mermaid

graph LR
    A[Query] --> B(Concept Predictor)
    B --> C{Concept Space}
    C --> D[Retrieve Similar Code]
    D --> E[Conditioned Generation]

Models Used (January 2026)

Component	Model	Description
Concept Encoder	`Salesforce/SFR-Embedding-Code-2B_R`	SOTA code embeddings (CoIR: 67.4)
Text Encoder	`Alibaba-NLP/gte-Qwen2-1.5B-instruct`	State-of-the-art text embedding
Concept Predictor	Custom MLP	Maps text queries to code concept space
Code LLM	`Qwen/Qwen2.5-Coder-32B-Instruct`	High-performance code generation

Files in this Repo

—concept_predictor.pt: PyTorch weights for the concept predictor MLP.
—concept_predictor.gguf: GGUF format for edge deployment (llama.cpp/LM Studio).
—concept_bank.pt: Pre-computed embeddings for the concept retrieval bank.

Usage

python

# Load the concept predictor
import torch
checkpoint = torch.load("concept_predictor.pt")
# ... (See Colab notebook for full implementation)

Datasets

Constructed from high-quality subsets of:

—MBPP
—Evol-Instruct-Code
—Magicoder-OSS-Instruct

Credits

Created by Core Subagent (Colab Composer) for Riley Seaburg.