KK9922/Forge-Gemma-3-27B-GGUF

Forge — Gemma 3 27B Code Assistant (Q4KM GGUF)

Forge is a QLoRA fine-tune of Google's Gemma 3 27B-IT base model, conditioned for structured code generation, multi-language debugging, and algorithmic problem solving across Python, JavaScript, Java, C++, C and SQL.

This repository contains the Q4_K_M GGUF weights for local inference on consumer hardware. The unquantized bfloat16 merge is ~56 GB; Q4KM compresses it to ~16 GB, runnable on a single 24 GB GPU or with CPU offloading on 12 GB VRAM cards.

🔗 Links & Resources

• —Training Code & Custom Eval Harness: GitHub Repo Link

Related Tools

[EvalSpy repo link](https://github.com/thesis09/EvalSpy-a-CLI-tool-) — a CLI that audits LLM eval pipelines for the exact bugs encountered building this model (0% HumanEval, 9% MBPP, 0 DebugBench samples).

\``bash pip install evalspy evalspy check your_eval.py \``

Author is open to LLM engineering roles (India onsite / worldwide remote). Contact: kaustubhkubitkar@gmail.com

Here is the working demo of the model:

(Note: The generation phase in the attached video is speed up 8x. Real-time inference on the 3060 took ~4 minutes.) this is the link to the video : https://youtu.be/3acwPjRmo74

Model Details

⚡ Local Inference Performance (RTX 3060 12GB)

Tested using llama.cpp with Q4KM quantization and partial GPU offloading.

Training Data

Three execution-verified sources, filtered and deduplicated before training:

*DebugBench was included in the pipeline but contributed 0 training samples due to a field-name mismatch (solution vs fixed_code) discovered during evaluation. Training effectively ran on ~33 K samples from the first two sources only.

Filtering applied: minimum 80-character responses, maximum 6 000-character sequences, removal of placeholder patterns (TODO, NotImplementedError, trailing ellipsis), Jaccard deduplication at 0.85 threshold.

Chat format: ChatML with a fixed Forge system prompt injected at every turn.

Evaluation Results

Evaluated on the merged bfloat16 model (full precision, no quantization artifacts) on an H100 80 GB using direct HuggingFace transformers inference.

HumanEval — context and caveats

98.78% on the full 164-problem set is a strong result. For reference, the base google/gemma-3-27b-it model scores approximately 84% on HumanEval, giving this fine-tune a ~15 percentage-point absolute improvement. That said:

• —HumanEval is a well-known benchmark and its problems appear in many training datasets. Self-oss-instruct and CodeAlpaca both contain HumanEval-adjacent content, so some of this score likely reflects training distribution overlap rather than pure generalization.
• —HumanEval+ (evalplus — 80× more test cases per problem) is planned; that score will be a more honest generalization signal.

MBPP — evaluation bug discovered and fixed

The initial MBPP run produced 9% pass@1. Root cause: MBPP test assertions hardcode a specific function name (e.g. assert min_cost(...) == 4) but the eval prompt gave the model no name hint, so it chose its own name and failed with NameError — even when the logic was correct. After fixing the eval to extract the expected name from the assertions and inject it into the prompt, a re-run is pending. The 9% figure is an eval bug, MBPP pass@1 of 73% is approximately equal to the base Gemma 3 27B-IT baseline (~72%), confirming no catastrophic forgetting — the fine-tune improved HumanEval-style generation without degrading general Python problem-solving ability.

DebugBench — metric limitation

The 78% DebugBench score uses a token-overlap proxy (Jaccard similarity between the first 30 tokens of the reference fix and all tokens in the model response, threshold 0.5). This is a directional signal, not an execution-based metric. Real debugging accuracy measured by running the fixed code against test cases would be a stricter and more meaningful number.

System Prompt

Forge is conditioned on this system prompt at inference time. For best results, inject it at the start of every conversation:

You are Forge, an elite precision coding assistant with deep expertise in Python,
JavaScript, Java, C++, C and SQL.

RESPONSE STRUCTURE:
1. One-sentence summary.
2. Complete code in a fenced block with language tag.
3. Brief explanation (3-6 bullet points).
4. Edge cases — at least 2.

When writing algorithms: state time and space complexity. Show a worked example.
When debugging: root cause in one sentence. Corrected code with # FIXED: comments.
Never write TODO, placeholder code, or incomplete functions.

How to Run Locally

llama.cpp (recommended)

# 12 GB VRAM card — offload as many layers as fit
./llama-cli \
  -m gemma3-forge-Q4_K_M.gguf \
  -n 2048 \
  --n-gpu-layers 20 \
  -c 4096 \
  --temp 0.1 \
  --top-k 40 \
  --top-p 0.95 \
  --repeat-penalty 1.1 \
  -p "<start_of_turn>user\nImplement a binary search tree in Python<end_of_turn>\n<start_of_turn>model\n"

Note: Use llama-cli not main — the main binary was renamed in llama.cpp after build b3000. If your build is older, replace llama-cli with ./main.

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="gemma3-forge-Q4_K_M.gguf",
    n_gpu_layers=20,    # increase if you have more VRAM
    n_ctx=4096,
)

system = "You are Forge, an elite precision coding assistant..."
prompt = f"<start_of_turn>user\n{system}\n\nWrite a Python function to find all prime numbers up to n using the Sieve of Eratosthenes.<end_of_turn>\n<start_of_turn>model\n"

output = llm(
    prompt,
    max_tokens=1024,
    temperature=0.1,
    top_k=40,
    top_p=0.95,
    repeat_penalty=1.1,
    stop=["<end_of_turn>"],
)
print(output["choices"][0]["text"])

VRAM requirements

Intended Use

• —Local code generation and completion (Python, JS, Java, C++, C, SQL)
• —Multi-language debugging assistance
• —Algorithmic problem solving with complexity analysis
• —Offline/air-gapped environments where cloud LLM APIs are unavailable

Out of Scope / Limitations

• —Not intended for general conversation — Forge's system prompt conditions it heavily toward structured code responses; open-ended chat quality is degraded.
• —Vision inputs ignored — vision tower weights were pruned during export; multimodal inputs will not work.
• —Context window — trained at 2 048 tokens; performance on very long contexts (>4 K) is untested.
• —Benchmark inflation — HumanEval scores should be read alongside MBPP and HumanEval+ results (pending) for a full generalization picture.
• —DebugBench metric is weak — 78% is a directional number, not a rigorous debugging accuracy claim.

License

This model is derived from google/gemma-3-27b-it and is therefore subject to the Gemma Terms of Use: https://ai.google.dev/gemma/terms

Fine-tuning code, eval harness, and the FastAPI server are released separately under MIT.

Citation

If you use Forge in research or projects, please cite the base model:

@misc{gemmateam2024gemma3,
  title  = {Gemma 3 Technical Report},
  author = {Gemma Team, Google DeepMind},
  year   = {2024},
  url    = {https://ai.google.dev/gemma}
}

license: apache-2.0