FathyElghoneimy/llama-3-8b-lora-code-gen

llama-3-8b-lora-code-gen

Use case

Instruction-following code generation assistant. Given a natural-language instruction (optionally with an input), the model produces the corresponding code. Fine-tuned as part of a structured AI/ML engineering roadmap (Day 23-25) to test whether LoRA fine-tuning improves code-generation quality over the base Llama-3-8B model.

Base model

`unsloth/llama-3-8b-bnb-4bit` (4-bit quantized Llama-3-8B)

Training data

`FathyElghoneimy/alpaca-code-generation-curated` — a curated, Alpaca-formatted code-generation dataset (80/10/10 train/validation/test split, 640 training examples).

Training procedure

• —Method: LoRA (r=16, alpha=32, dropout=0) via Unsloth + TRL's SFTTrainer
• —Target modules: qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
• —4 epochs, batch size 2 (grad accumulation 4 -> effective batch 8), lr 2e-4, linear schedule, 4-bit base weights, fp16

Evaluation

Evaluated on 50 held-out test prompts against the base model. Metrics: ROUGE-L, BERTScore-F1, and an LLM-as-judge rubric (Claude, correctness/completeness/quality, 1-5).

Fine-tuned model was judged better overall on 8.0% of test prompts.

Limitations

• —Trained on a relatively small curated dataset (640 examples); may not generalize to code domains outside the training distribution.
• —Evaluated with an LLM-as-judge, which itself is an approximation of true code quality — not a substitute for unit tests or human code review.
• —4-bit quantized base model trades some precision for lower VRAM use; full-precision behavior may differ.
• —No adversarial or safety-specific evaluation was performed; do not use for unreviewed production code generation.

How to use

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="FathyElghoneimy/llama-3-8b-lora-code-gen",
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)