Pradnya27/codegpt-lora-code-generation

CodeGPT LoRA Fine-tuned for Code Generation

Fine-tuned version of CodeGPT using LoRA (Low-Rank Adaptation) for Python code generation.

🔗 Links

• —Live Demo: https://huggingface.co/spaces/Pradnya27/code-generator
• —Full Fine-tuned Model: https://huggingface.co/Pradnya27/codegpt-finetuned-code-generation
• —GitHub: https://github.com/pradnyagundu/codegpt-finetuned-code-generation

Model Details

• —Base model: microsoft/CodeGPT-small-py (124M parameters)
• —Method: LoRA (Low-Rank Adaptation)
• —Trainable parameters: 589,824 (0.36% of total)
• —Model size: 2.36MB (vs 651MB for full fine-tuning)
• —Dataset: Rabinovich/Code-Generation-LLM-LoRA (5000 examples)

Training Details

• —Epochs: 2
• —Learning rate: 3e-4
• —Batch size: 8
• —LoRA rank: 16
• —LoRA alpha: 32
• —Hardware: Google Colab T4 GPU
• —Training time: ~9 minutes

Training Results

Comparison vs Full Fine-tuning

How to Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base_model = AutoModelForCausalLM.from_pretrained("microsoft/CodeGPT-small-py")
model = PeftModel.from_pretrained(base_model, "Pradnya27/codegpt-lora-code-generation")
tokenizer = AutoTokenizer.from_pretrained("microsoft/CodeGPT-small-py")
tokenizer.pad_token = tokenizer.eos_token

prompt = "Generate code: Write a function to check if a number is prime"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    outputs = model.generate(inputs["input_ids"], max_new_tokens=150)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• —Trained on competitive programming problems — works best for algorithmic tasks
• —Small base model (124M params) limits output quality
• —Full fine-tuning achieves lower loss on this dataset

Future Work

• —Train on full 34K dataset
• —Increase LoRA rank to r=32 or r=64
• —Evaluate on HumanEval benchmark