Sai-Nandu/Code-Completion-using-GPT-2-CodeXGLUE

GPT-2 Fine-Tuned for Python Code Completion

This repository contains a fine-tuned GPT-2 model for Python source code completion. The model was trained on the CodeXGLUE Python Code Completion dataset using the Hugging Face Transformers library and PyTorch.

Model Description

This model is designed to predict the next tokens in Python source code, enabling intelligent code completion for software development tasks.

• —Base Model: GPT-2
• —Task: Causal Language Modeling
• —Language: Python
• —Framework: PyTorch
• —Library: Hugging Face Transformers

Dataset

Dataset: CodeXGLUE – Python Code Completion

The dataset contains Python source code snippets used to train language models for next-token code prediction.

Note: A subset of approximately 13,000 training samples from the CodeXGLUE Python dataset was used for fine-tuning.

Training Configuration

Training was performed using the Hugging Face Trainer API.

Evaluation Results

The decreasing validation loss throughout training indicates successful adaptation of GPT-2 to the Python code completion task.

Training Progress

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/MODEL_NAME")
model = AutoModelForCausalLM.from_pretrained("YOUR_USERNAME/MODEL_NAME")

prompt = "def fibonacci(n):"

inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

• —Trained only on Python source code.
• —Intended for research and educational purposes.
• —May generate syntactically incorrect or incomplete code.
• —Does not guarantee production-quality code suggestions.

Technologies Used

• —Python
• —PyTorch
• —Hugging Face Transformers
• —Hugging Face Datasets
• —CodeXGLUE Dataset

Future Improvements

• —Fine-tune larger transformer models.
• —Train on larger subsets of CodeXGLUE.
• —Evaluate using additional code generation metrics.
• —Support multiple programming languages.
• —Deploy as an inference API.

Author

Sai Nandu Vajhala

GitHub: https://github.com/SaiNanduVajhala

LinkedIn: https://www.linkedin.com/in/sai-nandu-vajhala