CoolFace
Modelpublic

gandhiraketla277/gpt-neo-1.3b-dolly15k-lora

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
0likes11downloads
Model Card

GPT-Neo 1.3B LoRA Fine-tuned on Dolly-15k

This is a LoRA (Low-Rank Adaptation) fine-tuned version of EleutherAI/gpt-neo-1.3B on the Dolly-15k instruction-following dataset.

Model Details

  • —Base Model: EleutherAI/gpt-neo-1.3B
  • —Fine-tuning Method: LoRA (Low-Rank Adaptation)
  • —Dataset: Databricks Dolly-15k
  • —Training Date: 2025-08-10
  • —Trainable Parameters: 3,145,728 (0.24% of total parameters)

Training Results

  • —Initial Loss: 2.32
  • —Final Loss: 2.20
  • —Training: Stable convergence with no loss explosions
  • —Data Processing: Fixed padding issues for 4x efficiency gain

LoRA Configuration

python
LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

Usage

Loading the Model

python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/gpt-neo-1.3B",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapters
model = PeftModel.from_pretrained(base_model, "YOUR_USERNAME/gpt-neo-1.3b-dolly15k-lora")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

Generating Text

python
# For best results, use the instruction format:
prompt = "Instruction: Explain machine learning in simple terms. Input:  Answer:"

inputs = tokenizer(prompt, return_tensors="pt")
with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        do_sample=True,
        repetition_penalty=1.3,
        no_repeat_ngram_size=3
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response[len(prompt):].strip())

Training Format

This model works best with the Dolly-15k instruction format:

Instruction: [Your instruction here]
Input: [Optional context]
Answer: [Model generates response here]

Examples

Prompt: Instruction: List three benefits of cloud computing. Input: Answer:

Output: The model will provide a structured response listing three benefits of cloud computing.

Limitations

  • —Works best with instruction-formatted prompts
  • —May require specific generation parameters to avoid repetition
  • —Optimized for English instruction-following tasks

Training Details

  • —Framework: Transformers + PEFT
  • —Optimizer: AdamW
  • —Learning Rate: 1e-4
  • —Batch Size: 16 (effective)
  • —Epochs: 2
  • —Hardware: NVIDIA A100

Citation

bibtex
@misc{gpt-neo-dolly15k-lora-2025,
    title={GPT-Neo 1.3B LoRA Fine-tuned on Dolly-15k},
    author={Your Name},
    year={2025},
    publisher={Hugging Face},
    url={https://huggingface.co/YOUR_USERNAME/gpt-neo-1.3b-dolly15k-lora}
}

License

This model is released under the Apache 2.0 License, following the base model's license.