arif-butt/tinyllama-trl-gguf

🦙 TinyLlama GGUF - Quantized Model

Model Description

This is a GGUF quantized version of TinyLlama (1.1B parameters) fine-tuned using Unsloth and TRL with LoRA adapters. The model has been optimized for efficient CPU inference with minimal memory footprint.

Key Features:

• —GGUF Format: Optimized for llama.cpp and CPU inference
• —Quantized: Reduced memory usage without significant quality loss
• —Fine-tuned: Custom trained on specific dataset for improved performance
• —Efficient: Runs on CPU, Raspberry Pi, and mobile devices

Model Details:

• —Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• —Fine-tuning Method: LoRA (Low-Rank Adaptation) with Unsloth optimizations
• —Format: GGUF (GGML Universal Format)
• —Quantization: Q4KM (4-bit quantization)
• —Parameters: 1.1 Billion
• —Context Length: 2048 tokens

🛠️ Training Details

Base Model

Fine-tuning Configuration

LoRA Parameters

LORA_R       = 16        # Rank of LoRA matrices
LORA_ALPHA   = 32        # Scaling factor (alpha/r = 2.0)
LORA_DROPOUT = 0.05      # Dropout for regularization
TARGET_MODULES = [       # Layers where LoRA is applied
    "q_proj",            # Query projection
    "k_proj",            # Key projection  
    "v_proj",            # Value projection
    "o_proj",            # Output projection
    "gate_proj",         # Gate projection (MLP)
    "up_proj",           # Up projection (MLP)
    "down_proj"          # Down projection (MLP)
]

## 🚀 Usage

### Option 1: Using llama-cpp-python (Recommended)

Install llama-cpp-python

pip install llama-cpp-python

from llama_cpp import Llama

Load the model

modelpath = "arif-butt/tinyllama-trl-gguf" # or local path to .gguf file llm = Llama( modelpath=modelpath, nctx=2048, # Context length nthreads=4, # Number of CPU threads ngpu_layers=0, # Set >0 for GPU offloading verbose=False, )

Simple prompt

prompt = "Q: Name all the courses Arif butt teach?\nA:"

Generate response

output = llm( prompt, maxtokens=100, # Maximum tokens to generate temperature=0.2, # Lower = more deterministic topp=0.95, # Nucleus sampling repeat_penalty=1.1, # Penalize repetition stop=["Q:", "\nQ:"], # Stop sequences )

print(output["choices"][0]["text"])

── Simple Chat Interface for GGUF Model ───────────────────────────────────

from llama_cpp import Llama import sys

class TinyLlamaChat: def _init(self, modelpath="arif-butt/tinyllama-trl-gguf"): """Initialize the GGUF model""" print("Loading model...") self.llm = Llama( modelpath=modelpath, nctx=2048, nthreads=4, verbose=False, ) print("✅ Model loaded!")

def generate(self, prompt, maxtokens=100, temperature=0.7): """Generate response for a single prompt""" output = self.llm( prompt, maxtokens=maxtokens, temperature=temperature, topp=0.95, repeat_penalty=1.1, stop=["Q:", "\nQ:", "User:", "\nUser:", "Human:"], ) return output["choices"][0]["text"]

def chat(self): """Interactive chat mode""" print("\n💬 Chat Mode (type 'quit' to exit)") print("-" * 50)

while True: userinput = input("\n👤 You: ") if userinput.lower() in ['quit', 'exit', 'q']: break

prompt = f"Q: {user_input}\nA:" response = self.generate(prompt, temperature=0.7) print(f"🤖 Assistant: {response}")

Use the chat interface

if _name == "main_": chat = TinyLlamaChat() chat.chat()