trentzap/QTensor-TinyLlama-1.1B-Alpha

QTensor TinyLlama 1.1B (Alpha)

QTensor TinyLlama 1.1B is an extremely compressed, high-performance variant of TinyLlama/TinyLlama-1.1B-Chat-v1.0.

By utilizing the QTensor Architecture, all standard nn.Linear projection matrices (Q/K/V, MLP gates) have been compressed by 7.82x using a Hybrid Matrix Product Operator (MPO) decomposition paired with ternary (-1, 0, 1) INT8 weight packing and a bfloat16 LoRA error-cancellation adapter.

This model dynamically alters the PyTorch execution graph to utilize a custom Triton SRAM Fusion Engine that computes additive MPO operations directly inside GPU L1 Cache, cutting VRAM overhead down to sub-1GB while generating text at 30+ tokens/second.

📊 Compression & Hardware Telemetry

📊 Empirical Benchmarks & Evaluation

1. GGUF Baseline Comparison (RTX 5080)

We benchmarked QTensor-TinyLlama-1.1B against standard llama.cpp GGUF conversions (Q3_K_M and Q2_K from TheBloke) using identical temperature settings (T=0.7, top_p=0.9):

Key Finding: Standard 2-bit quantization (Q2_K) suffers catastrophic manifold collapse (PPL 1845.20). QTensor's LoRA Knowledge Distillation effectively heals the ternary noise, outperforming 3-bit GGUF in perplexity while maintaining sub-1GB VRAM execution.

2. Zero-Shot Downstream Retention (lm-evaluation-harness)

Using lm-evaluation-harness, QTensor was evaluated across standard reasoning benchmarks to verify downstream task retention against pristine FP16 weights:

🚀 Quickstart & Usage

Because QTensor employs a custom Triton execution graph, you must pass trust_remote_code=True when loading the model to pull the custom architecture patcher.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "trentzap/QTensor-TinyLlama-1.1B-Alpha"

# 1. Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# 2. Load Custom QTensor Architecture (Loads Triton JIT Kernels)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    trust_remote_code=True, 
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)

# 3. Generate Text!
prompt = "The capital of Australia is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))