julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer

Occitan Gemma 4 12B IT (RS-LoRA, QAT Base, Merged)

This repository contains a fine-tuned version of Google's Gemma 4 12B Instruct with a QAT-aware base (gemma-4-12B-it-qat-q4_0-unquantized), optimized for the Occitan language using RS-LoRA (Rank-Stabilized Low-Rank Adaptation) via SFTTrainer.

The QAT (Quantization-Aware Training) base model was originally quantized to Q4_0 and then dequantized back to bf16, meaning the weights carry QAT-aware characteristics that may improve robustness to post-training quantization.

Repository Structure

• —Root Directory: Full merged Safetensors weights (bfloat16, compatible with transformers)
• —`adapter/` Folder: The raw RS-LoRA adapter files (for use with PEFT)
• —`gguf/` Folder: Quantized GGUF versions for local inference via llama.cpp, LM Studio, Ollama, etc.

Usage

Using the merged model (Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "user", "content": "Escrivètz un cort paragraf en occitan sus la lenga occitana e son importància."},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using the RS-LoRA adapter with PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-12B-it-qat-q4_0-unquantized",
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12B-it-qat-q4_0-unquantized")

model = PeftModel.from_pretrained(base_model, "julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer", subfolder="adapter")
model.eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Escrivètz un cort paragraf en occitan."}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using GGUF quantizations (llama.cpp)

llama-cli -hf julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer:Q4_K_M \
  -p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
  -n 256 -e --temp 0.7

Or via the GGUF file directly:

llama-cli -m gguf/occitan-gemma-4-12b-it-rslora-qat-sfttrainer-Q4_K_M.gguf \
  -p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
  -n 256 -e --temp 0.7

Adapter Details

Training Details

Data

The model was trained on raw Occitan text from four categories:

• —Literary — Occitan literature texts
• —Journalistic — news articles in Occitan
• —Grammar — grammatical reference texts
• —Encyclopedic — encyclopedic entries in Occitan

All texts were chunked to 384-token blocks (causal language modeling, no chat templating applied during training).

Hyperparameters

Hardware & Quantisation

• —GPU: RTX 3060 12GB VRAM
• —Base Model: QAT-aware q4_0 (dequantized to bf16)
• —Training Quantisation: 4-bit fp4 (BitsAndBytes) — matches QAT base
• —Compute dtype: bfloat16
• —Gradient Checkpointing: enabled (use_reentrant=True)
• —Vision/Audio Embedders: stripped during training to reclaim VRAM (not needed for text-only fine-tuning)

Framework Versions

• —PEFT 0.19.1
• —TRL: 1.4.0
• —Transformers: 5.10.0.dev0
• —PyTorch: 2.7.0+cu128
• —Datasets: 4.8.5
• —Tokenizers: 0.22.2

GGUF Quantizations

Available in gguf/:

Training procedure

This model was trained with SFT (Supervised Fine-Tuning) using SFTTrainer on raw Occitan text. No chat templating was applied during training — the model learns language structure via causal language modeling on chunked text. The RS-LoRA variant helps stabilise training at higher ranks by scaling the LoRA update by alpha / sqrt(r) instead of alpha / r.

The QAT base model (gemma-4-12B-it-qat-q4_0-unquantized) was quantized to Q40 and dequantized back to bf16 before training, meaning the base weights carry quantization-aware characteristics. Combined with fp4 training quantization, this approach aims to produce a model that is more robust to post-training quantization (e.g., GGUF Q4KM or Q2K).

Framework versions

• —PEFT 0.19.1
• —TRL: 1.4.0
• —Transformers: 5.10.0.dev0
• —PyTorch: 2.7.0+cu128
• —Datasets: 4.8.5
• —Tokenizers: 0.22.2

Citations

@software{geyronneau2024trl,
  title   = {{TRL: Transformers Reinforcement Learning}},
  author  = {Geyronneau, Quentin and von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
  license = {Apache-2.0},
  url     = {https://github.com/huggingface/trl},
  year    = {2024}
}