CoolFace
Modelpublic

julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes91downloads
Model Card

Occitan Gemma 3 4B IT (RS-LoRA, Merged)

This repository contains a fine-tuned version of Google's Gemma 3 4B Instruct optimized for the Occitan language using RS-LoRA (Rank-Stabilized Low-Rank Adaptation).

Repository Structure

  • —Root Directory: Full merged Safetensors weights (bfloat16, compatible with transformers)
  • —`adapter/` Folder: The raw RS-LoRA adapter files (for use with PEFT)
  • —`gguf/` Folder: Quantized GGUF versions (f16, Q2K, Q4KM, Q5KM, Q80) for local inference via llama.cpp, LM Studio, Ollama, etc.

Usage

Using the merged model (Transformers)

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

messages = [
    {"role": "user", "content": "Escrivètz un cort paragraf en occitan sus la lenga occitana e son importància."},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using the RS-LoRA adapter with PEFT

python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-4b-it",
    device_map="auto",
    torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")

# Load adapter from the adapter/ subfolder
model = PeftModel.from_pretrained(base_model, "julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer", subfolder="adapter")
model.eval()

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Sètz un assistent utila que parla occitan."}],
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using GGUF quantizations (llama.cpp)

bash
llama-cli -hf julienp79/occitan-gemma-3-4b-it-rslora-sfttrainer:Q4_K_M \
  -p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
  -n 256 -e --temp 0.7

Or via the GGUF file directly:

bash
llama-cli -m gguf/occitan-gemma-3-4b-it-rslora-sfttrainer-Q4_K_M.gguf \
  -p "<start_of_turn>user\nEscrivètz un cort paragraf en occitan.<end_of_turn>\n<start_of_turn>model\n" \
  -n 256 -e --temp 0.7

Adapter Details

  • —Adapter Type: RS-LoRA (Rank-Stabilized LoRA, use_rslora=True)
  • —Base Model: google/gemma-3-4b-it (Google Gemma 3 4B Instruct)
  • —LoRA Rank (r): 32
  • —LoRA Alpha: 32
  • —Target Modules: qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
  • —Dropout: 0
  • —Training Method: SFT (Supervised Fine-Tuning) with SFTTrainer

Training Details

  • —Block Size: 768 tokens
  • —Learning Rate: 1e-4 (cosine schedule)
  • —Optimizer: Paged AdamW 8-bit
  • —Max Gradient Norm: 1.0
  • —Warmup Steps: 200
  • —Weight Decay: 0.01
  • —Epochs: 5
  • —Total Steps: ~1270
  • —Final Loss: ~1.18
  • —Hardware: RTX 3060 12GB (full bfloat16 base model)

Training procedure

This model was trained with SFT (Supervised Fine-Tuning) using SFTTrainer on raw Occitan text (literary, journalistic, grammar, and encyclopedic sources). No chat templating was applied during training — the model learns language structure via causal language modeling on chunked text.

Framework versions

  • —PEFT 0.19.1
  • —TRL: 1.4.0
  • —Transformers: 5.8.1
  • —Pytorch: 2.6.0+cu124
  • —Datasets: 4.8.5
  • —Tokenizers: 0.22.2