CoolFace
Modelpublic

julienp79/occitan-gemma-4-12b-it-lora

sourceHugging Facegemmaupdated 4mo agoView on Hugging Face
0likes185downloads
Model Card

Occitan Gemma-4-12B-IT (LoRA Merged)

This repository contains a fine-tuned version of Google's Gemma-4-12B-IT specifically optimized for the Occitan language.

The model was trained using LoRA (Low-Rank Adaptation) on a balanced corpus of Occitan texts. This is the largest Occitan Gemma model to date, offering significant improvements in reasoning and linguistic nuance.

🛠️ Training Engineering

Training a 12B model on an RTX 3060 (12GB VRAM) required surgical optimizations:

  • —Memory Management: Utilized paged_adamw_8bit to allow optimizer states to spill into system RAM. Vision and audio embedders were stripped to reclaim VRAM.
  • —Context Window: Block size set to 384 tokens to reduce activation memory overhead.
  • —Quantization: Loaded in 4-bit NormalFloat (NF4) with Double Quantization enabled.
  • —Gradient Checkpointing: Enabled with use_reentrant=True.
  • —Effective Batch Size: 8 (batch 1 × 8 gradient accumulation steps).

Training Details

ParameterValue
Base modelgoogle/gemma-4-12B-it
LoRA rank8
LoRA alpha16
LoRA dropout0
Target modules`.language_model.(q_proj\k_proj\v_proj\o_proj\gate_proj\up_proj\down_proj)`
Optimizerpaged_adamw_8bit
Learning rate5e-5
SchedulerCosine, 400 steps warmup
Epochs5
Weight decay0.01
Max grad norm1.0
Frameworktransformers + PEFT (no SFTTrainer, no unsloth)

Dataset

Balanced Occitan corpus across four categories:

  • —Literary — prose and poetry
  • —Journalistic — news and articles
  • —Grammar — grammatical examples and exercises
  • —Encyclopedic — factual and reference texts

Training was performed as raw causal language modeling (no chat wrapping).

📁 Repository Structure

  • —Root Directory: Full merged Safetensors weights (compatible with transformers, accelerate, etc.).
  • —`/gguf` Folder: Quantized versions for local inference via LM Studio, Ollama, or llama.cpp.
  • —`/adapter` Folder: The raw LoRA adapter files for researchers who wish to inspect the weights or perform their own merges.

🚀 How to Use

Using Transformers (Python)

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "julienp79/occitan-gemma-4-12b-it-lora"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="bfloat16",
)

messages = [
    {"role": "user", "content": "Pòdes m'ajudar a escriure un pichon tèxt en occitan?"},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

GGUF Quantisation

QuantSizeDescription
Q2_K~4.5 GBAggressive, for very constrained hardware
Q4KM~7 GBMain recommendation, best quality/size tradeoff
Q5KM~8 GBHigher quality, for users with more RAM
Q8_0~11 GBNear-lossless, for power users
f16~23 GBFull precision (source quant)

🤖 About Gemma 4

Gemma 4 is Google's latest family of lightweight open models, built from the same research and technology used to create the Gemini models. This 12B version offers strong reasoning capabilities while remaining suitable for local deployment.