Modelpublic

efficiencyx/Jun-LoRA-12B-v5-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes85downloads

Model Card

Jun-12B-GGUF

Merged GGUF builds of the latest Jun LoRA on Gemma 4 12B (QAT) — a fine-tune trained on a compact, heavily curated synthetic conversational dataset derived from the visual novel My Dystopian Robot Girlfriend. The model captures the personality, speech patterns, and emotional nuance of the character Jun while preserving the base model's general reasoning and instruction-following capabilities.

The adapter is merged into the base weights here — these are standalone models, no --lora flag needed.

Model Variants & Repositories

Repository	Format	Description
`efficiencyx/Jun-12B-GGUF`	GGUF (Q80 / Q6K / Q4KM)	Merged, quantized, for local inference
`efficiencyx/Jun-LoRA-12B-Adapter`	LoRA Adapter	The adapter merged into these builds
`efficiencyx/Jun-LoRA-12B-Adapter-Step60`	LoRA Adapter	Earlier checkpoint (step 60)
`efficiencyx/Jun-LoRA-v4-12B-GGUF`	GGUF	Previous generation (v4)

Quantization Guide

Quant	Size	Use Case
Q8_0	12.7 GB	Best quality, suggested ~16 GB VRAM
Q6_K	9.8 GB	High quality, minimal loss
Q4KM	7.4 GB	Fits 8 GB VRAM with acceptable quality loss

Sizes are measured, not estimated. The base model is QAT (quantization-aware trained), so lower quants hold up better than a standard FP16 export. All three are quantized from the same bf16 master — no requantization chain, no imatrix.

Usage

bash

llama-server -m Jun-12B-Q4_K_M.gguf --jinja -ngl 99 -c 8192

--jinja is required. Without it llama.cpp ignores the embedded chat template and tool calls come back as plain text instead of structured calls.

Intended Use

Conversational backend for Jun OS, an AI companion webapp:

—Character-consistent multi-turn conversation
—AI companion / interactive fiction applications
—Research into character-faithful fine-tuning on small, high-quality datasets

Limitations

—Specialized for a single character persona; not a general-purpose assistant.
—Outputs reflect fictional narrative tropes and are not factual information or advice.
—Performance degrades far outside the training distribution.
—Inherits any biases present in the Gemma 4 12B base weights.

Training Details

Parameter	Value
Base model	`unsloth/gemma-4-12B-it-qat-q4_0-unquantized`
Method	LoRA (rsLoRA)
LoRA rank	32
LoRA alpha	32
LoRA dropout	0.01
Target modules	q/k/v/o + gate/up/down projections, language tower
Learning rate	1e-4, cosine schedule, 10 warmup steps
Batch size	16 (no gradient accumulation)
Epochs	3
Max sequence length	2048
Weight decay	0.001
Optimizer	AdamW (8-bit)
Loss masking	completions only
Gradient checkpointing	Unsloth
Packing	off
Seed	3407
Fine-tuning framework	Unsloth
GGUF export pipeline	llama.cpp

Checkpoints were saved every 30 steps; `Step60` is published separately. These builds merge the final adapter.

Evaluation

Metric	Value
Final training loss	~0.6
Final eval loss	~0.07

Merge & Export

The adapter was merged directly on the safetensors as W += (B @ A) · scale in fp32 (rsLoRA scale 32/√32), cast back to bf16, exported with convert_hf_to_gguf.py --outtype bf16, then quantized with llama-quantize.

Acknowledgments

—Incontinent Cell for *My Dystopian Robot Girlfriend*, Jun's character
—Google for the Gemma 4 model family
—Unsloth for the efficient fine-tuning framework and the QAT base model