Modelpublic

ChatoyantAI/gemma-4-12b-it-lusy-sft-v4.0.1-step4300-lora

sourceHugging Faceapache-2.0updated 17d agoView on Hugging Face

0likes16downloads

Model Card

Lusy v4.0.1 — Gemma 4 12B IT LoRA, step 4300

This is checkpoint-4300, epoch 1.27220948 (approximately 1.27 epochs) from the Lusy v4.0.1 training run. It is not the exact first-epoch checkpoint, not the completed two-epoch model, and not the older roleplay-only experiment. These are adapter weights, not a standalone or merged model. The original google/gemma-4-12B-it base is required.

Training

—Base architecture: Gemma4UnifiedForConditionalGeneration.
—Text attention/MLP LoRA: rank32, alpha64, dropout0; 131,137,536 trainable parameters.
—Unsloth + PEFT; BF16 base, no weight quantization; 2 H100 GPUs.
—Fixed batch1/GPU, accumulation8, global batch16. Dynamic batching is disabled.
—Peak learning rate1e-4, cosine schedule, 10% warmup, gradient clipping1.
—Planned training: 2 epochs; this release is an intermediate checkpoint.
—Dataset v4.0.1: 44,275 roleplay + 9,800 general training records; validation: 891 roleplay + 200 general records. Data is not included.
—Only the final target Assistant response is supervised; earlier turns are context.
—Maximum configured sequence length65,536; no sequence packing.
—Last preceding combined validation loss: 1.05497244 at step4200. This is not a fresh evaluation of step4300 or an external benchmark result.

Loading

Use a Transformers build supporting Gemma4UnifiedForConditionalGeneration (training used Transformers5.10.2, PEFT0.18.1 and Unsloth2026.9.2).

python

import torch
from transformers import Gemma4UnifiedForConditionalGeneration, AutoTokenizer
from peft import PeftModel

adapter = "ChatoyantAI/gemma-4-12b-it-lusy-sft-v4.0.1-step4300-lora"
tokenizer = AutoTokenizer.from_pretrained(adapter)
base = Gemma4UnifiedForConditionalGeneration.from_pretrained(
    "google/gemma-4-12B-it", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, adapter)
model.eval()

Use the included checkpoint chat template. When present, preserve the sequence system -> assistant greeting -> user -> assistant history ... -> user. 65,536 is the training configuration, not a guarantee of long-context accuracy. The adapter targets text; multimodal behavior has not been evaluated after SFT.

Scope and limitations

The dataset includes roleplay and mature themes. Outputs can be inaccurate, biased, inappropriate, or memorized and require review. No independent preference, safety, or general-capability benchmark improvement is claimed. Verification covers adapter tensor shape/finiteness, asset integrity and tokenizer-template loading; no new generation benchmark is included with this release.

release-manifest.json records hashes and provenance. Optimizer states, training examples, credentials and raw logs are excluded. Apache2.0, following the base model metadata; see LICENSE, NOTICE and BASEMODELREADME.md. Not an official Google release.