ToastyPigeon/gemma-4-12b-full-cpt-itvec

Gemma-4-12B — full-CPT + instruct task vector

A Gemma-4-12B (base, non-instruct) derivative built in two steps:

1. 1.CPT merge: the gemma-4-12b-full-cpt LoRA (QLoRA r64, all projections, SWA layers, 1740 steps on a prose/story-heavy mix) folded into google/gemma-4-12b.
2. 2.Task-vector apply: the instruct task vector gemma-4-12b-it − gemma-4-12b added at weight 1.0 (normal scaling):

final = base + CPT-LoRA + (it − base)

The goal: keep the CPT prose/style gains while regaining the instruct model's instruction-following and formatting behavior.

Training data (CPT step)

Mix of story/prose and chat-log corpora (the named sets are public under the ToastyPigeon namespace): erotic-books-clone, new-story-dataset, some-erotica, steve-and-marvin, SpringDragon-chat, disco-chat, counter-signal — plus two other private datasets with a focus on WLW content and chat.

Content note: parts of the CPT mix are adult/erotic fiction. Intended for fiction-writing research and style experimentation by adults; not for production deployment or use by minors.

Intended use

• —Text-completion / storywriting experiments
• —Starting point for downstream RP/style finetunes that want both the CPT prose signal and instruct-style behavior

Notes

• —Unified multimodal architecture (Gemma4UnifiedForConditionalGeneration); the vision/audio towers are unchanged from base.
• —See the companion adapter repo gemma-4-12b-full-cpt for the CPT run details (hyperparameters, loss curve).