Modelpublic

BirdToast/Gemma-4-31B-glimmer-rp-v0.2

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes9downloads

Model Card

gemma4-31b-it-glimmer-rp-r16a32-v2

This model is a fine-tuned version of google/gemma-4-31B-it.

W&B run: https://wandb.ai/cooawoo-personal/Gemma4-31B/runs/cpm2god5

Training procedure

Hyperparameters

Parameter	Value
Learning rate	`1e-05`
LR scheduler	SchedulerType.CONSTANTWITHWARMUP
Per-device batch size	1
Gradient accumulation	4
Effective batch size	4
Epochs	1
Max sequence length	6144
Optimizer	OptimizerNames.PAGEDADAMW8BIT
Warmup ratio	0.05
Max gradient norm	1.0
Precision	bf16
Loss type	nll
Assistant-only loss	yes
Chunked cross-entropy	yes

LoRA configuration

Parameter	Value
Rank (r)	16
Alpha	32
Target modules	.*languagemodel\.layers\.\d+\.(selfattn\.(q	k	v	o)_proj	mlp\.(gate	up	down)_proj)$
Quantization	4-bit (nf4)

Dataset statistics

Dataset	Samples	Total tokens	Trainable tokens
writing_critique.jsonl	1,586	1,317,299	599,280
rpgenerationaesir_v2.jsonl	297	757,671	250,835
marvinstylebible.jsonl	5,137	22,329,966	21,120,065
rpgenerationmistral.jsonl	255	820,798	375,844
instruct.jsonl	932	902,452	810,983
rpgenerationlocal_300.jsonl	285	641,131	265,274
rpgenerationfinal.jsonl	261	1,015,876	475,576
rpgenerationtrans_stories.jsonl	368	880,474	840,128
rp_analysis.jsonl	244	725,062	177,829
Total	9,365	29,390,729	24,915,814

<details> <summary>Training config</summary>

yaml

model_name_or_path: google/gemma-4-31B-it
data_config: data_v2.yaml
prepared_dataset: prepared_packed_fixed
output_dir: gemma4-31b-it-glimmer-rp-r16a32-v2
chat_template_path: chat_template_with_channel_v4_train.jinja
attn_implementation: flex_attention
bf16: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
use_cce: true
chunked_mlp: true
chunked_mlp_chunks: 16
dataloader_num_workers: 2
dataloader_pin_memory: true
model_parallel: true
max_memory:
  0: 16GiB
  1: 24GiB
max_length: 6144
per_device_train_batch_size: 1
gradient_accumulation_steps: 4
pad_to_multiple_of: 128
use_peft: true
load_in_4bit: true
bnb_4bit_quant_type: nf4
lora_r: 16
lora_alpha: 32
lora_dropout: 0.0
use_rslora: false
lora_target_modules: .*language_model\.layers\.\d+\.(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj)$
learning_rate: 1.0e-05
lr_scheduler_type: constant_with_warmup
warmup_ratio: 0.05
weight_decay: 0.0
max_grad_norm: 1.0
optim: paged_adamw_8bit
num_train_epochs: 1
saves_per_epoch: 2
save_total_limit: 4
rolling_save_steps: 30
rolling_save_total_limit: 1
assistant_only_loss: true
full_mask_reasoning: true
logging_steps: 1
disable_tqdm: false
report_to: wandb
run_name: g4-31b-it-glimmer-rp-v2

</details>

<details> <summary>Data config</summary>

yaml

datasets:
- path: rp_generation_final.jsonl
  type: chat
  truncation_strategy: split
- path: rp_generation_mistral.jsonl
  type: chat
  truncation_strategy: split
- path: instruct.jsonl
  type: chat
  truncation_strategy: split
- path: rp_analysis.jsonl
  type: chat
  truncation_strategy: split
- path: writing_critique.jsonl
  type: chat
  truncation_strategy: split
- path: marvin_style_bible.jsonl
  type: chat
  truncation_strategy: split
- path: rp_generation_aesir_v2.jsonl
  type: chat
  truncation_strategy: split
- path: rp_generation_local_300.jsonl
  type: chat
  truncation_strategy: split
- path: rp_generation_trans_stories.jsonl
  type: chat
  truncation_strategy: split
shuffle_datasets: true
shuffle_combined: true
shuffle_seed: 42
eval_split: 0
split_seed: 42
assistant_only_loss: true

</details>

Framework versions

—PEFT 0.18.1
—Loft: 0.1.0
—Transformers: 5.5.4
—Pytorch: 2.6.0+cu124
—Datasets: 4.6.1
—Tokenizers: 0.22.2