Modelpublic

marcodsn/gemma-4-12b-it-flint-section-merged

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes8downloads

Model Card

gemma-4-12b-it-flint-section-merged

sft-gemma12b-flint-section-nosys arm of the caveman reasoning-compression ablation study: unsloth/gemma-4-12b-it fine-tuned (merged full model) on flint/data/flint-section-aware-gemma12b.jsonl (653 rows, 2 epochs, LoRA r=64).

The study asks whether compressed ("caveman") reasoning traces can train a model to reason in fewer tokens without losing accuracy — and which parts of a trace are compressible. See the run manifest below for the exact recipe; eval results live in the study's report.

Eval summary (t=0, max_tokens 8192)

Accuracy (avg reasoning tokens, loop rate) — this arm vs the original model it was fine-tuned from (unsloth/gemma-4-12b-it), same harness and prompts.

suite	this model	original gemma-4-12b-it
creative@t0.0	None (872.4 tok, loops 0.01)	None (992.2 tok, loops 0.01)
gsm8k@t0.0	0.86 (1679.3 tok, loops 0.04)	0.57 (3753.0 tok, loops 0.06)
humaneval@t0.0	0.57 (4776.8 tok, loops 0.08)	0.31 (6107.2 tok, loops 0.28)
loops:gsm8k@t0.0	0.72 (2931.9 tok, loops 0.06)	0.54 (4121.9 tok, loops 0.06)
loops:gsm8k@t0.6	0.76 (2319.4 tok, loops 0.02)	0.6 (3736.5 tok, loops 0.04)
loops:gsm8k@t1.0	0.96 (1421.3 tok, loops 0.0)	0.8 (2573.7 tok, loops 0.0)
math500@t0.0	0.49 (4882.5 tok, loops 0.19)	0.51 (5077.8 tok, loops 0.2)

Run manifest

json

{
  "arm": "sft-gemma12b-flint-section-nosys",
  "dataset": "flint/data/flint-section-aware-gemma12b.jsonl",
  "rows": 653,
  "dropped_overlong": 124,
  "epochs": 2,
  "system_prompts": false,
  "system_file": null,
  "lora": {
    "r": 64,
    "alpha": 128,
    "dropout": 0.0,
    "target": "all"
  },
  "train": {
    "epochs": 2,
    "lr": 0.0002,
    "batch_size": 1,
    "grad_accum": 16,
    "warmup_ratio": 0.03,
    "lr_scheduler": "cosine",
    "weight_decay": 0.01,
    "seed": 3407,
    "logging_steps": 10,
    "save_strategy": "epoch"
  },
  "model": {
    "name": "unsloth/gemma-4-12b-it",
    "max_seq_length": 8192,
    "load_in_4bit": true,
    "chat_template": "gemma4"
  },
  "train_runtime_s": 10738.3728,
  "final_loss": 0.2742718935012817,
  "log_history": [
    {
      "loss": 0.413497257232666,
      "grad_norm": 1.0575696229934692,
      "learning_rate": 0.000197166934004041,
      "epoch": 0.2450229709035222,
      "step": 10
    },
    {
      "loss": 0.3822723150253296,
      "grad_norm": 41.617244720458984,
      "learning_rate": 0.00018043165652707649,
      "epoch": 0.4900459418070444,
      "step": 20
    },
    {
      "loss": 0.3828407049179077,
      "grad_norm": 0.8829625844955444,
      "learning_rate": 0.00015114354791034225,
      "epoch": 0.7350689127105666,
      "step": 30
    },
    {
      "loss": 0.4039761066436768,
      "grad_norm": 2.487870931625366,
      "learning_rate": 0.00011387355319890685,
      "epoch": 0.9800918836140888,
      "step": 40
    },
    {
      "loss": 0.3301292657852173,
      "grad_norm": 4.181735992431641,
      "learning_rate": 7.443833675595255e-05,
      "epoch": 1.22052067381317,
      "step": 50
    },
    {
      "loss": 0.2838579177856445,
      "grad_norm": 4.314910888671875,
      "learning_rate": 3.899248539894757e-05,
      "epoch": 1.4655436447166923,
      "step": 60
    },
    {
      "loss": 0.2888139486312866,
      "grad_norm": 484.8851318359375,
      "learning_rate": 1.3067972556041752e-05,
      "epoch": 1.7105666156202144,
      "step": 70
    },
    {
      "loss": 0.2742718935012817,
      "grad_norm": 1.0065044164657593,
      "learning_rate": 7.10792629802659e-07,
      "epoch": 1.9555895865237365,
      "step": 80
    },
    {
      "train_runtime": 10738.3728,
      "train_samples_per_second": 0.122,
      "train_steps_per_second": 0.008,
      "total_flos": 2.723717760685179e+17,
      "train_loss": 0.34428255150957804,
      "epoch": 2.0,
      "step": 82
    }
  ]
}