jayshah5696/gemma4-e2b-humanize-rl-candidate-v1

Gemma 4 E2B — Humanize RL Candidate v1

Status: candidate-v1 — RL-tuned LoRA adapter, pending manual review before promotion to final.

A LoRA adapter trained with GRPO (TRL + Modal, A100-40GB, colocated vLLM) on top of `jayshah5696/gemma4-e2b-humanize-unsloth-merged`, which is itself an SFT-merged humanize model on Gemma 4 E2B IT.

The RL objective is a softened scalar reward combining a ridge-rubric humanness scorer (0.45 weight), a deterministic verifier (0.35 weight), and a clipped risk-compliance term (0.20 weight, penalty_cap=1.5). The dataset is a filtered mix of v01 (smoke), v02 (rewrite-heavy), and v03 (rich instructions + 6 modes) totaling 964 tasks (771 train / 90 validation / 103 test), balanced ≈ 41% v02 / 58% v03 / <1% v01 (v01 collapsed via text dedup vs v02).

Recipe

# Frozen from Slices 5–8 (see logbook / plan).
num_generations: 16
gradient_accumulation_steps: 16
per_device_train_batch_size: 1   # effective rollout batch = 256
learning_rate: 1.0e-5
optim: adamw_8bit
loss_type: bnpo
epsilon: 0.2
epsilon_high: 0.28
delta: 1.5
mask_truncated_completions: true
temperature: 1.0
seed: 3407
max_steps: 600
warmup_ratio: 0.1
max_grad_norm: 0.5

# Reward.
reward_mode: scalar_softened
penalty_cap: 1.5
ridge_weight: 0.45
deterministic_weight: 0.35
risk_weight: 0.20

# Reward normalization.
scale_rewards: "group"   # TRL default

# Sampling stratification.
stratify_batches: true
stratify_by: reward_profile

# vLLM (colocated).
use_vllm: true
vllm_mode: colocate
vllm_gpu_memory_utilization: 0.55

# LoRA.
lora_rank: 16
lora_alpha: 32

# Lengths.
max_seq_length: 2048
max_prompt_length: 512
max_completion_length: 768

Training wall-clock: 117 minutes (600 steps × ~11.7 s/step) on a single A100-40GB.

Training trajectory

Final EMAs at step 600:

The key delta vs the Slice 8 200-step pilot is risk_compliance: 0.20 → 0.72 (3.6×). This validates the penalty_cap=1.0 → 1.5 hypothesis from the full-run plan §3.2 — the previous training-time risk penalty was saturating at the softened-reward floor, killing the risk-side gradient.

Eval results

All eval rewards are the strict reward (raw penalties applied, no softening) — the user-facing metric, not the training-time shaped scalar.

Risk-penalty deltas (mean penalty improvement, larger is better):

Qualitative inspection

Sampled 16 validation tasks × 4 completions each at temperature=0.7. Manual side-by-side review of 4 random task pairs (compression and rewrite tasks): the adapter outputs are noticeably warmer and more conversational (e.g. opens with "Heads up", "hope you're having a good week"), while keeping required facts intact. Mean response length: 290 chars baseline → 272 chars adapter — slight tightening on compression tasks where it's warranted, no length collapse on generation tasks. No obvious style hacks ("always short to dodge penalty" pattern not observed).

Dataset mix

Built via scripts/rl/build_rl_task_mix.py + difficulty filter + local rebucket under scalar_softened_permissive mode (see scripts/rl/rebucket_difficulty.py).

Mode distribution in the train split:

Known limitations

1. 1.Reward saturation only partially addressed. Training-time penalty_rate/ema_50 dropped from Slice 8's 0.998 to 0.598 thanks to penalty_cap=1.5, but the strict-eval mean risk penalty barely moved (E1: −0.531 → −0.518, +0.013). Most eval gain came from style/structure improvements (ridge + deterministic), not from genuinely avoiding penalties. A next-iteration follow-up should add synonym-tolerant fact matching in _fact_is_present to widen the achievable risk-compliance ceiling.
2. 2.v03 author imbalance. The v03 task corpus is 57% GPT-5.4-mini / 43% Gemini Flash-Lite; the planned 40% Gemini Pro share is absent. The adapter therefore inherits any phrasing fingerprints from those two specific authors more than the v03 plan intended.
3. 3.No long-form ridge adapter. The ridge scorer was trained on short samples; long-form v03 tasks (200–600 word completions) may systematically under-score relative to short tasks. The E6 delta (+0.0408) is real but smaller than E1/E3/E4 deltas in relative terms.
4. 4.`scale_rewards="group"` (default), not `"batch"`. Slice 7 showed "batch" was a wash at 50 steps; we did not re-test at the 600-step horizon. The plan-default may be the right choice for a longer run; left as a follow-up ablation.
5. 5.`candidate-v1`, not final. No external benchmark verification yet; humanness is measured only on our own ridge + deterministic reward. A blind A/B against the SFT-only base is the next gate before promoting to final.

Reproducibility

Loading

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = "jayshah5696/gemma4-e2b-humanize-unsloth-merged"
adapter = "jayshah5696/gemma4-e2b-humanize-rl-candidate-v1"

processor = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype="bfloat16")
model = PeftModel.from_pretrained(model, adapter)
model.eval()

(Note: requires the vLLM 0.20.x Gemma 4 kv-shared knorm patch when serving with vLLM — see `patchvllmgemma4kvsharedk_norm` in the training script for the 4-line inline fix until upstream PR #40117 lands.)