CoolFace
Modelpublic

Mahesh111000/hanabi-training12-combined-skyrl

sourceHugging Faceupdated 8d agoView on Hugging Face
0likes24downloads
Model Card

hanabi-training12-combined-skyrl (final, merged fp32)

Final checkpoint of the RL run training12-combined-20260918T075329414627Z (W&B project hanabi_training12_combined_skyrl), merged into its base.

BaseMahesh111000/Qwen_merged @ cf390c83fb3214ee0d21f7fa0fd62d440942d666 (Qwen3-4B-Instruct-2507 lineage)
Trainedfresh rank-32 LoRA (alpha 32) on q/k/v/o/gate/up/down projections and lm_head
Rewardstate tracking (weight 1.0) + move rating (weight 1.0)
Data2,362 Hanabi prompts, 2 epochs, 148 optimizer steps, 32 groups x 16 rollouts per step (75,584 completions)
OptimizerAdam, lr 2e-5, betas (0.9, 0.95), no KL penalty, importance-sampling loss, temperature 1.0, 16,384 max new tokens
StackSkyRL self-hosted Tinker backend (Tinker SDK 0.24.1), FSDP, 2 x 8 A100-80GB
Training reward (mean of first 10 -> last 10 steps)total 0.62 -> 1.25; deduction score 0.19 -> 0.66; exact deduction 0.01 -> 0.22; move reward 0.43 -> 0.60; format 0.99 -> 1.00

Format

  • —fp32 safetensors, 4 shards (17.6 GB). Serve in half precision with --dtype bfloat16.
  • —`lm_head` is untied from `embed_tokens` (tie_word_embeddings: false). The adapter trains lm_head while the base ties it to the embedding; lm_head.weight = embed_tokens.weight + B@A, embed_tokens is unchanged. Do not use PEFT merge_and_unload() on the adapter: it writes the lm_head delta into the tied embedding and changes the model (logit error 0.4-0.8 measured here).
  • —Merge is done in fp32: the base is fp16 and max ||dW||/||W|| is 1.4e-3, so a half-precision merge rounds most of the update away.

Verification

training_run/verify_merge.py, verify_merge2.py: on three training examples (768 tokens each) this checkpoint matches PEFT(base fp32 + adapter) with mean |logit diff| 1e-5, max 1.3e-2 on logits of magnitude 55-59, per-position KL <= 4e-7, identical argmax at every position, identical data log-likelihood to 6 decimals; the adapter itself moves logits by up to 19.

Per-step adapters (steps 1-148): Mahesh111000/hanabi-training12-combined-skyrl-loras. Run config, manifest, metrics and the checkpoint index are in training_run/.