dharun2049/kaveri-stgrpo-0.5b

Kaveri ST-GRPO 0.5B

Kaveri ST-GRPO 0.5B is an experimental coding post-training checkpoint derived from `Qwen/Qwen2-0.5B-Instruct`.

It uses stock TRL GRPO with an experimental Signed Tensor GRPO (ST-GRPO) reward-shaping layer. Reward and punishment are maintained as separate signals. Third-order tensors encode interactions among triples of sampled solutions.

ST-GRPO

For each prompt, the trainer samples 8 rollouts, giving 56 unordered triples.

Positive tensor channels:

• —verified quality
• —quality-weighted structural/strategy diversity
• —complementary official-test coverage

Punishment tensor channels:

• —verifier-derived penalty
• —collapse of bad structurally similar solutions
• —common official-test failure
• —lexical repetition/degeneration

R_i = a*z(r_i+) - b*z(r_i-) + lp*z(C_i+) - ln*z(C_i-)

The signed scalar is returned to TRL, which performs normal group-relative advantage computation and policy optimization.

Training setup

• —Base model: Qwen/Qwen2-0.5B-Instruct
• —Dataset: open-r1/codeforces, verifiable-prompts
• —Language: C++17
• —Trainer: TRL GRPOTrainer
• —Generations per prompt: 8
• —Third-order triples per prompt: 56
• —Generation batch size: 16
• —Optimizer steps: 150
• —Max completion length: 256
• —LoRA rank: 8
• —LoRA alpha: 16
• —LoRA targets: q_proj, v_proj
• —Positive weight: 1.0
• —Punishment weight: 0.85
• —Tensor positive weight: 0.35
• —Tensor punishment weight: 0.45
• —Verifier: local g++ compile + official tests

Files

The repository root contains the merged standalone model. The original LoRA adapter is preserved in lora_adapter/.

Status

Experimental research checkpoint. Evaluate independently before drawing benchmark conclusions.

mmlu

43.94%