laion/grug-67b-a2b-sft-s3-agentic-step1903-repaired

Grug 67B-A2B SFT (Stage 3, AGENTIC opencode tool-calling) - step 1903

HF-BF16 (safetensors) export of the marin Grug 67B-A2B MoE model after SFT Stage 3 (agentic opencode tool-calling). Architecture GrugMoeForCausalLM (model_type: grug_moe); serve with the marin vLLM fork (--enable-expert-parallel), not upstream vLLM.

• —Total params: ~67B (A2B active MoE: 256 experts, 4 experts/token, 26 layers, hidden 2560, 20 heads / 5 KV heads).
• —Lineage: june-67b-a2b cooldown step-42150 -> SFT Stage 1 wildchat (step-257) -> SFT Stage 2 thinking (penfever/grug-67b-a2b-sft-s2-thinking-step630) -> SFT Stage 3 agentic (this model), weights-only init (fresh optimizer, step 0) from the Stage-2 step-630 endpoint, 5 packed epochs over 32 opencode serve-parity SFT datasets = 1903 steps, seq_len 32768, global batch 64, optimizer AdamH, LR 5e-6.
• —Chat template: the OPENCODE tools-aware Qwen-style jinja the model was trained with ({% generation %} completions-only mask; <tools> system block; structured <tool_call>/"arguments"; <tool_response> framing; <think>). Baked into tokenizer_config.json + chat_template.jinja at export (NOT a generic template).
• —Tokenizer: penfever/grug-67b-a2b-sft-s2-thinking-step630-tok (Llama-3 128256 vocab).

Export provenance

Exported from the native Levanter/Orbax checkpoint via marin's sanctioned GrugModelConfig.hf_checkpoint_converter().with_config_overrides({"dtype":"bfloat16"}) path (experiments/grug/moe/model.py), reproducing tests/vllm/e2e/test_june_67b_a2b_hf_bf16_export.py. pending_qb_betas is baked into the router bias before export (required for correct logits). All tensors BF16. Training chat template preserved explicitly on export (marin #7406).

Companion s3 export: s3://marin-us-east-02a/marin/exports/grug/june-67b-a2b-sft-s3-agentic/step-1903/hf-bf16-vllm/