thesilverheadengineer/llm-surgery-dark-arts-gpt-oss-40b-64a8-init

LLM Surgery Dark Arts: GPT-OSS 40B (64 Experts, Active 8) · BnB 4-bit

All Editions

Init checkpoints, mathematically identical to gpt-oss-20b at initialization. Designed as expanded-capacity bases for fine-tuning on complex reasoning and domain specialization tasks.

1. Context

The Bottleneck

gpt-oss-20b achieves remarkable performance from a compact architecture: 32 experts, 4 active per token, 24 layers. For general use this is sufficient. For research labs pushing domain-specific reasoning (mathematical competition, complex code generation, scientific inference), the expert pool becomes the limiting factor. The hidden dimension and depth are adequate; the number of distinct expert combinations the router can assign is not.

Width Over Depth

Adding layers increases KV cache consumption linearly, directly reducing throughput and maximum context length. Adding experts with the same layer count leaves the KV cache untouched.

• —Depth (more layers): linear KV cost, modest capacity gain
• —Width (more experts, same layers): zero KV impact, exponential routing growth

These models expand the expert pool while preserving the 24-layer architecture exactly. Attention, positional encoding, vocabulary, sliding window: all unchanged. Only the MoE routing space grows.

The quality of the resulting fine-tuned model depends on dataset preparation and training strategy, specifically on ensuring the expanded expert pool is utilized diversely rather than collapsing back to redundant configurations.

Train Wide, Deploy Narrow

The 64a8 and 96a12 configurations activate more experts per token than the original (8 or 12 vs 4). This is intentional for training: more active experts means each token provides gradient signal to more parameters, accelerating diversification.

After training, the active count can be reduced via router bias adjustment to recover gpt-oss-20b-class throughput while retaining the broader expert pool. Train wide, deploy narrow.

2. Mathematical Foundation: Silent Init

Principle

The expert pool is expanded by factor M (x2 for 64a8, x3 for 96a12). Each new expert is initialized from an existing one. The router is expanded with duplicated structure. The active expert count scales by the same factor M.

Under softmax normalization, the multiplicity cancels exactly.

Proof

Standard MoE forward pass:

FFN(h) = Σ_{i ∈ top-k(s)} p_i(h) · E_i(h)

where  s_i = W_r[i] · h + b_r[i]           (router logits)
       p_i = exp(s_i) / Σ_{j∈S} exp(s_j)   (softmax over selected set S)

After expansion with multiplier M (expert count E' = M·E, active count k' = M·k), top-k' selects exactly M copies of each original top-k expert. Within the softmax:

Σ_{copies of i} p'_copy = M · exp(s_i) / (M · Σ_{orig top-k} exp(s_j)) = p_i

Factor M in numerator and denominator cancels. The weighted expert outputs sum identically.

No approximation. No numerical error beyond floating-point identity.

3. Combinatorial Routing Analysis

The number of distinct expert subsets per token per layer is C(E, k). This bounds the model's capacity for input-dependent specialization.

Per-Layer Configurations

Over L Layers

Each layer routes independently. Total configuration space over the full network: C(E, k)^L.

These represent the theoretical configuration ceiling the optimizer can explore during fine-tuning. Practical utilization depends on dataset diversity and training strategy.

Interpretation

Activating 8 or 12 experts per token produces a richer per-token representation: each token is processed through more specialized views simultaneously. This is particularly relevant for tasks with high nonlinear reasoning demands.

Even after reduction to top-4 routing, the expanded models retain 17.7x to 92x more per-layer options than the original 20B, at equivalent inference cost.

4. Architecture

All editions share the same architecture. Only expert count, active count, and quantization format differ from gpt-oss-20b.

Unchanged: hiddensize (2880), numhiddenlayers (24), numattentionheads (64), numkeyvalueheads (8), headdim (64), slidingwindow (128), maxpositionembeddings (131072), vocab_size (201088), RoPE (YaRN, factor 32).

Changed:

Quantization format:

• —BnB editions: Expert MLP weights as individual Linear4bit modules (BitsAndBytes NF4). Attention, router, embeddings, norms in BF16. Each expert is a distinct module, directly addressable for LoRA or freezing.
• —MXFP4 editions: Expert MLP weights in MXFP4 (fused GptOssExperts packed tensors). Attention, router, embeddings, norms in BF16.

5. Usage

5.1 Choosing an Edition

The BnB editions decompose each expert into separate Linear4bit modules. Standard LoRA applies to any expert projection. Simpler path for most fine-tuning workflows.

The MXFP4 editions use GptOssExperts, a fused packed module. LoRA applies to attention projections; expert-level adaptation requires either full-parameter training with gradient control, or custom LoRA adapters operating on the packed tensor structure.

5.2 QLoRA via Unsloth

This edition is compatible with Unsloth's GPT-OSS support. Refer to the official Unsloth guide for setup, model loading, and LoRA configuration:

Unsloth: How to Run and Fine-Tune GPT-OSS

Replace the model name with the appropriate BnB edition:

• —khoinguyenbk/llm-surgery-dark-arts-gpt-oss-40b-64a8-init (64a8)
• —khoinguyenbk/llm-surgery-dark-arts-gpt-oss-60b-96a12-init (96a12)

5.3 MXFP4 Editions

For vLLM-native serving or Transformers + TRL fine-tuning with MXFP4 format, see the MXFP4 editions:

• —`gpt-oss-40b-64a8` (64a8)
• —`gpt-oss-60b-96a12` (96a12)

6. Training Considerations

Expert Preservation and Symmetry Breaking

Expanded experts begin in a symmetric state. Fine-tuning naturally breaks symmetry through stochastic gradient updates, but the process can be guided.

Original expert weights encode the full pretrained capability of gpt-oss-20b. Preserving this knowledge base during early training is the primary concern. Selective gradient control allows original experts to serve as a stability anchor while expanded experts diverge and specialize.

The router must remain trainable throughout. It is the mechanism through which diversification manifests.

In the BnB editions, selective freezing is straightforward: each expert lives at model.model.layers[L].mlp.experts[i] as a distinct module with its own parameters. Freezing original experts (indices 0-31) and training expanded experts (32-63) requires only setting requires_grad or applying gradient hooks per module.

Diversification Approaches

Several approaches to accelerating symmetry breaking, applicable individually or in combination:

• —Gradient-based selective freezing: protect originals, train expanded experts
• —Noise injection on expanded expert weights: seed divergence before training begins
• —Router-aware scheduling: controlling which experts receive gradient signal at which stages
• —Expert-specific regularization: auxiliary terms encouraging divergence

The appropriate combination depends on the target domain, dataset composition, and available compute.

Staged Training

The general principle: protect, diversify, refine.

Early stages prioritize stability of original knowledge. Middle stages allow broad divergence at the new experts. Late stages consolidate specialization with global refinement at reduced learning rate.

The attention layers in gpt-oss-20b are well-converged. Aggressive fine-tuning of attention carries degradation risk. Minimal learning rate or full freeze on attention is worth considering, depending on domain distance from the pretraining distribution.

Stage boundaries, learning rate schedules, and freeze transitions are task-dependent.

Active Expert Reduction

Post-diversification, the active count can be reduced (64a8 to 64a4, 96a12 to 96a4) via router bias adjustment. This recovers gpt-oss-20b inference throughput while retaining 17.7x to 92x more routing options.

The 64a8 and 96a12 configurations are designed for high nonlinear reasoning capacity. Reduction is optional. For competition-grade mathematics, multi-step code generation, and adversarial reasoning, the full active count is recommended.

7. Prerequisites

8. Lineage

openai/gpt-oss-20b              32 experts · top-4 · 24L · ~21B
    |
    +--> unsloth/gpt-oss-20b-unsloth-bnb-4bit      (BnB 4-bit quantization)
    |       |
    |       +--> x2 (64a8) BnB                      64 experts · top-8 · 24L · ~40B
    |       +--> x3 (96a12) BnB                     96 experts · top-12 · 24L · ~60B
    |
    +--> x2 (64a8) MXFP4                            64 experts · top-8 · 24L · ~40B
    +--> x3 (96a12) MXFP4                           96 experts · top-12 · 24L · ~60B
    |
openai/gpt-oss-120b             128 experts · top-4 · 36L · ~117B

License

Apache 2.0, inherited from `openai/gpt-oss-20b`

References

• —Unsloth: How to Run and Fine-Tune GPT-OSS
• —OpenAI GPT-OSS Fine-Tuning Cookbook (Transformers)
• —unsloth/gpt-oss-20b-unsloth-bnb-4bit
• —openai/gpt-oss-20b · openai/gpt-oss-120b
• —GPT-OSS Model Card, arXiv:2508.10925
• —Harmony Response Format
• —TRL · PEFT

Citation

@misc{llmsurgery2026gptoss,
  title={LLM Surgery Dark Arts: Silent-Init Expert Expansion for GPT-OSS-20B},
  author={Unnamed AI Lab},
  year={2026},
  note={Expert-expanded MoE init checkpoints: 64a8 and 96a12 configurations},
  url={https://huggingface.co/khoinguyenbk/llm-surgery-dark-arts-gpt-oss-40b-64a8-init}
}