eatang/Kimi-K2.5-2layer

Kimi-K2.5-2layer

A 2-layer INT4 slice of `moonshotai/Kimi-K2.5`, built as a GPU CI fixture for SkyRL. This is not a usable language model — two layers of a 61-layer network produce incoherent text. It exists so tests can exercise the real architecture, with real (truncated) weights, at a size that fits a single node.

Contents

• —language_model.model.layers.0 — dense MLP layer (first_k_dense_replace=1)
• —language_model.model.layers.1 — MoE layer with all 384 routed experts + 1 shared expert
• —language_model.model.embed_tokens, language_model.lm_head, language_model.model.norm
• —vision_tower.* and mm_projector.*, unchanged

Weights are byte-identical to the source release: the routed-expert weight_packed / weight_scale / weight_shape triples and its quantization_config (compressed-tensors pack-quantized INT4, group_size=32) are kept as they ship. Only text_config.num_hidden_layers is changed, to 2.

Companion

`eatang/Kimi-K2.5-2layer-BF16` is the same slice with its routed experts dequantized. SkyRL serves this INT4 checkpoint from the inference engine and loads the BF16 one as fake-INT4 QAT masters in the trainer (trainer.policy.model.fake_int4_qat.bf16_base_path), since Megatron-Bridge cannot load compressed-tensors.

Total: 16.4 GB.