Modelpublic

OpenMOSS-Team/Qwen3-8B-base-mla-topk8-rank256

sourceHugging Faceapache-2.0updated 10d agoView on Hugging Face

0likes41downloads

Model Card

Qwen3-8B-base-mla-topk8-rank256

This repository contains the attention-only incremental weights for an MLAfication conversion of Qwen/Qwen3-8B-Base. It is not a standalone full-model checkpoint.

Variant

Field	Value
Base model	`Qwen/Qwen3-8B-Base`
Base revision	`49e3418fbbbca6ecbdf9608b4d22e5a407081db4`
MLA latent rank	`256`
RoPE dimensions per KV head	`8`
Training checkpoint	step `4800`
Delta tensors	`216`
Delta size	`1.26 GiB`
SHA-256	`5a0024eb0c4dbee9af7bed7b0082a0843956a722fbced658f42375d1ae479c91`

The delta contains only q_proj, k_r_proj, low-rank kv_proj, and QK-Norm parameters. Embeddings, MLP/MoE, output projection, LM head, and other base-model parameters are intentionally omitted.

Loading

Use the MLAfication implementation to instantiate the MLAfication architecture from the base model, then load model.safetensors with strict=False:

python

from safetensors.torch import load_file

delta = load_file("model.safetensors")
load_result = mla_model.load_state_dict(delta, strict=False)

The architecture must be patched before loading the delta. Loading this repository directly with vanilla AutoModelForCausalLM.from_pretrained(...) is not supported. config.json includes the layer-wise RoPE indices and sanitized MLAfication metadata needed by the project integration.

Files

—model.safetensors: attention-only delta weights
—config.json: base architecture, RoPE indices, and MLAfication metadata
—manifest.json: provenance, byte size, tensor count, and checksum

License

The weights follow the Apache 2.0 license of the base Qwen3 model.