Modelpublic

OpenMOSS-Team/Qwen3-30B-A3B-base-mla-topk8-rank512

sourceHugging Faceapache-2.0updated 10d agoView on Hugging Face

0likes39downloads

Model Card

Qwen3-30B-A3B-base-mla-topk8-rank512

This repository contains the attention-only incremental weights for an MLAfication conversion of Qwen/Qwen3-30B-A3B-Base. It is not a standalone full-model checkpoint.

Variant

Field	Value
Base model	`Qwen/Qwen3-30B-A3B-Base`
Base revision	`1b75feb79f60b8dc6c5bc769a898c206a1c6a4f9`
MLA latent rank	`512`
RoPE dimensions per KV head	`8`
Training checkpoint	step `1200`
Delta tensors	`288`
Delta size	`0.90 GiB`
SHA-256	`eb39ad6eea4166053567648eba61c0a0c1b5987936e9b764233ec01eda500978`

The delta contains only q_proj, k_r_proj, low-rank kv_proj, and QK-Norm parameters. Embeddings, MLP/MoE, output projection, LM head, and other base-model parameters are intentionally omitted.

Loading

Use the MLAfication implementation to instantiate the MLAfication architecture from the base model, then load model.safetensors with strict=False:

python

from safetensors.torch import load_file

delta = load_file("model.safetensors")
load_result = mla_model.load_state_dict(delta, strict=False)

The architecture must be patched before loading the delta. Loading this repository directly with vanilla AutoModelForCausalLM.from_pretrained(...) is not supported. config.json includes the layer-wise RoPE indices and sanitized MLAfication metadata needed by the project integration.

Files

—model.safetensors: attention-only delta weights
—config.json: base architecture, RoPE indices, and MLAfication metadata
—manifest.json: provenance, byte size, tensor count, and checksum

License

The weights follow the Apache 2.0 license of the base Qwen3 model.