Modelpublic

AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-softmax-lr07

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes25downloads

Model Card

mobeqwen330ba3bb16

numB compression-stress sweep (B=16). MoBE-softmax arm at its LR optimum (lr=0.07). Shrinks the shared dictionary to 16 bases (tighter simplex / convex hull) to test whether the simplex constraint's cost grows with compression vs Sparse and no-softmax MoBE. See docs/findingslog.md.

—Method: mobe
—Base model: Qwen/Qwen3-30B-A3B
—Compressed: gate_proj, up_proj over layers 0-48 (down_proj kept dense)
—Settings: mobe, num_B=16, activation=tanh

See config.toml and training.json for the full run configuration and per-layer reconstruction stats.

Evaluation

No eval results attached.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-softmax-lr07",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/Qwen3-30B-A3B-MoBE16-softmax-lr07", trust_remote_code=True)

The factored experts are reconstructed at runtime via the bundled Qwen3MoBEForCausalLM.