lovedheart/Qwen3-Coder-Next-REAP-40B-A3B

Qwen3-Coder-Next-REAP-40B-A3B has the following specifications:

• —Type: Causal Language Models
• —Number of Parameters: 40B in total and 3B activated
• —Hidden Dimension: 2048
• —Number of Layers: 48
• —Hybrid Layout: 12 (3 (Gated DeltaNet -> MoE) -> 1 * (Gated Attention -> MoE))
• —Gated Attention:
• —Number of Attention Heads: 16 for Q and 2 for KV
• —Head Dimension: 256
• —Rotary Position Embedding Dimension: 64
• —Gated DeltaNet: Number of Linear Attention Heads: 32 for V and 16 for QK Head Dimension: 128
• —Mixture of Experts:
• —**Number of Experts: 256 (uniformly pruned from 512)
• —**Number of Activated Experts: 10
• —**Number of Shared Experts: 1
• —Context Length: 262,144 natively
• —Compression Method: REAP (Router-weighted Expert Activation Pruning)
• —Compression Ratio: 50% expert pruning