EXDai/attention-mechanism
The Attention Computation — Inside the Score Companion dataset for Episode 13 of EXD: the attention mechanism. One score matrix, one softmax, one weighted blend. This notebook loads Qwen3.6-35B-A3B, runs it up to a real full-attention layer, and reconstructs the entire attention computation by hand — QK Norm, RoPE, QKᵀ, causal mask, softmax, weighted sum, gate, and output projection — verified bit-for-bit against the model's own output. 📖 Article:… See the full description on the dataset page: https://huggingface.co/datasets/EXDai/attention-mechanism.
The Attention Computation — Inside the Score
Companion dataset for Episode 13 of EXD: the attention mechanism.
One score matrix, one softmax, one weighted blend. This notebook loads Qwen3.6-35B-A3B, runs it up to a real full-attention layer, and reconstructs the entire attention computation by hand — QK Norm, RoPE, QKᵀ, causal mask, softmax, weighted sum, gate, and output projection — verified bit-for-bit against the model's own output.
📖 Article: huggingface.co/blog/EXDai/attention-mechanism
Run the notebook: download attention_mechanism.ipynb and run it with a GPU.