mlnomad/goat-vvv-d12-fineweb-5x-pytorch
022
GOAT_VVV d=12 · FineWeb-Edu 5× Chinchilla (PyTorch)
GOATVVV attention: **no Q/K projections and no V projection**. Queries, keys, AND values are all the same RoPE'd `xheads (hence "VVV"). Only attention params per layer: c_proj` + per-head bias / ε scalars.
x_heads = RoPE(x.reshape(B, T, H, D))
dots = x_heads @ x_heads^T
dist² = ||x_i||² + ||x_j||² − 2·dots
scores = (dots + softplus(b))² / (dist² + softplus(ε))
scores = L1_normalize(scores) # strict causal j<i
y = scores @ x_heads # ← no V projection
y = c_proj(y)Architecture
Complexity vs GPT-2
Per-layer: GPT-2 = 12 n², GOAT_VVV = 9 n² (25% fewer params — dropped all of Q/K/V projections). Activation memory during forward is ~⅓ of GPT-2's for the attention block (no separate Q/K/V tensors). No KV cache needed — values are derived from the residual.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"mlnomad/goat-vvv-d12-fineweb-5x-pytorch",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
ids = tokenizer("The capital of France is", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=30, do_sample=True, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))License
Apache 2.0.