CoolFace
Modelpublic

Quazim0t0/Escarda-TriAtn-86M

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
1likes32downloads
Model Card

<img src="Escarda-LLMs.png" width="50%">

Escarda-TriAtn-86M

A TriAttention-enabled variant of [Escarda-86M-Identity](https://huggingface.co/Quazim0t0/Escarda-86M-Identity). Same weights, same identity-tuned chat behavior — plus a training-free KV-cache compression path (TriAttention) that bounds attention memory at long context. This repo does not replace Escarda-86M-Identity; it is a separate, drop-in model.

For contexts up to the KV budget (default 2048 tokens), this model's output is byte-for-byte identical to Escarda-86M-Identity (verified). TriAttention only changes behavior once the cache would exceed the budget, where it caps memory instead of growing.

What was changed vs Escarda-86M-Identity

Nothing was retrained — the transformer weights are identical. The additions are inference-side:

  • —`triattention.py` (new) — a per-layer KV-cache compressor implementing TriAttention: it scores cached keys in the pre-RoPE space using this model's Q/K concentration centers (a trigonometric series over Q–K distance) plus a Q/K-norm signal, and keeps the top-scoring keys (with attention-sink + recent-window retention). Streaming, position-aware eviction.
  • —Baked calibration — the per-layer pre-RoPE Q/K centers, concentration (mean-resultant-length R ≈ 0.77), and key-norm statistics are stored as buffers in model.safetensors. Calibrated offline on 256 FineWeb-Edu documents (the family's pretraining distribution). No weight updates.
  • —Config flags (config.py) — use_triattention (default True), tri_kv_budget (default 2048), tri_local_window, tri_sink_tokens, tri_future_offsets.
  • —Dynamic RoPE (model_v2.py) — the rotary tables now extend on demand, so generation can exceed the original 4096-position cap.

Helper scripts included: calibrate_triattention.py, generate_triattention.py, compare_triattention.py, chat_eval_triattention.py.

Why / when it helps

KV cache grows linearly with sequence length. TriAttention caps it at tri_kv_budget, cutting memory at long context (and stabilizing throughput) with little quality loss. On an 86M model this is a memory tool, not a speed tool — at short lengths the scoring overhead outweighs the savings; the benefit appears when the cache would otherwise grow large.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Escarda-TriAtn-86M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Escarda-TriAtn-86M", trust_remote_code=True)
# TriAttention is ON by default (budget 2048). To tune or disable:
#   ...from_pretrained(..., tri_kv_budget=1024)      # more compression
#   ...from_pretrained(..., use_triattention=False)  # exact original behavior

Prompts use ChatML (<|im_start|>role\n…<|im_end|>), generation starting after a trailing <|im_start|>assistant\n. Note: when using TriAttention with compression active you must pass explicit absolute position_ids during generation (see generate_triattention.py), because a compressed cache can no longer infer position from its length.

Measured behavior (this model)

  • —Short chat (≤ budget): identical to Escarda-86M-Identity.
  • —Long context (ChatML, ~3.3k tokens): coherent, on-topic answers down to budget ≈ 1024 (~3× KV reduction); quality degrades below that. Default 2048 leaves a safe margin.

Limitations

  • —Adaptation of TriAttention to MLA / partial-RoPE / MQA; pooled per-head scores onto the shared KV head. Validate on your own workload before relying on aggressive budgets.
  • —86M chat model: best on short, ChatML-formatted turns. Very long single-turn generations still degrade (a base-capacity limit, independent of compression).
  • —Custom SpikeTokenizer is not bf16-safe in the engram path; run in fp32/fp16.

Architecture, tokenizer, evaluation

Unchanged from Escarda-86M-Identity (SpikeWhaleLM, ~86M params, 16 layers, hidden 640, MLA + RoPE-16/NoPE-48 MQA, engram memory, hyper-connections, HRM refinement + JEPA, ChatML SpikeTokenizer). See the base repo for the full benchmark table.

Citation

bibtex
@misc{escardatriatn86m,
  title        = {Escarda-TriAtn-86M: TriAttention KV-compression variant of Escarda-86M-Identity},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Escarda-TriAtn-86M}},
  note         = {TriAttention: arXiv:2604.04921}
}

Update: format-blended SFT on the engram-repaired base

This revision applies the (behavior-preserving) engram repair, then a short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train (with '#### N' reasoning), and MMLU-style ('Answer: <letter>') examples -- so chat fluency improves while the benchmark output-formats are preserved rather than overwritten. Held-out (test-split) before->after:

MMLU acc 0.188->0.284, format 0.700->0.956; GSM8K '####' 0.530->0.800

Note: these are fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.