AbstractPhil/captionbert-8192-v2-b

captionbert-8192-b

A 58.3M standalone sentence encoder distilled from the geometric consensus of five BERT-family teachers. No expert models at inference: tokenizer + this model, 768-d L2-normalized output.

12 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. 0.53x bert-base.

This is the complete-corpus build: all 66 CC12M chunks, 31.9M rows. Its sibling `captionbert-8192-v2` trained on 54 chunks because ModernBERT was missing from 10 of them; those were repaired and gate-verified before this run.

📄 Technical companion: TECHNICAL.md — the full training recipe, per-task tables, geometry gauges, adapter-library numerics, and the laws this family earned. The long-context depth-extension arm carries its own companion at deep-arm/TECHNICAL.md. Week-in-review article: Geometric Memory FT5 — Agreement, Anchors, Addresses.

from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2-B", trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)
(emb[0] @ emb[1]).item()

model.attach_amoe()          # this repo's NATIVE arms -- see the warning below
emb = model.encode(["a cat on a windowsill"])

Benchmark

All ten models measured in one harness, same eight tasks, mean-pooled and L2-normalized, no task tuning. Spearman correlation; mean is the unweighted average over the eight.

all-MiniLM-L6-v2 was contrastively trained on 1B+ curated sentence pairs. It is listed for scale, not as a peer -- nothing here saw a similarity label.

The trunk beats every teacher it was distilled from, and the best of them (distilbert, .5883) by +.0194 -- at 13% of their combined 461M parameters, having never seen a similarity label. The margin comes mostly from STS12, where every teacher collapses to .31-.43 and the trunk holds .50.

With arms it clears the best teacher by +.14 and closes to within .063 of a model trained on a billion curated pairs.

Mean-pooled BERT-family encoders are known-weak sentence encoders -- that is the reason Sentence-BERT exists -- so beating them is an efficiency result rather than a state-of-the-art one. The MiniLM row is in the table to keep that honest.

Geometry

self_cos is the isotropy gauge: the mean cosine between unrelated sentences. Mean-pooled BERT-family embeddings sit in a narrow cone (+.61 to +.96), where cosine cannot discriminate. erank is the participation ratio -- how many of the 768 directions carry variance.

Both track capability almost perfectly across all ten models, and isotropy is the mechanism: no isotropy objective appears anywhere in the training stack. The arms then lift erank 36.6 -> 57.6, the first evidence in this line that adaptation adds usable directions rather than only rotating them.

More data bought nothing (and that is the finding)

-b trained on 19% more rows for 19% more steps than -v2. Head to head:

The entire -.0046 comes from BIOSSES, which is 100 rows -- a 0.4-sigma move. Everything else is a dead heat.

The ceiling is TEACHER AGREEMENT, not corpus size. The consensus target uses 28.7 of 768 directions: five BERT-family encoders only agree on ~29, and no amount of the same distribution raises that. The trunk reaches erank ~103 in domain but ~36 out of it -- the structure it builds on captions does not transfer. The next lever is heterogeneous teachers, measurable at the consensus stage before a single training step.

AMOE arms are TRUNK-BOUND -- use this repo's

Three 1.6M-parameter anchors on the frozen trunk, under a trained dispatch. Anchors toggle bit-exact, so one artifact serves both the unsupervised baseline and the adapted model.

An -only row is that arm as damped by the dispatch -- masking never renormalizes, so it reads lower than the same anchor trained alone.

Do not attach `captionbert-8192-v2`'s arms to this trunk. Measured:

Transferring the arms costs 31% of their gain. Re-training only the 1,536 routing keys recovers 29% of that; retraining the anchors recovers all of it. 71% of the loss is in the anchors themselves.

These two trunks are indistinguishable on eight STS tasks and on geometry, yet 1.6M adapter parameters tell them apart -- adapters read the residual stream and the task gauges read the pooled output, and the stream carries trunk identity the output does not. Budget one anchor set per trunk (~18 min).

attach_amoe() resolves this repo's own arms by default. Files are under amoe/b-collective/. See amoe-lora.

How it was built

1. 1.Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).
2. 2.One global whitened Procrustes map per teacher into bert-base's frame, fit on a stratified random sample and reported out-of-sample (worst arm retains 95% of its in-sample R@1 at 1,833x chance).
3. 3.Consensus = normalized centroid of the aligned teachers, per chunk.
4. 4.Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the consensus. Pure Adam, no weight decay. 31.9M rows, 62,312 steps at batch 2048, ~6.4 h on one RTX 6000 Pro.

The alignment maps in maps/ are the same maps v2 used -- refitting them would put the consensus targets in a different frame with no signal in the loss.

Known limits

• —Consensus rank ~28.7 of 768. The model's ceiling, and a property of teacher agreement rather than of this model.
• —Alignment quality varies by teacher. Out-of-sample cosine into the bert frame: distil .625, roberta .372, albert .331, modern .327 -- the ordering tracks architectural distance from bert-base.
• —Single seed. The AMOE results carry a measured seed spread of .003-.005; the trunk does not have one.
• —Trained on image captions; expect caption-like text to be its strongest domain.
• —BIOSSES is 100 rows. Treat any single-task delta there as noise.

Files

model.safetensors            the trunk, HF format
config.json                  AutoModel config (auto_map -> modeling_captionbert)
modeling_captionbert.py      CaptionBertV2Model + attach_amoe/detach_amoe
checkpoints/                 training checkpoints (final_model.pt is the ship)
maps/                        alignment maps -- SHARED with v2, do not refit
amoe/b-collective/           native anchors + dispatch + metrics

Output convention

geolip-captionbert-8192 (v1) returned the pooled embedding as last_hidden_state. If porting v1 code, use pooler_output.

deep-arm/ — long-context binding attachment (optional, detachable)

The base trunk's attribute binding is semantically alive to ~256 tokens (its trained position range) and collapses beyond it — measured with a minimal-pair battery ("a red cube on a blue sphere" vs swaps, ratio of own-attribute to other-attribute state movement at the noun positions; 1.0 = chance). deep-arm/ restores deep binding without touching the trunk: 4.98M trainable parameters distilled from allenai/longformer-base-4096 token states (span-resampled across tokenizers, mapped 768→512 by a whitened-Procrustes fit, out-of-sample cos .501 / retrieval R@1 .849 vs a dead shuffled null).

Construction: (1) position rows 256+ re-initialized by mod-256 tiling of the trained 0–255 table, then trained (rows 0–255 frozen); (2) one gated 16-slot relay adapter per encoder block (gates open monotonically with depth, .35–.51 after training); (3) per-token cosine distillation to the mapped Longformer states over long caption documents, deep-weighted.

Binding at depth (battery ratios, before → after; both alignment phases of the tiling shown):

(The tiled init alone restores the aligned depths; the trained deep rows repair the tile edges in a near-to-far wave; the relays amplify retro-binding wherever gradient reaches. The 13.1 cell is flagged pending an absolute-distance decomposition.)

The honest cost: with the attachment ENGAGED, short-input capability drops .6031 → .5655 on the 8-task STS mean and shallow isotropy degrades (self_cos +.003 → +.288) — the Longformer-mapped frame is anisotropic. The attachment is therefore a length-conditional mode: adapters are σ-gated wrappers and rows 0–255 are untouched, so with the wrappers removed (or gated off) short-input behavior is bit-identical to the stock trunk. Engage for inputs past ~256 tokens; run stock below.

Use: load the trunk as above; from deep-arm/deep1_arm_s0.pt copy pos_emb.weight, wrap each encoder.layers[i] with its block{i}.* relay (a residual adapter applied to the block output), or skip both to recover the stock model exactly. deep-arm/deep1_results.json carries the full battery and the fit report.

Citation

@misc{abstractphil2026captionbertb,
  title  = {captionbert-8192-b: consensus distillation on the complete CC12M corpus},
  author = {AbstractPhil},
  year   = {2026},
  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2-B}
}

MIT.