AbstractPhil/captionbert-8192-v2-b

1---2license: mit3language: [en]4library_name: transformers5pipeline_tag: feature-extraction6tags: [sentence-similarity, feature-extraction, consensus-distillation, geometric-deep-learning, amoe]7datasets: [AbstractPhil/conceptual-captions-12m-webdataset-berts]8base_model: [google-bert/bert-base-uncased, answerdotai/ModernBERT-base, FacebookAI/roberta-base, albert/albert-base-v2, distilbert/distilbert-base-uncased]9---10 11# captionbert-8192-b12 13A **58.3M** standalone sentence encoder distilled from the geometric **consensus**14of five BERT-family teachers. No expert models at inference: tokenizer + this15model, 768-d L2-normalized output.16 1712 layers, 512-d, 8 heads, FFN 2048, 8192 position capacity. **0.53x bert-base.**18 19This is the **complete-corpus** build: all 66 CC12M chunks, 31.9M rows. Its20sibling [`captionbert-8192-v2`](https://huggingface.co/AbstractPhil/captionbert-8192-v2)21trained on 54 chunks because ModernBERT was missing from 10 of them; those were22repaired and gate-verified before this run.23 24> 📄 **Technical companion:** [TECHNICAL.md](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B/blob/main/TECHNICAL.md) — the full training recipe, per-task tables, geometry gauges, adapter-library numerics, and the laws this family earned. The long-context depth-extension arm carries its own companion at [deep-arm/TECHNICAL.md](https://huggingface.co/AbstractPhil/captionbert-8192-v2-B/blob/main/deep-arm/TECHNICAL.md). Week-in-review article: [Geometric Memory FT5 — Agreement, Anchors, Addresses](https://huggingface.co/blog/AbstractPhil/geometric-memory-ft5).25 26```python27from transformers import AutoModel, AutoTokenizer28model = AutoModel.from_pretrained("AbstractPhil/captionbert-8192-v2-B", trust_remote_code=True)29tok   = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")30 31emb = model.encode(["a cat on a windowsill", "a feline by the window"])   # (2, 768)32(emb[0] @ emb[1]).item()33 34model.attach_amoe()          # this repo's NATIVE arms -- see the warning below35emb = model.encode(["a cat on a windowsill"])36```37 38## Benchmark39 40| model | params | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |41|---|---|---|---|---|---|---|---|---|---|---|42| bert-base | 109.5M | 0.4729 | 0.5865 | 0.3087 | 0.5988 | 0.4773 | 0.6029 | 0.6373 | 0.5469 | 0.5289 |43| ModernBERT-base | 149.0M | 0.4215 | 0.5479 | 0.3527 | 0.4247 | 0.3795 | 0.5349 | 0.4174 | 0.5630 | 0.4552 |44| roberta-base | 124.6M | 0.5436 | 0.6296 | 0.3211 | 0.5631 | 0.4522 | 0.6134 | 0.6198 | 0.5777 | 0.5401 |45| albert-base-v2 | 11.7M | 0.4784 | 0.5364 | 0.3101 | 0.4831 | 0.3809 | 0.5542 | 0.5491 | 0.4863 | 0.4723 |46| distilbert | 66.4M | 0.5717 | 0.6424 | 0.4344 | 0.6490 | 0.5410 | 0.6663 | 0.6854 | 0.5162 | 0.5883 |47| **captionbert-8192-b** | 58.3M | **0.5752** | **0.6548** | **0.5012** | **0.6037** | **0.5470** | **0.7146** | **0.6782** | **0.5500** | **0.6031** |48| **captionbert-8192-b + arms** | 63.2M | **0.7675** | **0.7374** | **0.6706** | **0.7381** | **0.6945** | **0.8109** | **0.7695** | **0.6472** | **0.7295** |49| captionbert-8192-v2 | 58.3M | 0.5747 | 0.6526 | 0.5051 | 0.5995 | 0.5452 | 0.7136 | 0.6776 | 0.5933 | 0.6077 |50| all-MiniLM-L6-v2 | 22.7M | 0.8203 | 0.7758 | 0.7237 | 0.8058 | 0.7559 | 0.8539 | 0.7899 | 0.8144 | 0.7925 |51 52All ten models measured in **one harness**, same eight tasks, **mean-pooled and53L2-normalized**, no task tuning. Spearman correlation; `mean` is the unweighted54average over the eight.55 56`all-MiniLM-L6-v2` was contrastively trained on 1B+ curated sentence pairs. It is57listed for scale, not as a peer -- nothing here saw a similarity label.58 59**The trunk beats every teacher it was distilled from**, and the best of them60(distilbert, .5883) by +.0194 -- at **13% of their combined 461M parameters**,61having never seen a similarity label. The margin comes mostly from STS12, where62every teacher collapses to .31-.43 and the trunk holds .50.63 64**With arms it clears the best teacher by +.14** and closes to within **.063** of65a model trained on a billion curated pairs.66 67Mean-pooled BERT-family encoders are known-weak sentence encoders -- that is the68reason Sentence-BERT exists -- so beating them is an efficiency result rather69than a state-of-the-art one. The MiniLM row is in the table to keep that honest.70 71### Geometry72 73| model | self_cos | erank |74|---|---|---|75| bert-base | +0.6071 | 32.8 |76| ModernBERT-base | +0.9001 | 26.1 |77| roberta-base | +0.9594 | 19.8 |78| albert-base-v2 | +0.7473 | 20.9 |79| distilbert | +0.6920 | 31.1 |80| **captionbert-8192-b** | +0.1411 | 36.1 |81| **captionbert-8192-b + arms** | +0.0984 | 55.5 |82| captionbert-8192-v2 | +0.1396 | 36.6 |83| all-MiniLM-L6-v2 | +0.0251 | 86.7 |84 85`self_cos` is the isotropy gauge: the mean cosine between unrelated sentences.86Mean-pooled BERT-family embeddings sit in a narrow cone (+.61 to +.96), where87cosine cannot discriminate. `erank` is the participation ratio -- how many of the88768 directions carry variance.89 90Both track capability almost perfectly across all ten models, and **isotropy is91the mechanism**: no isotropy objective appears anywhere in the training stack.92The arms then lift erank 36.6 -> 57.6, the first evidence in this line that93adaptation *adds* usable directions rather than only rotating them.94 95## More data bought nothing (and that is the finding)96 97`-b` trained on **19% more rows for 19% more steps** than `-v2`. Head to head:98 99| | v2 (54ch, 26.9M) | b (66ch, 31.9M) | delta |100|---|---|---|---|101| 8-task mean, bare | .6077 | .6031 | -.0046 |102| 7 tasks excluding BIOSSES | -- | -- | **+.0009** |103| erank (STS-B) | 36.6 | 36.1 | -0.5 |104| self_cos (STS-B) | +.1396 | +.1411 | +.0015 |105| 8-task mean, native arms | .7287 | **.7295** | **+.0008** |106 107The entire -.0046 comes from BIOSSES, which is 100 rows -- a 0.4-sigma move.108Everything else is a dead heat.109 110**The ceiling is TEACHER AGREEMENT, not corpus size.** The consensus target uses111**28.7 of 768 directions**: five BERT-family encoders only agree on ~29, and no112amount of the same distribution raises that. The trunk reaches erank ~103 *in113domain* but ~36 out of it -- the structure it builds on captions does not114transfer. The next lever is heterogeneous teachers, measurable at the consensus115stage before a single training step.116 117## AMOE arms are TRUNK-BOUND -- use this repo's118 119Three 1.6M-parameter anchors on the frozen trunk, under a trained dispatch.120Anchors toggle **bit-exact**, so one artifact serves both the unsupervised121baseline and the adapted model.122 123| mask | STS-B | SICK-R | mean (8 tasks) |124|---|---|---|---|125| OFF (bare trunk) | .5752 | .6548 | .6031 |126| `equiv` only | .7219 | .7200 | .6842 |127| `simplify` only | .5995 | .6603 | .6254 |128| `paraphrase` only | .6137 | .6612 | .6295 |129| **all three** | **.7675** | **.7374** | **.7295** |130 131An `-only` row is that arm **as damped by the dispatch** -- masking never132renormalizes, so it reads lower than the same anchor trained alone.133 134**Do not attach `captionbert-8192-v2`'s arms to this trunk.** Measured:135 136| configuration | mean |137|---|---|138| v2 arms on v2 | .7287 |139| v2 arms on **-b** | .6863 |140| + re-aligned routing keys | .6987 |141| **-b native anchors** | **.7295** |142 143Transferring the arms costs **31% of their gain**. Re-training only the 1,536144routing keys recovers 29% of that; retraining the anchors recovers all of it.145**71% of the loss is in the anchors themselves.**146 147These two trunks are indistinguishable on eight STS tasks and on geometry, yet1481.6M adapter parameters tell them apart -- adapters read the residual stream and149the task gauges read the pooled output, and the stream carries trunk identity150the output does not. Budget one anchor set per trunk (~18 min).151 152`attach_amoe()` resolves this repo's own arms by default. Files are under153`amoe/b-collective/`. See [amoe-lora](https://github.com/AbstractEyes/amoe-lora).154 155## How it was built156 1571. Five teachers embedded 33M CC12M llava-next captions (mean-pooled, 768-d).1582. One global **whitened Procrustes** map per teacher into `bert-base`'s frame,159   fit on a stratified random sample and **reported out-of-sample** (worst arm160   retains 95% of its in-sample R@1 at 1,833x chance).1613. Consensus = normalized centroid of the aligned teachers, per chunk.1624. Student trained from scratch: InfoNCE(T=0.07) + per-sample MSE against the163   consensus. Pure Adam, no weight decay. 31.9M rows, 62,312 steps at batch164   2048, ~6.4 h on one RTX 6000 Pro.165 166The alignment maps in `maps/` are **the same maps v2 used** -- refitting them167would put the consensus targets in a different frame with no signal in the loss.168 169## Known limits170 171- **Consensus rank ~28.7 of 768.** The model's ceiling, and a property of172  teacher agreement rather than of this model.173- **Alignment quality varies by teacher.** Out-of-sample cosine into the bert174  frame: distil .625, roberta .372, albert .331, modern .327 -- the ordering175  tracks architectural distance from bert-base.176- **Single seed.** The AMOE results carry a measured seed spread of .003-.005;177  the trunk does not have one.178- Trained on image captions; expect caption-like text to be its strongest domain.179- BIOSSES is 100 rows. Treat any single-task delta there as noise.180 181## Files182 183```184model.safetensors            the trunk, HF format185config.json                  AutoModel config (auto_map -> modeling_captionbert)186modeling_captionbert.py      CaptionBertV2Model + attach_amoe/detach_amoe187checkpoints/                 training checkpoints (final_model.pt is the ship)188maps/                        alignment maps -- SHARED with v2, do not refit189amoe/b-collective/           native anchors + dispatch + metrics190```191 192## Output convention193 194| field | shape | |195|---|---|---|196| `last_hidden_state` | (B, L, 512) | token states |197| `pooler_output` | (B, 768) | **the embedding**, L2-normalized |198| `embedding` | (B, 768) | alias |199 200`geolip-captionbert-8192` (v1) returned the pooled embedding as201`last_hidden_state`. If porting v1 code, use `pooler_output`.202 203 204## deep-arm/ — long-context binding attachment (optional, detachable)205 206The base trunk's attribute binding is semantically alive to ~256 tokens207(its trained position range) and collapses beyond it — measured with a208minimal-pair battery ("a red cube on a blue sphere" vs swaps, ratio of209own-attribute to other-attribute state movement at the noun positions;2101.0 = chance). `deep-arm/` restores deep binding **without touching the211trunk**: 4.98M trainable parameters distilled from212`allenai/longformer-base-4096` token states (span-resampled across213tokenizers, mapped 768→512 by a whitened-Procrustes fit, out-of-sample214cos .501 / retrieval R@1 .849 vs a dead shuffled null).215 216**Construction**: (1) position rows 256+ re-initialized by mod-256217tiling of the trained 0–255 table, then trained (rows 0–255 frozen);218(2) one gated 16-slot relay adapter per encoder block (gates open219monotonically with depth, .35–.51 after training); (3) per-token cosine220distillation to the mapped Longformer states over long caption221documents, deep-weighted.222 223**Binding at depth** (battery ratios, before → after; both alignment224phases of the tiling shown):225 226| payload depth | before | after |227|---|---|---|228| 10 | 1.59 / 2.01 | 2.69 / 2.27 |229| 480 (tile edge) | 1.17 / 1.13 | 1.63 / 3.50 |230| 1024 (aligned) | 3.12 / 2.92 | 2.66 / 3.96 |231| 1248 (tile edge) | 1.22 / 1.12 | 3.18 / 3.80 |232| 2048 (aligned) | 3.31 / 2.73 | 4.83 / 13.1 |233| 2288 (tile edge) | 1.13 / 0.98 | 1.63 / 1.58 |234 235(The tiled init alone restores the aligned depths; the trained deep236rows repair the tile edges in a near-to-far wave; the relays amplify237retro-binding wherever gradient reaches. The 13.1 cell is flagged238pending an absolute-distance decomposition.)239 240**The honest cost**: with the attachment ENGAGED, short-input capability241drops .6031 → .5655 on the 8-task STS mean and shallow isotropy degrades242(self_cos +.003 → +.288) — the Longformer-mapped frame is anisotropic.243The attachment is therefore a **length-conditional mode**: adapters are244σ-gated wrappers and rows 0–255 are untouched, so with the wrappers245removed (or gated off) short-input behavior is bit-identical to the246stock trunk. Engage for inputs past ~256 tokens; run stock below.247 248**Use**: load the trunk as above; from `deep-arm/deep1_arm_s0.pt` copy249`pos_emb.weight`, wrap each `encoder.layers[i]` with its `block{i}.*`250relay (a residual adapter applied to the block output), or skip both to251recover the stock model exactly. `deep-arm/deep1_results.json` carries252the full battery and the fit report.253 254## Citation255 256```bibtex257@misc{abstractphil2026captionbertb,258  title  = {captionbert-8192-b: consensus distillation on the complete CC12M corpus},259  author = {AbstractPhil},260  year   = {2026},261  url    = {https://huggingface.co/AbstractPhil/captionbert-8192-v2-B}262}263```264 265MIT.