shalyhinpavel/sovereign-jepa-lens

Sovereign Semantic Router v6 (JEPA lens)

A 41.4M-parameter latent trajectory predictor for multi-hop retrieval. Instead of generating text, it predicts the embedding of the next reasoning step directly in Harrier-270M space (dim 640), so retrieval becomes: predict → k-NN lookup → predict again.

It consumes vectors and returns a vector, so it drops into any existing retrieval stack. Inference runs on CPU in well under a millisecond.

Results

MuSiQue multi-hop retrieval — apples-to-apples against classical hybrid RAG on the same embedder, 2417 dev questions:

Complete Recall@10 means every supporting paragraph for a question landed in the top 10 — the metric that actually matters for multi-hop, and the one where single-shot retrieval fails.

No contamination. Training used MuSiQue split=train, evaluation uses dev. Verbatim text comparison: 0 overlapping questions, 0 of 2629 gold (supporting) dev paragraphs were ever encoded during training. Only distractor paragraphs (1.9%) and Wikipedia titles (45%) overlap.

Single-step latent prediction — on a corpus the model never saw during training, retrieving over 13,053 candidates:

Read this honestly: single-step prediction is a nearly linear problem, and a ridge regression fitted directly on the target corpus matches this model on hit@64. The router's edge is transfer (it reaches that level with zero exposure to the corpus) and ranking precision (hit@16, recall@64). Whether the non-linearity pays off in iterative multi-hop — where each step's input depends on the previous step's retrieval and errors compound — is the open question for v7.

Quick start

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("shalyhinpavel/sovereign-jepa-lens", trust_remote_code=True).eval()

# chain: (batch, steps, 640) — L2-normalized Harrier-270M embeddings
# step 0 = the query (encoded WITH the instruction prefix), steps 1+ = retrieved passages
chain = torch.nn.functional.normalize(torch.randn(1, 3, 640), dim=-1)

next_step = model.predict_next(chain)            # (1, 640), unit-norm — feed this to your k-NN index
all_steps = model(inputs_embeds=chain).last_hidden_state   # (1, 3, 640), prediction at every position

predict_next returns the predicted embedding of the following reasoning step: search your vector store with it, take what comes back, append it to the chain, repeat. That loop is the multi-hop retrieval this model exists for.

Verified on load: the wrapper reproduces the reference implementation bit-for-bit (max absolute deviation 0.0) and the reported cosine of 0.5320 on held-out data.

Input format

Position 0 is special. It holds a query encoded with the instruction prefix (Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: ...); positions 1+ hold raw texts encoded without it. This asymmetry mirrors production: stored documents are raw, incoming queries are instructed. Feeding uniformly-encoded vectors will degrade results.

Embeddings come from `shalyhinpavel/harrier-v1-270m-gguf` — this model consumes vectors, never text.

model.safetensors and semantic_router_v6.safetensors are identical weights under two names: the first is what from_pretrained looks for, the second is the filename requested by the Rust retrieval engine this router was built for.

Architecture

Transformer: 6 blocks, 10 heads, dim 640, RMSNorm, SwiGLU MLP (×4 expansion), learned positional embeddings (16 slots), 2-layer prediction head with a residual connection, L2-normalized output. Trained with InfoNCE (in-batch negatives) + Muon optimizer + an auxiliary two-hop MTP head that was dropped at checkpoint time.

The state dict contains an unused mask_token key — a training artifact, ignored at inference.

Limitations and provenance

Stated plainly, because these affect what you can reproduce:

• —The weights are canonical; the training run is not reproducible. The training script and the exact training corpus were lost (they lived on an ephemeral cloud instance). These weights are the only surviving copy. A reproducible v7 — corpus built by a versioned script, trainer in git, split by source chain — is in progress.
• —Do not cite earlier reported figures (Avg Sim 0.8393, Recall@10 95.7%, FRAMES 97%). Their measurement corpus did not survive and they do not reproduce on any available corpus; the numbers in the tables above are what has actually been re-measured, in August 2026, on this machine.
• —Embedder-specific. This model operates strictly in Harrier-270M GGUF space. Vectors produced by sentence-transformers are not compatible — measured degradation is severe (hit@64 drops from 0.527 to 0.114).
• —Quantization matters for reproducing the benchmarks. MuSiQue numbers were produced with the Q8_0 embedder build; the engine CLI defaults to Q4_K_M. Use Q8_0 to match.

Citation

@software{sovereign_jepa_lens_2026,
  author = {Shalyhin, Pavel},
  title  = {Sovereign Semantic Router v6: latent trajectory prediction for multi-hop retrieval},
  year   = {2026},
  url    = {https://huggingface.co/shalyhinpavel/sovereign-jepa-lens}
}