moebiusT7/mmv-wiki-index-ja

MOBIUS MMV — Wikipedia Index (Japanese, multilingual-e5-large) · v11b

### What this is, in three lines A ready-to-query semantic search index over Japanese Wikipedia. Point it at a question and it returns the Wikipedia passages that answer it, by meaning rather than by keyword — offline, with no API key, and without running the embedding pass yourself. That pass took hours on two GPUs. The usual use is the retrieval half of a RAG system, or a grounded-answer source for a local model. What is in the box: a FAISS index of 3,009,820 passage vectors (1.58 GB), the passages themselves as line-aligned JSONL (0.58 GB), and an offset table so that passage i is one seek away. Vectors are intfloat/multilingual-e5-large, 1024-d, so the query does not have to be in Japanese — ask in English and you still get Japanese passages back. What you need: faiss and sentence-transformers, about 2 GB of disk, and enough RAM for the ME5 encoder. The index is memory-mapped, so it does not need to fit in RAM. Working code is in the Use section below. This repository is the Japanese index. The other two languages are English at `moebiusT7/mmv-wiki-index` and Chinese at `moebiusT7/mmv-wiki-index-zh`. All three are built by the same pipeline from the same code and are documented identically. (The English repository has no language suffix for historical reasons — it was the first one published.)

Source: wikipedia_ja_all_mini_2026-06.zim (Kiwix all_mini dump of Wikipedia).

Every number in this card names the log it comes from. The logs, the build scripts and the review record are in eval/ in this repository. Anything not backed by a log is marked as such.

Retrieval on this artifact

Questions are generated by a local Gemma-4 26B-A4B from each article's lead text, one per article, with an instruction not to quote the title; the gold article must be recovered from the whole corpus (final_eval_ja.log).

Head to head against the previously published artifact

Restricted to gold articles present in both stores, nprobe=128, full corpus (final_eval_{ja,zh}.log block [B], and final_eval_en_h2h.log). Questions generated from the new extractor's text favour v11b by construction, so each control row regenerates them from what the old artifact could see:

This is an artifact-vs-artifact comparison, not an ablation: the dump date, the extractor, the chunker and the index type all changed together, and the dump change favours v11b in all three languages: Japanese is a 2026-06 store against a 2026-02 one, Chinese a 2026-05 store against a 2025-09 one eight months older, and English a 2026-06 store against a 2026-03 one. Two components can be priced separately from the same data: the index-type change is worth +7.2 MRR on a 206 k-vector Japanese pool (index_e2e.log), and v11b holds 3.21 chunks per gold article against the old store's 1.26 in Japanese and 3.08 against 1.30 in Chinese (fix_verification.log §F), which helps a metric that scores article-level containment.

English is measured differently. Its previous revision came from a cleaning pass whose script no longer exists and cannot be re-run, so its control questions are generated from the old store's own chunk text — exactly what the old index encoded — over 400 articles sampled by uniform stride from the both-store intersection. 5,328,397 of the old store's 5,351,447 titles are in v11b; the sampler then drops articles whose old-store text is 300 characters or shorter, leaving about 3.88 million to stride over, which removes the stubs the old store handled worst and so works against the result rather than for it. Every question is answerable from what the old artifact could see. v11b still wins by +22.8 MRR while searching a corpus 2.9 times larger, and this set leaks less than the other two controls: the gold title appears verbatim in 3.0 % of its questions against 7.6 % (ja) and 6.8 % (zh) in theirs.

What changed, and why

Three defects in the previous revision, each found by re-reading our own artifacts.

1. Chunks were cut by character count, so CJK text was silently truncated at embed time. ME5-large stops at 512 tokens. One token is 3.95 characters in English but 2.01 in Japanese and 1.98 in Chinese (measured over the published stores), so a 1,536-character slice is a different amount of model input in each language: 13.7 % of published Japanese chunks and 18.5 % of Chinese ones ran past the window, losing 21.7 % and 27.0 % of their tokens (window_footer_uniform.log, uniform stride sample of 8,000 chunks per store). The overflow is dropped at the encoder while the full text is still stored and shown to the user as evidence. v11b packs whole sentences to a 256-token budget with the title line reserved inside it.

2. The Kiwix licence footer was inside almost every chunk. The 184-character English licence string (remeasure_v2.log §A) was inside 97.1 % of the published Japanese chunks (1,505,103 of 1,550,503) and 92.9 % of the Chinese ones, against a mean chunk length of 635 and 707 characters. The English store did not carry it (0 of 5,458,524): it went through a cleaning pass whose script no longer exists. The same extractor emitted the article title two or three times, left HTML entities undecoded, and replaced inline tags with a space, which split words (<b>Ar</b>gentino → "Ar gentino") and injected spaces inside CJK sentences. v11b extracts from id="mw-content-text", removes the footer and navboxes as markup, decodes entities, and deletes inline tags without a separator.

3. The index type threw away half of the search. IndexIVFPQ(m=64) returned 45.4 % (trained on the head of the corpus, the arm that reproduces how the published index was built) to 47.1 % (uniform-trained) of the exact top-10 on 150 k real vectors (pq_loss.log); end to end it cost 7.8 MRR in Japanese and 10.7 in English against exact search (index_e2e.log). These vectors sit in a narrow cone — mean cosine 0.695 (en), 0.715 (ja), 0.753 (zh) over 20,000 disjoint random pairs from the whole shipped vector set (remeasure_v2.log §B). We did not measure the mechanism, and the deficit is not intrinsic to a 64-byte budget: OPQ64,IVF,PQ64 reaches 64.3 % at the same 64 bytes (index_types.log), so a learned rotation alone recovers most of the loss without the five-fold storage increase v11b chose. v11b uses `IVF4096,SQ4`: 4 bits per dimension, all 1024 dimensions kept.

SQ4 and SQ8 differ by 0.1 (ja) and 0.3 (en) MRR on these pools, with no paired significance test run, while SQ4 returns 87.6 % of the exact top-10 against SQ8's 98.2 % (index_types.log). The comparison was not run at shipping scale: the pools are 150 k–206 k vectors at three different partition settings (index_e2e.py uses nlist = 4√N, i.e. 1,590 and 1,814; index_types.py and pq_loss.py hard-code 1,024; the range-fit runs use the shipped 4,096). At shipping scale the same type measures 526 B/vector here, because the coarse quantiser is amortised over more vectors.

Quantiser range

FAISS fits the scalar quantiser's range to the min/max of the residuals from each coarse centroid, which a few outliers stretch. Clipping that range to the 0.1–99.9 percentile of the residual distribution costs nothing — same 4 bits, same decode, same file format — and raises top-10 agreement with exact search from 81.6 % to 85.2 % and MRR from 66.2 to 66.9 on 205,789 Japanese vectors at the shipped nlist=4096 (faiss_range_residual.log). A first attempt wrote raw-vector quantiles into the index and made it worse (59.6 %), because the index encodes residuals, not raw vectors. FAISS's own RS_optim scored 84.6 / 84.5 / 81.6 / 81.6 across four runs, two of them identical to the default; we have not diagnosed the disagreement and do not claim a number for it. Downstream the clip is worth +1.0 recall@1 and +0.7 MRR on 303 questions, which is well inside this benchmark's noise: the gain we can demonstrate is in set fidelity, not in answer quality.

Use

import faiss, numpy as np
from sentence_transformers import SentenceTransformer

ix = faiss.read_index("wiki_index_ivfpq_me5.faiss", faiss.IO_FLAG_MMAP)  # mmap keeps RAM flat
ix.nprobe = 128                                   # 32 is cheaper; see the table above
m  = SentenceTransformer("intfloat/multilingual-e5-large")
q  = m.encode(["query: ..."], normalize_embeddings=True)
D, I = ix.search(np.asarray(q, dtype="float32"), 5)

offs = np.load("line_offsets.npy")                # row i -> byte offset in the decompressed store

Line i of wiki_chunks.jsonl.gz corresponds to FAISS vector i. Fields: title, url, text, chunk_index, license, chunk_id. chunk_id is a 20-hex (80-bit) content hash (5 exact duplicates were dropped at build; the previous revision used 48 bits). Each chunk begins with the article title on its own line, so a retrieved passage is self-describing; infobox rows are linearised as key: value and kept in a separate chunk stream from prose. line_offsets.gzidx is an indexed_gzip seek index: with it, reading the 99th-percentile line takes 0.07 s (ja) / 0.13 s (en) instead of 5.7 s / 28.1 s (misc_measure.log §B).

Limits

• —One dump per language; all_mini ZIMs hold lead sections, not full articles.
• —The question set is synthetic, from one model, n ≈ 300–400 per language, with no human relevance judgements and no check that a question is answerable from its gold article. Per-question ranks were not retained, so the standard error of one MRR figure is reconstructed from the reported rates: 2.2–2.9 points depending on language (noise_floor.log). Treat "2 to 3 points" as a noise scale, not a test.
• —The gold title appears verbatim inside the question in 10.8 % (en) / 17.8 % (ja) / 12.3 % (zh) of the question set, and every v11b chunk begins with the title line, which the old store lacks (fix_verification.log §E). 27 of the 303 Japanese items contain no question mark.
• —All index-type and quantiser numbers come from 150 k–206 k-vector pools, not from this index.
• —The 256-token budget was swept coarsely (160/256/360/480) on pools of 6,303–6,400 articles and not optimised per language; 160 scored higher for English and Chinese at ~1.5× the chunk count (sweep_budget.log).
• —Articles whose cleaned text is under 25 characters are not indexed. Coverage on 2,000 sampled articles, previous extractor to v11b: 94.8 % → 94.2 % (resource_and_coverage.log §D).
• —The prose/infobox split is a 60-character heuristic. Of the lines it sends to the prose stream because they are 60 characters or longer, 22.9 % match a key: value pattern in this language — measured on the first 300,000 chunks in store order, not a random sample, and a pattern match does not by itself establish that those lines are misfiled (misc_measure.log §C). Reference lists and some MediaWiki template errors are indexed as content, and the url field is not percent-encoded, so titles containing / produce broken links.
• —Reviewed adversarially in three rounds before publication. Every reviewer was a separate model instance commissioned by the author, not an independent judge. Round 1 rejected the first rebuild; round 2 found sixteen unsupported statements in the draft release note; round 3 returned FAILS on the corrected document, found three measurements that round 2 had itself got wrong, and found that the acceptance script had not been re-run on two of the three shipped indexes after they were re-encoded (it was, on 2026-09-16, and both passed). All three rounds are in eval/REVIEW.md. No claim here is peer reviewed, certified, or independently replicated.

• —The rebuild is not uniformly better. Against the first rebuild, which our own reviewers rejected, the shipped artifacts score ja 54.3 → 56.3 and zh 54.8 → 55.5 MRR but en 62.7 → 60.9 on the same question sets. All three moves are inside the noise scale above.

Migration

chunk_id changed from a sequential integer to a content hash and the row count changed, so any stored offsets or ids from the previous revision are invalid. The previous revision remains retrievable by its commit SHA. The runtime reads this dataset through scripts/fetch_wiki_index.py in mobius-style/mmv.

License

CC BY-SA 4.0. Derived from Wikipedia; © Wikimedia Foundation and contributors. Index, offsets and embeddings are derivative artifacts of that text and inherit the same license. Build code: AGPL-3.0-or-later.