Sherlock-Comms/wikipedia-en-2026-07-01-faiss

English Wikipedia FAISS Indexes (2026-07-01)

Prebuilt FAISS indexes over the 17,473,199 Qwen3-Embedding-4B vectors, so you can query English Wikipedia locally without embedding or building anything.

Companion repos: embeddings · passages.

Files

• —ivfpq.faiss — 1.33 GB, compressed. OPQ64,IVF16384,PQ64x8, inner product. ~64 bytes/vector. Best when RAM is tight; re-rank its hits against the raw vectors or a text reranker for full accuracy.
• —hnsw_sq.faiss — 38 GB, exact-ish. IndexHNSWSQ (fp16), M=32, efConstruction=200, inner product. ~0.996 recall@10, sub-millisecond search; loads fully into RAM.
• —ids.txt — line i is the passage id for FAISS id i (194 MB).
• —offsets.sqlite — maps each passage id to its byte offset in the passages JSONL, for instant text fetch (391 MB).

Vectors are L2-normalized, so inner product equals cosine similarity. Embed queries with Qwen3-Embedding-4B using prompt_name="query".

Use

import faiss, numpy as np
index = faiss.read_index("ivfpq.faiss")          # or hnsw_sq.faiss
faiss.extract_index_ivf(index).nprobe = 256      # IVF-PQ only
ids = open("ids.txt", encoding="utf-8").read().splitlines()
scores, hits = index.search(query_vec, 8)        # query_vec: (1,1024) fp32, normalized
passage_ids = [ids[h] for h in hits[0]]

Fetch passage text via offsets.sqlite against the passages JSONL.

Recall (measured)

On the SimpleQA factoid set the exact index takes a 27B model from 15% to 56% correct and cuts wrong answers from 68% to 3%. Retriever ceiling ~68% answer-in- top-50 (body-prose-only corpus; infobox/table facts not indexed).

Licence

Built from CC BY-SA 4.0 English Wikipedia text; released under CC BY-SA 4.0. Attribute Wikipedia and this dataset on reuse.