sauerkrieger/dewiki-sqlite-zstd-fts5

German Wikipedia Compressed SQLite Database (FTS5 + Shared Zstd Frames)

A heavily compressed, fully queryable SQLite database containing the complete German Wikipedia (~2.66M articles). Designed specifically for offline-first RAG applications, local LLMs, and resource-constrained edge devices (Android, iOS, Raspberry Pi).

💡 Key Specifications

• —Articles: 2,660,620 articles in 13,504 shared zstd chunks.
• —Size: ~4.83 GB single .db file.
• —Full-Text Search: SQLite FTS5 index with trigram tokenizer (content='articles' external content table, zero duplicate storage).
• —Compression: Shared Zstandard frames (100–200 articles per frame, level 19, LDM window_log=23) for ultra-low memory usage during decompression.
• —Cleaned Data: Filtered out PR/advertising templates, stub pages, and reality-TV/influencer noise while strictly preserving historical knowledge ({{Veraltet}}).

🛠️ Build Pipeline & Code

This database was created using the open-source pipeline: 👉 GitHub Repository: Sauerkrieger/dewiki-sqlite-zstd-builder

🚀 Quick Usage (Python / SQLite)

Requires SQLite 3.34+ (for FTS5 trigram support).

import sqlite3
import zstandard as zstd

conn = sqlite3.connect("wikipedia_compressed.db")

# 1. Search via FTS5 Trigram
cur = conn.cursor()
cur.execute("SELECT rowid, title FROM fts_titles WHERE fts_titles MATCH 'Quantenphysik' LIMIT 5;")
results = cur.fetchall()

# 2. Fetch & Decompress Article
rowid = results[0][0]
article = conn.execute("SELECT title, chunk_id, offset, length FROM articles WHERE id = ?", (rowid,)).fetchone()
title, chunk_id, offset, length = article

blob = conn.execute("SELECT data FROM chunks WHERE id = ?", (chunk_id,)).fetchone()[0]
decompressed_chunk = zstd.ZstdDecompressor().decompress(blob)
article_text = decompressed_chunk[offset:offset + length].decode("utf-8")

print(f"--- {title} ---\n{article_text[:300]}...")