peterpausianlian/zolai-knowledge-vectors
Zolai Knowledge Vectors Pre-computed sentence embeddings for the Zolai-AI RAG Knowledge Brain -- a bilingual English-Zo (Tedim Chin) language preservation and learning system. 517,917 vectors from four knowledge sources, embedded with sentence-transformers/all-MiniLM-L6-v2 (384-dim). What is Zolai? Zolai (Tedim Zolai, ZVS 2018 orthography) is a Tibeto-Burman language spoken by the Zomi/Chin people of Myanmar and Northeast India. This dataset supports the Zolai-AI… See the full description on the dataset page: https://huggingface.co/datasets/peterpausianlian/zolai-knowledge-vectors.
Zolai Knowledge Vectors
Pre-computed sentence embeddings for the Zolai-AI RAG Knowledge Brain -- a bilingual English-Zo (Tedim Chin) language preservation and learning system.
517,917 vectors from four knowledge sources, embedded with sentence-transformers/all-MiniLM-L6-v2 (384-dim).
What is Zolai?
Zolai (Tedim Zolai, ZVS 2018 orthography) is a Tibeto-Burman language spoken by the Zomi/Chin people of Myanmar and Northeast India. This dataset supports the Zolai-AI project -- a RAG-first bilingual AI toolkit to preserve and teach the language.
Dataset Overview
Schema
Each row is a JSON object:
{
"text": "headword (pos): translation - example sentence",
"metadata": {
"source": "dictionary/dict_canonical_v1.jsonl",
"source_type": "dictionary",
"heading": "topa",
"chunk_type": "dictionary"
},
"embedding": [0.0123, -0.0456],
"embeddingModel": "sentence-transformers/all-MiniLM-L6-v2",
"embeddingDim": 384
}Fields
text content by source type
Data Sources
Embedding Model
- Model: `sentence-transformers/all-MiniLM-L6-v2`
- Dimensions: 384
- Normalization: L2-normalized (cosine similarity ready)
- Max sequence length: 512 tokens
- Build tool: `zolai-core/scripts/data/build_knowledge_index.py`
Usage
Load with Python
import json
vectors = []
with open("knowledge_vectors.jsonl") as f:
for line in f:
vectors.append(json.loads(line))
print(f"Loaded {len(vectors)} vectors")
entry = vectors[0]
print(f"Text: {entry['text'][:100]}...")
print(f"Embedding dim: {len(entry['embedding'])}")
print(f"Source: {entry['metadata']['source_type']}")Cosine similarity search
import numpy as np
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
query = "topa"
query_vec = model.encode([query], normalize_embeddings=True)[0]
results = []
for entry in vectors:
sim = np.dot(query_vec, entry["embedding"])
results.append((sim, entry))
results.sort(key=lambda x: -x[0])
for score, entry in results[:5]:
print(f"[{score:.4f}] {entry['text'][:120]}")With FAISS
import faiss
import json
import numpy as np
embeddings = []
with open("knowledge_vectors.jsonl") as f:
for line in f:
entry = json.loads(line)
embeddings.append(entry["embedding"])
matrix = np.array(embeddings, dtype="float32")
index = faiss.IndexFlatIP(matrix.shape[1])
index.add(matrix)
print(f"FAISS index: {index.ntotal} vectors")Language Notes
- ZVS 2018 orthography is enforced across all text content
- Key vocabulary differences vs Hakha/Falam:
pasian(God) !=pathian,topa(Lord) !=Pathian,vantung(heaven) !=van - `hiam` is the universal question marker (NOT
ze, which is emphatic) - SOV word order, ergative `in` marker
- 95.2% dictionary coverage on Genesis after dictionary consolidation
Reproducing
git clone https://github.com/Zolai-AI/zolai-core
cd zolai-core
python scripts/data/build_knowledge_index.py \
--data-dir /path/to/data \
--device cuda \
--batch-size 256
# Output: data/knowledge/knowledge_vectors.jsonl (~3.8 GB)License
MIT -- same as all Zolai-AI repositories.
Citation
@dataset{zolai_knowledge_vectors_2026,
title={Zolai Knowledge Vectors: English-Zo Bible + Dictionary + Wiki Embeddings},
author={Zolai-AI},
year={2026},
url={https://huggingface.co/datasets/peterpausianlian/zolai-knowledge-vectors}
}Links
- GitHub: github.com/Zolai-AI
- Website: zolai.space
- MCP Server: mcp.zolai.space/mcp
- Core Toolkit: github.com/Zolai-AI/zolai-core