Datasetpublic

Pika4028/low-resource-rag-indexes

Low-Resource RAG: Wikipedia FAISS Indexes (BGE-M3) FAISS indexes over Wikipedia 2023 passages for four languages, embedded with BAAI/bge-m3. Source corpus: CohereLabs/wikipedia-2023-11-embed-multilingual-v3. Each language folder has two files aligned row-by-row: index.faiss — FAISS IndexFlatIP, dim=1024, L2-normalized BGE-M3 vectors dataset/ — Arrow dataset with _id, title, text per passage Languages Language Code Passages Size Hindi hi TBD 2.6G… See the full description on the dataset page: https://huggingface.co/datasets/Pika4028/low-resource-rag-indexes.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes113downloads

Dataset Card

Low-Resource RAG: Wikipedia FAISS Indexes (BGE-M3)

FAISS indexes over Wikipedia 2023 passages for four languages, embedded with BAAI/bge-m3. Source corpus: CohereLabs/wikipedia-2023-11-embed-multilingual-v3.

Each language folder has two files aligned row-by-row:

—index.faiss — FAISS IndexFlatIP, dim=1024, L2-normalized BGE-M3 vectors
—dataset/ — Arrow dataset with _id, title, text per passage

Languages

Language	Code	Passages	Size
Hindi	`hi`	TBD	2.6G
Marathi	`mr`	TBD	975M
Nepali	`ne`	TBD	412M
Maithili	`mai`	TBD	66M
Bengali	`bn`	767,965	3.7G
Gujarati	`gu`	99,450	495M
Urdu	`ur`	434,269	2.0G
Santali	`sat`	43,500	209M
Tamil	`ta`	685,975	3.3G
Telugu	`te`	634,778	3.0G
Kannada	`kn`	309,085	1.6G
Malayalam	`ml`	384,795	1.9G
Punjabi	`pa`	188,091	914M
Assamese	`as`	76,093	376M

Usage

python

import faiss
from datasets import load_from_disk
from FlagEmbedding import BGEM3FlagModel

LANG = "hi"  # hi | mr | ne | mai
INDEX_DIR = "/path/to/indexes"

index = faiss.read_index(f"{INDEX_DIR}/{LANG}/index.faiss")
dataset = load_from_disk(f"{INDEX_DIR}/{LANG}/dataset")

model = BGEM3FlagModel("BAAI/bge-m3", use_fp16=True)
query_vec = model.encode(["your query"])["dense_vecs"]

scores, indices = index.search(query_vec, k=10)
results = dataset.select(indices[0])

Download

python

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Pika4028/low-resource-rag-indexes",
    repo_type="dataset",
    local_dir="/path/to/indexes",
    allow_patterns="hi/*",  # omit for all languages
)