Datasetpublic

Loctran123/vietnamese-evidence-corpus-chunked-e5-v2

Vietnamese Evidence Corpus - Chunked Chunked evidence corpus prepared for multilingual information retrieval, retrieval-augmented generation, and fact-checking experiments. Statistics 47,679 chunks from 13,572 source documents 38,603 Vietnamese chunks and 9,076 English chunks Maximum chunk length: 512 BGE-M3 tokenizer tokens Main fields chunk_id, doc_id, chunk_index token_start, token_end, token_count title, text, summary source, source_type… See the full description on the dataset page: https://huggingface.co/datasets/Loctran123/vietnamese-evidence-corpus-chunked-e5-v2.

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes20downloads

Dataset Card

Vietnamese Evidence Corpus - Chunked

Chunked evidence corpus prepared for multilingual information retrieval, retrieval-augmented generation, and fact-checking experiments.

Statistics

—47,679 chunks from 13,572 source documents
—38,603 Vietnamese chunks and 9,076 English chunks
—Maximum chunk length: 512 BGE-M3 tokenizer tokens

Main fields

—chunk_id, doc_id, chunk_index
—token_start, token_end, token_count
—title, text, summary
—source, source_type, domain, document_type
—language, country, publish_date, crawl_date, url, author
—metadata, quality

For embedding, use title + "\n\n" + text as the embedding input and keep the remaining fields as filter/citation metadata.

Loading

python

from datasets import load_dataset

dataset = load_dataset(
    "Loctran123/vietnamese-evidence-corpus-chunked-e5-v2",
    data_files="data/corpus_v1_chunked_e5_v2.json",
    split="train",
)

The repository does not grant additional rights over the original source articles. Users are responsible for following the terms of each source.