bkai-foundation-models/NewsSapo

Vietnamese NewsSapo Dataset

The Vietnamese NewsSapo dataset was constructed to train sentence/passage embeddings. Our dataset is structured in a "title-abstract-contents" format, where each news article is represented by a tuple of (title, abstract, content). The content is the main text body of the article and has been processed to remove images, videos, and other non-textual elements. The dataset contains 31,728,183 triples.

To build this dataset, we followed a two-step process:

Step 1: Collect news data from 2021-11/2023. Combine with Binhvq News Corpus to form a unified dataset.

Step 2: Extract title-sapo-content for each article.

Please cite our manuscript if this dataset is used for your work

  @article{duc2024towards,
    title={Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models},
    author={Nguyen Quang Duc, Le Hai Son, Nguyen Duc Nhan, Nguyen Dich Nhat Minh, Le Thanh Huong, Dinh Viet Sang},
    journal={arXiv preprint arXiv:2403.01616},
    year={2024}
  }