mamei16/wikipedia_paragraphs
Description This dataset consists of English Wikipedia articles, which first have been split by paragraph breaks and subsequently by spaces. For each resulting token, there is a corresponding binary ner_tag, which is 1 if a token was followed by paragraph break in the original text. There are two deliberate exceptions to this, which can be seen in the dataset generation code: The text is not split if a paragraph break is preceded by a colon (":"), to avoid lists being separated… See the full description on the dataset page: https://huggingface.co/datasets/mamei16/wikipedia_paragraphs.
Description
This dataset consists of English Wikipedia articles, which first have been split by paragraph breaks and subsequently by spaces. For each resulting token, there is a corresponding binary ner_tag, which is 1 if a token was followed by paragraph break in the original text. There are two deliberate exceptions to this, which can be seen in the dataset generation code:
- The text is not split if a paragraph break is preceded by a colon (":"), to avoid lists being separated into different chunks.
- A paragraph must be longer than one token.
The purpose of this dataset is to train chunking models for use in RAG applications.