Datasetpublic

agentlans/c4-en-tokenized

C4 English Tokenized Samples This dataset contains tokenized English samples from the C4 (Colossal Clean Crawled Corpus) dataset for natural language processing (NLP) tasks. The first 125 000 entries from the en split of allenai/c4 were tokenized using spaCy's en_core_web_sm model. Tokens joined with spaces. Features text: Original text from C4 tokenized: The tokenized and space-joined text num_tokens: Number of tokens after tokenization num_punct_tokens: Number… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/c4-en-tokenized.

sourceHugging Faceodc-byupdated 2y agoView on Hugging Face

0likes8downloads

Dataset Card

C4 English Tokenized Samples

This dataset contains tokenized English samples from the C4 (Colossal Clean Crawled Corpus) dataset for natural language processing (NLP) tasks.

The first 125 000 entries from the en split of allenai/c4 were tokenized using spaCy's en_core_web_sm model. Tokens joined with spaces.

Features

—text: Original text from C4
—tokenized: The tokenized and space-joined text
—num_tokens: Number of tokens after tokenization
—num_punct_tokens: Number of punctuation tokens after tokenization

Example

json

{
  "text": "ALDUS MANUTIUS AND HIS THESAURUS CORNUCOPIAE OF 1496.\nSyracuse (1958) . 7.5 x 4.25, cloth, 32 pp, a v.g. copy [...]",
  "tokenized": "ALDUS MANUTIUS AND HIS THESAURUS CORNUCOPIAE OF 1496 . \n Syracuse ( 1958 ) . 7.5 x 4.25 , cloth , 32 pp , a v.g . copy [...]",
  "num_tokens": 84,
  "num_punct_tokens": 19
}

Usage

This dataset can be useful for:

—Text classification tasks
—Language modeling
—Sentiment analysis
—Other NLP applications requiring tokenized English text

Researchers and developers can use this dataset to jumpstart their projects without the need for initial tokenization steps.

Licence

This dataset is licensed under the ODC-BY (Open Data Commons Attribution) licence.