CoolFace
Datasetpublic

TemryL/tokenized_wikipedia_20220301.en_train_512

Tokenized English Wikipedia Dataset Dataset Description This dataset contains tokenized chunks of text from the English Wikipedia dump of March 1, 2022. Each entry in the dataset represents a chunk of text from Wikipedia, with information about which document and position within the document it comes from. Dataset Creation Source Dataset: Wikipedia (20220301.en) Tokenizer: BERT base uncased Chunk Size: 512 tokens (including special tokens)… See the full description on the dataset page: https://huggingface.co/datasets/TemryL/tokenized_wikipedia_20220301.en_train_512.

sourceHugging Facecc-by-sa-3.0updated 2y agoView on Hugging Face
0likes258downloads
Dataset Card

Tokenized English Wikipedia Dataset

Dataset Description

This dataset contains tokenized chunks of text from the English Wikipedia dump of March 1, 2022. Each entry in the dataset represents a chunk of text from Wikipedia, with information about which document and position within the document it comes from.

Dataset Creation

  • —Source Dataset: Wikipedia (20220301.en)
  • —Tokenizer: BERT base uncased
  • —Chunk Size: 512 tokens (including special tokens)

Processing Steps

  1. 1.The raw Wikipedia dataset was loaded using the Hugging Face datasets library.
  2. 2.Each article was tokenized using the BERT base uncased tokenizer.
  3. 3.The tokenized articles were processed to create chunks of exactly 512 tokens each:
  4. 4.Each chunk starts with the [CLS] token and ends with the [SEP] token.
  5. 5.Up to 510 tokens from the article content are included in each chunk.
  6. 6.If a chunk is shorter than 512 tokens, it is padded with the [PAD] token to reach the full length.
  7. 7.An attention mask is created for each chunk, with 1 for real tokens and 0 for padding tokens.
  8. 8.Information about the document index and chunk index within the document is preserved for each chunk.
  9. 9.All chunks, including padded ones, are retained in the final processed dataset.

Code

The following Python code was used to create this dataset:

python
import os
from datasets import load_dataset
from transformers import AutoTokenizer
from transformers.utils import logging


def chunk_doc(examples, indices, tokenizer, chunk_length):
    chunks = []
    doc_idxs = []
    chunk_idxs = []
    attention_masks = []
    
    pad_token_id = tokenizer.pad_token_id
    cls_id = tokenizer.cls_token_id
    sep_id = tokenizer.sep_token_id
    
    for example, doc_idx in zip(examples['text'], indices):
        tokens = tokenizer.encode(example, add_special_tokens=False)
        
        for idx, i in enumerate(range(0, len(tokens), chunk_length - 2)):
            chunk = tokens[i:i + chunk_length - 2]
            
            # Add CLS and SEP tokens
            padded_chunk = [cls_id] + chunk + [sep_id]
            
            # Pad if necessary
            padding_length = chunk_length - len(padded_chunk)
            if padding_length > 0:
                padded_chunk += [pad_token_id] * padding_length
            attention_mask = [1] * (chunk_length - padding_length) + [0] * padding_length
            
            assert len(padded_chunk) == chunk_length
            assert len(attention_mask) == chunk_length
            
            chunks.append(padded_chunk)
            doc_idxs.append(doc_idx)
            chunk_idxs.append(idx)
            attention_masks.append(attention_mask)
    
    return {
        'token_ids': chunks,
        'attention_mask': attention_masks,
        'doc_idx': doc_idxs,
        'chunk_idx': chunk_idxs
    }


def main():
    CHUNK_LENGTH = 512
    TOKENIZER_NAME = 'bert-base-uncased'

    # Load the tokenizer
    logging.set_verbosity(40)
    tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_NAME)

    # Load the dataset
    dataset = load_dataset("wikipedia", "20220301.en", split="train", trust_remote_code=True)
    dataset = dataset.remove_columns([col for col in dataset.column_names if col != "text"])
    
    # Create the dataset using map function
    tokenized_dataset = dataset.map(
        lambda examples, indices: chunk_doc(examples, indices, tokenizer, MAX_LENGTH),
        remove_columns=dataset.column_names,
        batched=True,
        with_indices=True,
        num_proc=os.cpu_count(),
        desc="Creating tokenized dataset"
    )

    # Push the dataset to the Hugging Face Hub
    dataset_name = f"tokenized_wikipedia_20220301.en_train_{MAX_LENGTH}"
    tokenized_dataset.push_to_hub(dataset_name, token=os.environ.get("HF_TOKEN"))
    print(f"Dataset pushed to Hugging Face Hub: {dataset_name}")


if __name__ == '__main__':
    main()

Dataset Structure

Each row in the dataset corresponds to a chunk of text from a Wikipedia document and contains the following fields:

  • —token_ids: A list of 512 integer token IDs corresponding to a chunk of text from a document.
  • —attention_mask: A list of 512 integers (0 or 1) indicating which tokens are padding (0) and which are not (1).
  • —doc_idx: The index of the document this chunk comes from.
  • —chunk_idx: The index of this chunk within its document.

For example, a row might look like this:

python
{
  'token_ids': [101, 2054, 2003, ..., 0, 0],  # 512 tokens, potentially including padding
  'attention_mask': [1, 1, 1, ..., 0, 0]  # 1 for real tokens, 0 for padding
  'doc_idx': 42,  # This chunk is from the 43rd document
  'chunk_idx': 3,  # This is the 4th chunk in that document
}

Usage / Limitations / Ethical Considerations

For usage, limitations and ethical considerations, please refer to the Wikipedia dataset.

Citation

If you use this dataset in your research, please cite:

bibtex
@dataset{tokenized_wikipedia_20220301,
  author = {Mery, Tom},
  title = {Tokenized English Wikipedia Dataset (March 2022)},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Data Repository},
  howpublished = {\url{https://huggingface.co/datasets/TemryL/tokenized_wikipedia_20220301.en_train_512}}
}