CoolFace
Datasetpublic

Rubin-Wei/enwiki-dec2021-preprocessed-mistral

Dataset Description This dataset is a preprocessed version of the English Wikipedia snapshot from December 2021, processed using the preprocess_dataset.py script provided in the repository below. Paper: MLP Memory: A Retriever-Pretrained Memory for Large Language Models GitHub: https://github.com/Rubin-Wei/MLPMemory Dataset Source: English Wikipedia (December 2021) Tokenizer: Mistral-7B-v0.3 Two key preprocessing parameters used are: block_size: 2048 stride: 1024… See the full description on the dataset page: https://huggingface.co/datasets/Rubin-Wei/enwiki-dec2021-preprocessed-mistral.

sourceHugging Faceapache-2.0updated 11mo agoView on Hugging Face
0likes2.2kdownloads
Dataset Card

Dataset Description

This dataset is a preprocessed version of the English Wikipedia snapshot from December 2021, processed using the preprocess_dataset.py script provided in the repository below.

  • —Tokenizer: Mistral-7B-v0.3

Two key preprocessing parameters used are:

  • —block_size: 2048
  • —stride: 1024

Data Splits

The dataset is organized into train and test splits as follows:

├── test
├── train
├── dataset_dict.json
└── dstore_summary.json

The train/test split was created using the following code:

python
from datasets import load_dataset

dataset = load_dataset('json', data_files=dataset_path)
split_dataset = dataset['train'].train_test_split(
    test_size=0.002,
    seed=42,
    shuffle=True
)
train_dataset = split_dataset['train']
test_dataset = split_dataset['test']
raw_datasets = DatasetDict({"train": train_dataset, "test": test_dataset})

Citation

bibtex
@inproceedings{Wei2025MLPMA,
  title={MLP Memory: A Retriever-Pretrained Memory for Large Language Models},
  author={Rubin Wei and Jiaqi Cao and Jiarui Wang and Jushi Kai and Qipeng Guo and Bowen Zhou and Zhouhan Lin},
  year={2025},
  url={https://api.semanticscholar.org/CorpusID:281658735}
}

Contact

For questions and discussions, feel free to email: weirubinn@gmail.com