lm1b
Datasets
All datasets matching “lm1b”lm1bA benchmark corpus to be used for measuring progress in statistical language modeling. This has almost one billion words in the training data.lm1blm1blm1blm1b
LM1B - One Billion Word Benchmark
Dataset Description
The One Billion Word Benchmark is a large language modeling dataset.
It contains approximately one billion words of training data derived from news articles.
How was this dataset built?
We download the full LM1B dataset from TensorFlow Datasets (TFDS) and convert it to HuggingFace format automatically. The full script is in lm1b.py. The required environment is:
tensorflow==2.20.0
tensorflow-datasets==4.9.9… See the full description on the dataset page: https://huggingface.co/datasets/FrankCCCCC/lm1b.lm1b-dep-wrapped