datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
lm1bA benchmark corpus to be used for measuring progress in statistical language modeling. This has almost one billion words in the training data.lm1b
LM1B - One Billion Word Benchmark
Dataset Description
The One Billion Word Benchmark is a large language modeling dataset.
It contains approximately one billion words of training data derived from news articles.
How was this dataset built?
We download the full LM1B dataset from TensorFlow Datasets (TFDS) and convert it to HuggingFace format automatically. The full script is in lm1b.py. The required environment is:
tensorflow==2.20.0
tensorflow-datasets==4.9.9… See the full description on the dataset page: https://huggingface.co/datasets/FrankCCCCC/lm1b.lm1bA benchmark corpus to be used for measuring progress in statistical language modeling. This has almost one billion words in the training data.