MaLA-LM/mala-monolingual-dedup

MaLA Corpus: Massive Language Adaptation Corpus This is a deduplicated version after minhash and exact hash deduplication. Dataset Summary The MaLA Corpus (Massive Language Adaptation) is a comprehensive, multilingual dataset designed to support the continual pre-training of large language models. It covers 939 languages and consists of over 74 billion tokens, making it one of the largest datasets of its kind. With a focus on improving the representation of… See the full description on the dataset page: https://huggingface.co/datasets/MaLA-LM/mala-monolingual-dedup.