datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
MADLAD-400
MADLAD-400
Dataset and Introduction
MADLAD-400 (Multilingual Audited Dataset: Low-resource And Document-level) is
a document-level multilingual dataset based on Common Crawl, covering 419
languages in total. This uses all snapshots of CommonCrawl available as of August
1, 2022. The primary advantage of this dataset over similar datasets is that it
is more multilingual (419 languages), it is audited and more highly filtered,
and it is document-level. The main… See the full description on the dataset page: https://huggingface.co/datasets/allenai/MADLAD-400.madlad-400_urls_noisy
Dataset Card for madlad-400_urls_noisy
This dataset provides the URLs and top-level domains associated with training records in allenai/MADLAD-400 (noisy variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/madlad-400_urls_noisy.madlad-400_urls_clean
Dataset Card for madlad-400_urls_clean
This dataset provides the URLs and top-level domains associated with training records in allenai/MADLAD-400 (clean variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/madlad-400_urls_clean.madlad-400_vi
MADLAD-400
Dataset and Introduction
MADLAD-400 (Multilingual Audited Dataset: Low-resource And Document-level) is
a document-level multilingual dataset based on Common Crawl, covering 419
languages in total. This uses all snapshots of CommonCrawl available as of August
1, 2022. The primary advantage of this dataset over similar datasets is that it
is more multilingual (419 languages), it is audited and more highly filtered,
and it is document-level. The main disadvantage… See the full description on the dataset page: https://huggingface.co/datasets/Symato/madlad-400_vi.MADLAD-400_Persian-Clain.Finaly
MADLAD-400 Persian Cleaned (Final)
این دیتاست نسخه تمیزسازیشده و بهینهسازیشده از زیرمجموعه زبان فارسی دیتاست MADLAD-400 است.
مشخصات دیتاست:
زبان: فارسی (fa)
تعداد کل رکوردها: 24,109,659
فرمت ذخیرهسازی: Apache Parquet (Snappy)
فیلد داده: text (متن تمیز فارسی)
نحوه استفاده:
from datasets import load_dataset
dataset = load_dataset("mahan13900/MADLAD-400_Persian-Clain.Finaly")
print(dataset["train"][0])
madlad_cleaned_version
Sinhala Cleaned Sentences - MADLAD-400
Cleaned and deduplicated Sinhala sentences derived from Minuri/sinhala-corpus-madlad400, produced through a multi-stage cleaning pipeline. This repo was used as pipeline storage across cleaning stages, with the final output being stage10_final_corpus_deduped.csv.
Final Output
File
Rows
Description
stage10_final_corpus_deduped.csv
5,033,732
Final cleaned and deduplicated sentences
Dataset Structure (final… See the full description on the dataset page: https://huggingface.co/datasets/Minuri/madlad_cleaned_version.sinhala-corpus-madlad400
Sinhala Raw Sentences - MADLAD-400
Raw Sinhala sentences extracted and sentence-split from the allenai/MADLAD-400 dataset. This is an intermediate dataset used in the construction of Minuri/diverse_sinhala_dataset.
Dataset Structure
Column
Description
text
Raw Sinhala sentence
source
Source identifier (madlad)
Split
Rows
train
7,281,026
Pipeline Position
allenai/MADLAD-400 → this repo → Minuri/madlad_cleaned_version →… See the full description on the dataset page: https://huggingface.co/datasets/Minuri/sinhala-corpus-madlad400.madlad-91-african
MADLAD Afro
African-language subset of allenai/MADLAD-400 (ODC-BY),
repackaged for the africa-corpus-builder.
Every file is plain verse-style CSV at the repo root:
Subset suffix
File name
Columns
clean
{Name}_{code}_v1.csv
verse_key, version_id, local
noisy
{Name}_{code}_v2.csv
verse_key, version_id, local
85 languages, each with clean and noisy as separate subsets (e.g. sw_clean, sw_noisy).
Very large splits are chunked into multiple _v{id}.csv files that… See the full description on the dataset page: https://huggingface.co/datasets/AfriSpeech/madlad-91-african.