datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
c4
C4
Dataset Summary
A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".
This is the processed version of Google's C4 dataset
We prepared five variants of the data: en, en.noclean, en.noblocklist, realnewslike, and multilingual (mC4).
For reference, these are the sizes of the variants:
en: 305GB
en.noclean: 2.3TB
en.noblocklist: 380GB
realnewslike: 15GB
multilingual (mC4): 9.7TB (108 subsets, one… See the full description on the dataset page: https://huggingface.co/datasets/allenai/c4.c4-10k
Dataset Card for "c4-10k"
More Information needed
c4-pt-randMore35M-part04-deduplicated-128000-no-digit-split-mask-train-15003771-lines
Dataset Card for "c4-pt-randMore35M-part04-deduplicated-128000-no-digit-split-mask-train-15003771-lines"
More Information needed
tokenized_C4c4-nanochatbpe-10B
c4-nanochatbpe-10B
C4 (en) (from allenai/c4), pre-tokenized with the nanochatbpe tokenizer (vocab 65,536)
and packaged as flat uint16 token-id .bin files for fast memmap training.
file
split
tokens
train.bin
train
10,000,000,000
val.bin
val
168,272,017
train and val are disjoint held-out partitions. Each .bin is a raw
little-endian uint16 stream (no header); token count = filesize / 2, and
train.meta.json / val.meta.json carry the full metadata. The tokenizer/files… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/c4-nanochatbpe-10B.c4-en-html-with-metadatadetails_CohereForAI__c4ai-command-r7b-arabic-02-2025_v2
Dataset Card for Evaluation run of CohereForAI/c4ai-command-r7b-arabic-02-2025
Dataset automatically created during the evaluation run of model CohereForAI/c4ai-command-r7b-arabic-02-2025.
The dataset is composed of 116 configuration, each one coresponding to one of the evaluated task.
The dataset has been created from 4 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing… See the full description on the dataset page: https://huggingface.co/datasets/OALL/details_CohereForAI__c4ai-command-r7b-arabic-02-2025_v2.chinese-c4
Introduction
Chinese-C4 is a clean Chinese internet dataset based on Common Crawl. The dataset is 46.29GB and has undergone multiple cleaning strategies, including Chinese filtering, heuristic cleaning based on punctuation, line-based hashing for deduplication, and repetition removal.
The dataset is open source and free for commercial use, and you are welcome to use the data and the cleaning strategies provided and contribute your cleaning strategies.
You can find the cleaning… See the full description on the dataset page: https://huggingface.co/datasets/shjwudp/chinese-c4.c4-subsets
Dataset Summary
Various subsets of C4 with different numbers of tokens measured with the GPT2Tokenizer.
This data is used in the paper Scaling Data-Constrained Language Models.
Please refer to our GitHub repository for more details.
@article{muennighoff2023scaling,
title={Scaling Data-Constrained Language Models},
author={Muennighoff, Niklas and Rush, Alexander M and Barak, Boaz and Scao, Teven Le and Piktus, Aleksandra and Tazi, Nouamane and Pyysalo, Sampo and Wolf, Thomas and… See the full description on the dataset page: https://huggingface.co/datasets/datablations/c4-subsets.c4_urls_en.noblocklist
Dataset Card for c4_urls_en.noblocklist
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English no blocklist variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.noblocklist.c4p0
c4p0 (consolidated)
Back-translation / round-trip translation traces over C4 source text,
produced by LLM-based MT prompts. Each row records the forward translation
(source -> target) plus the back-translation
(target -> target_backto_source) so the data can be used for noisy
parallel-corpus mining, distillation, or evaluation of translation
robustness.
This is the consolidated release. Earlier work was published as 46
separate repos (alvations/c4p0, alvations/c4p0-<pair>… See the full description on the dataset page: https://huggingface.co/datasets/alvations/c4p0.SeahorseDB-C4-10M
Example
dataset = load_dataset("dnotitia/SeahorseDB-C4-10M", trust_remote_code=True)
c4-en-html-with-metadata-ppl-cleanFile list:
"c4-en-html_cc-main-2019-18_pq00-000.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-001.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-002.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-003.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-004.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-005.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-006.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-007.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-008.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-009.jsonl.gz"… See the full description on the dataset page: https://huggingface.co/datasets/masoudjs/c4-en-html-with-metadata-ppl-clean.C4-Hindi-Cleaned
Dataset Card for "C4-Hindi-Cleaned"
More Information needed
c4-en-10kThis is a small subset representing the first 10K records of the original C4 dataset, "en" subset - created for testing. The records were extracted after having been shuffled.
The full 1TB+ dataset is at https://huggingface.co/datasets/c4.c4-en-html-with-training_metadata_allarabic-ocr-labelsc4-pro
📚 c4-pro
ArXiv | Models | Code
c4 is refined from c4 using the ProX refining framework.
It contains about 40B high quality tokens, ready for general language model pre-training.
License
c4 is based on c4, which is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: https://commoncrawl.org/terms-of-use/. We do not alter the license of any of the underlying data.
Citation
@article{zhou2024programming… See the full description on the dataset page: https://huggingface.co/datasets/gair-prox/c4-pro.c4_tr_fineweb_plusallenai-c4🧠 ALLENAI C4 - English Train Split (Prepared Version)
This repository contains the preprocessed and ready-to-use version of the ALLENAI C4 (Colossal Clean Crawled Corpus) English train split. It has been downloaded and optionally transformed for downstream NLP tasks such as pretraining large language models or text-based retrieval systems.
📦 Dataset Details
Original Source: allenai/c4
Language: English (en)
Split: train
License: Google C4 License
⚠️ Note: This version only includes the train… See the full description on the dataset page: https://huggingface.co/datasets/amanpreet7/allenai-c4.c4-clusters
Dataset Card for "c4-clusters"
More Information needed
c4-filter
Dataset Card for "c4-dedup"
More Information needed
c4-bnC4-Eval
C4-Eval
C4-Eval is the evaluation set for C4 Bench, a Chengyu-based benchmark for measuring whether multimodal language models can understand cross-concept creativity. The release contains the original images, the corresponding idiom answers, and the complete task-specific questions used for evaluation.
221 base items: 37 human-designed seed figures and 184 bridge-controlled synthetic figures.
1,105 evaluation instances: five task formulations for every base item.
Language:… See the full description on the dataset page: https://huggingface.co/datasets/sci-m-wang/C4-Eval.c4-chinese-zhtw
Dataset Card for "c4-chinese-zhtw"
內容
Common Crawl 是一個非營利組織，負責抓取網路並向公眾免費提供其檔案和資料集。Common Crawl 的網路檔案包含自 2008 年以來收集的 PB 級資料。它一般每月完成一次抓取。
Common Crawl 的爬蟲程式遵守 nofollow 和 robots.txt 政策。用於處理 Common Crawl 資料集的開源程式碼是公開可用的。
這個繁中的數據來是來自 Common Crawl 2023-14 的 data archive 下載并進行清理 。
這是 jed351 準備的版本，託管在這個位址：
https://huggingface.co/datasets/jed351/Traditional-Chinese-Common-Crawl-Filtered
支援的任務
C4主要用於預訓練語言模型(pretrain language model)。
範例
一個樣本的範例:
{… See the full description on the dataset page: https://huggingface.co/datasets/erhwenkuo/c4-chinese-zhtw.C4c4_urls_en
Dataset Card for c4_urls_en
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.c4
Dataset Card for "c4"
More Information needed
c4-rewritten-14b-retok-smollm360mc4_urls_multilingual
Dataset Card for c4_urls_multilingual
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (multilingual variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_multilingual.