datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
c4
C4
Dataset Summary
A colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org".
This is the processed version of Google's C4 dataset
We prepared five variants of the data: en, en.noclean, en.noblocklist, realnewslike, and multilingual (mC4).
For reference, these are the sizes of the variants:
en: 305GB
en.noclean: 2.3TB
en.noblocklist: 380GB
realnewslike: 15GB
multilingual (mC4): 9.7TB (108 subsets, one… See the full description on the dataset page: https://huggingface.co/datasets/allenai/c4.c4_urls_en.noblocklist
Dataset Card for c4_urls_en.noblocklist
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English no blocklist variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.noblocklist.c4p0
c4p0 (consolidated)
Back-translation / round-trip translation traces over C4 source text,
produced by LLM-based MT prompts. Each row records the forward translation
(source -> target) plus the back-translation
(target -> target_backto_source) so the data can be used for noisy
parallel-corpus mining, distillation, or evaluation of translation
robustness.
This is the consolidated release. Earlier work was published as 46
separate repos (alvations/c4p0, alvations/c4p0-<pair>… See the full description on the dataset page: https://huggingface.co/datasets/alvations/c4p0.allenai-c4🧠 ALLENAI C4 - English Train Split (Prepared Version)
This repository contains the preprocessed and ready-to-use version of the ALLENAI C4 (Colossal Clean Crawled Corpus) English train split. It has been downloaded and optionally transformed for downstream NLP tasks such as pretraining large language models or text-based retrieval systems.
📦 Dataset Details
Original Source: allenai/c4
Language: English (en)
Split: train
License: Google C4 License
⚠️ Note: This version only includes the train… See the full description on the dataset page: https://huggingface.co/datasets/amanpreet7/allenai-c4.c4-chinese-zhtw
Dataset Card for "c4-chinese-zhtw"
內容
Common Crawl 是一個非營利組織，負責抓取網路並向公眾免費提供其檔案和資料集。Common Crawl 的網路檔案包含自 2008 年以來收集的 PB 級資料。它一般每月完成一次抓取。
Common Crawl 的爬蟲程式遵守 nofollow 和 robots.txt 政策。用於處理 Common Crawl 資料集的開源程式碼是公開可用的。
這個繁中的數據來是來自 Common Crawl 2023-14 的 data archive 下載并進行清理 。
這是 jed351 準備的版本，託管在這個位址：
https://huggingface.co/datasets/jed351/Traditional-Chinese-Common-Crawl-Filtered
支援的任務
C4主要用於預訓練語言模型(pretrain language model)。
範例
一個樣本的範例:
{… See the full description on the dataset page: https://huggingface.co/datasets/erhwenkuo/c4-chinese-zhtw.c4_urls_en
Dataset Card for c4_urls_en
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.c4_urls_multilingual
Dataset Card for c4_urls_multilingual
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (multilingual variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_multilingual.loong_c4A filtered subset of C4-en containing 3,584,358 pages that are at least 16,000 characters long, useful for training models with longer context windows.
long_c4A filtered subset of C4-en containing 13,688,429 pages that are at least 8,000 characters long, useful for training models with longer context windows.
c4_200m\
GEC Dataset Generated from C4yentinglin-zh_TW_c4
Language Models for Taiwanese Culture
✍️ Online Demo
•
🤗 HF Repo • 🐦 Twitter • 📃 [Paper Coming Soon]
• 👨️ Yen-Ting Lin
Overview
Taiwan-LLaMa is a full parameter fine-tuned model based on LLaMa 2 for Traditional Mandarin applications.
Taiwan-LLaMa v1.0 pretrained on over 5 billion tokens and instruction-tuned on over 490k conversations both in traditional mandarin.
Demo
A live demonstration of the model can… See the full description on the dataset page: https://huggingface.co/datasets/botp/yentinglin-zh_TW_c4.c4_urls_en.noclean
Dataset Card for c4_urls_en.noclean
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English no clean variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.noclean.c4-ko-cleaned-2이전 데이터셋에서 아쉬운 점이 많이 보여 조금 개선한 데이터셋 입니다.
원본 데이터셋: c4
파일 크기: 약 10gb
데이터 수: 2261464
c4_200m
C4 200M
Dataset Summary
C4 200M is a collection of 183,894,319 synthetic sentence pairs generated from the cleaned English portion of the C4 corpus for grammatical error correction (GEC).
This repository is a Parquet conversion of the original liweili/c4_200m dataset. The original dataset relied on a loading script, which is incompatible with recent versions of the 🤗 Datasets library. This version stores the data in Apache Parquet format, enabling efficient… See the full description on the dataset page: https://huggingface.co/datasets/martinsr/c4_200m.c4-zhtw
Dataset Card for C4-zhtw
Traditional Chinese subset of the C4 dataset.
De-duplicated with MinHash.
Is suggested to filter the dataset with NLU models before any serious use.
Raw-C4-ko-500MT
⚠️ Warning ⚠️
This dataset was created by the owner of this repository to validate the capabilities of Nari-C4-ko-500MT. It is not recommended to use this for LLM training.
Lumia101/Raw-C4-ko-500MT
This dataset is a dataset from which 500M tokens were extracted from the allenai/c4 without any additional filtering.
This dataset was created to measure the effect of filtering applied to Lumia101/Nari-C4-ko-500MT, so if you intend to use this dataset for LLM training, it is… See the full description on the dataset page: https://huggingface.co/datasets/Lumia101/Raw-C4-ko-500MT.looong_c4A filtered subset of C4-en containing 835,400 pages that are at least 32,000 characters long, useful for training models with longer context windows.
Nari-C4-ko-500MT
Lumia101/Nari-C4-ko-500MT
This dataset is a modified version of C4 dataset(multilingual, ko subset) made more useful for LLM training by applying additional filtering.
Since the number of tokens is only about 500M, it is recommended to mix it with other high-quality datasets.
Additional filtering methods used
Phase 1: Text normalization
Phase 2: Remove HTML-filled junk documents
Phase 3: Remove documents containing a lot of broken characters
Phase 4: Remove documents… See the full description on the dataset page: https://huggingface.co/datasets/Lumia101/Nari-C4-ko-500MT.c4-faqs
Dataset Card for [Dataset Name]
Dataset Summary
This dataset comprises of open-domain question-answer pairs obtained from extracting 150K FAQ URLs from C4 dataset. Please refer to the original paper and dataset card for more details.
You can load C4-FAQs as follows:
from datasets import load_dataset
c4_faqs_dataset = load_dataset("vishal-burman/c4-faqs")
Supported Tasks and Leaderboards
C4-FAQs is mainly intended for open-domain end-to-end question… See the full description on the dataset page: https://huggingface.co/datasets/vishal-burman/c4-faqs.myanmar-c4-datasetPlease visit to the GitHub repository for other Myanmar Langauge datasets.
Myanmar C4 Dataset
A preprocessed subset of the C4 dataset containing only Myanmar language text, with consistent Unicode encoding.
Dataset Description
This dataset is derived from the Colossal Clean Crawled Corpus (C4) created by AllenAI. It contains only the Myanmar language portion of the original C4 dataset, with additional preprocessing to standardize text encoding.
Preprocessing… See the full description on the dataset page: https://huggingface.co/datasets/chuuhtetnaing/myanmar-c4-dataset.c4-en-64tokenc4_urls_realnewslike
Dataset Card for c4_urls_realnewslike
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (realnewslike variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_realnewslike.color-filtered-c4
CoLoR-Filtered C4
This repo contains two datasets: color-filtered-c4-books and color-filtered-c4-down associated with the CoLoR-Filter paper.
Each dataset is a 64x filtered version of the C4 dataset from Raffel et al., 2019 that has been selected using the CoLoR-Filter algorithm for data selection.
Each dataset has about 2.7b tokens when using the allenai/eleuther-ai-gpt-neox-20b-pii-special tokenizer.
color-filtered-c4-books was selected to target books based on a small (25m token)… See the full description on the dataset page: https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4.C4-200m-550k-DeterminerThis dataset is a subset of the C4_200M dataset from kaggle, containing 550k pairs of incorrect and correct sentences of Determiner category.
Link for the original dataset: https://www.kaggle.com/datasets/felixstahlberg/the-c4-200m-dataset-for-gec
The original dataset is also a synthetic dataset, containing input[incorrect sentence] and output[correct_sentence].
To generate the corruption, a tagged corruption model was first trained. This model is trained on existing datasets by taking as… See the full description on the dataset page: https://huggingface.co/datasets/abhinavsarkar/C4-200m-550k-Determiner.c4-zhtw
Dataset Card for c4-zhtw
c4-zhtw 是一個繁體中文（zh-tw）版本之 C4 子集之小型實驗用語料集，合計 2,783 筆，於原始文本之外額外附帶分類器預測（pred_label / pred_conf）與人工標註（label / reason）欄位，用於訓練或評估「繁中高品質文本分類器」（例如用於篩選 c4 / fineweb 繁中子集之 classifier）。
Dataset Details
Dataset Description
繁體中文之高品質預訓練語料長期不足，過濾現有大規模語料（如 C4、FineWeb）成為常見解法。本資料集為其中一個小型實驗子集，記錄了：
原始 C4 之 zh 子集中之文本、時間戳、URL；
分類器自動預測之標籤（pred_label 與信心 pred_conf）；
人工覆核之最終標籤（label）與標註理由（reason）；
文本長度之 word / token 計數。
主要用途為訓練或評估「繁中品質分類器」之… See the full description on the dataset page: https://huggingface.co/datasets/lianghsun/c4-zhtw.c4-ko-cleaned학교 점심시간 때 할 거 없어서 만든 c4를 정제한 데이터입니다. 다 하면 컴퓨터가 감당 못 할 거 같아서 전체 데이터의 1/10만 진행하였으며 아마 품질은 안 좋을 겁니다.
파일 크기: 약 3gb
데이터 수: 1847023
c4-benchfilter-nano
crumb/c4-benchfilter-nano
A 278k sample derivation of the first 3M samples from the C4 dataset for a cheap and short continued pretraining for language models to optimize for benchmark scores without sacrificing generalization and generative modelling unrelated to chat or 'instruct' data.
The estimated top 10% of highest estimated length normalized ngram (mean of tri, quad, and penta-gram) overlaps for each of the
selected benchmark datasets (arc, truthful_qa, hellaswag, mmlu… See the full description on the dataset page: https://huggingface.co/datasets/crumb/c4-benchfilter-nano.msm-v2-shared-c4-36k
MSM v2 shared C4 36k
Training-ready inputs for the second MSM run. Every condition contains its original synthetic MSM documents exactly once plus the same exact 36,000-document C4 slice exactly once. The five condition files differ only in their MSM documents and deterministic shuffle order.
Synthetic rows begin with <DOCTAG>\n and declare the same string in mask_prefix; C4 rows are untagged and declare an empty mask_prefix. The trainer must mask only the declared prefix tokens… See the full description on the dataset page: https://huggingface.co/datasets/GaloisTheory123/msm-v2-shared-c4-36k.opencode_openmath_60k_c40m60
opencode_openmath_60k_c40m60
60k instruction mixture used for RELAY / Fast-dLLM-style SFT ablations:
24,000 code rows from nvidia/OpenCodeInstruct (40%)
36,000 math rows from nvidia/OpenMathInstruct-2 (train_1M, 60%)
Seed 42; code rows require a Python def; length-filtered to ≤2048 tokens under a chat template used only for filtering
Format
LMFlow-style conversation JSON (not model-specific weights or LLaDA ChatML):
{"type": "conversation", "instances":… See the full description on the dataset page: https://huggingface.co/datasets/brozonoyer/opencode_openmath_60k_c40m60.c4
C4
This repository hosts a copy of the widely used C4 dataset, a variant of the Colossal Clean Crawled Corpus designed for training and evaluating Large Language Models (LLMs) on news-like text.
C4 consists of cleaned web data from Common Crawl, specifically curated to contain more news-style content. This dataset is commonly used in language modeling tasks, text generation, and research focused on news and article-like content.
Contents
c4.json (or your actual… See the full description on the dataset page: https://huggingface.co/datasets/S3IC/c4.