datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
tokenized_C4c4-nanochatbpe-10B
c4-nanochatbpe-10B
C4 (en) (from allenai/c4), pre-tokenized with the nanochatbpe tokenizer (vocab 65,536)
and packaged as flat uint16 token-id .bin files for fast memmap training.
file
split
tokens
train.bin
train
10,000,000,000
val.bin
val
168,272,017
train and val are disjoint held-out partitions. Each .bin is a raw
little-endian uint16 stream (no header); token count = filesize / 2, and
train.meta.json / val.meta.json carry the full metadata. The tokenizer/files… See the full description on the dataset page: https://huggingface.co/datasets/alexkstern/c4-nanochatbpe-10B.c4-en-html-with-metadatachinese-c4
Introduction
Chinese-C4 is a clean Chinese internet dataset based on Common Crawl. The dataset is 46.29GB and has undergone multiple cleaning strategies, including Chinese filtering, heuristic cleaning based on punctuation, line-based hashing for deduplication, and repetition removal.
The dataset is open source and free for commercial use, and you are welcome to use the data and the cleaning strategies provided and contribute your cleaning strategies.
You can find the cleaning… See the full description on the dataset page: https://huggingface.co/datasets/shjwudp/chinese-c4.c4-subsets
Dataset Summary
Various subsets of C4 with different numbers of tokens measured with the GPT2Tokenizer.
This data is used in the paper Scaling Data-Constrained Language Models.
Please refer to our GitHub repository for more details.
@article{muennighoff2023scaling,
title={Scaling Data-Constrained Language Models},
author={Muennighoff, Niklas and Rush, Alexander M and Barak, Boaz and Scao, Teven Le and Piktus, Aleksandra and Tazi, Nouamane and Pyysalo, Sampo and Wolf, Thomas and… See the full description on the dataset page: https://huggingface.co/datasets/datablations/c4-subsets.c4-en-html-with-metadata-ppl-cleanFile list:
"c4-en-html_cc-main-2019-18_pq00-000.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-001.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-002.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-003.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-004.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-005.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-006.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-007.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-008.jsonl.gz",
"c4-en-html_cc-main-2019-18_pq00-009.jsonl.gz"… See the full description on the dataset page: https://huggingface.co/datasets/masoudjs/c4-en-html-with-metadata-ppl-clean.c4-en-html-with-training_metadata_allC4-Eval
C4-Eval
C4-Eval is the evaluation set for C4 Bench, a Chengyu-based benchmark for measuring whether multimodal language models can understand cross-concept creativity. The release contains the original images, the corresponding idiom answers, and the complete task-specific questions used for evaluation.
221 base items: 37 human-designed seed figures and 184 bridge-controlled synthetic figures.
1,105 evaluation instances: five task formulations for every base item.
Language:… See the full description on the dataset page: https://huggingface.co/datasets/sci-m-wang/C4-Eval.c4-ko-cleaned-2이전 데이터셋에서 아쉬운 점이 많이 보여 조금 개선한 데이터셋 입니다.
원본 데이터셋: c4
파일 크기: 약 10gb
데이터 수: 2261464
c4_pretrain_zhtw
Dataset Card for "yuhuanstudio/c4_pretrain_zhtw"
資料集摘要
本資料集基於 C4（Colossal Clean Crawled Corpus）原始數據，並經過以下處理步驟，轉換為適用於大型語言模型（LLM）預訓練的格式：
資料清理：去除非中文內容、重複文本及不必要的 HTML 標籤，並使用pangu格式化中文語句間隔，提升語言模型的訓練品質。
格式化：將數據重新整理為適合 LLM 預訓練的結構，便於高效載入與處理。
內容說明
數據來源：Colossal Clean Crawled Corpus (C4)
語言：繁體中文
資料格式：JSON 格式，適用於 LLM 預訓練
資料數量：包含大量經過清理和格式化的繁體中文文本
使用說明
此資料集適用於：
大型語言模型的預訓練
自然語言處理（NLP）研究
繁體中文語言理解與分析
資料集結構
{
"text": "台北故事館 雲門特展 As Lomo aslomo 天空部落 TIAN… See the full description on the dataset page: https://huggingface.co/datasets/yuhuanstudio/c4_pretrain_zhtw.c4tasky_v2website_metadata_c4The dataset is in the form of a json lines file with 1,20,000 examples, where an example consists of text (extracted from C4 English dataset) and metadata fields (website description extracted from Wikipedia).
Example:
{
"text": "US10289222B2 - Handling of touch events in a browser environment - Google Patents\nHandling of touch events in a browser environment Download PDF\nUS10289222B2\nUS10289222B2 US13/857,848 US201313857848A US10289222B2 US 10289222 B2 US10289222 B2 US 10289222B2 US… See the full description on the dataset page: https://huggingface.co/datasets/bs-modeling-metadata/website_metadata_c4.c4-text-zstd-v2This dataset contains two configurations: c4 and msmarco, both stored in zstd format
diffusion_data_constraint_c4subsetsc4allc4-chunksCohereForAI__c4ai-command-r-plus-details
Dataset Card for Evaluation run of CohereForAI/c4ai-command-r-plus
Dataset automatically created during the evaluation run of model CohereForAI/c4ai-command-r-plus
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/CohereForAI__c4ai-command-r-plus-details.c4-en-propella
C4 English Text Annotated Using Propella
This dataset contains over 200 000 rows of the en split from allenai/c4, richly annotated using the ellamind/propella-1-0.6b model.
🎯 Use Cases
Dataset Distillation: Train and distill smaller, efficient classifiers based on Propella's outputs.
Document Classification: Fine-tune models for automated content moderation, quality estimation, structural tagging, and domain categorization.
📂 Dataset Structure… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/c4-en-propella.c4taskycleaned_C4CohereForAI__c4ai-command-r-v01-details
Dataset Card for Evaluation run of CohereForAI/c4ai-command-r-v01
Dataset automatically created during the evaluation run of model CohereForAI/c4ai-command-r-v01
The dataset is composed of 44 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the latest results.… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/CohereForAI__c4ai-command-r-v01-details.c4-subsetwebsite_metadata_c4_toyA smaller version (100 samples) of https://huggingface.co/datasets/bs-modeling-metadata/website_metadata_c4
CohereForAI__c4ai-command-r7b-12-2024-details
Dataset Card for Evaluation run of CohereForAI/c4ai-command-r7b-12-2024
Dataset automatically created during the evaluation run of model CohereForAI/c4ai-command-r7b-12-2024
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to the… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/CohereForAI__c4ai-command-r7b-12-2024-details.color-filtered-c4
CoLoR-Filtered C4
This repo contains two datasets: color-filtered-c4-books and color-filtered-c4-down associated with the CoLoR-Filter paper.
Each dataset is a 64x filtered version of the C4 dataset from Raffel et al., 2019 that has been selected using the CoLoR-Filter algorithm for data selection.
Each dataset has about 2.7b tokens when using the allenai/eleuther-ai-gpt-neox-20b-pii-special tokenizer.
color-filtered-c4-books was selected to target books based on a small (25m token)… See the full description on the dataset page: https://huggingface.co/datasets/davidbrandfonbrener/color-filtered-c4.CohereForAI__c4ai-command-r-plus-08-2024-details
Dataset Card for Evaluation run of CohereForAI/c4ai-command-r-plus-08-2024
Dataset automatically created during the evaluation run of model CohereForAI/c4ai-command-r-plus-08-2024
The dataset is composed of 38 configuration(s), each one corresponding to one of the evaluated task.
The dataset has been created from 1 run(s). Each run can be found as a specific split in each configuration, the split being named using the timestamp of the run.The "train" split is always pointing to… See the full description on the dataset page: https://huggingface.co/datasets/open-llm-leaderboard/CohereForAI__c4ai-command-r-plus-08-2024-details.MPT-7b-c4mx7-c48a20e1f3fineweb2hq-vs-c4This dataset includes 5000 rows per language from each of two sources: the higher-quality epfml/FineWeb2-HQ
and the lower-quality allenai/c4. The data is split 80/20 into training and test sets.
Languages were carefully chosen to ensure balanced representation across both splits:
Arabic, Chinese, Czech, Danish, Dutch, French, German, Greek, Hungarian, Indonesian, Italian, Japanese, Persian, Polish, Portuguese, Russian, Spanish, Swedish, Turkish, and Vietnamese.
c4-ko-cleaned학교 점심시간 때 할 거 없어서 만든 c4를 정제한 데이터입니다. 다 하면 컴퓨터가 감당 못 할 거 같아서 전체 데이터의 1/10만 진행하였으며 아마 품질은 안 좋을 겁니다.
파일 크기: 약 3gb
데이터 수: 1847023