datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
common-crawl-character-countsTraditional-Chinese-Common-Crawl-NOT-CleanedCommon Crawl Dumps that were briefly filtered by keywords to remove bad words and simplified Chinese.
The hash based cleaned dataset can be found here.
Files here are for future usage (downloading from Common Crawl and keyword filtering are very slow)
mind2web-utg-crawl-outputsTraditional-Chinese-Common-Crawl-Filtered
Traditional Chinese C4
Dataset Summary
Data obtained from 2013～2025 Common Crawl.
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant dataset contains both simplified and traditional Chinese, which could be found here.
It was then filtered using a modified list of simplified Chinese characters to obtain this traditional Chinese dataset.
Unfortunately, I don't have enough funding to run a deduplication across… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Traditional-Chinese-Common-Crawl-Filtered.common-crawl-sample
Common Crawl sample
A small unofficial random subset of the famous Common Crawl dataset.
60 random segment WET files were downloaded from Common Crawl on 2024-05-12.
Lines between 500 and 5000 characters long (inclusive) were kept.
Only unique texts were kept.
No other filtering.
Languages
Each text was assigned to one of the language codes using the GCLD3 Python package.
The Chinese texts were classified as either simplified, traditional, or Cantonese using the… See the full description on the dataset page: https://huggingface.co/datasets/agentlans/common-crawl-sample.wdc-common-crawl-embedded-jsonldweb-crawl-2026
Web Crawl 2026
A large-scale web crawl dataset for language model pretraining, collected by the OpenTransformer project.
Dataset Description
This dataset contains text extracted from web pages crawled directly from the internet using custom high-throughput crawlers. All data is freshly scraped.
Data Format
Each record is a JSON line (gzipped) with fields:
text: extracted text content (200-200,000 chars)
url: source URL
domain: source domain
timestamp: crawl… See the full description on the dataset page: https://huggingface.co/datasets/OpenTransformer/web-crawl-2026.3D-dungeon-crawler-video-v2
3D Dungeon Crawler Video v2
32,000 deterministic 28-second observational Unity episodes.
The canonical split contains 16,000 pretrain, 14,000 training,
1,000 test, and 1,000 evaluation episodes.
Unity renders at 512x288 for supersampling. Videos are stored at
256x144, 30 fps, H.264. Training samples every third frame,
yielding 280 frames and an 18x32 visual-token grid per episode.
manifest.jsonl is authoritative for asset paths. Each record points to one MP4 and one
NPZ… See the full description on the dataset page: https://huggingface.co/datasets/osazuwa/3D-dungeon-crawler-video-v2.crawl-my-websiteCrawlPT_dedup
CrawlPT (deduplicated)
CrawlPT is a generic Portuguese corpus extracted from various web pages.
This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022).
The raw version is also available here.
Dataset Details
Dataset is composed by three corpora:
brWaC, C100-PT, OSCAR-2301.
brWaC: a web corpus for Brazilian Portuguese from 120,000 different websites.
C100-PT: Portuguese subset from CC-100. C100 was… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/CrawlPT_dedup.3D-dungeon-crawler-stratified-continuations-v1
Stratified conditional-continuation benchmark (v1)
Complete and frozen.
A frozen evaluation cohort of 4,000 matched pairs (8,000 episodes)
for estimating and comparing
P(M,Y,R | do(X=1), A=1) P(R=1 | do(X=1), A=1)
from a common set of pre-X contexts shared by every model arm and by the
simulator reference. No arm may select a different prefix set.
manifest.jsonl is immutable and audit.json records the result of every
audit required by issue #53, including the exact-quota… See the full description on the dataset page: https://huggingface.co/datasets/osazuwa/3D-dungeon-crawler-stratified-continuations-v1.mirror-eduagarcia__CrawlPT_dedup
CrawlPT (deduplicated)
CrawlPT is a generic Portuguese corpus extracted from various web pages.
This version is deduplicated using MinHash algorithm and Locality Sensitive Hashing, following the approach of Lee et al. (2022).
The raw version is also available here.
Dataset Details
Dataset is composed by three corpora:
brWaC, C100-PT, OSCAR-2301.
brWaC: a web corpus for Brazilian Portuguese from 120,000 different websites.
C100-PT: Portuguese subset from CC-100.… See the full description on the dataset page: https://huggingface.co/datasets/leeaandrob/mirror-eduagarcia__CrawlPT_dedup.Cantonese_Common_Crawl_Filtered
Cantonese Chinese C4
Dataset Summary
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant traditional Chinese dataset can be found here.
This dataset contains data processed with CantoneseDetect.
In CantoneseDetect, you can choose whether to include quotes (i.e. categorise the data as Cantonese even if Cantonese appeared only in quotes).
And I found that a lot of entries came from Wikipedia and LIHKG. If you… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Cantonese_Common_Crawl_Filtered.CrawlPT
CrawlPT
CrawlPT is a generic Portuguese corpus extracted from various web pages.
This is the raw version. Deduplicated version is available here.
Dataset Details
Dataset is composed by three corpora:
brWaC, C100-PT, OSCAR-2301.
brWaC: a web corpus for Brazilian Portuguese from 120,000 different websites.
C100-PT: Portuguese subset from CC-100. C100 was created for training the multilingual Transformer XLM-R, containing two terabytes of cleaned data from 2018 snapshots of… See the full description on the dataset page: https://huggingface.co/datasets/eduagarcia/CrawlPT.CrawlSinger-OS
CrawlSinger-OS
CrawlSinger-OS is a large-scale, open-source singing corpus constructed for
score-native singing voice synthesis. It contains more than 2,300 hours of
processed singing data from multiple public song and singing collections, with
a unified annotation scheme for lyrics, MIDI pitches, symbolic note values,
lyric-to-note alignment, and global tempo.
VocalRender paper
VocalRender code
VocalRender checkpoints
Why CrawlSinger-OS
Modern singing… See the full description on the dataset page: https://huggingface.co/datasets/pymaster/CrawlSinger-OS.common_crawl_pointers_by_collectionChinese-Common-Crawl-Filtered
Traditional Chinese C4
Dataset Summary
Data obtained from 2025-18 and 2025-13 Common Crawl.
Downloaded and processed using code based on another project attempting to recreate the C4 dataset.
The resultant dataset contains both simplified and traditional Chinese.
It was then filtered using a modified list of simplified Chinese characters to obtain another traditional Chinese dataset.
I am still ironning out the process of filtering.
The 2025-13 dataset was… See the full description on the dataset page: https://huggingface.co/datasets/jed351/Chinese-Common-Crawl-Filtered.v2-crawler3D-dungeon-crawler-video-v2-leaky-xor-supplementCrawlPT_dedup_Cleaned📚 CrawlPT Clean — High-Quality Portuguese Corpus
Versão limpa, filtrada e refinada do dataset CrawlPT_dedup
🧼 Visão Geral
Este repositório fornece uma versão limpa, filtrada e padronizada do dataset:
➡️ eduagarcia/CrawlPT_dedup
https://huggingface.co/datasets/eduagarcia/CrawlPT_dedup
A limpeza tem como objetivo criar um corpus de alta qualidade para:
pré-treino contínuo de modelos LLM (Qwen, Mistral, LLaMA, Phi etc.)
melhora de fluência e coerência em português
pesquisas em NLP
geração de… See the full description on the dataset page: https://huggingface.co/datasets/tiagoloeblein/CrawlPT_dedup_Cleaned.common_crawl_pointer_indicesFRENCH-ONLY-Common-Crawl-2026-25ai-crawler-index
AI Crawler Index
150 web crawlers and AI user agents from 74 operators — what each one is for,
what blocking it costs you, and the IP ranges its operator publishes.
Plus a compiled user-agent regex and the union of 1997 IPv4 and 1062 IPv6
prefixes from 15 operator-published range files.
Home: https://www.pathwren.workers.dev/c/huggingface-datasets/ · CC0 · no signup, no key.
What this is, plainly
This is an independent, non-commercial automated project. It is run… See the full description on the dataset page: https://huggingface.co/datasets/pathwren/ai-crawler-index.common-crawl-2026-21
Common Crawl SEO & AEO/GEO Dataset — CC-MAIN-2026-21
Web pages from the May 2026 Common Crawl (CC-MAIN-2026-21) that mention SEO, Answer Engine Optimization (AEO), or Generative Engine Optimization (GEO) — filtered to English content and ranked by Common Crawl Web Graph harmonic centrality and PageRank.
🔎 Interactive Explorer
A Gradio app to search, filter, and analyze this dataset:
https://huggingface.co/spaces/metehan777/cc-seo-explorer
Tabs: SEO raw search ·… See the full description on the dataset page: https://huggingface.co/datasets/metehan777/common-crawl-2026-21.sea-vl_crawling
SEA-VL: A Multicultural Vision-Language Dataset for Southeast Asia
Paper: Crowdsource, Crawl, or Generate? Creating SEA-VL, A Multicultural Vision-Language Dataset for Southeast Asia
Dataset: SEA-VL Collection on HuggingFace
Code: SEA-VL Experiment | SEA-VL Image Collection
What is SEA-VL?
Following the success of our SEACrowd project, we’re excited to announce SEA-VL, a new open-source initiative to create high-quality vision-language datasets specifically for… See the full description on the dataset page: https://huggingface.co/datasets/SEACrowd/sea-vl_crawling.common_crawl_meta_indexes3D-dungeon-crawler-video
3D Dungeon Crawler Video
Deterministic 23-second Unity episodes for observational world-model training. The 10,000 episodes are split into 8,000 train, 1,000 validation, and 1,000 test episodes.
Unity renders at 512x288 for supersampling; each clips/*.mp4 is area-downscaled and stored at 256x144 and 30 fps. Training samples every third frame, yielding 230 frames and an 18x32 tokenizer grid. Matching arrays/*.npz files contain dag_ticks, action_tokens, and final_state.
All… See the full description on the dataset page: https://huggingface.co/datasets/osazuwa/3D-dungeon-crawler-video.newsxlm-mhtml
MHTML Sources for NewsXLM dataset
Columns Explanation
mhtml: MHTML snapshots of pages where wtl-uid and wtl-parent-uid attributes have been added to every element in <body>, following the WTL algorithm.
crawl-astroawanicrawl-google-image
Crawl Google Image
Crawl Google Image using Malay keywords, total 2046313 rows. Done by https://github.com/kurkurzz
Source code at https://github.com/mesolitica/malaysian-dataset/tree/master/crawl/google-image