hotchpotch/multilingual_cc_news
hotchpotch/multilingual_cc_news Dataset Summary This dataset republishes multilingual CC-News data in a Hugging Face friendly layout with one subset per language. Source and transformation Original source datasets on the Hugging Face Hub: CloverSearch/cc-news-mutlilingual intfloat/multilingual_cc_news The intfloat version provides a loading script, but it can be difficult to use directly via the datasets library because it pulls raw JSONL files… See the full description on the dataset page: https://huggingface.co/datasets/hotchpotch/multilingual_cc_news.
hotchpotch/multilingualccnews
Dataset Summary
This dataset republishes multilingual CC-News data in a Hugging Face friendly layout with one subset per language.
Source and transformation
Original source datasets on the Hugging Face Hub:
The intfloat version provides a loading script, but it can be difficult to use directly via the datasets library because it pulls raw JSONL files and relies on a custom builder. This repo republishes the same article-level content as pre-sharded Parquet subsets for easier loading. The text content is not semantically modified; the main change is packaging/layout.
Data Fields
- title: string
- maintext: string
- url: string
- date_publish: string
How to use this dataset
Each language is a dataset config. Load one language at a time:
from datasets import load_dataset
# Single language
train = load_dataset("hotchpotch/multilingual_cc_news", "af", split="train")
# Another language
train_ja = load_dataset("hotchpotch/multilingual_cc_news", "ja", split="train")License
This dataset follows the license and usage terms of the original CC-News sources. The immediate upstream dataset cards are CloverSearch/cc-news-mutlilingual and intfloat/multilingual_cc_news. Because the content is derived from Common Crawl news data and source news articles, downstream users should also respect the applicable source-site and Common Crawl terms.
References
No dedicated paper is listed by the source dataset cards. For the CC-News dataset announcement, see: