ruggsea/infini-news-corpus

INFINI-NEWS Corpus

🔎 Search this corpus online: query it with sub-second full-text search and n-gram counts — in the browser or via a public, keyless REST API, no download required — at [infini-news.uni-graz.at](https://infini-news.uni-graz.at/) (API reference).

A multilingual news corpus extracted from Common Crawl CC-News WARC files. One row per article, with body text extracted via trafilatura, WARC provenance, and derived metadata (publish date, language, topic, byte hashes) in a single flat schema. Covers Aug 2016 – Apr 2026.

At a glance

Articles per year

Top languages

Detected with two independent identifiers — GlotLID (Kargaran et al. 2023) and CommonLingua (Pleias / GSMA 2026); counts below are from CommonLingua (334-class, byte-level).

Schema

Quick start

from datasets import load_dataset
ds = load_dataset("ruggsea/infini-news-corpus",
                  data_files="data/year=2022/month=06/part-*.parquet",
                  split="train", streaming=True)
# text_xxhash64 is uint64; HF datasets can't auto-cast it to int64.
ds = ds.remove_columns(["text_xxhash64"])
for row in ds.take(3):
    print(row["title"], "|", row["lang"], "|", row["iptc_topic"])

INSTALL httpfs; LOAD httpfs;
-- Use the hf:// scheme so the part-*.parquet glob resolves (the
-- generic https:// URL forbids globs unless you SET
-- allow_asterisks_in_http_paths = true).
SELECT iptc_topic, COUNT(*) AS n
FROM read_parquet(
  'hf://datasets/ruggsea/infini-news-corpus/data/year=2022/month=06/part-*.parquet'
)
WHERE lang = 'eng' AND text_length > 500
GROUP BY iptc_topic ORDER BY n DESC LIMIT 5;

Companion FM-index

Pre-built FM-index over the same corpus, with sub-10 ms exact substring count over all 1.36 B articles: `ruggsea/infini-news-index`.

Dataset creation

Curation rationale

A broad, multilingual news corpus that is queryable in milliseconds via the companion FM-index (`ruggsea/infini-news-index`), with stable provenance back to the original WARCs and lightweight derived metadata inline (language, topic) so downstream filtering doesn't require a secondary classifier pass. Together the corpus + index let you do exact substring search and count over 1.36 B articles without scanning the parquet.

Source data

The corpus is built from Common Crawl News (CC-News) WARC files — the weekly archival snapshots that Common Crawl publishes for the news subset of its crawls. WARC (Web ARChive, ISO 28500) is the standard binary container format used by web archives; each WARC file holds a stream of HTTP responses with their full headers and payloads, exactly as the crawler observed them. CC-News covers Aug 2016 — Apr 2026 in this release. Common Crawl re-crawls a curated seed list of news domains plus URLs discovered in its main crawls. The raw WARC files live at `commoncrawl/CC-NEWS/`; each row in this dataset preserves the exact warc_filename (file the article was read from), warc_target_uri (the URL crawled), warc_date (capture timestamp), and warc_payload_digest (SHA-1 of the original response body).

Processing pipeline

1. 1.Stream WARC payloads from the CC-News S3 bucket.
2. 2.Extract body text with trafilatura.
3. 3.Tag languages with GlotLID (Kargaran et al. 2023) and CommonLingua (Pleias / GSMA 2026) — both labels stored independently, no winner picked.
4. 4.Classify each article into the IPTC top-17 Media Topic taxonomy with the XLM-RoBERTa-large model from Kuzman & Ljubešić 2025.
5. 5.Write monthly parquet shards with merged metadata, partitioned year=YYYY/month=MM.

Full code: `git.ksolovev.com/kirill/infini-news`.

Terms of use

Access to the corpus is gated. By accepting the access request you commit to using the corpus for academic research purposes only.

The underlying articles were extracted from CommonCrawl CC-News and remain subject to the original publishers' copyright; we redistribute them on the legal basis of research exceptions recognised in most jurisdictions, mirroring the access model used by comparable web-derived corpora. Derived metadata columns (language, topic, extraction provenance) are released under CC-BY-4.0.

The companion FM-index at `ruggsea/infini-news-index` is offered ungated as the open-access query interface; the bulk article text is only available through this gated corpus repo.

Operators of news outlets who want their content excluded from future releases can open a discussion on the Hugging Face dataset page citing the affected url_hostname.

FAIR compliance

This release follows the FAIR data principles (Wilkinson et al. 2016):

• —Findable. Indexed on the Hugging Face Hub, assigned a persistent DOI `10.57967/hf/8606`, and accompanied by this dataset card describing schema, coverage and processing.
• —Accessible. Articles are retrievable over HTTPS through the huggingface_hub client. Access requires institutional affiliation and a stated research purpose (see Terms of use); FAIR explicitly permits authentication. The dataset card itself remains publicly accessible regardless of gating, and the companion FM-index is open for queries without downloading the corpus.
• —Interoperable. Data are in Apache Parquet with the explicit per-column schema above. Identifiers use standard vocabularies: ISO 639-3 language codes, ISO 15924 script codes, the IPTC top-17 Media Topic taxonomy, and the original WARC URNs.
• —Reusable. Provenance is preserved row-by-row (warc_filename, warc_target_uri, warc_payload_digest, warc_date); pipeline versions are captured in trafilatura_version, extraction_version, extraction_timestamp, iptc_method and lang_method. The derived-metadata columns carry CC-BY-4.0; reuse of the article bodies is governed by the Terms of use above.

Citation

@misc{lazzaroni2026infininews,
  author    = {Lazzaroni, Ruggero Marino and Lasser, Jana and Solovev, Kirill},
  title     = {{INFINI-NEWS Corpus}},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/ruggsea/infini-news-corpus},
  doi       = {10.57967/hf/8606}
}