Datasetpublic

ruggsea/infini-news-corpus

INFINI-NEWS Corpus 🔎 Search this corpus online: query it with sub-second full-text search and n-gram counts — in the browser or via a public, keyless REST API, no download required — at infini-news.uni-graz.at (API reference). A multilingual news corpus extracted from Common Crawl CC-News WARC files. One row per article, with body text extracted via trafilatura, WARC provenance, and derived metadata (publish date, language, topic, byte hashes) in a single flat schema. Covers… See the full description on the dataset page: https://huggingface.co/datasets/ruggsea/infini-news-corpus.

sourceHugging Facecc-by-4.0updated 10d agoView on Hugging Face

39likes43kdownloads

50 commits on main

dbac12210d ago

Point the code link at git.ksolovev.com (repo moved; codeberg is now a mirror)

ruggsea

5b781993mo ago

docs: add live search API reference (infini-news.uni-graz.at) to top of card

ruggsea

ebda3fe5mo ago

Drop trafilatura paper citation from opening sentence

ruggsea

9ec27a25mo ago

Expand Source Data paragraph: explain WARC format (ISO 28500) for readers unfamiliar with it

ruggsea

f8fdc175mo ago

Opening line: extracted from Common Crawl CC-News WARC files

ruggsea

fdb9db15mo ago

BibTeX: reorder authors — Kirill last (senior)

ruggsea

af249395mo ago

Slim Terms to research-purpose-only; fix BibTeX authors to Lazzaroni/Solovev/Lasser

ruggsea

64397f65mo ago

Slim Terms of use to attribution + no-redistribute + takedown (pure CC-BY)

ruggsea

f9d8ed95mo ago

Split Languages row: 1172 GlotLID + 331 CommonLingua (two identifiers, two counts)

ruggsea

951916d5mo ago

Add Terms of use + FAIR compliance sections (manuscript-aligned)

ruggsea

411732b5mo ago

Fix: Languages detected = 1172 (GlotLID ISO 639-3), not 331 (CommonLingua)

ruggsea

ee098d25mo ago

Add dataset_info to YAML: authoritative row counts (HF datasets-server failed 6 of 12 configs)

ruggsea

4d078015mo ago

Fix Quick Start: drop uint64 hash for datasets, use hf:// scheme for DuckDB glob

ruggsea

762b8725mo ago

Rationale emphasises FM-index queryability; per-column schema; drop hash step from pipeline

ruggsea

e9a507d5mo ago

Drop non-standard text-mining task_category

ruggsea

78a212b5mo ago

Expand YAML metadata + add Dataset Creation section

ruggsea

2504cf15mo ago

README: drop redundant "all columns inline" line

ruggsea

58b8dfd5mo ago

README: keep only own bibtex in Citation; inline-cite tools where relevant

ruggsea

7b1fbc75mo ago

README: concrete stats (hostnames, tokens, top languages), reproduction one-liner, citations only

ruggsea

4ed56c85mo ago

README: treat lang+topic as ordinary columns (no special framing)

ruggsea

ad47b005mo ago