JoeyLLM/uk-dataset-1b

🇬🇧 UK Web Text — 1B-token Sample ☕

A 1-billion-token representative sample of a much larger cleaned United Kingdom web-text corpus derived from Common Crawl. This sample is released alongside the JoeyLLM project's ongoing research into regional English language models. 🌐

The full 735B-token corpus is not publicly released due to its size, operational cost, and intended controlled use in research and model development. 🔒 Researchers seeking access to the full corpus for non-commercial testing or academic collaboration may contact the project leads using the details in the Dataset Card Contact section.

📊 Dataset Summary 📌

The sample represents approximately 0.14% of the full internal UK corpus by token count, while preserving coverage across the Common Crawl dumps and years represented in the full cleaned corpus. ⏱️

🎯 Intended Uses

💡 Direct Use

• —Pre-training and continued pre-training of language models on UK-domain text. 🤖
• —Domain-adaptation experiments involving British English usage, place names, institutions, and topics. 📝
• —Research into Common Crawl-derived language-model datasets. 🔬
• —Inspection and reproducibility of JoeyLLM data pipeline outputs. ⚙️

🚫 Out-of-Scope Use

• —Extracting personal information or deanonymizing individuals. 🕵️‍♂️
• —Using the unreleased 735B-token corpus without explicit authorization. 💼
• —Training models for malicious use, hate speech, harassment, or other harmful applications. 🛑

🧱 Dataset Structure 🗂️

Each row represents one cleaned web document or document-like text segment.

🏗️ Dataset Creation

🔍 Curation Rationale

Regional linguistic nuances — including British spelling, place names, institutions, public services, media references, and local web conventions — are often diluted in global web-scale datasets. JoeyLLM provides targeted regional English web-text samples to support research into foundation models with stronger UK and regional coverage. 🏙️

🌍 Source

The dataset was derived from Common Crawl using a FineWeb-style web-text processing pipeline.

Documents were selected as United Kingdom web text by the upstream country-attribution stage of the JoeyLLM pipeline. Country attribution may use signals such as top-level domains, URL/domain features, crawl metadata, and content-derived features.

This dataset should be interpreted as UK-attributed web text, not necessarily text authored by UK citizens or officially published in the United Kingdom.

🎲 Sampling Methodology

The sample was produced using stratified random sampling by Common Crawl dump, proportional to each dump's token count.

• —A per-dump token quota was allocated proportional to each dump's share of total corpus tokens.
• —Documents were selected in a reproducible random order.
• —Documents were appended in their entirety to avoid mid-text truncation.
• —Because documents are not truncated, the realised token count may slightly overshoot the 1B-token target.

🧹 Cleaning Pipeline

The dataset was processed using a FineWeb-style pipeline including:

• —Language filtering: documents retained only when classified as English with sufficient language confidence. ✨
• —Quality filtering: heuristics to reduce low-quality pages, boilerplate, repetitive text, navigation text, and obvious extraction artefacts. 🧽
• —Country attribution: targeted selection using signals such as .uk domains, URL features, crawl metadata, and UK-specific content signals. 📍
• —Deduplication: MinHash-style deduplication applied at the corpus level. ✂️

🛡️ Personal and Sensitive Information

This dataset is derived from public web crawls and may contain names, contact details, opinions, offensive content, copyrighted text, or other sensitive material. No dedicated PII masking was performed. ⚠️

Users should apply additional filtering, redaction, and safety review before using this dataset in production systems or public-facing models.

🚀 Loading the Dataset 💻

from datasets import load_dataset

# Load the full 1B-token sample
ds = load_dataset("JoeyLLM/UK-dataset-1b", split="train")

print(ds)
print(ds[0]["text"][:500])

For streaming, which is recommended for rapid inspection:

from datasets import load_dataset

ds = load_dataset("JoeyLLM/UK-dataset-1b", split="train", streaming=True)

for ex in ds.take(3):
    print(ex["url"], ex["token_count"])

⚠️ Limitations and Known Issues

This dataset is derived from public web crawl data and inherits the usual limitations of Common Crawl-derived corpora.

Known limitations include:

• —Heuristic country attribution. UK attribution is based on automated signals and may contain false positives or non-UK content.
• —Web-text noise. Boilerplate, navigation text, advertisements, duplicate fragments, low-quality pages, and formatting artefacts may remain.
• —Residual duplication. Deduplication may not remove all near-duplicates.
• —Potential personal information. Public web data may contain personal names, contact details, or other sensitive material.
• —Copyright and source terms. The underlying text originates from public web pages and may remain subject to the rights and terms of the original publishers.
• —Not balanced by domain or genre. The dataset reflects the distribution of selected web crawl data rather than a deliberately balanced linguistic corpus.

📜 License ⚖️

The dataset card, metadata, selection, and processing outputs are released under CC BY 4.0.

The underlying text is derived from publicly crawled web pages via Common Crawl and may remain subject to the rights, licences, and terms of the original publishers. Users are responsible for ensuring that their downstream use complies with applicable law and source terms.

📚 Citation ✒️

A citation entry for the JoeyLLM project paper will be added once available. Until then, please cite this dataset card by URL:

@misc{joeyllm_uk_1b,
  title        = {UK Web Text -- 1B-token Sample},
  author       = {JoeyLLM Team},
  year         = {2026},
  howpublished = {https://huggingface.co/datasets/JoeyLLM/UK-dataset-1b}
}

🙏 Acknowledgements 🤝

Built using Common Crawl data and a FineWeb-style processing pipeline. While dataset selection, cleaning, sampling, and publication were carried out by the JoeyLLM team, this project would not have been possible without the invaluable tools, feedback, and ongoing support of the broader open-source AI community. We extend our deepest gratitude to all open-source contributors and researchers whose collaborative efforts continue to drive this field forward. 🌟

<a id="contact"></a>

📬 Dataset Card Contact ✉️

For inquiries regarding research access to the full 735B-token UK corpus for non-commercial testing or academic collaboration, please contact:

Matthew Altenburg AI Scientist & Lead Researcher, JoeyLLM matthew.altenburg@anu.edu.au Backup: mattaltenburg@gmail.com