JoeyLLM/canada-dataset-1b

🇨🇦 Canada Web Text — 1B-token Sample 🍁

A 1-billion-token representative sample of a much larger cleaned Canadian web-text corpus derived from Common Crawl. This sample is released alongside the JoeyLLM project's ongoing research into regional English language models. 🌐

The full 210B-token corpus is not publicly released due to its size, operational cost, and intended controlled use in research and model development. 🔒 Researchers seeking access to the full corpus for non-commercial testing or academic collaboration may contact the project leads using the details in the Dataset Card Contact section.

📊 Dataset Summary 📌

The sample represents approximately 0.47% of the full internal Canadian corpus by token count, while preserving coverage across the Common Crawl dumps and years represented in the full cleaned corpus. ⏱️

🎯 Intended Uses

💡 Direct Use

• —Pre-training and continued pre-training of language models on Canadian-domain text. 🤖
• —Domain-adaptation experiments involving Canadian English usage, place names, institutions, and topics. 📝
• —Research into Common Crawl-derived language-model datasets. 🔬
• —Inspection and reproducibility of JoeyLLM data pipeline outputs. ⚙️

🚫 Out-of-Scope Use

• —Extracting personal information or deanonymizing individuals. 🕵️‍♂️
• —Using the unreleased 210B-token corpus without explicit authorization. 💼
• —Training models for malicious use, hate speech, harassment, or other harmful applications. 🛑

🧱 Dataset Structure 🗂️

Each row represents one cleaned web document or document-like text segment.

🏗️ Dataset Creation

🔍 Curation Rationale

Regional linguistic nuances — including Canadian spelling, place names, institutions, public services, media references, and local web conventions — are often diluted in global web-scale datasets. JoeyLLM provides targeted regional English web-text samples to support research into foundation models with stronger Canadian and regional coverage. 🏙️

🌍 Source

The dataset was derived from Common Crawl using a FineWeb-style web-text processing pipeline.

Documents were selected as Canadian web text by the upstream country-attribution stage of the JoeyLLM pipeline. Country attribution may use signals such as top-level domains, URL/domain features, crawl metadata, and content-derived features.

This dataset should be interpreted as Canada-attributed web text, not necessarily text authored by Canadians or officially published in Canada.

🎲 Sampling Methodology

The sample was produced using stratified random sampling by Common Crawl dump, proportional to each dump's token count.

• —A per-dump token quota was allocated proportional to each dump's share of total corpus tokens.
• —Documents were selected in a reproducible random order.
• —Documents were appended in their entirety to avoid mid-text truncation.
• —Because documents are not truncated, the realised token count may slightly overshoot the 1B-token target.

🧹 Cleaning Pipeline

The dataset was processed using a FineWeb-style pipeline including:

• —Language filtering: documents retained only when classified as English with sufficient language confidence. ✨
• —Quality filtering: heuristics to reduce low-quality pages, boilerplate, repetitive text, navigation text, and obvious extraction artefacts. 🧽
• —Country attribution: targeted selection using signals such as .ca domains, URL features, crawl metadata, and Canadian-specific content signals. 📍
• —Deduplication: MinHash-style deduplication applied at the corpus level. ✂️

🛡️ Personal and Sensitive Information

This dataset is derived from public web crawls and may contain names, contact details, opinions, offensive content, copyrighted text, or other sensitive material. No dedicated PII masking was performed. ⚠️

Users should apply additional filtering, redaction, and safety review before using this dataset in production systems or public-facing models.

🚀 Loading the Dataset 💻

from datasets import load_dataset

# Load the full 1B-token sample
ds = load_dataset("JoeyLLM/Canada-dataset-1b", split="train")

print(ds)
print(ds[0]["text"][:500])

For streaming, which is recommended for rapid inspection:

from datasets import load_dataset

ds = load_dataset("JoeyLLM/Canada-dataset-1b", split="train", streaming=True)

for ex in ds.take(3):
    print(ex["url"], ex["token_count"])

⚠️ Limitations and Known Issues

This dataset is derived from public web crawl data and inherits the usual limitations of Common Crawl-derived corpora.

Known limitations include:

• —Heuristic country attribution. Canada attribution is based on automated signals and may contain false positives or non-Canadian content.
• —Web-text noise. Boilerplate, navigation text, advertisements, duplicate fragments, low-quality pages, and formatting artefacts may remain.
• —Residual duplication. Deduplication may not remove all near-duplicates.
• —Potential personal information. Public web data may contain personal names, contact details, or other sensitive material.
• —Copyright and source terms. The underlying text originates from public web pages and may remain subject to the rights and terms of the original publishers.
• —Not balanced by domain or genre. The dataset reflects the distribution of selected web crawl data rather than a deliberately balanced linguistic corpus.

📜 License ⚖️

The dataset card, metadata, selection, and processing outputs are released under CC BY 4.0.

The underlying text is derived from publicly crawled web pages via Common Crawl and may remain subject to the rights, licences, and terms of the original publishers. Users are responsible for ensuring that their downstream use complies with applicable law and source terms.

📚 Citation ✒️

A citation entry for the JoeyLLM project paper will be added once available. Until then, please cite this dataset card by URL:

@misc{joeyllm_canada_1b,
  title        = {Canada Web Text -- 1B-token Sample},
  author       = {JoeyLLM Team},
  year         = {2026},
  howpublished = {https://huggingface.co/datasets/JoeyLLM/Canada-dataset-1b}
}

🙏 Acknowledgements 🤝

Built using Common Crawl data and a FineWeb-style processing pipeline. While dataset selection, cleaning, sampling, and publication were carried out by the JoeyLLM team, this project would not have been possible without the invaluable tools, feedback, and ongoing support of the broader open-source AI community. We extend our deepest gratitude to all open-source contributors and researchers whose collaborative efforts continue to drive this field forward. 🌟

<a id="contact"></a>

📬 Dataset Card Contact ✉️

For inquiries regarding research access to the full 210B-token Canadian corpus for non-commercial testing or academic collaboration, please contact:

Matthew Altenburg AI Scientist & Lead Researcher, JoeyLLM matthew.altenburg@anu.edu.au Backup: mattaltenburg@gmail.com