wordingone/ember-corpus

Ember Corpus

The off-device mirror of the Ember training corpus: what was acquired, what was admitted, and what was tokenized. It exists so that a single disk failure on the training host does not destroy the corpus and every token derived from it.

Layout

The top level is the stage of the pipeline, and every artifact sits at exactly one stage.

tokens/ is keyed by tokenizer digest and then dataset-version digest, so shards produced under a different tokenizer or a different admitted set never share a directory. The shard ledger sits beside the shards it describes, because shards without their ledger are bytes with no provenance.

Licences

This dataset is not under a single licence. Redistribution obligations vary by source, and they differ between what is acquired here and what is tokenized here. Both are stated, because conflating them is how a share-alike obligation goes missing.

What is acquired (sources/)

The source's own declared licence, read from its acquisition manifest.

licenses/ATTRIBUTION.md carries the per-source attribution the CC-BY and CC-BY-SA rows require.

What is tokenized (tokens/)

Licence is resolved per object, not per source, for all 116,366 staged objects — 100% covered, nothing assumed and nothing inherited.

No share-alike material is in the tokenized set. The stackexchange CC-BY-SA-4.0 bytes are acquired and mirrored but were not admitted into this dataset version, so no shard span carries a share-alike obligation.

How an object's licence is selected, stated because the rule is load-bearing and an earlier version of it was wrong. Identical bytes legitimately appear in several repositories under different declarations — 949 of these objects are in exactly that state, declared one way by one repository and another way by another. A content digest therefore does not identify a licence. Each object's SPDX identifier is read from the partition receipt that object's own staging row is bound to, never from whichever receipt happens to declare the same digest. Conflicting declarations are recorded rather than collapsed, and an object whose own binding receipt does not declare it is reported as uncovered rather than given a borrowed value.

Known gaps, stated rather than smoothed over

• —Token shards do not yet carry a per-span licence field. Licence is a resolved property of every staged object, but once text becomes tokens the span itself cannot name its source's obligation without walking back to the staging manifest. Making licence a recorded property of every ledger span is tracked work, not a completed property of this mirror.
• —One acquired source has no declared licence in its manifest and is marked unresolved above rather than given an assumed one. It is not part of the tokenized set.
• —`ember-corpus-v1-lane-285/` is a June-era snapshot of scripts, docs, and run receipts. It is source code rather than corpus data and does not belong in a dataset repository; it is retained only until its removal is confirmed.

Held-out evaluation material is separated by path

objects/ carries the content-addressed blob stores behind the admitted catalog. Four of its eleven tranches are protected evaluation partitions, and they live under objects/heldout/ while training tranches live under objects/train/.

This is a structural separation, not a naming convention. A backup that stores evaluation material beside training material invites precisely the mistake the leakage assertion exists to prevent, and a consumer reading the tree should not have to read this file to avoid it. Anything under objects/heldout/ is not training data and must not be admitted into a training mixture.

Each tranche is one gzip archive rather than loose blobs. The stores hold roughly 600,000 sha256-named files between them, which is past what a repository handles usefully at rest and past what the commit endpoint will accept in one operation. The trade is that a restore untars rather than reading blobs in place; state/hf-mirror-tier1-manifest.json on the training host records each archive's sha256 and member count so a restore is verified rather than assumed.

Two of the four blob stores are byte-identical. Blobs are sha256-named, so name-set equality is content equality, and the two github-license-partition stores have 213,723 names each with a symmetric difference of zero in both directions. Only one is mirrored.

Scope

Text is the only modality currently materialized into shards. Audio is acquired (sources/librispeech-seed) but not admitted.

Language composition, measured 2026-09-09 on a seeded 4,000-document sample of the admitted set (method and artifacts: langdetect, seeded, per document from a 20 KB prefix):

Two things this measurement makes plain, and both belong here rather than in a footnote:

• —52.7% of the admitted documents are not natural-language prose — source code, numeric tables, generated bindings, materials and configuration files. Any figure computed over the whole set is a figure about a majority-non-prose population.
• —A raw language distribution over this set overstates non-English content. A per-document detector assigns a language to anything it is handed: the largest apparent Catalan document, at confidence 1.000, is a tab-separated table of floats. Genuine non-English prose is a thin tail of single-digit document counts, so no multilingual claim should be drawn from this corpus.

Writing systems are measured, languages are not. A Unicode script census over all 116,366 staged objects covers 806,771,068 script-bearing characters: 99.640% Latin, 0.247% Han, 0.084% other, 0.018% Thai, 0.006% Cyrillic, 0.001% Greek, 0.001% Hangul (9,020 characters across 31 files), with Hiragana, Katakana, Arabic, Hebrew and Devanagari present only in trace amounts.

Latin script covers dozens of languages, so 99.640% Latin is an upper bound on English and not a measurement of it — the language table above measures inside that bound, and the two are consistent: 95-98% English against a 99.6% Latin ceiling.