zesen-kth/tiny-llm

tiny-llm retained training evidence

1782 seed/horizon records across 594 configurations: cosine-to-zero and WSD schedules, synchronous and four- and eight-worker decentralized training, at 20 to 160 global tokens per parameter, for 20.4M-parameter models on C4. Snapshot 2026-09-17.

This mirrors doc/data/current-training/ in WangZesen/tiny-llm. The website built from it is at <https://wangzesen.github.io/tiny-llm/>.

Layout

Retained artifacts are grouped into gzipped JSONL containers, one line per artifact:

{"path":"logs/cosine/sync/h20/lr-0.01-b1-0.9-b2-0.99/s42/metrics.jsonl",
  "bytes":71410,"sha256":"...","text":"<the artifact verbatim>"}

sources.json is the index: it maps every artifact to its container and records the digest of the original file. Metrics containers concatenate one gzip member per line, so offset and compressedBytes address a single run's log directly; every other container is a single stream. Placement is a pure rule, so a misfiled container is detectable.

What is retained, and what is not

9836 artifacts are byte-identical to the files written during training. The 1296 metrics.jsonl logs had two members deleted from every train event: local_grad_norms, local_losses. These are per-worker diagnostics for the packed models; no published figure, table or curve reads them, and they were 37% of the logged bytes.

The transform is a deletion, not a re-serialization: no other byte of any line, and no line, was changed, added or reordered. sources.json.sha256 remains the digest of the original file and can no longer be recomputed from this copy for those logs — but for the cosine runs it stays independently attested by the retained receipt.json, which records the trainer's own hash of metrics.jsonl and is checked on every verification run. The complete pre-repack bundle, with the arrays intact, is this repo at revision per-file-original.

Verify and regenerate

uv run python scripts/export_publication.py --check --bundle hf://zesen-kth/tiny-llm

No GPU, campaign directory or checkpoint is needed. Verification checks every retained byte against sources.json and every derived file against checksums.json.