fffoivos/apertus-tokenizer-extension

Apertus Greek Tokenizer Extension

This repo has five front-stage artifacts.

Everything else is under supporting-material/.

Greek Extension Tokenizer

greek-extension-tokenizer/ contains ModernGreek-148k, the selected tokenizer for these experiments:

• —base Apertus vocab: 131072;
• —added modern Greek C3 tokens: 17408;
• —total vocab: 148480;
• —tokenizer.json SHA-256: 358ae3f29ac17c99769d6d437339e28657d5fcaed3486f8550feed3d6adfc394.

The original Apertus tokenizer is only used by the Vanilla-* checkpoints as a control. The optional polytonic tokenizer lives under supporting-material/optional-tokenizers/.

Production Modern + Polytonic Tokenizer

greek-modern-polytonic-tokenizer/ contains the production tokenizer selected on 2026-07-29 for the bibliography-cleaned Greek CPT corpus:

• —base Apertus vocabulary: 131072;
• —modern Greek continuation: 17408;
• —selected polytonic continuation: 512;
• —actual vocabulary: 148992 = 256 x 582;
• —external padding entries: 0;
• —tokenizer SHA-256: bbb08e71929b519c5c2362338b0fc6a0e99955cb8fdbf0729ae1311117e6561b.

The appended vocabulary is merge-native: all 512 new entries are produced by 512 dependency-safe, ID-sequential BPE merges. No orphan or dummy alignment entries exist. release_audit.json records the structural and Transformers runtime gates; selection.json and suspicious_token_review.json record the selection and manual suspicious-token review.

Load it with:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "fffoivos/apertus-tokenizer-extension",
    subfolder="greek-modern-polytonic-tokenizer",
)

CPT Training Dataset

cpt-training-dataset/ describes CPT-7B-mix, built from:

• —fffoivos/glossapi-greek-nanochat-pretraining-dataset;
• —nanochat internal dedup metadata;
• —Apertus-overlap drop overlay from fffoivos/apertus-c3-dedup-audit-dedup-20260519t010924z;
• —non-Greek replay, code, and math.

Bulk recipe: 70% Greek, 24% non-Greek replay, 4% code, 2% math.

Measured source-token count with the selected tokenizer:

See cpt-training-dataset/token-counts.json for the exact run metadata.

Experiment Checkpoints

experiment-checkpoints/ contains one folder per checkpoint we care about. Each has a manifest.json + README.md with the source Clariden path. 5B is the bakeoff final endpoint; 2B and 3.5B are iso-token snapshots.

Large model weights are uploaded to Hugging Face in these folders. They are not mirrored in the GitHub source repository.

Benchmark Evals

The current result anchor is:

benchmark-evals/bakeoff-final/    # 5.0B endpoint, canonical headline
benchmark-evals/3.5B-comparison/  # 3.5B iso-token (Vanilla/ReTok/TD)
benchmark-evals/native-greek-suite/ # vetted native-Greek decision suite

Loss-reading rule: raw Megatron lm loss is per-token cross entropy and is not comparable between the original 131,072-token Vanilla tokenizer and the 148,480-token extended tokenizer arms. Cross-arm loss conclusions use heldout BPB from the tokenizer-fair eval jobs plus downstream benchmark scores. Older files may call BPB BPC; that is a legacy bits-per-byte label, not bits per character. Raw training loss is only a health and within-arm trace unless dense bpb training logs are present.

Greek aggregate rule: the Greek-specific headline now uses benchmark-evals/native-greek-suite/, which includes vetted native MCQ tasks and excludes explicit MT diagnostics. The older bakeoff-final/ Greek aggregate is a fallback lm-eval slice and should not be treated as the native-Greek selection headline.

At 5.0B (bakeoff-final/):

Reading for the older fallback suite: TokenDistil-5B leads all three downstream aggregates over Vanilla-5B. Vanilla-5B retains tokenizer-fair heldout BPB leadership; gap narrowing (0.110 → 0.027 over the bakeoff).

Native-Greek suite reading:

For Greek-specific selection, Vanilla is ahead of TokenDistil on the native MCQ headline, while Apertus-Base remains above all continued checkpoints. Full native-suite tables are in benchmark-evals/native-greek-suite/.

Caveat — the bakeoff was not rule-bound. The pre-commit decision-rule thresholds from old_experiments_plan.md v0.12 §10 Q8 (X / Mprogress / Mext / M_van / T) were never locked before results came in. The 5B headline above is an honest description of the numbers, not an adjudicated winner. See:

supporting-material/provenance/decisions/PLAN_VS_RESULTS_RECONCILIATION_20260526.md
supporting-material/provenance/decisions/CPT_MASTER_20260526.md

Supporting Material

Source

Runnable scripts live in GitHub:

https://github.com/fffoivos/glossapi-tokenizer-extension/tree/main/subprojects/03_apertus_extension_and_embedding_adaptation