Abzalbek89/kk-tokenizer-fertility-baseline
Kazakh Tokenizer Fertility Baseline Reproducible fertility benchmark of subword tokenizers on the Kazakh language. Companion artifact for the paper "Tokenizer Optimization for Kazakh Small Language Models" (in preparation, target: ACM TALLIP). Headline numbers Tokenizer Fertility 🥇 Best overall kk-bpe-32k 1.679 🚨 Worst GPT-4 (cl100k) 5.895 GPT-4 penalty GPT-4 (cl100k) is 3.51× worse than the best Kazakh-trained tokenizer → The custom Kazakh… See the full description on the dataset page: https://huggingface.co/datasets/Abzalbek89/kk-tokenizer-fertility-baseline.
Exp4 MorphScore (morpheme-boundary faithfulness)
Exp4 MorphScore (morpheme-boundary faithfulness)
Exp4 MorphScore (morpheme-boundary faithfulness)
Exp4 MorphScore (morpheme-boundary faithfulness)
Exp4 MorphScore (morpheme-boundary faithfulness)
KazNERD metrics unigram
KazNERD metrics morphbpe
KazNERD metrics morphbpe
KazNERD metrics bpe
KazNERD metrics bpe
KazNERD metrics unigram
add PUSH_MODEL flag for multi-seed runs
KazQAD metrics bpe
KazQAD metrics unigram
KazQAD metrics morphbpe
KazQAD metrics unigram
KazQAD metrics morphbpe
KazQAD metrics bpe
upload finetune_kazqad.py
KazNERD metrics unigram
KazNERD metrics bpe
KazNERD metrics morphbpe
upload finetune_kaznerd_v2 (loads from GitHub)
upload finetune_kaznerd.py
Upload pretrain_metrics.json with huggingface_hub
Upload pretrain_metrics.json with huggingface_hub
Upload pretrain_metrics.json with huggingface_hub
Add bpe pretrain metrics
v2 streamlined pretrain (200K docs, 25K steps, streaming tokenize)
Add KazNERD finetune script
Fix: num_proc=1 to avoid multiprocessing deadlock
Fix Trainer tokenizer arg for transformers 4.46+
Add Exp 3 pretraining script
Update master README with full ranking and methodology
Add v3 morphology-aware results
Fix last len(generator) bug at line 389
Speed up segmentation: dict lookup instead of per-call viterbi
Cache trained Morfessor model
Update master README with full ranking and methodology
Fix len(generator) bug + cache morfessor.bin on HF
Add experiment 2 (morph-aware)
Fix: re-measure SentencePiece tokenizers with SentencePieceProcessor directly
Add fix script for SP tokenizers
Upload status.json with huggingface_hub
Upload folder using huggingface_hub
Upload status.json with huggingface_hub
Upload status.json with huggingface_hub
Upload status.json with huggingface_hub
Upload status.json with huggingface_hub
Upload status.json with huggingface_hub