sliderforthewin/lt-asr-lm-corpora

Lithuanian ASR Language Model Training Corpora

Text corpora for building token-level n-gram language models for Lithuanian ASR beam search decoding. Used with sliderforthewin/parakeet-tdt-lt.

Corpora

Best LM recipe

The domain 5-gram (Wikipedia + training manifests only) achieves 9.51% WER on CV25 LT — better than using all 61M sentences. Domain match matters more than corpus size.

Adding Europarl helps VoxPopuli (parliamentary speech benchmark).

See in the finetuneparakeet repo for the full source guide and build instructions.

Usage

--- Building KenLM tools (one-time setup) --- -- Looking for clockgettime in c -- Looking for clockgettime in c - found -- Configuring done (4.4s) -- Generating done (0.2s) -- Build files have been written to: /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/tools/kenlm/build [ 98%] Linking CXX executable ../../bin/lmplz [ 98%] Built target lmplz 12 warnings generated. [100%] Linking CXX executable ../../bin/countngrams [100%] Built target countngrams Built: /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/tools/kenlm/build/bin/lmplz === Build parakeet token-level 5-gram LM from HF corpora === repo: sliderforthewin/lt-asr-lm-corpora output: data/lm/lttoken5gram_v2.arpa lmplz: /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/tools/kenlm/build/bin/lmplz lmplz mem: 50% prune: 0 2 2 2 2

--- Step 1: Downloading corpora from HuggingFace --- /Users/jasontitus/experiments/finetuneparakeet/finetuneparakeet/data/lm/hf-corpora Downloaded to data/lm/hf-corpora/corpora/

--- Step 2: Decompressing and concatenating corpora --- Combining into data/lm/ltcombinedhf.txt... +cc100.sentences.txt.zst... 56892247 lines +opensubtitles.sentences.txt.zst... 1269060 lines +wikipedia.sentences.txt.zst... 2938541 lines Total: 61099848 lines, 10801 MB

--- Step 3: Stream-tokenize + build 5-gram LM via lmplz --- Piping 61099848 sentences through parakeet BPE into lmplz. Memory: ~4 GB (tokenizer) + 50% (lmplz sort buffers)