Modelpublic

leuconoe/litert-lm-unity-quantized

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes68downloads

Model Card

Community Release — Project-Quantized LiteRT Models (2026-07-23)

Quantized ASR/LLM artifacts produced by the LiteRT-LM-Unity project, prepared for upstream contribution (e.g., litert-community on Hugging Face). All files were validated before inclusion; validation data below.

Toolchain / provenance

—Quantizer: ai-edge-quantizer 0.8.0 (post-training, dynamic-range).
—Recipes:
—i8 = recipe.dynamic_wi8_afp32() (int8 weights channelwise, fp32 activations).
—i4 = recipe.dynamic_wi4b64_afp32() (int4 weights, blockwise-64, fp16 scales) with i8 overrides on sensitive scopes (encoder / embedding-logits table / selected projections). Pure full-scope int4 was tested and REJECTED for several models (Korean transcription errors); the mixed recipes below are what passed.
—int4 channelwise (wi4c) and blockwise-32 (wi4b32) are not used in this collection: in our tests wi4c degraded accuracy sharply and wi4b32 produced an immediate EOS. Do not regenerate with them.
—Whisper medium / large-v3 / large-v3-turbo tflite graphs were exported by this project from the openai checkpoints (30 s fixed window, encode/decode split signatures matching the litert-community whisper-tiny/base graph interface; large-v3 family = 128 mel bins / 51866 vocab). f32 source exports (3–6 GB) are retained separately and available on request.
—Qwen2.5 .litertlm files were produced by unpack → quantize → repack: litert-lm-peek dump → dynamic_wi4b64_afp32 on the f32 tflite section → rebuild with litert-lm-builder 0.14.0 (tokenizer + LlmMetadata mirrored from the official q8 release). Source: official litert-community f32 releases.
—Runtime compatibility: .litertlm format 1.5.0; validated on LiteRT-LM v0.14.0 (Windows x86_64 CPU/GPU and Android arm64, Snapdragon 865-class device).

Files and validation

Accuracy-claim scope: "exact" below means the specific gate clips used during quantization validation (sentence-length Korean/English test audio), not general Korean capability. Per-tier CER across the full 10-clip matrix — including short voice commands, where small tiers degrade sharply — is published in the project repo's docs/benchmarks/asr-model-matrix.md.

Validation clips: Korean "2025년 3월 5일 전술평가 결과 보고" sentence + English equivalent + short Korean voice commands ("볼륨 업" etc.). CER computed against reference transcripts (punctuation-normalized). Full matrix: docs/benchmarks/asr-model-matrix.md in the LiteRT-LM-Unity repo.

File	Size	Base model	Recipe	Validation
`whisper-tiny/whisper_tiny_30s_i4.tflite`	35 MB	openai/whisper-tiny (via litert-community f32)	wi4b64 + i8 encoder+embed table	en gate clips exact; lower Korean accuracy — 3/9 exact, CER ko 0.281 (misses `볼륨 업`→`별념`, `음량 증가`→`능량 증가`). Not recommended for Korean; see the ASR matrix in the project repo
`whisper-base/whisper_base_30s_i8.tflite`	74 MB	openai/whisper-base (via litert-community f32)	wi8	transcripts byte-identical to f32; ~1.6× faster
`whisper-base/whisper_base_30s_i4.tflite`	44 MB	same	wi4b64 (full scope OK for base)	ko/en gate clips exact (sentence-length audio); sub-1.2 s clips remain unreliable at this tier
`whisper-medium/whisper_medium_30s_i8.tflite`	794 MB	openai/whisper-medium (project export)	wi8 (DRQ)	7/9 exact, CER ko 0.042 / en 0.000
`whisper-medium/whisper_medium_30s_i4.tflite`	634 MB	same	wi4b64 mixed (L1)	same transcripts as i8
`whisper-large-v3/whisper_large_v3_30s_i8.tflite`	1.6 GB	openai/whisper-large-v3 (project export, 128-mel)	wi8 (DRQ)	CER 0.000/0.000, 7/9 exact (spacing-only misses)
`whisper-large-v3/whisper_large_v3_30s_i4.tflite`	1.1 GB	same	wi4b64 mixed (mixD)	CER 0.000/0.000; ~20 % faster than i8
`whisper-large-v3-turbo/whisper_large_v3_turbo_30s_i8.tflite`	1.1 GB	openai/whisper-large-v3-turbo (project export, 128-mel)	wi8	5–6/9 exact, CER ko 0.000
`whisper-large-v3-turbo/whisper_large_v3_turbo_30s_i4.tflite`	721 MB	same	wi4b64 mixed (mixD)	best tier overall: 8/9 exact, CER 0.000/0.000; device-validated (Snapdragon 865, shape-based decode binding)
`qwen2.5-0.5b-instruct/Qwen2.5-0.5B-Instruct_wi4b64_ekv1280.litertlm`	265 MB	litert-community/Qwen2.5-0.5B-Instruct f32	wi4b64 full scope	inference-validated Windows + Android (35.5 tok/s device CPU, +38 % vs official q8)
`qwen2.5-1.5b-instruct/Qwen2.5-1.5B-Instruct_wi4b64_ekv4096.litertlm`	790 MB	litert-community/Qwen2.5-1.5B-Instruct f32	wi4b64 full scope	inference-validated Windows CPU (59.7 prefill / 11.8 decode tok/s, Korean QA correct)

Related: ACFT short-window Whisper models

Fixed 5 s/10 s/30 s short-window TFLite graphs (encode-signature-sized mel window; method: futo-org/whisper-acft) live in two dedicated repos:

—Pure ACFT (TFLite conversions of the futo-org checkpoints, no retraining; tiny / base / small × multilingual / .en, dynamic-range int8): [litert-community/whisper-acft](https://huggingface.co/litert-community/whisper-acft).
—Korean ACFT (ACFT-KO, Korean-trained for on-device Korean voice commands; tiny / base / medium / large-v3-turbo, dynamic-range int8): [leuconoe/whisper-acft-ko](https://huggingface.co/leuconoe/whisper-acft-ko). Training data zeroth-korean (OpenSLR SLR40, CC-BY-4.0) + google/fleurs en_us (CC-BY-4.0).

Known caveats (disclose in model cards)

—Whisper 30 s graphs; decoder is fixed-length re-run (no KV cache) matching the litert-community tiny/base interface. large-v3 family needs a 128-mel frontend and vocab 51866 (<|transcribe|>=50360, <|notimestamps|>=50364).
—tiny/base i8 showed sub-1.2 s clip sensitivity on one Snapdragon 865 device (mel/STFT numerics, ~0.1 % energy delta vs desktop); turbo/large-v3 unaffected.
—Qwen2.5 i4 litertlm metadata carries lowercase model_type (tf_lite_prefill_decode) vs uppercase in official files — loads fine on v0.14.0, noted for completeness.
—Licenses: whisper weights MIT (OpenAI); Qwen2.5 Apache-2.0. Quantized derivatives inherit the base licenses.