wundr-ai/echoread-gemma4-e4b-v3.2.1-litertlm-gpu

EchoRead Gemma 4 E4B — v3.2.1 (GPU-clean .litertlm re-export)

A GPU-delegable .litertlm re-export of the fine-tuned EchoRead Gemma 4 E4B checkpoint, for the wundr-echo on-device reading-coach app. Same weights as wundr-ai/echoread-gemma4-e4b-v3.2.1-litertlm; this bundle is re-converted so LiteRT-LM's Metal GPU backend can fully delegate text prefill/decode and the vision encoder on iOS.

Weights are unchanged from the v3.2.1 merged checkpoint. The -gpu suffix names the conversion, not a new checkpoint — no retraining, no DPO/LoRA changes.

Why this exists

The prior .litertlm (June 2026) failed GPU engine creation on iOS. Root cause: exactly one MUL(per_layer_projection_weight [10752,2560] fp32, scalar 1/√2560) op per text signature (prefill_128, prefill_1024, decode) — a MUL(const, const). The Metal delegate rejects MUL with two const inputs, and LiteRT-LM requires full delegation on GPU or engine creation returns NULL. The fp32 weight also escaped INT4 quantization (~94 MB waste).

Mechanism: experimental_lightweight_conversion=True stores consts as MLIR resources, so MLIR constant folding is skipped and the scalar never folds into the weight. Disabling it (LIGHTWEIGHT=0 below) runs full folding → the offending MUL disappears and the weight quantizes normally.

Verification (this bundle)

• —GPU-delegability gate — PASS. MUL(const, const) audit across every TFLite section (mul_audit_all.py): 0 offenders, all sections (text prefill/decode/embedder + vision encoder/adapter + per-layer embedder). The prior export had 3.
• —Structural peek — PASS (litertlm_peek): bundle format 1.5.0; LlmMetadata has start_token=<bos>, EOS (id 1) + <end_of_turn> (id 106) stop tokens, sampler_params: TOP_P k=64 p=0.95 temperature=1.0 (matches the wire sampling contract), max_num_tokens=4096, llm_model_type: gemma4 (multimodal, <|image>/<image|> tokens), the litert-community Gemma 4 jinja chat template; HF tokenizer bundled (HF_Tokenizer_Zlib); 5 TFLite sections present incl. tf_lite_vision_encoder + tf_lite_vision_adapter.
• —Generative equivalence / wire-smoke — NOT run on-pod. The litert-torch nightly used here ships no LiteRT-LM runtime (no experimental/validation, no litert_lm python), so no tokens were generated during conversion. Full generative validation ([READING_SESSION] / [FEEDBACK] parseable JSON, output-quality, vision on GPU) is deferred to the on-device pass, which is also the true GPU-load + runtime-compat proof.

Conversion recipe (reproducible)

⚠️ You MUST set `LIGHTWEIGHT=0`. The convert driver defaults to LIGHTWEIGHT=1, which reproduces the GPU-broken bundle. LIGHTWEIGHT=0 sets experimental_lightweight_conversion=False.

Lineage

• —LoRA SFT v3.2 + DPO v3.2.1 → merged: wundr-ai/echoread-gemma4-e4b-merged-v3.2.1
• —Base: litert-community/gemma-4-E4B-it-litert-lm (chat template + runtime target)
• —Prior (CPU-only) bundle: wundr-ai/echoread-gemma4-e4b-v3.2.1-litertlm

Known gaps

• —On-device generative + GPU-load pass pending (see above). A bundle built by 1.5.0 tooling must still be confirmed to load on the app's LiteRT-LM v0.13.1 runtime.
• —Audio tower not included (did not survive conversion; separate roadmap).

Gotchas reference: docs/plans/2026-06-22-litertlm-conversion-recipe.md, docs/plans/2026-07-07-litertlm-gpu-reexport.md.

Run C — GPU composites + PR-1099 + recipe-parity (2026-07-09)

Two additional bundles were added to this repo for a device A/B, on top of the model.litertlm Plan B bundle documented above. The weights are still the v3.2.1 merged checkpoint — R1 only adds GPU-composite kernels; R2 additionally re-quantizes the embedding/lm_head tables (a conversion change, not a retrain).

Toolchain

• —Converter: litert-torch-nightly==0.10.0.dev20260709.
• —Composites provenance (reproducibility caveat): that nightly's wheel is MISSING the experimental/composites subpackage that core/attention.py/core/cache.py import — even import export fails as-shipped. The 3 composite modules (sdpa.py, cache_update.py, runtime_batched_matmul.py) were dropped in from google-ai-edge/litert-torch main (2026-07-09; PR #1096 merged 07-08). Neither bundle can be rebuilt from `pip install litert-torch-nightly==0.10.0.dev20260709` alone — re-apply that drop-in.
• —use_bool_mask=True, apply_gpu_composites=True (consumed via extra_kwargs on this nightly).
• —PR #1099 patch applied (optimization_barrier) → experimental_lightweight_conversion=True is GPU-safe (no MUL(const,const)).
• —All other export settings identical to the Plan B recipe table above (task=image_text_to_text, externalize_embedder, export_vision_encoder, prefill_lengths=[128,1024], cache_length=4096, litert-community chat template).

R1 — composites + patch (default dynamic_wi4b32_afp32)

• —Size 4,763,848,016 B. GPU-clean gate: 0 `MUL(const,const)`, all sections.
• —Composites verified present (not just requested): 1,039 STABLEHLO_COMPOSITE ops (odml.rms_norm 791, odml.runtime_bmm 176, odml.cache_update 72). Plan B has none. Op counts differ from Plan B (prefill 1342/214 vs 1363/213; decode 1970/328 vs 1995/327).
• —Same weight values as Plan B → zero quality risk. This isolates the composite-kernel speedup.

R2 — mobile-parity recipe (2-bit embed/PLE/lm_head, 8-bit per-layer)

Custom recipe (authored on-pod; aiedgequantizer scopes ops by output-tensor name and matches re.search, last-match-wins; 2-bit/8-bit tiers require `CHANNELWISE` — `BLOCKWISE_32` is rejected by the INTEGER-compute policy for `EMBEDDING_LOOKUP`/8-bit and silently falls back to INT4):

[
 {"regex": ".*",                          "num_bits": 4, "granularity": "BLOCKWISE_32"},
 {"regex": "Linear_per_layer_input_gate", "num_bits": 8, "granularity": "CHANNELWISE"},
 {"regex": "Linear_per_layer_projection", "num_bits": 8, "granularity": "CHANNELWISE"},
 {"regex": "ScaledWordEmbedding",         "num_bits": 2, "granularity": "CHANNELWISE"},
 {"regex": "logits_output",               "num_bits": 2, "granularity": "CHANNELWISE"}
]

• —Size 3,500,100,832 B (~3.50 GB). GPU-clean gate: 0 `MUL(const,const)`; composites present (1,039). Per-table: lm_head, externalized embedder, and the 1.41 GB PLE table all INT2.
• —dtype histogram vs the official litert-community E4B bundle — near parity:

• —⚠️ QUALITY UNVERIFIED. The official 2-bit map is 2-bit-QAT-trained; ours is q40-QAT PTQ'd down to 2 bits — this MAY degrade scaffold/content/vision quality. **No generative eval was run:** the `dev20260709` nightly ships no LiteRT-LM runtime (only the `litertlmbuilder` packager, no `experimental/validation`), and LiteRT-LM's releases publish only macOS/iOS runners — no Linux x8664 binary — so no tokens could be generated on the converter box. R2 is published for on-device A/B only; if quality visibly degrades vs R1, ship R1 and wait for the Q3 direct QAT-mobile conversion.

A/B guidance

R1 = safe kernel-win baseline (quality-anchor). R2 = size/RAM/decode-bandwidth win, quality-gated. See docs/plans/pod-scripts/RUNC_REPORT.md in the app repo for the full run log.