Modelpublic

Tdamre/Bonsai-27B-litert-lm

sourceHugging Faceapache-2.0updated 4d agoView on Hugging Face

3likes195downloads

Model Card

Bonsai-27B LiteRT-LM — source repair and experimental conversion

23 September 2026: source reconstruction is repaired and tested; a repaired full-model Android artifact is not yet available. The existing Bonsai-slice-4L-* files contain only four layers, not the complete 64-layer Bonsai-27B model. The word fixed in their historical filenames is not an end-to-end quality certification.

Current validation status

Check	Result	Scope
Original Q1 source SHA-256	Pass	Pinned 3,803,452,480-byte GGUF
Full source reconstruction	Pass	All 64 layers, all 851 tensors, no missing weights
Value-level regression tests	Pass	21 tests, including exhaustive bit/mask checks
Full source generation smoke tests	Recorded in validation report	Deterministic host PyTorch generation, not a benchmark
Packed-Q1 numerical prototype	Pass	Small host LiteRT CPU test only
Hardware-accelerated Android setup	Pass	x86_64, 8,192 MiB RAM, 30G data partition, WHPX
Official Edge Gallery 1.0.19	Install and launch pass	No repaired-model inference performed
Repaired full `.litertlm` export	Not completed	Export adapter positional-encoding incompatibility
Full-model generation in Gallery	Not run	End-to-end release validation does not pass

The validated source implementation and reproduction commands are in quality/README.md. Machine-readable evidence is in validation/2026-09-23/.

What was wrong

The previous GGUF reconstruction could produce correctly shaped matrices with scrambled values: reshaping flat GGUF storage into its fast-dimension-first dimensions and then transposing is not the correct inverse layout operation. The repair interprets the flat values directly in Hugging Face row-major shape.

The conversion also needs to reverse Qwen's zero-centered RMSNorm transform, restore A_log from the stored negative exponential, and undo the linear-attention value-head permutation. The gated linear-attention norm must not receive the zero-centered norm subtraction. These transforms now have value-level regression tests.

The Q1 sign encoding is byte-major and little-endian, with one FP16 scale for each 128 weights. That bit order was not the cause of the reconstruction failure.

Full-model source repair

The quality/ loader retains the original packed sign bits and scales rather than expanding all 26.90 billion parameters to FP32 or requantizing them to INT8. Embeddings gather only requested rows before decoding; linear layers decode bounded row chunks. Unknown, duplicate, missing or shape-mismatched tensors fail explicitly.

The initial full-source smoke run correctly answered four deterministic prompts, including arithmetic, a capital-city question and exact output. Its parameter/buffer storage was 4,216,748,288 bytes and peak CUDA allocation was 4,642,257,408 bytes on an RTX 4090. These measurements do not establish Android memory fit. The published source validator adds twelve named cases and records actual answers and token IDs; consult its report for the observed result.

The separate small LiteRT prototype passed six numerical comparisons with maximum absolute error 1.049041748046875e-05. A successful tiny graph is not evidence that the complete model exports or runs in Gallery.

Historical artifacts — unchanged

Files	Contents	Status
`experimental_artifacts/Qwen3.5-0.8B-hybrid-exact-c2048.litertlm` and c32k/c64k variants	Qwen3.5-0.8B hybrid graph experiments	Not Bonsai-27B
`experimental_artifacts/Bonsai-slice-4L-Q1dequant-fixed-c2048.litertlm` and c32k/c64k variants	Four-layer Bonsai slices	Legacy graph experiments; not full-model quality releases
Other `Bonsai-slice-4L-*` files	Earlier four-layer experiments	Historical, not recommended for evaluating Bonsai-27B chat quality

No existing weight artifact is replaced by this source-repair commit. The old experimental/ scripts are preserved for provenance, but their transpose Linears reconstruction is superseded by the strict loader in quality/.

Required before an Android release

The repaired source must pass a compatible full-model export, complete-graph LiteRT parity, tokenizer/stop-token checks, measured memory on the 8 GB x86_64 AVD, and actual generation in official Edge Gallery using the exact exported artifact hash. App installation and launch alone cannot satisfy these gates.