litert-community/FastContext-1.0-4B-SFT

Measured on device (edge-compat, fastcontext-1.0-4b-sft-model-block128): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 7.5 tok/s · prefill 86 tok/s · TTFT 2.47 s · all 1674 ops delegated (2026-08-24); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.5 tok/s · prefill 10 tok/s · TTFT 26.04 s (2026-09-02). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/fastcontext-1.0-4b-sft-model-block128/CARD.md

Measured on device (edge-compat, fastcontext-4b): Mac Studio M4 Max · LiteRT-LM 0.14.0 · GPU · decode 74.5 tok/s · prefill 964 tok/s · TTFT 279 ms (2026-07-23); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 7.1 tok/s · prefill 37 tok/s · TTFT 5.61 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 1.4 tok/s · prefill 10 tok/s · TTFT 28.46 s (2026-09-02). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/fastcontext-4b/CARD.md

FastContext-1.0-4B-SFT — LiteRT-LM (blockwise int4)

microsoft/FastContext-1.0-4B-SFT converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

FastContext is a lightweight repository-exploration sub-agent for coding agents — it specializes in repo discovery and evidence gathering via parallel tool calls (READ / GLOB / GREP). The 4B backbone is Qwen3-4B-Instruct (Qwen3ForCausalLM), so it rides the existing Qwen3 converter and runtime directly.

Which file?

Use `model.litertlm` (block 32) unless decode latency dominates — it is full parity with bf16 (see below) and loads on iPhone, Android and desktop alike. The block-128 build trades −6 pt accuracy for ~40% faster decode (block 128 stores ¼ the scales → lighter GPU dequant; this is the granularity Google's official Gemma block-128 bundles use).

Usage

Run with the LiteRT-LM runtime:

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "List the files you would read to understand a Python project's entry point."

The .litertlm bundle carries the tokenizer and prompt template (Qwen3 ChatML — <|im_start|>role\n…<|im_end|>, stop token <|im_end|>), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this on a phone is the official [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) app:

1. 1.Install a recent Gallery (package com.google.ai.edge.gallery, APK from the repo's releases — 1.0.15+ supports .litertlm).
2. 2.Download model.litertlm and push it:

   adb push model.litertlm /sdcard/Download/

1. 1.In the app tap + (bottom-right), pick the file, choose the GPU backend.
2. 2.Chat — the bundle already carries the tokenizer and Qwen3 chat template.

A 4B int4 build needs ~2.5 GB free RAM; reboot the phone first if memory is tight.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/FastContext-1.0-4B-SFT model.litertlm fastcontext-1.0-4b-sft
litert-lm run fastcontext-1.0-4b-sft     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Run on iPhone

Verified on iPhone 17 Pro with the LiteRT-LM Swift runtime (swift-litert-lm): both files load and generate on-device (block 32 ~10 tok/s, block 128 ~14 tok/s). Note: this 4B's main weights section is ~2.11 GiB for block 32 / ~1.94 GiB for block 128 — both load on iPhone 17 Pro, so externalizing the embedder (below) is sufficient; no further size reduction is required to fit iOS.

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The desktop rows are the shipped model.litertlm (block 32); the block-128 build was not re-measured, so the figures for it in “Which file?” above are the older ship-gate numbers. The iPhone figures come from the 2026-06-27 on-device platform matrix for this model, one per build; that record does not state a run count or prompt.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, identical prompt and answer-extraction for every row).

The block-32 build is at full parity (−0 pt vs bf16) — the OCTAV + blockwise-32 recipe leaves int4 indistinguishable from bf16 here. (FastContext is a tool-calling agent, not a math model, so GSM8K is an indirect capability measure; the bf16-vs-int4 delta is nonetheless the correct test for "did int4 quantization degrade the model" — and at block 32 it did not.) Passes the local quality gate 8/8 (no degeneracy).

Identity: asked "what is your name?", the model answers "I am Qwen…". FastContext is fine-tuned from Qwen3-4B-Instruct and the SFT does not override the base identity — this is inherited from the base model (the bf16 original behaves identically), not a conversion artifact.

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with the official `litert-torch` converter — FastContext is a standard Qwen3ForCausalLM, so it uses the existing Qwen3 path with no custom graph code. Recipe: blockwise int4 + OCTAV (INT4 weights, block 32 or 128, symmetric, OCTAV optimal-clipping) with embeddings kept at INT8, KV cache 4096. Blockwise (not the tool's default channelwise) int4 is what preserves accuracy.

from litert_torch.generative.export_hf.export import export
export(
    model="microsoft/FastContext-1.0-4B-SFT",
    output_dir="out",
    quantization_recipe="qwen3_int4_block32_octav.json",  # blockwise-32 int4 + OCTAV, int8 embeddings
    cache_length=4096,
    externalize_embedder=True,  # embedding → its own section (dedups tied matrix)
)

externalize_embedder=True writes the (tied) embedding as its own .litertlm section and dedups the tied matrix, shrinking the main weights section — the generic equivalent of Gemma's per-layer-embedding mmap.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with `litert-lm benchmark` 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License

MIT, inherited from the base model microsoft/FastContext-1.0-4B-SFT (itself built on Qwen3-4B-Instruct).