litert-community/SmolLM3-3B

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Measured on device (edge-compat, smollm3-3b): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 14.8 tok/s · prefill 268 tok/s · TTFT 820 ms · all 1476 ops delegated (2026-08-24); Pixel 8a · LiteRT-LM 0.16.0 · GPU · decode 7.7 tok/s · prefill 12 tok/s · TTFT 1.56 s · all 1476 ops delegated (2026-08-17); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 9.8 tok/s · prefill 148 tok/s · TTFT 3.09 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/smollm3-3b/CARD.md

Measured on device (edge-compat, smollm3-3b-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · decode 14.4 tok/s · prefill 312 tok/s · TTFT 750 ms · all 1548 ops delegated (2026-08-24); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 2.0 tok/s · prefill 19 tok/s · TTFT 14.04 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/smollm3-3b-int8/CARD.md

SmolLM3-3B — LiteRT-LM (blockwise int4)

HuggingFaceTB/SmolLM3-3B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

SmolLM3 is a fully-open 3B decoder (Apache-2.0) with GQA, a NoPE attention schedule, multilingual support, and long-context training — a strong small reasoner.

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

Usage

Run with the LiteRT-LM runtime:

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path SmolLM3-3B_q4_block32_ekv4096.litertlm \
  --backend gpu \
  --input_prompt "Explain on-device AI in one sentence."

The .litertlm bundle carries the tokenizer and the prompt template (ChatML — <|im_start|>role / <|im_end|>, stop token <|im_end|>), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The easiest way to try this model on a phone is the official [Google AI Edge Gallery](https://github.com/google-ai-edge/gallery) app — it runs .litertlm models fully on-device and can import your own:

1. 1.Install a recent Gallery (package com.google.ai.edge.gallery, APK from the repo's releases — 1.0.15+ supports .litertlm). Older 1.0.x builds (package com.google.aiedge.gallery) only accept the legacy MediaPipe .task format and reject .litertlm.
2. 2.Download SmolLM3-3B_q4_block32_ekv4096.litertlm from this repo and push it to the device:

   adb push SmolLM3-3B_q4_block32_ekv4096.litertlm /sdcard/Download/

1. 1.In the app, tap the + button (bottom-right), pick the file, and choose the GPU backend (CPU also works).
2. 2.Chat. Nothing else to configure — the .litertlm bundle already carries the tokenizer and ChatML prompt template.

See the Gallery Importing Local Models) guide for details. To embed the model in your own Android app instead, use the LiteRT-LM Kotlin API (Gradle artifact com.google.ai.edge.litertlm:litertlm-android, getting started).

Measured on an 8 GB phone (added 2026-08-17): driving litert_lm_main directly on a Pixel 8a (Tensor G3, Mali-G715, 8 GB RAM), the graph runs entirely on the OpenCL delegate — 1476/1476 nodes in the 128-token prefill graph and 1308/1308 in decode, zero rejected ops — and answers correctly.

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/SmolLM3-3B SmolLM3-3B_q4_block32_ekv4096.litertlm smollm3-3b
litert-lm run smollm3-3b     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The iPhone row is one cold run through the LiteRTDemo harness on iOS 27.0 (prompt "Explain on-device AI in one short sentence.", 512-token budget, no warm-up turn), read back from its run log. Its prefill figure is measured on that short prompt, so it reflects fixed per-turn overhead rather than prefill throughput and is not comparable to the 256-token desktop column.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought asking for #### <n>, identical prompt and answer-extraction for both rows — only the quantization differs).

LiteRT int4 is fully at parity — 0.0 pt vs the bf16 reference. The blockwise-32 + OCTAV recipe with a 4096 KV cache preserves reasoning accuracy exactly at n=100. The model produces visible step-by-step chain-of-thought in the answer body and terminates cleanly at <|im_end|> (no rambling).

Galaxy S26 — GPU backend

Both published bundles run on the Android GPU backend and generate.

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. In SmolLM3-3B_q4_block32_ekv4096.litertlm, XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

No speed rows, on purpose. On this handset the GPU backend wins prefill and does not win decode, so a GPU throughput figure only means something beside a CPU row from the same handset, and no S26 CPU row exists for this model yet.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion

Converted with `litert-torch` via its generic export_hf path. SmolLM3ForCausalLM rides the existing converter with no custom code: the NoPE attention schedule (rotary disabled on every 4th layer, no_rope_layer_interval=4) lowers to generic ops with no custom kernel. The int4 recipe is blockwise (block 32) + OCTAV optimal-clipping with the embedding kept at INT8; the embedding is externalized into its own bundle section so the main weights section stays under the iOS ~2 GiB single-mmap limit. Blockwise (not channelwise) int4 plus OCTAV is what holds reasoning accuracy at parity.

Training data & PII

This is a weights-exact format conversion of HuggingFaceTB/SmolLM3-3B; no new training was performed. SmolLM3 was trained by Hugging Face on ~11T tokens of publicly documented data — web (FineWeb-Edu, DCLM), code (StarCoder-family), math, and multilingual sources — plus public SFT/preference sets. Being web-derived it may incidentally contain PII; none was deliberately collected and this format conversion adds none. Apply your own content/PII filtering before deployment. See the base model card for the full data mixture.

2026-08-28 — start_token fix (weights unchanged)

The bundle's LlmMetadata start_token held the literal string "None". This tokenizer has no BOS, and the LiteRT-LM engine resolved that string to a real vocabulary token — so every prompt began with the word None, which the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: the ## Metadata header (knowledge cutoff, today's date, Reasoning Mode: /think) followed by the instructions You are a helpful AI assistant named SmolLM, trained by Hugging Face…. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in SmolLM3-3B.litertlm, SmolLM3-3B_q4_block32_ekv4096.litertlm now emits the block exactly once when no system message is given. In SmolLM3-3B.litertlm, a system message you pass is wrapped in the same metadata header as upstream, and /no_think in it switches reasoning off. In SmolLM3-3B_q4_block32_ekv4096.litertlm, the block is not emitted when you pass a system message; the upstream template also wraps a caller's system message in its own preamble, and this file passes it through unchanged, exactly as it did before. The previous SmolLM3-3B.litertlm also forced an empty <think></think> before every answer (/no_think); upstream's default is /think, and the template now follows it. The restored block adds 240 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself now renders through the upstream metadata header in SmolLM3-3B.litertlm; in SmolLM3-3B_q4_block32_ekv4096.litertlm it renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

2026-08-30 — tokenizer section replaced (weights unchanged)

The tokenizer in SmolLM3-3B.litertlm was a SentencePiece conversion of the model's BPE tokenizer, and the conversion lost the byte-level semantics: a standalone accented letter or symbol (é, ñ, ü, °, ·, …) was encoded to the id of a single-byte token instead of the token the upstream tokenizer uses, and any character without a whole-character vocabulary entry (emoji, most of Latin Extended-A) became the token the conversion had reused as UNK — the end-of-text token for this vocabulary. For this file that token was <|im_end|>, so every turn's end marker arrived at the model as six spelled-out tokens instead of the single <|im_end|> id, and an emoji in a message arrived as an end-of-turn. SmolLM3-3B.litertlm now embeds the upstream tokenizer.json (the same HF tokenizer path most bundles in this collection use).

Tokenizer-only change: every section of the bundle except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token — only the file's own sha256 differs. Verified on the LiteRT-LM runtime: the default turn, 7 probe strings, the 223 standalone characters U+00A1–U+017F and every special token now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). Prompts containing accented letters, symbols or emoji reach the model differently from before, so individual answers to such prompts can change. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file, and any on-device rows were measured on the previous file too — the on-device gate has not been re-run on this one (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30, re-download.

2026-08-31 — thought channel declared (metadata only, weights unchanged)

SmolLM3-3B.litertlm, SmolLM3-3B_q4_block32_ekv4096.litertlm now declare the reasoning channel in their metadata (LlmMetadata.channels: channel name thought, markers <think>…</think> exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with `litert-lm benchmark` 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

License

Apache-2.0, inherited from the base model HuggingFaceTB/SmolLM3-3B.