musk12/apple-fastvlm-cpu-inference-models

FastVLM CPU Inference (Vision Projector + Qwen2 GGUF)

A complete, CPU-only FastVLM inference pipeline: an ONNX vision projector for image encoding, paired with a quantized Qwen2 GGUF language model for text generation. This repo includes the model weights plus the full loading/serving code, so you can run the pipeline end to end without a GPU.

Pipeline overview

Image → [ONNX Vision Projector] → image embeddings ↓ [fastvlmserver (llama.cpp-based C binary)] ↓ Streamed text response (Qwen2 GGUF, Q4K_M)

1. 1.An image is preprocessed and passed through vision_projector_v1_standalone.onnx (ONNX Runtime, CPU) to produce image embeddings.
2. 2.The embeddings are written to a temporary binary file and handed to a persistent fastvlm_server process (a custom llama.cpp-based binary) that keeps the GGUF language model loaded in memory.
3. 3.The language model generates a response token-by-token, streamed back over HTTP.

Files in this repo

Requirements

• —Python 3.9+
• —onnxruntime, fastapi, uvicorn, pillow, numpy, python-multipart

pip install onnxruntime fastapi uvicorn pillow numpy python-multipart

Loading and running

Download all files in this repo into one local directory, keeping them all together (stream_api.py looks for the .onnx, .gguf, fastvlm_server, and .so files relative to its own location), then:

python stream_api.py

This starts a FastAPI server on http://0.0.0.0:8000. On startup it:

• —loads the ONNX vision encoder via ONNX Runtime (CPU),
• —launches fastvlm_server as a persistent subprocess with the GGUF model loaded and kept in memory, so the model isn't reloaded per request.

Endpoints

• —GET / — basic status info
• —GET /health — reports whether the ONNX session, GGUF file, and server binary are all present and loaded
• —POST /predict — send an image + optional prompt, get a streamed text response

Example request:

curl -X POST http://localhost:8000/predict \
  -F "image=@your_image.jpg" \
  -F "prompt=Describe this image in detail." \
  --no-buffer

The response streams as plain text until generation completes.

How it stays fast

fastvlm_server is started once at API startup and kept alive as a long-running subprocess — the GGUF model is loaded into memory a single time. Each /predict request sends the embedding file path and prompt over the process's stdin and reads generated tokens back from stdout until a ---END--- sentinel, avoiding reload cost per request. Requests are serialized (one at a time), since the underlying server is single-threaded.

Eval results

Benchmarks were run on a 6-core Intel CPU, comparing the two input resolution variants of the vision pipeline (512×512 vs 1024×1024).

LLaVA-Wild average TTFT @ 512: 1,382 ms (6-core Intel CPU)

Notes

• —512×512 uses fewer visual tokens (64) and is significantly faster to first token, making it a good fit for latency-sensitive use cases and general VQA-style tasks (POPE, GQA).
• —1024×1024 uses more visual tokens (256) for finer-grained visual detail, at the cost of higher TTFT — better suited for tasks requiring precise text/detail recognition (TextVQA, OCRBench v2).
• —TTFT (time to first token) figures are CPU-only, measured on a 6-core Intel CPU, and will vary with hardware.

Base models

Derived from:

• —apple/FastVLM-0.5B (vision projector)
• —Qwen/Qwen2-0.5B (language backbone, quantized to GGUF)

Related

A hosted Gradio demo of this pipeline is available at: musk12/FastVLM-CPU-Inference-demo