waltgrace/llama-cpp-expert-sniper

llama.cpp Expert Sniper — madvise prefetch for MoE inference

~65 lines of C++ that enables MoE models larger than RAM on llama.cpp.

Stock llama.cpp thrashes indefinitely. This build generates tokens.

Results

On GPU machines with abundant RAM (A100 251GB, RTX 3090 31GB), stock llama.cpp is faster — the OS page cache handles it. madvise helps specifically when system RAM < model size and layers are on CPU (ngl 0).

How it works

MoE models activate 8 of 128+ experts per token. Consecutive tokens share ~87% of active experts. Stock llama.cpp uses mmap but the OS has no idea which expert pages are hot — it evicts them randomly, causing a page fault storm.

Our patch hooks llama.cpp's eval callback, intercepts every ggml_mul_mat_id operation, reads the router's top-k expert selection from t->src[2], and calls madvise(MADV_WILLNEED) on each active expert's memory range. This tells the kernel which pages to prefetch before the compute needs them.

Zero allocation. Zero memcpy. Zero mutex. One syscall per expert slice.

What we learned

1. madvise beats LRU cache everywhere. We first built a 460-line LRU cache. It was 2.4x slower than 15 lines of madvise (0.24 vs 0.57 tok/s on 8GB MacBook). The cache stole 5GB from the OS page cache for duplicate data. Don't fight the OS page cache — coach it.

2. Even a no-op callback prevents thrashing. Just hooking the eval callback and inspecting tensor pointers (without any madvise) produces 0.46 tok/s where stock produces zero. The callback inadvertently warms mmap pages through pointer inspection.

3. Device pointer bug. All prior GPU benchmarks were silently invalid — the callback dereferenced t->src[2]->data without checking ggml_backend_buffer_is_host(). On GPU layers this was a CUDA device pointer. Fixed.

4. On abundant RAM, do nothing. The OS page cache is remarkably good when it has enough room. Any intervention is pure overhead when RAM exceeds model size.

Build

git clone https://github.com/walter-grace/mac-code
cd mac-code/research/expert-sniper/llama-cpp

# macOS (Metal)
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-server

# NVIDIA GPU (CUDA)
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-server

Usage

# Enable madvise prefetch
./build/bin/llama-server \
  -m Qwen3.5-35B-A3B-UD-IQ2_M.gguf \
  -ngl 0 \
  --expert-cache-size 1 \
  --port 8201

# Test no-op mode (isolate callback overhead)
EXPERT_CACHE_NOOP=1 ./build/bin/llama-server \
  -m model.gguf \
  -ngl 0 \
  --expert-cache-size 1 \
  --port 8201

Files changed vs stock llama.cpp

New files (~430 lines):

Patched files (~30 lines across 5 files):

The research journey

460-line LRU cache → 0.24 tok/s (stole RAM from OS page cache)
15-line madvise    → 0.57 tok/s (coached the OS page cache)
no-op callback     → 0.46 tok/s (accidental page warming)

The cache was the experiment. madvise was the answer.

Full three-way benchmark (8 GB MacBook Air)

Gemma 4-26B-A4B — MoE Sparsity Benchmark

Google Gemma 4 has 128 experts with top-8 routing (4B active of 26B total). Tested at multiple quantization levels on Apple Silicon:

All results: stock llama.cpp with mmap, no madvise. Canberra verified on all configs.

Finding: Gemma 4's low activation ratio (15.4%) lets the OS page cache handle memory pressure without explicit madvise. The madvise sniper is most valuable for denser MoE models (Qwen 35B) where the per-token working set overwhelms the page cache.

Related

• —MLX Expert Sniper (Apple Silicon, 5.4 tok/s on 35B): huggingface.co/waltgrace/mlx-expert-sniper
• —Full research + code: github.com/walter-grace/mac-code/tree/main/research/expert-sniper