waltgrace/llama-cpp-expert-sniper

1---2library_name: gguf3tags:4  - moe5  - expert-prefetch6  - madvise7  - llama-cpp8  - apple-silicon9  - cuda10  - on-device11license: mit12pipeline_tag: text-generation13---14 15# llama.cpp Expert Sniper — madvise prefetch for MoE inference16 17~65 lines of C++ that enables MoE models larger than RAM on llama.cpp.18 19Stock llama.cpp thrashes indefinitely. This build generates tokens.20 21## Results22 23| Hardware | RAM | Model | Stock llama.cpp | madvise build |24|----------|-----|-------|-----------------|---------------|25| M2 MacBook Air | 8 GB | Qwen3.5-35B-A3B IQ2_M (10.6 GB) | 0 tok/s (thrash) | **0.57 tok/s** |26| M2 MacBook Air | 8 GB | Same model, no-op callback only | 0 tok/s (thrash) | 0.46 tok/s |27 28On GPU machines with abundant RAM (A100 251GB, RTX 3090 31GB), stock llama.cpp is faster — the OS page cache handles it. madvise helps specifically when **system RAM < model size** and **layers are on CPU (ngl 0)**.29 30## How it works31 32MoE models activate 8 of 128+ experts per token. Consecutive tokens share ~87% of active experts. Stock llama.cpp uses mmap but the OS has no idea which expert pages are hot — it evicts them randomly, causing a page fault storm.33 34Our patch hooks llama.cpp's eval callback, intercepts every `ggml_mul_mat_id` operation, reads the router's top-k expert selection from `t->src[2]`, and calls `madvise(MADV_WILLNEED)` on each active expert's memory range. This tells the kernel which pages to prefetch before the compute needs them.35 36Zero allocation. Zero memcpy. Zero mutex. One syscall per expert slice.37 38## What we learned39 40**1. madvise beats LRU cache everywhere.** We first built a 460-line LRU cache. It was 2.4x slower than 15 lines of madvise (0.24 vs 0.57 tok/s on 8GB MacBook). The cache stole 5GB from the OS page cache for duplicate data. Don't fight the OS page cache — coach it.41 42**2. Even a no-op callback prevents thrashing.** Just hooking the eval callback and inspecting tensor pointers (without any madvise) produces 0.46 tok/s where stock produces zero. The callback inadvertently warms mmap pages through pointer inspection.43 44**3. Device pointer bug.** All prior GPU benchmarks were silently invalid — the callback dereferenced `t->src[2]->data` without checking `ggml_backend_buffer_is_host()`. On GPU layers this was a CUDA device pointer. Fixed.45 46**4. On abundant RAM, do nothing.** The OS page cache is remarkably good when it has enough room. Any intervention is pure overhead when RAM exceeds model size.47 48## Build49 50```bash51git clone https://github.com/walter-grace/mac-code52cd mac-code/research/expert-sniper/llama-cpp53 54# macOS (Metal)55cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release56cmake --build build -j$(nproc) --target llama-server57 58# NVIDIA GPU (CUDA)59cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release60cmake --build build -j$(nproc) --target llama-server61```62 63## Usage64 65```bash66# Enable madvise prefetch67./build/bin/llama-server \68  -m Qwen3.5-35B-A3B-UD-IQ2_M.gguf \69  -ngl 0 \70  --expert-cache-size 1 \71  --port 820172 73# Test no-op mode (isolate callback overhead)74EXPERT_CACHE_NOOP=1 ./build/bin/llama-server \75  -m model.gguf \76  -ngl 0 \77  --expert-cache-size 1 \78  --port 820179```80 81## Files changed vs stock llama.cpp82 83**New files (~430 lines):**84 85| File | Purpose |86|------|---------|87| `src/llama-expert-cache-ctx.cpp` | Eval callback, madvise prefetch, tensor identification |88| `src/llama-expert-cache-ctx.h` | Context struct and declarations |89| `src/llama-expert-cache.cpp` | LRU cache (deprecated, retained for reference) |90| `src/llama-expert-cache.h` | Cache class definition |91 92**Patched files (~30 lines across 5 files):**93 94| File | Change |95|------|--------|96| `src/CMakeLists.txt` | Added new source files to build |97| `src/llama-context.h` | Added expert cache context member |98| `common/common.h` | Added `expert_cache_size` parameter |99| `common/common.cpp` | Cache init + eval callback registration |100| `common/arg.cpp` | `--expert-cache-size` CLI flag |101 102## The research journey103 104```105460-line LRU cache → 0.24 tok/s (stole RAM from OS page cache)10615-line madvise    → 0.57 tok/s (coached the OS page cache)107no-op callback     → 0.46 tok/s (accidental page warming)108 109The cache was the experiment. madvise was the answer.110```111 112## Full three-way benchmark (8 GB MacBook Air)113 114| Config | tok/s | Mechanism |115|--------|-------|-----------|116| Stock llama.cpp | 0 (thrash) | OS blind LRU, no domain knowledge |117| No-op callback | 0.46 | Accidental page warming from tensor inspection |118| madvise prefetch | 0.57 | Explicit kernel prefetch hints |119| LRU cache (5 GB) | 0.24 | Duplicate data in user-space heap |120 121## Gemma 4-26B-A4B — MoE Sparsity Benchmark122 123Google Gemma 4 has 128 experts with top-8 routing (4B active of 26B total). Tested at multiple quantization levels on Apple Silicon:124 125| Hardware | Quant | Model size | RAM | Speed | Notes |126|----------|-------|-----------|-----|-------|-------|127| M2 MacBook Air | IQ2_M | 9.3 GB | 8 GB | **1.37 tok/s** | Model exceeds RAM, MoE sparsity prevents thrash |128| M4 Mac Mini | IQ2_M | 9.3 GB | 16 GB | **36.5 tok/s** | Fits in RAM, full GPU speed |129| M4 Mac Mini | Q4_K_M | 16.9 GB | 16 GB | **5.18 tok/s** | Exceeds RAM, still runs smoothly |130| M4 Mac Mini | Q8_0 | 26.9 GB | 16 GB | **0 tok/s (thrash)** | CPU_REPACK doubles memory to 51 GB, can't load |131 132All results: stock llama.cpp with mmap, no madvise. Canberra verified on all configs.133 134**Finding:** Gemma 4's low activation ratio (15.4%) lets the OS page cache handle memory pressure without explicit madvise. The madvise sniper is most valuable for denser MoE models (Qwen 35B) where the per-token working set overwhelms the page cache.135 136## Related137 138- **MLX Expert Sniper** (Apple Silicon, 5.4 tok/s on 35B): [huggingface.co/waltgrace/mlx-expert-sniper](https://huggingface.co/waltgrace/mlx-expert-sniper)139- **Full research + code**: [github.com/walter-grace/mac-code/tree/main/research/expert-sniper](https://github.com/walter-grace/mac-code/tree/main/research/expert-sniper)140