Manojb/CUA_benchmark_local_small_models

1# Three-Model Comparison: Qwen3.5-9B vs Gemma4 E4B vs Bonsai-8B2 3**Date:** 2026-04-07 | **Hardware:** Mac Mini M4 16GB | **Vision:** Falcon Perception v24 5---6 7## Test Results8 9```10┌────┬───────────────────┬────────┬─────────────────┬─────────────────┬─────────────────┐11│    │                   │        │   Qwen3.5-9B    │   Gemma4 E4B    │   Bonsai-8B     │12│  # │ Task              │ Diff.  │ llama.cpp       │ mlx_vlm+proxy   │ prism llama.cpp │13├────┼───────────────────┼────────┼─────────────────┼─────────────────┼─────────────────┤14│  1 │ Wikipedia extract  │ Easy   │  51s  1T  ⚠️    │  30s  4T  ✅    │  33s  1T  ⚠️    │15│  2 │ DDG search         │ Medium │ 180s  6T  ⚠️    │  60s  6T  ✅    │  53s 16T  ❌    │16│  3 │ HN top story       │ Easy   │  32s  2T  ✅    │  30s  2T  ✅    │   4s  1T  ⚠️    │17│  4 │ Cat vision (FP)    │ Medium │  34s  2T  ✅    │  40s  3T  ✅    │   5s  1T  ❌    │18│  5 │ Form filling       │ Medium │ 237s  6T  ✅    │  60s  1T  ❌    │   4s  1T  ❌    │19│  6 │ reCAPTCHA          │ Hard   │ 300s  6T  ⚠️    │ 120s 13T  ⚠️    │   4s  1T  ❌    │20├────┼───────────────────┼────────┼─────────────────┼─────────────────┼─────────────────┤21│    │ PASS / PARTIAL     │        │  3✅ 2⚠️ 1❌    │  4✅ 1⚠️ 1❌    │  0✅ 2⚠️ 4❌    │22│    │ Total time         │        │     834s        │     340s        │     103s        │23│    │ Total tool calls   │        │      23         │      29         │      21         │24└────┴───────────────────┴────────┴─────────────────┴─────────────────┴─────────────────┘25 26T = tool calls | ✅ = pass | ⚠️ = partial | ❌ = fail27```28 29## Speed Comparison30 31```32Generation speed (tok/s):33 34Bonsai-8B    ████████████████████████████████████████████████  48.835Gemma4 E4B   ██████████████████████████████████               35.036Qwen3.5-9B   ██████████                                       10.037 38Total suite time:39 40Bonsai-8B    ██████                                  103s (1.7 min)41Gemma4 E4B   █████████████████████                   340s (5.7 min)42Qwen3.5-9B   █████████████████████████████████████████████████████  834s (13.9 min)43```44 45## Model Specs46 47```48┌──────────────────┬──────────────┬──────────────┬──────────────┐49│                  │ Qwen3.5-9B   │ Gemma4 E4B   │ Bonsai-8B    │50├──────────────────┼──────────────┼──────────────┼──────────────┤51│ Architecture     │ Dense        │ MoE (4B act) │ Dense 1-bit  │52│ Total params     │ 9B           │ 9B           │ 8B           │53│ Active params    │ 9B           │ 4B           │ 8B           │54│ Quantization     │ Q4_K_XL      │ 4-bit MLX    │ Q1_0_g128    │55│ Disk size        │ 5.6 GB       │ ~5 GB        │ 1.15 GB      │56│ Memory (peak)    │ ~6.5 GB      │ ~5.4 GB      │ ~1.5 GB      │57│ Generation tok/s │ ~10          │ ~35          │ ~49          │58│ Base model       │ Qwen3.5      │ Gemma4       │ Qwen3        │59│ Vision           │ ✅ mmproj     │ ✅ native     │ ❌ text only  │60│ Tool calling     │ ✅ native     │ ✅ patched    │ ✅ native     │61│ Proxy needed     │ ❌            │ ✅            │ ❌            │62│ Server           │ llama.cpp    │ mlx_vlm      │ PrismML fork │63└──────────────────┴──────────────┴──────────────┴──────────────┘64```65 66## Task-by-Task Analysis67 68### T1: Wikipedia Extract (Easy)69```70Qwen:  1 tool (scrape_url), got some info          ⚠️ Partial71Gemma: 4 tools (navigate+extract), all 3 answers   ✅ Pass72Bonsai: 1 tool (navigate), stopped too early        ⚠️ Partial73```74 75### T2: DuckDuckGo Search (Medium)76```77Qwen:  6 tools, hit 180s timeout                   ⚠️ Partial (slow)78Gemma: 6 tools (go+type+click+scroll+extract)      ✅ Pass79Bonsai: 16 tools but 21 idle turns, confused        ❌ Fail80```81 82### T3: HN Top Story (Easy)83```84Qwen:  2 tools, clean stop_loop                    ✅ Pass85Gemma: 2 tools, found title                        ✅ Pass86Bonsai: 1 tool, 4s but didn't extract properly     ⚠️ Partial87```88 89### T4: Cat Vision with Falcon (Medium)90```91Qwen:  2 tools incl vision_detect                  ✅ Pass92Gemma: 3 tools, 2x vision_detect, found 6 cats     ✅ Pass93Bonsai: 1 tool, never called vision_detect          ❌ Fail94```95 96### T5: Form Filling (Medium)97```98Qwen:  6 tools (go+input+click x3), stop_loop      ✅ Pass — ONLY WINNER99Gemma: 1 tool, stuck thinking about form fields     ❌ Fail100Bonsai: 1 tool, only navigated                      ❌ Fail101```102 103### T6: reCAPTCHA (Hard)104```105Qwen:  6 tools, 3 vision, 1 batch click, timeout   ⚠️ Partial106Gemma: 13 tools, 9 vision, 5 batch clicks, timeout ⚠️ Partial (most active)107Bonsai: 1 tool, only navigated                      ❌ Fail108```109 110## Scoring Matrix111 112```113┌───────────────────┬───────┬───────┬───────┐114│ Task              │ Qwen  │ Gemma │Bonsai │115├───────────────────┼───────┼───────┼───────┤116│ T1 Wikipedia      │  0.5  │  1.0  │  0.5  │117│ T2 DDG Search     │  0.5  │  1.0  │  0.0  │118│ T3 HN Story       │  1.0  │  1.0  │  0.5  │119│ T4 Cat Vision     │  1.0  │  1.0  │  0.0  │120│ T5 Form Fill      │  1.0  │  0.0  │  0.0  │121│ T6 reCAPTCHA      │  0.5  │  0.5  │  0.0  │122├───────────────────┼───────┼───────┼───────┤123│ TOTAL             │  4.5  │  4.5  │  1.0  │124│ Speed factor      │  1x   │  3.5x │  4.9x │125│ Score/minute      │  0.32 │  0.79 │  0.58 │126└───────────────────┴───────┴───────┴───────┘127 128Score/minute = total_score / suite_time_minutes129```130 131## Memory Budget on 16GB132 133```134┌──────────────────────┬──────────┬──────────┬──────────┐135│ Component            │ +Qwen    │ +Gemma4  │ +Bonsai  │136├──────────────────────┼──────────┼──────────┼──────────┤137│ LLM model            │ 6.5 GB   │ 5.4 GB   │ 1.5 GB   │138│ Falcon Perception    │ 1.5 GB   │ 1.5 GB   │ 1.5 GB   │139│ GUA_Blazor + Browser │ 0.8 GB   │ 0.8 GB   │ 0.8 GB   │140│ Proxy                │ —        │ 0.1 GB   │ —        │141│ OS + system          │ 3.0 GB   │ 3.0 GB   │ 3.0 GB   │142├──────────────────────┼──────────┼──────────┼──────────┤143│ TOTAL                │ 11.8 GB  │ 10.8 GB  │ 6.8 GB   │144│ HEADROOM             │ 4.2 GB   │ 5.2 GB   │ 9.2 GB ★ │145└──────────────────────┴──────────┴──────────┴──────────┘146 147Bonsai leaves 9.2 GB free — enough to run ANOTHER model simultaneously!148```149 150## Verdict151 152```153┌────────────────────────────────────────────────────────────────┐154│                                                                │155│  🏆 BEST OVERALL: Gemma4 E4B                                  │156│     4.5 score, 3.5x speed, best score/minute (0.79)          │157│     Wins: Wikipedia, DDG search, HN, Cat vision              │158│                                                                │159│  🥈 MOST RELIABLE: Qwen3.5-9B                                │160│     4.5 score, slowest but ONLY model that fills forms       │161│     Wins: Form filling (exclusive), HN, Cat vision           │162│                                                                │163│  🥉 FASTEST BUT WEAKEST: Bonsai-8B                           │164│     1.0 score, 4.9x speed, but can't do multi-step tasks    │165│     Only 1.15 GB — could pair with another model             │166│                                                                │167│  IDEAL SETUP:                                                  │168│  Bonsai-8B (1.5 GB) + Qwen3.5-9B (6.5 GB) = 8 GB            │169│  Route simple→Bonsai, complex→Qwen                            │170│  Both fit on 16GB simultaneously!                              │171│                                                                │172│  OR: Gemma4 E4B alone (best score/minute, most versatile)     │173│                                                                │174└────────────────────────────────────────────────────────────────┘175```176 177---178 179*All tests run on Mac Mini M4 16GB with identical GUA_Blazor agent loop,*180*Falcon Perception v2 vision backend, and same prompts.*181