Datasetpublic
YuvrajSingh9886/jetson-non-reasoning-benchmark-ollama-7w
Tiny LLM Benchmark — Jetson Orin Nano Super 8GB Date: 2026-06-09 02:38Backends: ollamaSweep: prompt ∈ {128,512,1024,2048} tok × gen ∈ {64,128,256} tokArtifacts: /home/yuvrajsingh/Desktop/benchmark/smolbenchmark/non-reasoning-models/artifacts/blog-all-20260607-0403-7w Full Results — ollama Power = VDD_CPU_GPU_CV avg over aiperf window. Model Quant ISL OSL OSL mis% TTFT avg p50 p90 p99 T2T avg p50 p90 p99 ITL avg p50 p90 p99 Tok/s Req/s E2E avg p50 p90 p99… See the full description on the dataset page: https://huggingface.co/datasets/YuvrajSingh9886/jetson-non-reasoning-benchmark-ollama-7w.
0likes495downloads
Dataset Card
Tiny LLM Benchmark — Jetson Orin Nano Super 8GB
Date: 2026-06-09 02:38 Backends: ollama Sweep: prompt ∈ {128,512,1024,2048} tok × gen ∈ {64,128,256} tok Artifacts: /home/yuvrajsingh/Desktop/benchmark/smolbenchmark/non-reasoning-models/artifacts/blog-all-20260607-0403-7w
Full Results — ollama
Power = VDD\CPU\GPU\_CV avg over aiperf window.
| Model | Quant | ISL | OSL | OSL mis% | TTFT avg | p50 | p90 | p99 | T2T avg | p50 | p90 | p99 | ITL avg | p50 | p90 | p99 | Tok/s | Req/s | E2E avg | p50 | p90 | p99 | RL avg | p50 | p90 | p99 | Prefill avg | p50 | p90 | p99 | Power (W) | **Tok/J** |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| gemma3-1b | Q4KM | 128 | 667.3 | 942.66 | 766.7 | 760.6 | 786.3 | 802.0 | 105.90 | 104.98 | 107.88 | 120.26 | 101.63 | 101.59 | 101.75 | 102.54 | 9.84 | 0.015 | 9.68 | 9.75 | 9.78 | 9.79 | 68480.8 | 70043.7 | 90250.7 | 104517.8 | 167.0 | 168.3 | 171.0 | 172.7 | 2.03 | 4.849 |
| gemma3-1b | Q4KM | 512 | 838.2 | 1209.69 | 1423.4 | 1424.0 | 1448.6 | 1468.5 | 105.61 | 104.28 | 108.79 | 126.40 | 101.86 | 101.74 | 101.91 | 103.87 | 9.82 | 0.012 | 9.66 | 9.68 | 9.72 | 9.75 | 86663.7 | 84273.0 | 118797.6 | 127062.2 | 359.8 | 359.6 | 366.9 | 369.6 | 2.03 | 4.838 |
| gemma3-1b | Q4KM | 1024 | 848.4 | 1225.62 | 2314.7 | 2308.1 | 2346.8 | 2358.4 | 104.67 | 103.79 | 106.61 | 122.89 | 102.03 | 102.04 | 102.17 | 102.29 | 9.80 | 0.011 | 9.55 | 9.55 | 9.61 | 9.65 | 88766.5 | 84340.0 | 115394.8 | 135929.3 | 442.4 | 443.7 | 446.3 | 448.0 | 2.03 | 4.830 |
| gemma3-1b | Q4KM | 2048 | 868.0 | 1256.25 | 4108.7 | 4108.7 | 4135.0 | 4148.7 | 103.70 | 103.15 | 107.30 | 118.84 | 102.29 | 102.27 | 102.37 | 103.03 | 9.78 | 0.010 | 9.33 | 9.32 | 9.47 | 9.52 | 92789.6 | 85619.2 | 128385.5 | 148911.4 | 498.5 | 498.5 | 501.7 | 502.7 | 2.03 | 4.818 |
| gemma3-1b | Q4KM | 128 | 654.0 | 410.94 | 763.6 | 768.1 | 780.0 | 787.8 | 105.05 | 104.82 | 110.00 | 116.03 | 101.49 | 101.46 | 101.71 | 102.70 | 9.85 | 0.015 | 9.63 | 9.76 | 9.78 | 9.80 | 67071.2 | 71669.0 | 86612.9 | 92955.4 | 167.7 | 166.6 | 172.4 | 174.2 | 2.03 | 4.856 |
| gemma3-1b | Q4KM | 512 | 826.2 | 545.51 | 1432.9 | 1425.5 | 1458.2 | 1475.9 | 105.01 | 105.03 | 107.22 | 109.51 | 101.73 | 101.75 | 101.87 | 101.97 | 9.83 | 0.012 | 9.67 | 9.67 | 9.71 | 9.72 | 85382.9 | 85018.6 | 113417.7 | 116843.8 | 357.4 | 359.2 | 362.9 | 366.2 | 2.03 | 4.844 |
| gemma3-1b | Q4KM | 1024 | 885.5 | 591.80 | 2318.9 | 2315.5 | 2339.1 | 2373.1 | 104.23 | 104.83 | 107.33 | 115.13 | 102.02 | 102.00 | 102.21 | 102.30 | 9.80 | 0.011 | 9.56 | 9.57 | 9.61 | 9.64 | 92553.6 | 90159.6 | 115218.1 | 118175.2 | 441.6 | 442.2 | 446.2 | 448.1 | 2.03 | 4.830 |
| gemma3-1b | Q4KM | 2048 | 891.2 | 596.29 | 4116.3 | 4111.4 | 4148.7 | 4200.9 | 105.87 | 104.52 | 112.06 | 118.07 | 102.40 | 102.36 | 102.59 | 103.52 | 9.77 | 0.010 | 9.34 | 9.30 | 9.47 | 9.53 | 95259.6 | 84162.5 | 125780.9 | 149322.6 | 497.6 | 498.1 | 500.8 | 502.3 | 2.03 | 4.813 |
| gemma3-1b | Q4KM | 128 | 711.5 | 177.91 | 763.7 | 764.9 | 789.1 | 819.5 | 105.62 | 104.75 | 107.59 | 117.93 | 101.44 | 101.46 | 101.62 | 101.75 | 9.86 | 0.014 | 9.70 | 9.77 | 9.80 | 9.80 | 72844.7 | 76269.9 | 90235.7 | 115879.1 | 167.7 | 167.3 | 173.0 | 175.0 | 2.03 | 4.858 |
| gemma3-1b | Q4KM | 512 | 850.9 | 232.38 | 1427.2 | 1426.6 | 1457.8 | 1468.7 | 103.86 | 104.92 | 106.72 | 107.81 | 101.65 | 101.64 | 101.81 | 101.84 | 9.84 | 0.011 | 9.68 | 9.68 | 9.72 | 9.72 | 87823.0 | 90867.1 | 110898.9 | 113031.5 | 358.8 | 358.9 | 365.4 | 366.9 | 2.03 | 4.848 |
| gemma3-1b | Q4KM | 1024 | 891.4 | 248.20 | 2322.5 | 2317.0 | 2383.1 | 2385.6 | 105.04 | 104.89 | 107.77 | 113.78 | 101.94 | 101.93 | 102.09 | 102.23 | 9.81 | 0.011 | 9.56 | 9.57 | 9.63 | 9.64 | 93083.8 | 95107.6 | 118444.7 | 129214.7 | 441.0 | 442.0 | 446.5 | 449.3 | 2.03 | 4.834 |
| gemma3-1b | Q4KM | 2048 | 969.2 | 278.60 | 4102.0 | 4099.7 | 4136.8 | 4146.0 | 105.19 | 105.36 | 107.35 | 108.38 | 102.33 | 102.32 | 102.49 | 102.88 | 9.77 | 0.010 | 9.37 | 9.35 | 9.48 | 9.54 | 103200.7 | 89499.8 | 146287.1 | 162677.7 | 499.3 | 499.5 | 502.4 | 504.7 | 2.03 | 4.816 |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| gemma3-4b | Q4KM | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.35 | — |
| lfm2.5-1.2b | Q4KM | 128 | 441.1 | 589.14 | 537.4 | 537.7 | 558.8 | 581.3 | 100.85 | 98.82 | 113.32 | 117.97 | 101.34 | 102.02 | 103.94 | 104.93 | 9.89 | 0.022 | 9.78 | 9.69 | 9.82 | 11.53 | 45056.7 | 39377.1 | 63696.1 | 110454.3 | 238.5 | 238.0 | 247.2 | 256.1 | 2.04 | 4.836 |
| lfm2.5-1.2b | Q4KM | 512 | 631.2 | 886.33 | 1256.7 | 1251.5 | 1280.1 | 1287.6 | 102.13 | 99.33 | 112.37 | 119.91 | 100.96 | 101.89 | 102.32 | 102.98 | 9.93 | 0.015 | 9.71 | 9.63 | 9.72 | 11.50 | 64691.1 | 64475.4 | 94411.4 | 110982.6 | 407.5 | 409.1 | 411.7 | 416.6 | 2.04 | 4.854 |
| lfm2.5-1.2b | Q4KM | 1024 | 700.0 | 993.67 | 2251.3 | 2247.5 | 2279.0 | 2297.2 | 101.44 | 99.00 | 111.82 | 120.64 | 101.32 | 102.40 | 102.88 | 103.48 | 9.89 | 0.014 | 9.55 | 9.50 | 9.63 | 11.42 | 72779.1 | 70213.8 | 104572.1 | 133522.2 | 454.9 | 455.6 | 460.1 | 460.5 | 2.04 | 4.838 |
| lfm2.5-1.2b | Q4KM | 2048 | 633.0 | 888.98 | 4239.9 | 4237.9 | 4262.9 | 4274.7 | 99.14 | 97.60 | 109.71 | 118.66 | 101.43 | 102.37 | 103.10 | 103.34 | 9.88 | 0.015 | 9.20 | 9.22 | 9.35 | 10.77 | 68330.9 | 73569.8 | 94588.8 | 100814.8 | 483.0 | 483.3 | 485.3 | 486.8 | 2.04 | 4.830 |
| lfm2.5-1.2b | Q4KM | 128 | 421.6 | 229.41 | 531.3 | 527.0 | 560.2 | 588.8 | 106.57 | 102.99 | 128.30 | 143.39 | 101.86 | 102.68 | 104.93 | 106.41 | 9.84 | 0.023 | 9.73 | 9.64 | 9.81 | 11.46 | 43533.8 | 41593.2 | 62703.6 | 76829.5 | 241.4 | 242.9 | 248.8 | 260.9 | 2.04 | 4.813 |
| lfm2.5-1.2b | Q4KM | 512 | 657.6 | 413.79 | 1254.4 | 1254.3 | 1273.0 | 1280.1 | 100.85 | 99.10 | 115.55 | 121.70 | 100.94 | 101.93 | 102.59 | 102.83 | 9.93 | 0.015 | 9.71 | 9.65 | 9.72 | 11.57 | 67240.1 | 62122.3 | 100248.0 | 103010.0 | 408.2 | 408.2 | 414.4 | 418.3 | 2.04 | 4.855 |
| lfm2.5-1.2b | Q4KM | 1024 | 682.6 | 433.28 | 2251.8 | 2250.2 | 2270.4 | 2281.3 | 101.45 | 99.25 | 113.76 | 116.11 | 101.15 | 101.97 | 102.74 | 103.25 | 9.91 | 0.014 | 9.53 | 9.49 | 9.65 | 11.14 | 71263.4 | 74849.2 | 104731.2 | 117716.0 | 454.8 | 455.1 | 459.1 | 460.6 | 2.04 | 4.844 |
| lfm2.5-1.2b | Q4KM | 2048 | 751.8 | 487.30 | 4245.2 | 4241.3 | 4277.8 | 4284.0 | 95.43 | 95.35 | 104.64 | 108.29 | 101.47 | 102.39 | 103.19 | 103.80 | 9.88 | 0.012 | 9.34 | 9.29 | 9.39 | 10.83 | 80516.3 | 81503.1 | 98365.9 | 109592.8 | 482.4 | 482.9 | 485.0 | 486.4 | 2.04 | 4.829 |
| lfm2.5-1.2b | Q4KM | 128 | 451.1 | 76.19 | 531.0 | 529.0 | 552.4 | 557.6 | 104.56 | 102.34 | 117.95 | 124.65 | 101.06 | 102.06 | 102.67 | 103.41 | 9.92 | 0.022 | 9.80 | 9.71 | 9.79 | 11.67 | 45944.4 | 53800.1 | 70000.3 | 74804.1 | 241.2 | 242.0 | 248.9 | 250.9 | 2.04 | 4.850 |
| lfm2.5-1.2b | Q4KM | 512 | 620.5 | 142.40 | 1262.0 | 1262.0 | 1280.2 | 1292.7 | 104.35 | 103.42 | 116.87 | 120.54 | 101.05 | 101.89 | 102.68 | 102.95 | 9.92 | 0.016 | 9.71 | 9.63 | 9.71 | 11.49 | 63814.8 | 61579.6 | 94077.1 | 112557.0 | 405.8 | 405.7 | 410.7 | 420.1 | 2.04 | 4.850 |
| lfm2.5-1.2b | Q4KM | 1024 | 656.0 | 156.23 | 2242.7 | 2244.3 | 2255.9 | 2260.9 | 102.32 | 99.90 | 114.48 | 122.53 | 101.00 | 101.95 | 102.67 | 102.93 | 9.92 | 0.015 | 9.56 | 9.45 | 9.66 | 11.33 | 68327.9 | 62525.3 | 104609.0 | 133945.5 | 456.6 | 456.3 | 459.9 | 460.9 | 2.04 | 4.852 |
| lfm2.5-1.2b | Q4KM | 2048 | 684.5 | 167.38 | 4244.0 | 4248.7 | 4264.0 | 4271.5 | 101.26 | 100.17 | 110.06 | 126.73 | 101.48 | 102.33 | 103.41 | 103.76 | 9.87 | 0.014 | 9.21 | 9.22 | 9.45 | 10.04 | 73943.5 | 70156.6 | 110606.5 | 123854.5 | 482.6 | 482.0 | 485.4 | 487.0 | 2.04 | 4.829 |
| lfm2.5-350m | Q4KM | 128 | 241.4 | 277.27 | 301.0 | 298.9 | 322.0 | 330.5 | 65.51 | 63.47 | 76.56 | 96.12 | 63.39 | 63.96 | 65.94 | 68.66 | 15.86 | 0.064 | 15.00 | 15.31 | 15.64 | 19.75 | 15594.5 | 11409.5 | 29607.5 | 47544.3 | 426.2 | 428.2 | 448.0 | 451.7 | 1.71 | 9.262 |
| lfm2.5-350m | Q4KM | 512 | 359.4 | 461.56 | 555.8 | 556.9 | 573.7 | 585.7 | 63.78 | 60.74 | 75.67 | 92.16 | 63.27 | 63.89 | 64.95 | 65.66 | 15.87 | 0.043 | 15.21 | 15.33 | 15.53 | 18.45 | 23411.8 | 20824.3 | 36216.8 | 63100.5 | 922.0 | 919.4 | 954.9 | 978.3 | 1.71 | 9.268 |
| lfm2.5-350m | Q4KM | 1024 | 376.4 | 488.05 | 904.2 | 903.1 | 922.8 | 928.8 | 64.78 | 66.48 | 74.23 | 80.18 | 63.64 | 64.08 | 65.65 | 66.07 | 15.78 | 0.041 | 14.23 | 14.88 | 15.40 | 19.13 | 24560.7 | 26741.3 | 47073.1 | 51144.2 | 1132.8 | 1133.8 | 1153.6 | 1166.0 | 1.71 | 9.212 |
| lfm2.5-350m | Q4KM | 2048 | 426.2 | 566.02 | 1617.3 | 1612.5 | 1651.6 | 1666.3 | 61.80 | 61.37 | 71.95 | 79.03 | 63.88 | 64.25 | 66.09 | 66.68 | 15.72 | 0.035 | 13.75 | 14.75 | 15.11 | 18.53 | 28607.8 | 31250.3 | 46340.3 | 56664.3 | 1266.6 | 1270.1 | 1288.1 | 1303.5 | 1.71 | 9.176 |
| lfm2.5-350m | Q4KM | 128 | 216.1 | 68.79 | 307.5 | 308.9 | 320.4 | 344.3 | 67.52 | 63.64 | 81.03 | 83.72 | 63.29 | 63.50 | 65.29 | 66.17 | 15.82 | 0.071 | 15.03 | 15.39 | 15.72 | 15.81 | 13951.9 | 13226.3 | 33892.3 | 35609.3 | 417.1 | 414.3 | 441.4 | 449.9 | 1.71 | 9.238 |
| lfm2.5-350m | Q4KM | 512 | 330.6 | 158.24 | 556.7 | 555.7 | 577.8 | 582.5 | 63.35 | 62.25 | 79.75 | 81.37 | 63.07 | 63.74 | 65.46 | 69.14 | 15.95 | 0.047 | 14.49 | 15.21 | 15.69 | 20.08 | 21254.3 | 24774.5 | 33551.5 | 41115.5 | 920.4 | 921.4 | 947.7 | 981.3 | 1.71 | 9.312 |
| lfm2.5-350m | Q4KM | 1024 | 388.1 | 203.24 | 906.7 | 904.8 | 934.1 | 946.5 | 63.77 | 63.86 | 72.73 | 78.87 | 63.52 | 63.94 | 65.24 | 66.41 | 15.79 | 0.039 | 14.93 | 15.15 | 15.39 | 17.43 | 25677.8 | 30560.5 | 40048.0 | 41727.8 | 1129.8 | 1131.8 | 1158.1 | 1169.2 | 1.71 | 9.220 |
| lfm2.5-350m | Q4KM | 2048 | 438.3 | 242.42 | 1604.8 | 1603.1 | 1641.5 | 1650.8 | 62.70 | 63.04 | 74.20 | 79.27 | 63.30 | 64.43 | 65.02 | 65.68 | 15.87 | 0.034 | 14.23 | 14.67 | 15.06 | 19.07 | 29324.0 | 30372.0 | 46395.1 | 52672.6 | 1276.4 | 1277.5 | 1293.7 | 1309.3 | 1.71 | 9.269 |
| lfm2.5-350m | Q4KM | 128 | 261.8 | 2.25 | 297.1 | 295.5 | 314.3 | 332.2 | 61.73 | 61.36 | 69.43 | 72.32 | 63.70 | 63.25 | 65.91 | 74.13 | 15.77 | 0.059 | 14.61 | 15.58 | 15.77 | 17.73 | 16771.4 | 14925.3 | 31120.4 | 45684.6 | 432.2 | 433.1 | 458.0 | 465.3 | 1.71 | 9.209 |
| lfm2.5-350m | Q4KM | 512 | 352.8 | 37.81 | 553.4 | 551.0 | 571.4 | 580.1 | 62.90 | 58.81 | 77.81 | 85.10 | 63.28 | 64.16 | 64.82 | 65.14 | 15.90 | 0.045 | 15.41 | 15.20 | 15.49 | 20.63 | 22329.1 | 22457.2 | 38467.0 | 39703.8 | 925.8 | 929.3 | 948.8 | 977.4 | 1.71 | 9.282 |
| lfm2.5-350m | Q4KM | 1024 | 422.9 | 65.20 | 908.7 | 904.8 | 925.0 | 947.3 | 63.97 | 64.29 | 77.34 | 86.25 | 63.32 | 64.10 | 64.76 | 67.38 | 15.88 | 0.036 | 14.55 | 15.14 | 15.38 | 19.12 | 27770.5 | 27141.5 | 46200.7 | 68561.7 | 1127.3 | 1131.8 | 1143.5 | 1175.1 | 1.71 | 9.272 |
| lfm2.5-350m | Q4KM | 2048 | 409.9 | 60.14 | 1607.3 | 1601.7 | 1626.5 | 1639.6 | 60.33 | 60.46 | 73.03 | 76.56 | 63.71 | 64.62 | 65.17 | 65.61 | 15.77 | 0.036 | 14.21 | 14.70 | 14.94 | 19.10 | 27506.5 | 27571.8 | 38974.3 | 57381.4 | 1274.4 | 1278.6 | 1289.4 | 1300.4 | 1.71 | 9.208 |
| llama3.2-1b | Q4KM | 128 | 331.6 | 418.20 | 726.8 | 731.6 | 742.5 | 749.0 | 83.89 | 84.44 | 85.04 | 85.39 | 82.99 | 82.80 | 83.54 | 84.66 | 12.05 | 0.035 | 11.53 | 11.81 | 11.90 | 11.94 | 28133.2 | 28916.1 | 50468.7 | 58325.4 | 176.2 | 175.0 | 180.7 | 182.4 | 2.26 | 5.324 |
| llama3.2-1b | Q4KM | 512 | 356.3 | 456.72 | 1464.1 | 1462.1 | 1476.7 | 1498.8 | 85.20 | 85.56 | 86.24 | 86.50 | 83.09 | 83.03 | 83.30 | 83.41 | 12.04 | 0.032 | 11.08 | 11.40 | 11.71 | 11.79 | 31006.8 | 25704.7 | 52665.9 | 81416.2 | 349.7 | 350.2 | 352.4 | 354.3 | 2.26 | 5.317 |
| llama3.2-1b | Q4KM | 1024 | 531.9 | 731.09 | 2428.8 | 2429.6 | 2445.2 | 2465.7 | 84.97 | 85.22 | 85.62 | 86.12 | 83.70 | 83.64 | 83.86 | 84.37 | 11.95 | 0.021 | 11.26 | 11.29 | 11.55 | 11.64 | 46879.0 | 42101.3 | 69419.8 | 109031.3 | 421.6 | 421.5 | 425.2 | 426.8 | 2.26 | 5.278 |
| llama3.2-1b | Q4KM | 2048 | 446.5 | 597.70 | 4400.2 | 4400.9 | 4421.7 | 4427.5 | 86.27 | 86.46 | 86.88 | 87.19 | 84.62 | 84.57 | 84.79 | 84.90 | 11.82 | 0.019 | 10.52 | 10.53 | 10.88 | 11.01 | 42110.6 | 39333.2 | 54841.5 | 66181.6 | 465.4 | 465.4 | 467.7 | 468.7 | 2.26 | 5.221 |
| llama3.2-1b | Q4KM | 128 | 285.3 | 122.89 | 732.0 | 728.1 | 752.8 | 772.0 | 84.45 | 84.63 | 85.67 | 86.52 | 82.70 | 82.66 | 82.83 | 83.27 | 12.09 | 0.041 | 11.45 | 11.71 | 11.89 | 11.90 | 24259.0 | 20538.5 | 40627.5 | 60722.5 | 175.0 | 175.8 | 179.9 | 180.7 | 2.26 | 5.342 |
| llama3.2-1b | Q4KM | 512 | 478.4 | 273.71 | 1474.0 | 1476.3 | 1491.5 | 1509.8 | 85.47 | 85.73 | 86.34 | 86.78 | 83.15 | 83.07 | 83.50 | 83.77 | 12.03 | 0.024 | 11.53 | 11.59 | 11.69 | 11.79 | 41218.8 | 37067.4 | 61191.7 | 101338.3 | 347.4 | 346.8 | 352.2 | 355.1 | 2.26 | 5.313 |
| llama3.2-1b | Q4KM | 1024 | 496.9 | 288.24 | 2425.6 | 2427.4 | 2437.4 | 2474.5 | 85.23 | 85.31 | 86.17 | 86.66 | 83.68 | 83.64 | 83.87 | 84.18 | 11.95 | 0.023 | 11.18 | 11.33 | 11.43 | 11.64 | 43961.5 | 44060.7 | 56045.8 | 121396.5 | 422.2 | 421.8 | 425.2 | 426.0 | 2.26 | 5.279 |
| llama3.2-1b | Q4KM | 2048 | 490.6 | 283.28 | 4402.2 | 4395.9 | 4435.6 | 4447.3 | 86.30 | 86.35 | 86.90 | 87.95 | 84.66 | 84.58 | 84.93 | 85.56 | 11.81 | 0.022 | 10.49 | 10.58 | 11.10 | 11.24 | 45886.3 | 40882.5 | 73418.4 | 92718.7 | 465.2 | 465.9 | 467.4 | 467.5 | 2.26 | 5.218 |
| llama3.2-1b | Q4KM | 128 | 299.4 | 16.95 | 730.8 | 731.1 | 756.3 | 775.7 | 83.78 | 84.29 | 84.79 | 85.04 | 82.78 | 82.71 | 83.01 | 83.39 | 12.08 | 0.039 | 11.60 | 11.75 | 11.89 | 11.90 | 25430.2 | 25263.7 | 37245.3 | 42585.0 | 175.3 | 175.1 | 181.5 | 184.4 | 2.26 | 5.337 |
| llama3.2-1b | Q4KM | 512 | 486.8 | 90.16 | 1466.5 | 1470.5 | 1479.5 | 1492.7 | 85.18 | 84.78 | 86.67 | 91.09 | 83.19 | 83.15 | 83.54 | 83.55 | 12.02 | 0.024 | 11.53 | 11.59 | 11.76 | 11.79 | 41904.2 | 37963.4 | 64974.4 | 89978.6 | 349.2 | 348.2 | 353.5 | 356.9 | 2.26 | 5.310 |
| llama3.2-1b | Q4KM | 1024 | 596.7 | 133.09 | 2435.2 | 2438.4 | 2456.5 | 2476.9 | 85.25 | 85.38 | 85.94 | 86.00 | 83.75 | 83.70 | 84.16 | 84.34 | 11.94 | 0.019 | 11.07 | 11.24 | 11.55 | 11.68 | 52405.8 | 40618.1 | 73761.3 | 163229.6 | 420.5 | 419.9 | 425.8 | 428.4 | 2.26 | 5.275 |
| llama3.2-1b | Q4KM | 2048 | 546.1 | 113.34 | 4399.7 | 4401.2 | 4430.9 | 4448.1 | 86.20 | 86.43 | 87.18 | 87.45 | 84.63 | 84.62 | 84.82 | 84.93 | 11.82 | 0.020 | 10.53 | 10.61 | 11.05 | 11.42 | 50567.5 | 43111.4 | 69645.7 | 137439.6 | 465.5 | 465.3 | 468.7 | 469.3 | 2.26 | 5.221 |
| qwen2.5-0.5b | Q4KM | 128 | 222.8 | 248.09 | 510.3 | 503.7 | 533.8 | 554.0 | 61.44 | 61.44 | 62.25 | 62.53 | 58.26 | 58.12 | 58.81 | 58.87 | 17.17 | 0.030 | 16.05 | 16.24 | 16.78 | 16.86 | 13485.9 | 7513.1 | 35587.5 | 42412.2 | 251.1 | 254.1 | 258.6 | 263.3 | 1.96 | 8.757 |
| qwen2.5-0.5b | Q4KM | 512 | 521.1 | 714.26 | 879.6 | 881.0 | 903.1 | 911.3 | 62.02 | 62.23 | 62.86 | 63.11 | 59.03 | 58.98 | 59.27 | 60.34 | 16.94 | 0.013 | 15.41 | 16.38 | 16.61 | 16.68 | 31779.5 | 23242.8 | 47486.6 | 136120.3 | 582.4 | 581.2 | 599.0 | 604.4 | 1.96 | 8.642 |
| qwen2.5-0.5b | Q4KM | 1024 | 622.6 | 872.89 | 1396.4 | 1389.1 | 1414.3 | 1440.9 | 62.57 | 62.54 | 63.43 | 63.54 | 59.66 | 59.61 | 60.11 | 60.32 | 16.76 | 0.020 | 14.82 | 15.77 | 16.41 | 16.43 | 38658.5 | 21306.2 | 99527.4 | 132797.0 | 733.5 | 737.2 | 745.2 | 748.1 | 1.96 | 8.551 |
| qwen2.5-0.5b | Q4KM | 2048 | 796.9 | 1145.13 | 2566.3 | 2561.1 | 2601.0 | 2610.3 | 62.90 | 62.90 | 63.34 | 64.96 | 60.69 | 60.64 | 60.95 | 61.10 | 16.48 | 0.014 | 15.04 | 15.35 | 16.03 | 16.05 | 50962.0 | 35902.4 | 101312.0 | 114917.7 | 798.1 | 799.7 | 804.0 | 808.3 | 1.96 | 8.406 |
| qwen2.5-0.5b | Q4KM | 128 | 252.5 | 97.27 | 509.8 | 511.0 | 525.4 | 529.3 | 61.67 | 61.75 | 62.15 | 62.28 | 58.27 | 58.18 | 58.75 | 58.98 | 17.16 | 0.065 | 16.35 | 16.47 | 16.70 | 16.83 | 15221.0 | 11635.3 | 19823.8 | 71188.3 | 251.3 | 250.5 | 260.0 | 267.8 | 1.96 | 8.755 |
| qwen2.5-0.5b | Q4KM | 512 | 428.6 | 234.85 | 872.0 | 866.6 | 899.2 | 905.0 | 62.28 | 62.34 | 62.90 | 63.47 | 59.10 | 58.95 | 59.67 | 59.97 | 16.92 | 0.022 | 15.68 | 16.09 | 16.45 | 16.66 | 26294.0 | 14962.4 | 42595.8 | 141635.8 | 587.4 | 590.8 | 598.5 | 605.8 | 1.96 | 8.633 |
| qwen2.5-0.5b | Q4KM | 1024 | 757.3 | 491.65 | 1400.5 | 1402.6 | 1415.8 | 1423.8 | 62.32 | 62.80 | 63.35 | 63.76 | 59.68 | 59.56 | 60.22 | 60.31 | 16.76 | 0.014 | 15.60 | 15.92 | 16.41 | 16.42 | 46781.6 | 24629.4 | 112218.4 | 138101.1 | 731.2 | 730.1 | 740.2 | 744.5 | 1.96 | 8.548 |
| qwen2.5-0.5b | Q4KM | 2048 | 874.9 | 583.51 | 2567.2 | 2565.5 | 2592.4 | 2600.6 | 62.67 | 63.05 | 63.71 | 68.54 | 60.81 | 60.86 | 61.07 | 61.18 | 16.44 | 0.013 | 15.22 | 15.44 | 16.04 | 16.14 | 55777.0 | 45190.7 | 108392.2 | 146881.4 | 797.8 | 798.3 | 804.0 | 808.8 | 1.96 | 8.389 |
| qwen2.5-0.5b | Q4KM | 128 | 294.6 | 15.10 | 505.6 | 501.7 | 525.4 | 542.0 | 61.46 | 61.75 | 62.77 | 63.64 | 58.28 | 58.22 | 58.59 | 59.29 | 17.16 | 0.056 | 16.15 | 16.50 | 16.78 | 16.86 | 17742.4 | 10724.3 | 22662.2 | 106889.6 | 253.4 | 255.1 | 259.4 | 264.6 | 1.96 | 8.754 |
| qwen2.5-0.5b | Q4KM | 512 | 350.1 | 36.76 | 872.9 | 873.2 | 890.2 | 907.3 | 62.34 | 62.20 | 63.40 | 64.22 | 59.00 | 58.84 | 59.44 | 59.55 | 16.95 | 0.019 | 15.80 | 16.11 | 16.53 | 16.65 | 21501.8 | 16792.5 | 41505.7 | 54350.8 | 586.7 | 586.3 | 596.1 | 610.0 | 1.96 | 8.647 |
| qwen2.5-0.5b | Q4KM | 1024 | 551.5 | 115.44 | 1391.2 | 1387.0 | 1411.2 | 1420.6 | 62.19 | 62.72 | 63.08 | 63.24 | 59.54 | 59.38 | 60.07 | 60.16 | 16.79 | 0.015 | 15.47 | 15.56 | 16.38 | 16.42 | 34323.4 | 17701.8 | 86487.6 | 96269.1 | 736.1 | 738.3 | 746.0 | 750.6 | 1.96 | 8.568 |
| qwen2.5-0.5b | Q4KM | 2048 | 629.9 | 146.05 | 2559.0 | 2559.5 | 2574.8 | 2583.8 | 63.12 | 63.19 | 63.39 | 63.55 | 60.71 | 60.76 | 61.01 | 61.32 | 16.47 | 0.020 | 14.77 | 15.05 | 15.88 | 16.07 | 40794.8 | 28868.4 | 74707.2 | 113939.5 | 800.3 | 800.2 | 806.1 | 811.1 | 1.96 | 8.403 |
| qwen3-0.6b | Q8_0 | 128 | 791.4 | 1136.48 | 515.8 | 512.2 | 538.6 | 587.8 | 64.26 | 64.96 | 66.80 | 67.50 | 64.44 | 64.41 | 64.98 | 65.18 | 15.52 | 0.019 | 15.37 | 15.38 | 15.44 | 15.46 | 51535.5 | 48919.0 | 69884.8 | 78526.8 | 248.6 | 249.9 | 256.0 | 264.3 | 2.08 | 7.473 |
| qwen3-0.6b | Q8_0 | 512 | 783.4 | 1124.06 | 1013.5 | 1008.2 | 1027.3 | 1107.6 | 67.67 | 67.60 | 69.05 | 71.25 | 65.96 | 65.98 | 66.84 | 66.95 | 15.16 | 0.019 | 14.84 | 14.87 | 14.91 | 15.04 | 52752.9 | 52126.7 | 80044.2 | 83287.6 | 505.6 | 507.8 | 520.4 | 520.9 | 2.08 | 7.302 |
| qwen3-0.6b | Q8_0 | 1024 | 743.4 | 1061.48 | 1659.6 | 1662.9 | 1677.3 | 1678.0 | 70.48 | 69.32 | 73.62 | 82.84 | 67.91 | 68.01 | 68.55 | 68.86 | 14.73 | 0.019 | 14.22 | 14.26 | 14.31 | 14.32 | 52193.6 | 51440.8 | 75073.4 | 88673.0 | 617.1 | 615.8 | 623.8 | 626.9 | 2.08 | 7.092 |
| qwen3-0.6b | Q8_0 | 2048 | 816.5 | 1175.78 | 3081.4 | 3072.8 | 3104.5 | 3170.3 | 74.20 | 73.72 | 77.64 | 83.68 | 71.45 | 71.50 | 71.70 | 71.79 | 14.00 | 0.010 | 13.13 | 13.19 | 13.47 | 13.67 | 61438.3 | 52636.6 | 88928.5 | 166043.0 | 664.7 | 666.5 | 669.6 | 670.5 | 2.08 | 6.741 |
| qwen3-0.6b | Q8_0 | 128 | 724.0 | 465.66 | 515.3 | 510.9 | 534.2 | 570.2 | 65.98 | 65.51 | 68.36 | 75.44 | 64.31 | 64.30 | 64.90 | 65.31 | 15.55 | 0.021 | 15.39 | 15.40 | 15.47 | 15.50 | 47113.3 | 43084.0 | 67206.4 | 82788.5 | 248.7 | 250.5 | 256.2 | 261.5 | 2.08 | 7.489 |
| qwen3-0.6b | Q8_0 | 512 | 920.0 | 618.75 | 1003.0 | 1003.6 | 1015.1 | 1028.1 | 68.25 | 67.99 | 69.72 | 77.04 | 66.24 | 66.18 | 66.96 | 67.68 | 15.10 | 0.016 | 14.83 | 14.85 | 14.91 | 14.92 | 62084.1 | 60678.3 | 91467.5 | 112267.0 | 510.5 | 510.2 | 518.5 | 520.5 | 2.08 | 7.271 |
| qwen3-0.6b | Q8_0 | 1024 | 825.4 | 544.84 | 1667.0 | 1659.9 | 1690.5 | 1745.8 | 70.29 | 69.55 | 71.70 | 82.17 | 68.02 | 68.09 | 68.50 | 69.05 | 14.70 | 0.017 | 14.24 | 14.28 | 14.32 | 14.36 | 57863.8 | 60990.4 | 74171.3 | 95722.6 | 614.5 | 616.9 | 622.6 | 627.8 | 2.08 | 7.080 |
| qwen3-0.6b | Q8_0 | 2048 | 1071.9 | 737.46 | 3075.3 | 3074.1 | 3098.6 | 3126.0 | 73.84 | 74.43 | 76.74 | 77.31 | 71.58 | 71.66 | 71.81 | 71.90 | 13.97 | 0.010 | 13.29 | 13.38 | 13.60 | 13.67 | 79820.4 | 74712.0 | 126697.5 | 159048.5 | 666.0 | 666.2 | 670.0 | 674.2 | 2.08 | 6.728 |
| qwen3-0.6b | Q8_0 | 128 | 708.6 | 176.80 | 511.3 | 513.5 | 529.1 | 532.0 | 64.64 | 65.26 | 66.76 | 68.00 | 64.26 | 64.20 | 64.83 | 65.13 | 15.56 | 0.022 | 15.40 | 15.41 | 15.45 | 15.47 | 46056.3 | 44670.8 | 64438.2 | 74721.5 | 250.5 | 249.3 | 261.1 | 264.0 | 2.08 | 7.495 |
| qwen3-0.6b | Q8_0 | 512 | 729.5 | 184.94 | 1002.8 | 1001.1 | 1016.0 | 1025.6 | 68.45 | 67.75 | 70.56 | 81.82 | 65.88 | 65.96 | 66.47 | 66.68 | 15.18 | 0.020 | 14.85 | 14.87 | 14.89 | 14.90 | 49095.8 | 47559.9 | 72279.3 | 75919.0 | 510.6 | 511.4 | 518.2 | 521.0 | 2.08 | 7.311 |
| qwen3-0.6b | Q8_0 | 1024 | 853.7 | 233.49 | 1660.6 | 1655.9 | 1678.9 | 1700.0 | 69.42 | 69.59 | 71.68 | 72.19 | 68.13 | 68.06 | 68.78 | 69.47 | 14.68 | 0.014 | 14.23 | 14.28 | 14.30 | 14.34 | 59910.5 | 61028.2 | 82081.5 | 101982.0 | 616.7 | 618.4 | 621.7 | 624.4 | 2.08 | 7.069 |
| qwen3-0.6b | Q8_0 | 2048 | 1163.9 | 354.64 | 3092.2 | 3071.1 | 3106.5 | 3301.8 | 75.42 | 73.70 | 81.05 | 95.18 | 71.57 | 71.56 | 71.79 | 71.84 | 13.97 | 0.008 | 13.31 | 13.32 | 13.66 | 13.70 | 86424.5 | 65032.1 | 157392.4 | 174863.9 | 662.6 | 666.9 | 670.4 | 671.2 | 2.08 | 6.729 |
| smollm2-135m | Q4KM | 128 | 461.2 | 620.70 | 224.0 | 224.2 | 242.3 | 249.8 | 28.79 | 29.41 | 31.90 | 34.24 | 25.96 | 25.92 | 26.60 | 27.02 | 38.52 | 0.081 | 36.92 | 37.45 | 38.04 | 38.43 | 12340.7 | 5837.3 | 34564.0 | 49976.0 | 574.5 | 571.2 | 638.1 | 645.0 | 1.81 | 21.329 |
| smollm2-135m | Q4KM | 512 | 286.1 | 347.03 | 410.2 | 409.4 | 423.0 | 427.5 | 28.95 | 29.81 | 31.02 | 32.43 | 26.22 | 26.17 | 26.56 | 26.77 | 38.14 | 0.126 | 33.18 | 36.08 | 37.16 | 37.25 | 7883.9 | 6580.8 | 17384.0 | 23268.6 | 1248.8 | 1250.7 | 1281.3 | 1285.2 | 1.81 | 21.116 |
| smollm2-135m | Q4KM | 1024 | 501.9 | 684.14 | 700.2 | 699.3 | 707.5 | 729.1 | 29.59 | 30.22 | 30.54 | 30.94 | 26.59 | 26.51 | 26.79 | 27.41 | 37.61 | 0.070 | 33.25 | 34.87 | 36.06 | 36.33 | 14172.1 | 8844.5 | 19366.1 | 65200.4 | 1462.7 | 1464.2 | 1483.2 | 1489.3 | 1.81 | 20.822 |
| smollm2-135m | Q4KM | 2048 | 468.9 | 632.66 | 1377.6 | 1380.8 | 1389.1 | 1396.9 | 31.14 | 31.21 | 32.20 | 32.81 | 27.50 | 27.50 | 27.67 | 27.98 | 36.37 | 0.070 | 30.86 | 32.34 | 34.57 | 34.92 | 14270.0 | 11837.9 | 29149.7 | 38708.6 | 1486.8 | 1483.2 | 1501.3 | 1504.6 | 1.81 | 20.134 |
| smollm2-135m | Q4KM | 128 | 515.3 | 302.59 | 215.3 | 213.6 | 230.8 | 267.2 | 28.43 | 29.55 | 31.10 | 37.58 | 25.66 | 25.55 | 25.96 | 26.92 | 38.98 | 0.043 | 37.99 | 38.17 | 38.40 | 38.49 | 13684.3 | 7518.3 | 15662.5 | 87884.1 | 597.7 | 599.3 | 635.7 | 643.7 | 1.81 | 21.579 |
| smollm2-135m | Q4KM | 512 | 324.6 | 153.59 | 413.3 | 414.6 | 423.1 | 426.3 | 29.48 | 29.98 | 30.62 | 32.19 | 26.13 | 26.08 | 26.20 | 26.72 | 38.28 | 0.112 | 35.97 | 36.40 | 37.16 | 37.37 | 8881.5 | 9640.1 | 14378.1 | 15797.4 | 1239.2 | 1235.0 | 1270.3 | 1287.3 | 1.81 | 21.192 |
| smollm2-135m | Q4KM | 1024 | 757.0 | 491.41 | 701.5 | 697.6 | 715.6 | 732.6 | 29.85 | 30.24 | 30.69 | 31.18 | 26.72 | 26.62 | 27.22 | 27.41 | 37.43 | 0.047 | 33.71 | 35.82 | 36.30 | 36.47 | 21106.4 | 13493.2 | 51507.9 | 82200.4 | 1460.1 | 1467.9 | 1484.4 | 1491.3 | 1.81 | 20.722 |
| smollm2-135m | Q4KM | 2048 | 442.8 | 245.94 | 1379.9 | 1372.7 | 1385.0 | 1466.8 | 30.51 | 31.10 | 31.56 | 32.05 | 27.53 | 27.48 | 27.72 | 28.01 | 36.33 | 0.074 | 30.21 | 32.54 | 34.18 | 34.67 | 13554.2 | 12668.8 | 25649.0 | 32868.4 | 1484.6 | 1491.9 | 1499.4 | 1503.0 | 1.81 | 20.113 |
| smollm2-135m | Q4KM | 128 | 656.5 | 156.46 | 209.3 | 205.4 | 218.8 | 241.7 | 28.43 | 29.29 | 29.81 | 30.46 | 25.57 | 25.42 | 25.80 | 27.17 | 39.13 | 0.057 | 37.40 | 37.93 | 38.49 | 38.72 | 17647.8 | 5785.6 | 20726.8 | 149417.2 | 613.5 | 623.1 | 647.4 | 651.0 | 1.81 | 21.664 |
| smollm2-135m | Q4KM | 512 | 829.3 | 223.95 | 415.0 | 411.6 | 431.0 | 444.4 | 29.80 | 29.77 | 30.89 | 32.27 | 26.14 | 26.01 | 26.79 | 27.24 | 38.26 | 0.031 | 36.62 | 36.95 | 37.55 | 37.56 | 22514.0 | 11197.9 | 56289.4 | 128891.5 | 1234.8 | 1243.9 | 1276.0 | 1288.1 | 1.81 | 21.183 |
| smollm2-135m | Q4KM | 1024 | 524.5 | 104.90 | 702.3 | 700.4 | 710.3 | 719.5 | 29.82 | 30.17 | 30.58 | 32.64 | 26.56 | 26.51 | 26.79 | 27.23 | 37.65 | 0.068 | 34.82 | 35.27 | 36.21 | 36.35 | 14725.3 | 10353.8 | 19631.8 | 68722.3 | 1458.2 | 1462.0 | 1475.4 | 1481.4 | 1.81 | 20.845 |
| smollm2-135m | Q4KM | 2048 | 513.9 | 100.72 | 1376.3 | 1377.5 | 1385.2 | 1395.6 | 30.40 | 31.20 | 31.78 | 32.86 | 27.47 | 27.45 | 27.58 | 27.75 | 36.40 | 0.064 | 31.75 | 32.09 | 34.85 | 35.03 | 15502.0 | 11206.5 | 34336.2 | 49551.9 | 1488.1 | 1486.8 | 1502.4 | 1504.0 | 1.81 | 20.152 |
| smollm2-360m | Q8_0 | 128 | 517.8 | 709.06 | 266.8 | 265.7 | 275.8 | 287.6 | 45.77 | 46.11 | 46.75 | 47.21 | 42.49 | 42.08 | 42.76 | 46.14 | 23.55 | 0.043 | 22.70 | 23.12 | 23.36 | 23.37 | 23302.7 | 9604.2 | 35177.8 | 150619.0 | 480.2 | 481.8 | 496.0 | 507.3 | 0.79 | 30.000 |
| smollm2-360m | Q8_0 | 512 | 1072.6 | 1575.99 | 635.0 | 632.2 | 651.3 | 658.4 | 47.01 | 46.96 | 47.39 | 49.71 | 44.51 | 44.21 | 46.89 | 46.96 | 22.49 | 0.017 | 21.24 | 21.85 | 22.15 | 22.19 | 49896.9 | 22430.3 | 150655.6 | 160871.7 | 806.4 | 809.9 | 817.8 | 821.2 | 0.79 | 28.647 |
| smollm2-360m | Q8_0 | 1024 | 854.8 | 1235.55 | 1140.8 | 1138.3 | 1153.2 | 1168.3 | 50.14 | 49.77 | 51.88 | 54.09 | 45.98 | 45.69 | 47.30 | 47.55 | 21.75 | 0.024 | 18.49 | 20.42 | 21.02 | 21.12 | 41189.6 | 15343.4 | 123342.5 | 168977.3 | 897.7 | 899.6 | 905.4 | 906.5 | 0.79 | 27.708 |
| smollm2-360m | Q8_0 | 2048 | 354.6 | 454.03 | 2350.6 | 2336.1 | 2394.1 | 2433.9 | 51.87 | 51.41 | 53.26 | 67.82 | 47.86 | 47.70 | 48.46 | 48.66 | 20.89 | 0.035 | 17.53 | 18.07 | 19.01 | 19.96 | 19291.9 | 16746.9 | 25344.0 | 65592.6 | 871.4 | 876.7 | 879.4 | 881.1 | 0.79 | 26.614 |
| smollm2-360m | Q8_0 | 128 | 344.4 | 169.10 | 268.7 | 266.7 | 278.5 | 293.6 | 45.22 | 46.07 | 47.93 | 50.48 | 42.30 | 42.10 | 42.46 | 45.21 | 23.65 | 0.065 | 22.71 | 22.94 | 23.30 | 23.35 | 15278.2 | 7962.3 | 26022.4 | 109045.4 | 477.0 | 479.9 | 493.5 | 503.9 | 0.79 | 30.120 |
| smollm2-360m | Q8_0 | 512 | 469.5 | 266.78 | 631.9 | 627.3 | 646.2 | 655.4 | 46.99 | 46.99 | 47.96 | 50.22 | 43.55 | 43.21 | 44.54 | 46.27 | 22.97 | 0.032 | 21.41 | 22.11 | 22.19 | 22.20 | 21533.6 | 13413.5 | 37992.3 | 106157.1 | 810.5 | 816.2 | 823.0 | 826.0 | 0.79 | 29.261 |
| smollm2-360m | Q8_0 | 1024 | 932.4 | 628.41 | 1142.6 | 1141.2 | 1150.3 | 1159.2 | 48.67 | 49.35 | 50.25 | 51.56 | 45.97 | 45.55 | 47.44 | 47.56 | 21.76 | 0.018 | 20.33 | 20.75 | 21.00 | 21.12 | 44784.2 | 20051.1 | 149115.5 | 157588.4 | 896.2 | 897.3 | 904.3 | 906.0 | 0.79 | 27.720 |
| smollm2-360m | Q8_0 | 2048 | 529.4 | 313.59 | 2340.1 | 2337.9 | 2353.5 | 2356.1 | 51.05 | 51.54 | 51.97 | 52.23 | 47.80 | 47.70 | 48.21 | 48.37 | 20.92 | 0.036 | 17.25 | 18.30 | 19.66 | 20.34 | 27718.1 | 18272.7 | 43819.0 | 129225.8 | 875.2 | 876.0 | 880.5 | 881.4 | 0.79 | 26.651 |
| smollm2-360m | Q8_0 | 128 | 305.9 | 19.51 | 268.5 | 265.5 | 278.6 | 291.8 | 44.87 | 46.11 | 46.84 | 47.17 | 42.25 | 42.09 | 42.50 | 44.20 | 23.67 | 0.044 | 22.88 | 23.05 | 23.30 | 23.38 | 13340.9 | 9018.2 | 24932.6 | 59420.0 | 477.3 | 482.2 | 495.2 | 497.2 | 0.79 | 30.152 |
| smollm2-360m | Q8_0 | 512 | 513.9 | 100.74 | 636.6 | 626.8 | 669.6 | 705.6 | 47.30 | 47.01 | 49.19 | 54.32 | 43.83 | 43.52 | 45.21 | 46.93 | 22.83 | 0.029 | 20.25 | 21.30 | 22.20 | 22.32 | 23917.6 | 9366.5 | 44626.3 | 151827.2 | 805.4 | 816.8 | 826.9 | 830.0 | 0.79 | 29.080 |
| smollm2-360m | Q8_0 | 1024 | 500.5 | 95.51 | 1143.2 | 1144.5 | 1151.6 | 1156.8 | 48.93 | 49.21 | 50.34 | 51.75 | 45.62 | 45.56 | 46.24 | 46.39 | 21.92 | 0.029 | 19.57 | 20.88 | 21.16 | 21.16 | 24076.9 | 22834.7 | 46320.0 | 50612.3 | 895.8 | 894.7 | 904.3 | 906.7 | 0.79 | 27.923 |
| smollm2-360m | Q8_0 | 2048 | 353.1 | 37.95 | 2338.7 | 2333.8 | 2353.8 | 2366.1 | 50.44 | 51.58 | 51.88 | 52.30 | 47.75 | 47.68 | 48.05 | 48.20 | 20.94 | 0.052 | 17.55 | 18.06 | 19.33 | 19.80 | 19196.7 | 16351.8 | 32504.8 | 51060.8 | 875.7 | 877.5 | 881.1 | 882.4 | 0.79 | 26.673 |
Best Output Tok/J per Model
| Backend | Model | Quant | Best Tok/J | ISL | OSL | Tok/s | Power (W) |
|---|---|---|---|---|---|---|---|
| ollama | gemma3-1b | Q4KM | 4.858 | 128 | 711.5 | 9.86 | 2.03 |
| ollama | lfm2.5-1.2b | Q4KM | 4.855 | 512 | 657.6 | 9.93 | 2.04 |
| ollama | lfm2.5-350m | Q4KM | 9.312 | 512 | 330.6 | 15.95 | 1.71 |
| ollama | llama3.2-1b | Q4KM | 5.342 | 128 | 285.3 | 12.09 | 2.26 |
| ollama | qwen2.5-0.5b | Q4KM | 8.757 | 128 | 222.8 | 17.17 | 1.96 |
| ollama | qwen3-0.6b | Q8_0 | 7.495 | 128 | 708.6 | 15.56 | 2.08 |
| ollama | smollm2-135m | Q4KM | 21.664 | 128 | 656.5 | 39.13 | 1.81 |
| ollama | smollm2-360m | Q8_0 | 30.152 | 128 | 305.9 | 23.67 | 0.79 |
Thermal Summary
| Backend | Model | Avg Power (W) | Avg CPU (°C) | Avg GPU (°C) | Peak TJ (°C) | Throttled |
|---|---|---|---|---|---|---|
| ollama | gemma3-1b | 2.03 | 47.6 | 49.4 | 50.3 | No |
| ollama | gemma3-4b | 1.35 | 44.7 | 46.2 | 49.6 | No |
| ollama | lfm2.5-1.2b | 2.04 | 48.2 | 49.9 | 50.5 | No |
| ollama | lfm2.5-350m | 1.71 | 46.7 | 48.4 | 49.4 | No |
| ollama | llama3.2-1b | 2.26 | 48.1 | 49.9 | 50.8 | No |
| ollama | qwen2.5-0.5b | 1.96 | 46.1 | 47.9 | 49.9 | No |
| ollama | qwen3-0.6b | 2.08 | 47.2 | 49.0 | 50.9 | No |
| ollama | smollm2-135m | 1.81 | 43.4 | 45.1 | 46.8 | No |
| ollama | smollm2-360m | 0.79 | 43.4 | 45.2 | 49.9 | No |
Generated by `bench-non-reasoning.sh` on 2026-06-09 02:38:16 108 rows | 1 backends | 9 models
License & citation
- —Benchmark results and artifacts (aiperf exports, server logs,
tegrastatslogs, generated reports): CC BY 4.0. Reuse and adaptation allowed, including commercially, provided you credit Yuvraj Singh, link the license, and indicate changes. - —Harness code that produced these results: Apache-2.0.
If you use these results, please cite:
bibtex
@misc{singh2026smolperfbenchmark,
title={smolperfbenchmark: On-Device LLM Leaderboard},
author={Yuvraj Singh},
year={2026},
howpublished={\url{https://github.com/YuvrajSingh-mist/smolperfbenchmark}},
}<!-- ad-footer --> ---
Released freely — support more experiments like it:
 
More: <https://huggingface.co/YuvrajSingh9886>