Datasetpublic

YuvrajSingh9886/jetson-non-reasoning-benchmark-maxn

Tiny LLM Benchmark — Jetson Orin Nano Super 8GB Date: 2026-05-26 18:18Backend: llama.cpp CUDA (-ngl 99) Concurrency: 1Sweep: prompt in {128,512,1024,2048} tok × gen in {64,128,256} tok Artifacts: https://huggingface.co/datasets/YuvrajSingh9886/jetson-non-reasoning-benchmark-maxn Skipped / Failed Models gemma3-4b (OOM — server failed to start) Full Results Cells marked — = OOM (server crashed or skipped). Power = VDD_CPU_GPU_CV average over… See the full description on the dataset page: https://huggingface.co/datasets/YuvrajSingh9886/jetson-non-reasoning-benchmark-maxn.

sourceHugging Facecc-by-4.0updated 10d agoView on Hugging Face

0likes448downloads

Dataset Card

Tiny LLM Benchmark — Jetson Orin Nano Super 8GB

Date: 2026-05-26 18:18 Backend: llama.cpp CUDA (-ngl 99) Concurrency: 1 Sweep: prompt in {128,512,1024,2048} tok × gen in {64,128,256} tok Artifacts: https://huggingface.co/datasets/YuvrajSingh9886/jetson-non-reasoning-benchmark-maxn

Skipped / Failed Models

—gemma3-4b (OOM — server failed to start)

Full Results

Cells marked — = OOM (server crashed or skipped). Power = VDD\CPU\GPU\_CV average over aiperf run window (CPU+GPU+CV rail, milliwatts → watts).

Model	Quant	ISL (tok)	OSL (tok)	OSL mismatch (%)	TTFT avg (ms)	TTFT p50	TTFT p90	TTFT p99	T2T avg (ms)	T2T p50	T2T p90	T2T p99	ITL avg (ms)	ITL p50	ITL p90	ITL p99	Tok/s (server)	Req/s	E2E tok/s avg	E2E p50	E2E p90	E2E p99	Req lat avg (ms)	Req lat p50	Req lat p90	Req lat p99	Prefill tok/s avg	Prefill p50	Prefill p90	Prefill p99	Power (W)	Tok/J
gemma3-1b	Q4KM	128	63.4	-1.02	118.4	117.1	117.9	136.7	24.02	24.78	24.89	24.95	22.39	22.38	22.43	22.44	44.67	0.658	41.83	41.88	41.94	41.95	1514.1	1527.4	1530.1	1544.7	1083.0	1092.9	1095.7	1095.8	6.22	7.184
gemma3-1b	Q4KM	512	64.0	0.08	282.2	275.9	280.2	366.6	24.91	24.85	25.18	25.46	22.37	22.38	22.43	22.47	44.70	0.589	37.85	37.96	37.99	38.44	1692.6	1686.2	1691.9	1781.1	1824.0	1855.9	1858.0	1860.9	6.22	7.189
gemma3-1b	Q4KM	1024	64.0	0.00	489.7	487.9	493.5	507.9	25.12	25.10	25.32	25.56	22.59	22.59	22.63	22.71	44.26	0.522	33.45	33.48	33.55	33.56	1913.1	1911.5	1918.8	1929.4	2091.5	2098.7	2103.5	2108.0	6.22	7.118
gemma3-1b	Q4KM	2048	64.0	0.00	900.7	899.4	904.8	909.0	25.19	25.13	25.45	25.67	22.62	22.61	22.65	22.68	44.21	0.429	27.52	27.53	27.57	27.58	2325.6	2324.5	2333.4	2336.1	2273.8	2277.0	2280.0	2282.6	6.22	7.110
gemma3-1b	Q4KM	128	126.8	-0.98	117.4	116.9	117.5	123.0	24.77	24.76	24.84	25.20	22.35	22.34	22.38	22.63	44.73	0.341	43.28	43.31	43.36	43.50	2928.5	2954.3	2959.2	2964.0	1090.9	1094.6	1096.9	1098.1	6.22	7.194
gemma3-1b	Q4KM	512	128.0	0.00	276.3	275.2	278.8	286.6	24.89	24.87	25.02	25.58	22.38	22.37	22.45	22.49	44.69	0.320	41.05	41.07	41.12	41.27	3118.4	3116.9	3132.3	3133.8	1853.5	1860.5	1864.3	1866.5	6.22	7.186
gemma3-1b	Q4KM	1024	128.0	0.00	488.2	486.9	492.2	495.5	25.09	25.05	25.23	25.92	22.53	22.51	22.61	22.73	44.39	0.298	38.22	38.26	38.31	38.56	3349.0	3345.3	3359.8	3365.1	2097.7	2102.9	2105.7	2107.2	6.22	7.139
gemma3-1b	Q4KM	2048	128.0	-0.04	896.9	895.7	901.6	906.5	24.87	24.87	25.05	25.24	22.44	22.40	22.59	22.62	44.56	0.267	34.16	34.23	34.25	34.26	3745.8	3739.8	3763.4	3776.1	2283.3	2286.5	2290.3	2291.3	6.22	7.166
gemma3-1b	Q4KM	128	245.5	-4.10	117.4	117.0	117.6	123.0	24.57	24.58	24.68	24.81	22.23	22.20	22.30	22.34	44.99	0.180	44.10	44.30	44.33	44.33	5551.5	5777.7	5808.3	5813.6	1090.5	1093.8	1096.2	1098.6	6.22	7.236
gemma3-1b	Q4KM	512	256.0	0.00	277.3	276.4	279.9	284.7	24.90	24.90	25.04	25.24	22.43	22.44	22.45	22.46	44.57	0.167	42.68	42.67	42.71	42.72	5998.1	5998.9	6001.3	6002.7	1846.3	1852.5	1856.0	1856.7	6.22	7.168
gemma3-1b	Q4KM	1024	255.9	-0.02	487.5	486.8	490.3	494.4	24.99	25.02	25.09	25.12	22.51	22.49	22.56	22.71	44.43	0.160	41.11	41.14	41.19	41.25	6225.6	6222.4	6238.2	6265.5	2100.4	2103.7	2108.3	2110.6	6.22	7.145
gemma3-1b	Q4KM	2048	255.9	-0.02	900.2	899.2	904.3	908.9	25.10	25.13	25.25	25.69	22.60	22.59	22.68	22.72	44.24	0.150	38.42	38.44	38.49	38.55	6662.5	6659.5	6685.3	6695.7	2275.0	2277.5	2281.0	2283.0	6.22	7.115
lfm2.5-1.2b	Q4KM	128	64.0	0.00	118.3	117.8	118.9	123.7	20.00	19.99	20.23	20.71	18.70	18.70	18.76	18.78	53.48	0.769	49.37	49.38	49.52	49.58	1296.3	1295.9	1300.2	1303.5	1082.4	1086.2	1090.7	1092.5	8.02	6.667
lfm2.5-1.2b	Q4KM	512	64.0	-0.08	303.8	302.4	303.1	325.5	20.12	20.11	20.35	20.79	18.76	18.75	18.81	18.97	53.31	0.672	43.08	43.14	43.25	43.28	1484.5	1483.5	1487.2	1504.5	1686.1	1693.1	1697.6	1699.4	8.02	6.647
lfm2.5-1.2b	Q4KM	1024	63.9	-0.16	539.2	538.6	540.1	547.8	20.07	20.07	20.34	20.60	18.85	18.84	18.96	19.13	53.05	0.578	37.04	37.08	37.19	37.27	1724.9	1725.4	1731.6	1732.3	1899.3	1901.3	1904.9	1905.6	8.02	6.613
lfm2.5-1.2b	Q4KM	2048	64.0	-0.08	1012.3	1011.9	1013.7	1020.7	20.35	20.38	20.61	20.66	19.09	19.08	19.14	19.35	52.38	0.451	28.88	28.92	28.93	28.95	2214.2	2213.3	2219.2	2219.6	2023.1	2023.8	2026.7	2027.2	8.02	6.530
lfm2.5-1.2b	Q4KM	128	127.8	-0.16	117.9	117.3	118.4	124.3	19.90	19.91	20.15	20.22	18.72	18.68	18.85	18.96	53.42	0.400	51.29	51.39	51.47	51.49	2491.7	2490.1	2498.6	2505.7	1086.1	1090.9	1095.5	1096.6	8.02	6.660
lfm2.5-1.2b	Q4KM	512	127.8	-0.12	303.1	302.8	303.7	310.3	19.96	19.98	20.11	20.18	18.75	18.72	18.85	18.95	53.33	0.372	47.67	47.71	47.83	47.87	2681.8	2680.6	2688.6	2691.8	1689.2	1691.0	1697.5	1698.5	8.02	6.649
lfm2.5-1.2b	Q4KM	1024	127.8	-0.12	539.8	539.3	541.4	546.1	20.09	20.16	20.22	20.28	18.85	18.83	18.90	19.01	53.06	0.341	43.63	43.65	43.73	43.76	2930.7	2932.0	2936.8	2938.0	1897.1	1898.6	1902.5	1904.0	8.02	6.615
lfm2.5-1.2b	Q4KM	2048	127.8	-0.12	1010.0	1009.4	1011.3	1018.7	20.28	20.28	20.51	20.66	19.06	19.05	19.17	19.21	52.46	0.291	37.30	37.33	37.38	37.40	3428.0	3428.4	3432.7	3433.4	2027.7	2028.9	2031.4	2031.6	8.02	6.541
lfm2.5-1.2b	Q4KM	128	243.6	-4.86	118.1	117.6	118.7	123.9	20.00	19.97	20.31	20.33	18.70	18.70	18.74	18.75	53.48	0.215	52.23	52.40	52.47	52.57	4652.5	4884.5	4891.1	4896.9	1084.5	1088.9	1091.1	1091.9	8.02	6.668
lfm2.5-1.2b	Q4KM	512	255.7	-0.12	303.7	303.2	304.8	310.2	20.04	20.09	20.22	20.36	18.75	18.74	18.83	18.86	53.33	0.197	50.34	50.37	50.45	50.52	5079.6	5079.1	5087.5	5092.0	1685.9	1688.8	1693.2	1695.1	8.02	6.649
lfm2.5-1.2b	Q4KM	1024	255.9	-0.02	540.1	539.4	541.6	548.1	20.10	20.12	20.23	20.35	18.85	18.84	18.90	18.99	53.05	0.187	47.88	47.92	47.98	48.08	5346.0	5342.5	5357.5	5370.7	1896.0	1898.4	1902.2	1903.0	8.02	6.614
lfm2.5-1.2b	Q4KM	2048	256.0	0.00	1008.0	1007.2	1009.3	1017.9	20.17	20.15	20.59	20.91	19.00	18.99	19.03	19.15	52.63	0.171	43.74	43.76	43.79	43.88	5853.2	5850.7	5860.7	5889.5	2031.7	2033.4	2035.6	2036.8	8.02	6.562
lfm2.5-350m	Q4KM	128	64.0	0.00	52.0	51.6	52.3	58.1	9.60	9.49	10.27	10.40	8.37	8.33	8.41	8.82	119.49	1.713	110.48	110.99	111.28	111.36	579.4	576.6	582.2	608.8	2461.7	2479.0	2494.9	2503.2	5.51	21.684
lfm2.5-350m	Q4KM	512	63.1	-1.41	128.9	126.0	127.9	170.4	9.86	9.90	10.14	10.93	8.50	8.48	8.51	8.78	117.69	1.512	96.08	96.88	97.25	97.32	656.7	660.1	662.8	723.2	3995.4	4061.9	4080.9	4084.1	5.51	21.358
lfm2.5-350m	Q4KM	1024	64.0	0.00	217.8	217.2	219.2	224.0	9.88	9.93	10.12	10.21	8.61	8.61	8.65	8.67	116.11	1.307	84.17	84.23	84.54	84.64	760.4	759.9	763.2	768.1	4701.6	4713.8	4727.0	4740.6	5.51	21.071
lfm2.5-350m	Q4KM	2048	62.6	-2.11	396.5	396.1	397.7	401.9	10.04	10.07	10.34	10.70	8.88	8.87	8.94	9.09	112.59	1.054	66.18	67.02	67.19	67.34	944.0	954.8	959.9	968.1	5165.5	5170.8	5181.7	5186.1	5.51	20.433
lfm2.5-350m	Q4KM	128	113.0	-11.76	52.6	52.2	53.5	58.6	9.53	9.55	9.89	10.23	8.45	8.39	8.64	9.00	118.37	1.001	108.20	114.42	115.12	115.93	992.5	1115.8	1123.0	1146.5	2433.7	2452.3	2484.5	2491.9	5.51	21.482
lfm2.5-350m	Q4KM	512	122.3	-4.41	126.5	126.1	127.6	133.2	9.71	9.75	10.07	10.30	8.44	8.43	8.50	8.58	118.43	0.865	104.73	107.04	107.30	107.42	1150.4	1194.7	1202.3	1218.5	4049.1	4061.3	4094.0	4101.7	5.51	21.493
lfm2.5-350m	Q4KM	1024	127.9	-0.08	217.1	216.5	218.1	223.9	9.87	9.82	10.19	10.71	8.56	8.55	8.63	8.66	116.83	0.764	98.14	98.30	98.56	98.90	1303.3	1302.1	1310.5	1313.6	4717.3	4728.8	4744.9	4748.3	5.51	21.202
lfm2.5-350m	Q4KM	2048	119.0	-7.07	395.6	394.7	397.6	403.9	9.94	9.92	10.29	10.46	8.82	8.81	8.88	8.93	113.41	0.694	81.13	84.49	84.82	85.01	1435.0	1512.6	1518.4	1536.5	5177.4	5188.6	5199.1	5205.3	5.51	20.581
lfm2.5-350m	Q4KM	128	229.8	-10.21	52.1	51.6	53.1	57.5	9.54	9.51	9.70	10.47	8.39	8.37	8.46	8.62	119.13	0.505	115.30	116.96	117.55	117.66	1973.3	2180.8	2196.9	2255.0	2459.8	2479.5	2507.3	2513.3	5.51	21.620
lfm2.5-350m	Q4KM	512	223.8	-12.58	125.8	125.2	126.6	133.1	9.61	9.58	9.86	10.59	8.45	8.44	8.47	8.54	118.37	0.496	109.02	112.27	112.62	112.72	2007.2	2276.0	2280.5	2284.9	4070.1	4089.5	4098.9	4099.7	5.51	21.482
lfm2.5-350m	Q4KM	1024	251.0	-1.95	217.3	216.8	218.1	224.5	9.71	9.72	10.20	10.72	8.59	8.59	8.63	8.70	116.35	0.421	106.01	106.36	106.59	106.74	2366.0	2404.9	2409.6	2424.3	4711.7	4723.9	4738.2	4744.6	5.51	21.116
lfm2.5-350m	Q4KM	2048	244.8	-4.36	396.0	395.4	397.5	403.0	9.90	9.93	10.03	10.13	8.86	8.86	8.90	8.92	112.87	0.390	94.12	96.48	96.68	96.80	2555.8	2652.2	2664.4	2671.1	5171.7	5179.8	5193.2	5198.0	5.51	20.484
llama3.2-1b	Q4KM	128	64.0	0.00	96.2	95.8	96.5	101.2	19.45	19.38	19.60	20.45	18.32	18.29	18.36	18.74	54.58	0.797	51.18	51.28	51.35	51.38	1250.6	1248.0	1252.7	1281.9	1331.2	1336.4	1338.9	1342.0	5.00	10.909
llama3.2-1b	Q4KM	512	64.0	0.00	249.1	248.7	249.2	255.9	19.78	19.81	19.97	20.14	18.59	18.61	18.66	18.75	53.78	0.702	45.05	45.03	45.26	45.28	1420.5	1421.1	1424.1	1437.0	2055.3	2059.0	2062.0	2063.3	5.00	10.750
llama3.2-1b	Q4KM	1024	64.0	0.00	457.1	456.4	457.8	464.4	19.93	19.93	20.20	20.29	18.71	18.70	18.75	18.77	53.45	0.610	39.13	39.14	39.18	39.21	1635.8	1635.0	1638.4	1645.3	2240.5	2243.8	2245.8	2247.2	5.00	10.683
llama3.2-1b	Q4KM	2048	63.9	-0.16	886.7	885.7	888.4	895.4	20.01	20.03	20.12	20.21	19.09	19.06	19.09	19.58	52.38	0.478	30.61	30.67	30.70	30.71	2087.4	2087.0	2089.4	2097.0	2309.8	2312.3	2314.7	2315.7	5.00	10.471
llama3.2-1b	Q4KM	128	127.8	-0.12	97.1	96.5	99.1	101.8	19.52	19.53	19.65	19.84	18.46	18.45	18.59	18.59	54.18	0.409	52.43	52.47	52.61	52.75	2438.4	2438.9	2441.2	2450.1	1318.4	1326.6	1329.9	1333.8	5.00	10.830
llama3.2-1b	Q4KM	512	127.9	-0.08	249.4	248.9	249.8	255.5	19.87	19.76	20.05	21.51	18.62	18.60	18.72	18.88	53.70	0.382	48.96	49.03	49.10	49.12	2612.4	2610.6	2617.1	2630.7	2052.9	2057.4	2060.5	2060.8	5.00	10.734
llama3.2-1b	Q4KM	1024	128.0	0.00	458.1	457.7	458.6	463.5	20.05	20.08	20.18	20.28	18.80	18.79	18.81	18.92	53.20	0.351	44.99	45.01	45.04	45.07	2845.2	2844.0	2847.5	2861.5	2235.1	2237.3	2239.4	2241.4	5.00	10.635
llama3.2-1b	Q4KM	2048	127.9	-0.08	888.3	887.9	890.3	894.3	20.21	20.23	20.38	20.49	19.15	19.15	19.23	19.40	52.21	0.301	38.54	38.54	38.69	38.70	3318.9	3320.9	3329.9	3339.3	2305.5	2306.4	2311.1	2311.8	5.00	10.436
llama3.2-1b	Q4KM	128	246.2	-3.83	96.9	96.7	97.1	101.5	19.54	19.56	19.70	19.74	18.48	18.45	18.60	18.64	54.13	0.216	53.19	53.32	53.41	53.49	4626.8	4799.6	4808.7	4834.7	1320.7	1324.3	1329.0	1332.1	5.00	10.819
llama3.2-1b	Q4KM	512	256.1	0.02	249.7	249.1	250.6	255.1	19.77	19.70	19.95	20.76	18.54	18.53	18.60	18.66	53.92	0.201	51.42	51.47	51.52	51.64	4979.5	4975.2	4993.2	5013.6	2050.7	2055.4	2058.2	2060.0	5.00	10.778
llama3.2-1b	Q4KM	1024	255.9	-0.02	458.4	457.6	460.0	464.4	19.98	19.95	20.18	20.24	18.76	18.72	18.88	18.90	53.29	0.191	48.82	48.95	48.97	49.00	5242.4	5230.0	5272.6	5273.5	2234.0	2237.7	2241.1	2242.5	5.00	10.652
llama3.2-1b	Q4KM	2048	255.9	-0.02	890.5	890.5	892.4	893.6	20.49	20.33	21.10	21.17	19.27	19.26	19.37	19.42	51.89	0.172	44.10	44.14	44.21	44.30	5804.2	5798.2	5831.1	5841.6	2299.9	2299.8	2304.2	2308.2	5.00	10.371
qwen2.5-0.5b	Q4KM	128	41.3	-35.47	56.2	55.8	56.8	60.4	11.38	11.37	11.45	11.87	9.65	9.60	9.80	9.90	103.61	2.214	87.98	94.67	97.71	98.02	443.0	527.7	660.1	676.1	2278.4	2293.3	2302.2	2305.7	2.12	48.806
qwen2.5-0.5b	Q4KM	512	57.1	-10.78	140.4	140.1	140.7	145.1	11.58	11.48	11.93	12.05	9.64	9.59	9.79	9.97	103.73	1.459	81.35	85.95	86.32	86.44	679.9	743.3	747.2	765.0	3647.9	3655.1	3664.7	3671.0	2.12	48.859
qwen2.5-0.5b	Q4KM	1024	55.1	-13.83	252.7	252.1	253.6	258.8	11.61	11.56	11.90	12.35	9.77	9.74	9.86	10.03	102.38	1.268	68.96	73.47	74.28	74.39	781.0	861.3	869.3	871.5	4052.9	4061.2	4069.0	4085.1	2.12	48.226
qwen2.5-0.5b	Q4KM	2048	51.0	-20.39	484.1	483.5	484.9	491.1	11.74	11.67	12.07	12.13	10.01	9.92	10.27	10.37	99.92	1.012	48.72	57.61	57.83	57.94	981.0	1106.9	1111.8	1116.7	4231.0	4235.5	4242.1	4243.5	2.12	47.067
qwen2.5-0.5b	Q4KM	128	86.6	-32.34	56.3	55.8	56.8	61.4	11.36	11.34	11.70	11.71	9.63	9.57	9.82	10.02	103.84	1.131	93.22	99.69	101.21	101.96	876.1	1257.8	1272.3	1302.5	2277.1	2295.3	2319.3	2329.6	2.12	48.914
qwen2.5-0.5b	Q4KM	512	113.9	-11.02	140.3	139.8	141.1	145.2	11.52	11.46	11.79	12.00	9.58	9.56	9.66	9.83	104.37	0.815	91.03	94.45	95.02	95.21	1220.5	1351.1	1358.9	1366.6	3649.6	3662.3	3670.1	3676.9	2.12	49.162
qwen2.5-0.5b	Q4KM	1024	90.8	-29.02	252.2	251.9	252.9	257.8	11.56	11.54	11.79	12.32	9.69	9.69	9.75	9.89	103.21	0.885	76.55	83.75	86.73	87.09	1121.0	1287.5	1480.9	1485.9	4059.8	4065.4	4074.6	4080.5	2.12	48.614
qwen2.5-0.5b	Q4KM	2048	110.4	-13.75	484.0	483.9	484.8	488.6	11.73	11.69	12.09	12.43	9.92	9.89	9.99	10.23	100.82	0.636	66.81	73.56	73.81	74.01	1566.2	1738.5	1743.6	1747.4	4231.2	4232.7	4239.5	4242.9	2.12	47.491
qwen2.5-0.5b	Q4KM	128	147.3	-42.46	56.2	55.7	56.6	61.2	11.31	11.34	11.64	11.71	9.65	9.60	9.85	10.15	103.61	0.680	94.87	100.93	102.84	103.23	1459.4	1541.9	2498.6	2526.3	2280.2	2299.5	2312.9	2315.4	2.12	48.806
qwen2.5-0.5b	Q4KM	512	163.3	-36.19	140.5	140.2	141.2	145.8	11.48	11.44	11.81	11.99	9.65	9.63	9.71	9.97	103.68	0.584	89.37	96.53	98.77	98.87	1702.8	1837.7	2600.4	2605.3	3643.8	3651.0	3670.4	3676.2	2.12	48.838
qwen2.5-0.5b	Q4KM	1024	163.5	-36.13	252.6	252.1	253.1	258.8	11.63	11.60	11.88	12.29	9.78	9.75	10.01	10.04	102.24	0.542	80.58	91.88	93.84	94.11	1834.7	2347.2	2741.3	2744.3	4054.6	4062.2	4068.1	4076.8	2.12	48.159
qwen2.5-0.5b	Q4KM	2048	175.4	-31.46	484.8	484.5	485.6	490.9	11.68	11.69	11.89	12.11	9.95	9.94	10.03	10.18	100.50	0.449	70.17	84.15	84.96	85.60	2215.6	2894.6	3017.5	3020.2	4224.4	4226.9	4235.6	4245.5	2.12	47.338
qwen3-0.6b	Q8_0	128	62.0	-3.20	101.9	101.5	102.7	107.6	15.79	15.77	15.85	16.17	15.83	15.80	15.89	16.05	63.17	0.934	58.08	58.14	58.33	58.36	1066.7	1066.3	1068.6	1082.5	1256.1	1261.4	1267.6	1269.0	2.98	21.208
qwen3-0.6b	Q8_0	512	62.0	-3.05	229.1	224.6	225.7	295.8	16.50	16.50	16.59	16.79	16.48	16.49	16.52	16.61	60.66	0.807	50.24	50.39	50.45	51.11	1235.5	1230.4	1233.0	1309.0	2246.2	2279.7	2283.4	2286.3	2.98	20.365
qwen3-0.6b	Q8_0	1024	62.0	-3.12	376.6	376.2	377.2	382.5	17.20	17.20	17.28	17.47	17.17	17.19	17.21	17.23	58.24	0.700	43.54	43.51	43.65	43.77	1424.1	1424.9	1427.1	1428.5	2719.3	2721.9	2726.4	2731.7	2.98	19.550
qwen3-0.6b	Q8_0	2048	62.0	-3.12	703.3	702.8	703.9	712.1	18.56	18.57	18.62	18.66	18.56	18.56	18.58	18.58	53.89	0.544	33.78	33.80	33.82	33.85	1835.2	1834.6	1837.5	1843.9	2912.1	2914.1	2917.7	2921.8	2.98	18.092
qwen3-0.6b	Q8_0	128	126.0	-1.56	102.2	101.4	104.8	108.1	15.76	15.74	15.86	16.03	15.86	15.85	15.92	16.00	63.03	0.478	60.42	60.50	60.57	60.59	2085.3	2082.6	2093.7	2102.8	1253.3	1262.2	1266.7	1267.5	2.98	21.161
qwen3-0.6b	Q8_0	512	126.0	-1.60	225.6	224.7	226.4	234.1	16.53	16.51	16.72	16.87	16.52	16.49	16.63	16.70	60.53	0.436	55.01	55.13	55.20	55.27	2289.7	2285.5	2303.4	2312.7	2270.2	2279.0	2285.0	2287.8	2.98	20.322
qwen3-0.6b	Q8_0	1024	126.0	-1.56	376.4	376.0	376.8	382.8	17.23	17.21	17.44	17.56	17.18	17.17	17.20	17.33	58.20	0.395	49.92	49.94	49.99	50.04	2524.2	2522.9	2529.3	2542.4	2720.4	2723.1	2726.3	2730.3	2.98	19.539
qwen3-0.6b	Q8_0	2048	126.0	-1.56	703.1	703.1	704.1	710.7	18.58	18.56	18.74	18.76	18.55	18.55	18.57	18.58	53.92	0.330	41.70	41.69	41.78	41.80	3021.6	3022.2	3024.5	3025.6	2912.6	2912.8	2922.1	2925.7	2.98	18.100
qwen3-0.6b	Q8_0	128	253.8	-0.88	102.1	101.6	103.0	108.0	15.81	15.76	15.94	16.19	16.02	16.03	16.11	16.17	62.41	0.241	61.12	61.12	61.38	61.45	4151.7	4155.9	4160.1	4167.5	1253.9	1259.2	1263.0	1265.1	2.98	20.953
qwen3-0.6b	Q8_0	512	253.8	-0.84	224.8	224.5	226.2	231.0	16.44	16.41	16.56	16.80	16.49	16.50	16.56	16.72	60.65	0.227	57.77	57.75	58.09	58.20	4394.0	4398.6	4414.0	4427.5	2277.8	2280.2	2292.5	2294.4	2.98	20.360
qwen3-0.6b	Q8_0	1024	253.8	-0.86	375.2	374.6	376.5	383.2	17.09	17.09	17.21	17.25	17.14	17.11	17.22	17.39	58.35	0.212	53.91	53.98	54.13	54.22	4707.6	4705.3	4731.0	4733.2	2729.4	2733.8	2741.1	2743.4	2.98	19.589
qwen3-0.6b	Q8_0	2048	253.9	-0.82	700.3	699.0	701.9	711.5	18.51	18.47	18.60	18.85	18.45	18.43	18.48	18.60	54.19	0.186	47.31	47.37	47.39	47.39	5366.9	5362.1	5372.6	5397.9	2924.3	2930.1	2932.7	2935.0	2.98	18.194
smollm2-135m	Q4KM	128	62.4	-2.58	37.0	36.0	37.4	49.2	8.03	8.02	8.39	8.56	5.46	5.44	5.52	5.76	183.33	2.654	167.74	168.87	170.45	171.73	371.7	378.2	382.3	416.9	3482.4	3552.0	3593.8	3608.9	4.32	42.426
smollm2-135m	Q4KM	512	59.5	-7.03	93.6	92.7	95.5	99.7	8.27	8.30	8.72	8.92	5.66	5.59	5.73	6.26	176.86	2.337	138.33	143.61	144.30	144.37	422.9	444.0	448.6	471.7	5474.7	5521.4	5553.3	5584.4	4.32	40.929
smollm2-135m	Q4KM	1024	57.8	-9.77	166.6	165.4	168.2	178.8	8.45	8.36	9.14	10.67	5.96	5.84	6.02	7.59	168.66	1.983	111.54	120.02	121.47	122.10	499.0	531.3	544.1	563.7	6150.3	6192.5	6227.6	6241.6	4.32	39.031
smollm2-135m	Q4KM	2048	64.0	-0.08	319.6	318.9	320.9	327.4	8.86	8.74	9.18	9.92	6.35	6.33	6.43	6.67	157.60	1.382	88.95	89.12	89.73	89.90	719.1	717.7	721.5	746.4	6409.2	6422.7	6437.3	6473.0	4.32	36.472
smollm2-135m	Q4KM	128	105.2	-17.81	36.3	35.9	36.7	41.8	8.03	7.90	8.17	9.93	5.48	5.47	5.57	5.71	182.60	1.636	169.30	175.06	176.70	178.60	604.8	728.5	735.7	747.2	3530.9	3563.3	3600.5	3614.6	4.32	42.256
smollm2-135m	Q4KM	512	123.8	-3.32	93.5	92.8	95.4	98.9	8.55	8.47	8.98	9.51	5.59	5.58	5.64	5.90	178.91	1.273	157.93	159.76	161.15	161.60	779.7	800.5	810.4	847.9	5477.3	5517.0	5540.7	5555.1	4.32	41.402
smollm2-135m	Q4KM	1024	124.4	-2.81	165.9	165.4	167.0	172.3	8.42	8.43	8.76	9.06	5.79	5.78	5.83	5.91	172.80	1.129	140.80	142.21	142.99	143.34	880.0	898.6	905.6	916.0	6174.2	6190.4	6233.1	6252.6	4.32	39.988
smollm2-135m	Q4KM	2048	128.2	0.20	319.8	319.4	321.8	326.0	8.83	8.84	9.15	9.21	6.28	6.27	6.34	6.45	159.22	0.889	114.61	114.76	115.70	116.55	1119.1	1115.9	1124.8	1142.3	6404.7	6411.5	6443.4	6463.1	4.32	36.846
smollm2-135m	Q4KM	128	209.7	-18.11	36.8	36.1	37.3	44.3	7.94	7.96	8.26	8.34	5.46	5.45	5.51	5.66	183.33	0.847	176.44	179.45	182.07	183.29	1172.3	1423.4	1427.6	1459.4	3489.3	3543.2	3593.1	3604.0	4.32	42.425
smollm2-135m	Q4KM	512	229.3	-10.43	93.2	92.7	93.5	98.6	8.24	8.19	8.60	8.69	5.57	5.58	5.61	5.64	179.67	0.729	167.30	168.06	170.88	174.00	1363.9	1515.8	1524.1	1526.0	5497.0	5520.9	5544.3	5556.1	4.32	41.578
smollm2-135m	Q4KM	1024	215.1	-15.98	165.6	165.3	166.2	172.0	8.39	8.36	8.72	9.11	5.94	5.85	5.95	7.18	168.78	0.701	141.43	154.22	154.74	156.41	1418.6	1656.1	1662.2	1668.3	6185.1	6194.8	6229.8	6237.8	4.32	39.058
smollm2-135m	Q4KM	2048	249.2	-2.68	319.5	318.8	322.0	325.4	8.72	8.71	8.92	9.74	6.28	6.27	6.29	6.48	159.38	0.531	132.59	133.33	133.80	135.77	1876.5	1918.9	1922.8	1925.3	6411.2	6425.1	6444.9	6466.8	4.32	36.883
smollm2-360m	Q8_0	128	58.5	-8.59	51.4	50.9	52.7	55.9	12.54	12.54	12.87	13.35	9.81	9.72	9.79	11.03	102.08	1.623	93.40	96.21	96.80	97.28	610.7	663.3	667.2	667.3	2493.3	2514.3	2530.5	2541.1	5.24	19.477
smollm2-360m	Q8_0	512	56.6	-11.48	146.2	145.5	147.7	153.7	13.17	13.19	13.39	13.54	10.15	10.04	10.33	11.23	98.62	1.406	76.79	82.26	82.53	82.62	706.3	777.0	780.7	792.4	3501.7	3518.8	3528.7	3537.2	5.24	18.815
smollm2-360m	Q8_0	1024	47.5	-25.70	268.2	267.3	270.3	276.5	13.58	13.62	13.76	14.08	10.77	10.52	10.95	13.23	93.20	1.305	56.84	68.81	69.08	69.27	758.9	926.9	930.9	955.0	3818.5	3830.8	3843.0	3845.0	5.24	17.782
smollm2-360m	Q8_0	2048	61.5	-3.83	528.3	527.7	529.6	536.0	13.97	13.97	14.38	14.56	11.20	11.16	11.32	11.43	89.33	0.825	50.73	51.94	52.15	52.22	1205.7	1230.7	1236.7	1239.8	3876.5	3880.9	3890.2	3892.8	5.24	17.043
smollm2-360m	Q8_0	128	92.7	-27.58	51.2	50.9	51.2	56.7	12.51	12.52	12.83	12.91	9.77	9.75	9.94	10.02	102.37	1.050	96.13	99.28	99.96	100.04	944.0	1279.9	1288.9	1289.3	2500.7	2514.5	2531.7	2541.7	5.24	19.531
smollm2-360m	Q8_0	512	90.8	-29.10	146.5	145.7	149.4	153.2	12.90	12.99	13.27	13.33	10.13	9.99	10.55	10.99	98.77	0.950	77.56	89.96	90.76	91.08	1043.3	1406.2	1417.5	1424.5	3494.6	3514.3	3536.6	3540.9	5.24	18.844
smollm2-360m	Q8_0	1024	97.5	-23.83	267.3	266.8	268.0	274.6	13.35	13.35	13.56	14.02	10.67	10.46	10.84	12.94	94.04	0.778	68.45	79.97	80.52	80.77	1277.2	1589.5	1596.6	1601.0	3831.2	3837.7	3846.2	3853.7	5.24	17.942
smollm2-360m	Q8_0	2048	109.0	-14.84	528.0	527.4	529.6	535.9	14.11	14.03	14.58	14.86	11.16	11.14	11.27	11.39	89.63	0.574	61.48	65.47	66.05	66.10	1732.8	1937.6	1957.8	1973.1	3879.0	3883.4	3891.3	3901.0	5.24	17.100
smollm2-360m	Q8_0	128	146.9	-42.60	51.3	50.8	51.5	58.3	12.50	12.50	12.79	12.85	9.79	9.78	9.85	9.93	102.20	0.670	97.70	99.27	100.42	100.62	1479.7	1437.4	2549.5	2555.8	2498.5	2521.5	2531.6	2533.8	5.24	19.499
smollm2-360m	Q8_0	512	162.6	-36.50	145.8	145.5	146.0	151.5	12.89	12.91	13.16	13.20	10.11	10.10	10.17	10.54	98.91	0.560	85.96	92.91	94.14	95.03	1773.3	2105.9	2723.9	2729.3	3512.6	3518.8	3532.4	3539.8	5.24	18.870
smollm2-360m	Q8_0	1024	175.4	-31.46	267.5	266.9	268.8	274.5	13.42	13.45	13.70	13.81	10.58	10.47	10.81	11.34	94.54	0.475	73.07	86.93	87.36	87.41	2093.5	2929.3	2937.9	2939.6	3828.8	3836.8	3852.0	3857.5	5.24	18.038
smollm2-360m	Q8_0	2048	188.3	-26.45	528.3	528.0	530.1	534.8	13.90	13.93	14.27	14.54	11.18	11.16	11.26	11.53	89.42	0.380	66.60	75.67	75.96	76.35	2617.4	3357.9	3378.7	3385.0	3876.3	3879.0	3889.0	3896.2	5.24	17.061

Best Tok/J per Model

Model	Quant	Best Tok/J	ISL (tok)	OSL (tok)	Tok/s	Power (W)
gemma3-1b	Q4KM	7.236	128	245.5	44.99	6.22
lfm2.5-1.2b	Q4KM	6.668	128	243.6	53.48	8.02
lfm2.5-350m	Q4KM	21.684	128	64.0	119.49	5.51
llama3.2-1b	Q4KM	10.909	128	64.0	54.58	5.00
qwen2.5-0.5b	Q4KM	49.162	512	113.9	104.37	2.12
qwen3-0.6b	Q8_0	21.208	128	62.0	63.17	2.98
smollm2-135m	Q4KM	42.426	128	62.4	183.33	4.32
smollm2-360m	Q8_0	19.531	128	92.7	102.37	5.24

Thermal Summary

Model	Avg Power (W)	Avg CPU (°C)	Avg GPU (°C)	Peak TJ (°C)	Throttled
gemma3-1b	6.22	59.4	61.6	67.0	No
lfm2.5-1.2b	8.02	55.9	58.6	63.5	No
lfm2.5-350m	5.51	49.5	51.9	56.8	No
llama3.2-1b	5.00	54.8	57.0	69.5	No
qwen2.5-0.5b	2.12	50.2	52.1	62.8	No
qwen3-0.6b	2.98	50.0	52.0	64.0	No
smollm2-135m	4.32	48.4	50.5	53.2	No
smollm2-360m	5.24	50.1	52.3	56.8	No

Generated by `bench_all.sh` on 2026-05-26 18:18:09

License & citation

—Benchmark results and artifacts (aiperf exports, server logs, tegrastats logs, generated reports): CC BY 4.0. Reuse and adaptation allowed, including commercially, provided you credit Yuvraj Singh, link the license, and indicate changes.
—Harness code that produced these results: Apache-2.0.

If you use these results, please cite:

bibtex

@misc{singh2026smolperfbenchmark,
      title={smolperfbenchmark: On-Device LLM Leaderboard},
      author={Yuvraj Singh},
      year={2026},
      howpublished={\url{https://github.com/YuvrajSingh-mist/smolperfbenchmark}},
}

---

Released freely — support more experiments like it:

![Support me on Ko-fi](https://ko-fi.com/O7W120DR8R) ![GitHub Sponsors](https://github.com/sponsors/YuvrajSingh-mist)

More: <https://huggingface.co/YuvrajSingh9886>