Datasetpublic

YuvrajSingh9886/bonsai-jetson-benchmark-7w

Bonsai Jetson Benchmark: 7W Platform: NVIDIA Jetson Orin Nano Super 8GB · Power mode: 7W Backend: llama.cpp build-jetson · CUDA · -ngl 99 Sweep: prompt in {256, 512, 1024, 2048} tok x gen in {128, 256, 512} tok · 20 reqs/combo Context: 2560 tok · Concurrency: 1 Part of smolperfbenchmark, a public on-device LLM benchmark leaderboard. Headline metric is output tok/J (tokens per joule), computed over the decode phase. Files Bonsai-*, Ternary-Bonsai-*: per-combo… See the full description on the dataset page: https://huggingface.co/datasets/YuvrajSingh9886/bonsai-jetson-benchmark-7w.

sourceHugging Facecc-by-4.0updated 10d agoView on Hugging Face

0likes789downloads

Dataset Card

Bonsai Jetson Benchmark: 7W

Platform: NVIDIA Jetson Orin Nano Super 8GB · Power mode: 7W Backend: llama.cpp build-jetson · CUDA · -ngl 99 Sweep: prompt in {256, 512, 1024, 2048} tok x gen in {128, 256, 512} tok · 20 reqs/combo Context: 2560 tok · Concurrency: 1

Part of smolperfbenchmark, a public on-device LLM benchmark leaderboard. Headline metric is output tok/J (tokens per joule), computed over the decode phase.

Files

—Bonsai-*, Ternary-Bonsai-*: per-combo aiperf exports.
—*-server.log: llama.cpp server logs.
—tegrastats.log: 1 Hz power and thermal samples.
—model_timing.log: per-combo timing.
—report.md: generated summary of the tables below.

Results

Date: 2026-06-04 02:15 Backend: llamacpp Context: 2560 tokens Concurrency: 1 Sweep: prompt in {256,512,1024,2048} gen in {128,256,512} Artifacts: artifacts/llamacpp/bonsai-all-20260528-0328-7W

Full Results

Model	Bits	Prompt (tok)	Gen (tok)	TTFT avg (ms)	ITL avg (ms)	Tok/s	Power (W)	Tok/J
Bonsai-1.7B	1-bit	256	128	798	146.61	6.82	1.54	4.4308
Bonsai-1.7B	1-bit	512	128	1427	147.89	6.76	1.54	4.3924
Bonsai-1.7B	1-bit	1024	128	2725	149.82	6.67	1.54	4.3359
Bonsai-1.7B	1-bit	2048	128	5419	153.87	6.50	1.54	4.2219
Bonsai-1.7B	1-bit	256	256	799	146.91	6.81	1.54	4.4218
Bonsai-1.7B	1-bit	512	256	1429	147.97	6.76	1.54	4.3900
Bonsai-1.7B	1-bit	1024	256	2725	149.92	6.67	1.54	4.3330
Bonsai-1.7B	1-bit	2048	256	5418	153.74	6.50	1.54	4.2253
Bonsai-1.7B	1-bit	256	512	799	147.28	6.79	1.54	4.4107
Bonsai-1.7B	1-bit	512	512	1429	148.38	6.74	1.54	4.3778
Bonsai-1.7B	1-bit	1024	512	2725	150.61	6.64	1.54	4.3132
Bonsai-1.7B	1-bit	2048	512	5420	154.28	6.48	1.54	4.2104
Bonsai-4B	1-bit	256	128	1924	276.40	3.62	1.60	2.2583
Bonsai-4B	1-bit	512	128	3479	277.70	3.60	1.60	2.2477
Bonsai-4B	1-bit	1024	128	6808	280.34	3.57	1.60	2.2266
Bonsai-4B	1-bit	2048	128	13380	285.57	3.50	1.60	2.1858
Bonsai-4B	1-bit	256	256	1914	276.47	3.62	1.60	2.2577
Bonsai-4B	1-bit	512	256	3447	278.10	3.60	1.60	2.2445
Bonsai-4B	1-bit	1024	256	6660	280.62	3.56	1.60	2.2244
Bonsai-4B	1-bit	2048	256	14117	285.93	3.50	1.60	2.1830
Bonsai-4B	1-bit	256	512	1980	277.23	3.61	1.60	2.2516
Bonsai-4B	1-bit	512	512	3615	278.64	3.59	1.60	2.2402
Bonsai-4B	1-bit	1024	512	6480	281.15	3.56	1.60	2.2202
Bonsai-4B	1-bit	2048	512	12967	286.38	3.49	1.60	2.1796
Bonsai-8B	1-bit	256	128	3204	269.12	3.72	2.11	1.7615
Bonsai-8B	1-bit	512	128	5727	270.38	3.70	2.11	1.7533
Bonsai-8B	1-bit	1024	128	10978	273.07	3.66	2.11	1.7360
Bonsai-8B	1-bit	2048	128	21752	278.76	3.59	2.11	1.7006
Bonsai-8B	1-bit	256	256	3182	268.99	3.72	2.11	1.7624
Bonsai-8B	1-bit	512	256	5725	270.38	3.70	2.11	1.7533
Bonsai-8B	1-bit	1024	256	10976	273.20	3.66	2.11	1.7352
Bonsai-8B	1-bit	2048	256	21731	278.82	3.59	2.11	1.7002
Bonsai-8B	1-bit	256	512	3184	269.55	3.71	2.11	1.7587
Bonsai-8B	1-bit	512	512	5724	271.05	3.69	2.11	1.7489
Bonsai-8B	1-bit	1024	512	10975	273.80	3.65	2.11	1.7314
Bonsai-8B	1-bit	2048	512	21726	279.47	3.58	2.11	1.6963
Ternary-Bonsai-1.7B	1.58-bit	256	128	913	103.06	9.70	1.90	5.1161
Ternary-Bonsai-1.7B	1.58-bit	512	128	1642	103.91	9.62	1.90	5.0745
Ternary-Bonsai-1.7B	1.58-bit	1024	128	3111	106.05	9.43	1.90	4.9721
Ternary-Bonsai-1.7B	1.58-bit	2048	128	6159	110.18	9.08	1.90	4.7855
Ternary-Bonsai-1.7B	1.58-bit	256	256	915	103.06	9.70	1.90	5.1161
Ternary-Bonsai-1.7B	1.58-bit	512	256	1640	104.02	9.61	1.90	5.0687
Ternary-Bonsai-1.7B	1.58-bit	1024	256	3111	106.05	9.43	1.90	4.9719
Ternary-Bonsai-1.7B	1.58-bit	2048	256	6157	110.22	9.07	1.90	4.7838
Ternary-Bonsai-1.7B	1.58-bit	256	512	914	103.53	9.66	1.90	5.0930
Ternary-Bonsai-1.7B	1.58-bit	512	512	1640	104.54	9.57	1.90	5.0435
Ternary-Bonsai-1.7B	1.58-bit	1024	512	3111	106.57	9.38	1.90	4.9479
Ternary-Bonsai-1.7B	1.58-bit	2048	512	6158	110.71	9.03	1.90	4.7625
Ternary-Bonsai-4B	1.58-bit	256	128	2261	231.55	4.32	1.96	2.1981
Ternary-Bonsai-4B	1.58-bit	512	128	4108	232.72	4.30	1.96	2.1870
Ternary-Bonsai-4B	1.58-bit	1024	128	8104	235.64	4.24	1.96	2.1599
Ternary-Bonsai-4B	1.58-bit	2048	128	19228	241.23	4.15	1.96	2.1099
Ternary-Bonsai-4B	1.58-bit	256	256	2297	231.53	4.32	1.96	2.1983
Ternary-Bonsai-4B	1.58-bit	512	256	4136	232.87	4.29	1.96	2.1856
Ternary-Bonsai-4B	1.58-bit	1024	256	8612	235.98	4.24	1.96	2.1568
Ternary-Bonsai-4B	1.58-bit	2048	256	16026	241.86	4.13	1.96	2.1045
Ternary-Bonsai-4B	1.58-bit	256	512	2311	232.20	4.31	1.96	2.1919
Ternary-Bonsai-4B	1.58-bit	512	512	4438	233.72	4.28	1.96	2.1777
Ternary-Bonsai-4B	1.58-bit	1024	512	9017	237.01	4.22	1.96	2.1476
Ternary-Bonsai-4B	1.58-bit	2048	512	15845	242.04	4.13	1.96	2.1028

Per-Model Best Tok/J

Model	Bits	Best Tok/J	Prompt (tok)	Gen (tok)	Tok/s	Power (W)
Bonsai-1.7B	1-bit	4.4308	256	128	6.82	1.54
Bonsai-4B	1-bit	2.2583	256	128	3.62	1.60
Bonsai-8B	1-bit	1.7624	256	256	3.72	2.11
Ternary-Bonsai-1.7B	1.58-bit	5.1161	256	128	9.70	1.90
Ternary-Bonsai-4B	1.58-bit	2.1983	256	256	4.32	1.96

Thermal Summary

Model	Avg Power (W)	Avg CPU (C)	Avg GPU (C)	Peak TJ (C)	Throttled
Bonsai-1.7B	1.54	54.2	55.4	56.4	No
Bonsai-4B	1.60	53.0	54.3	55.8	No
Bonsai-8B	2.11	56.1	57.4	58.3	No
Ternary-Bonsai-1.7B	1.90	54.8	56.1	57.0	No
Ternary-Bonsai-4B	1.96	54.5	55.7	56.8	No

Generated by `benchmark_all_bonsai.sh` (llamacpp) on 2026-06-04 02:15:00

Notes

—Results cover 5 model/quant configs: Bonsai 1.7B/4B/8B at Q10, and Ternary-Bonsai 1.7B/4B at Q20.
—A Ternary-Bonsai-8B-server.log is included, but that model produced no timing rows in this run.
—Power is the VDD_CPU_GPU_CV rail (CPU + GPU + CV) from tegrastats, averaged over each aiperf run window.

License & citation

—Benchmark results and artifacts (aiperf exports, server logs, tegrastats logs, generated reports): CC BY 4.0. Reuse and adaptation allowed, including commercially, provided you credit Yuvraj Singh, link the license, and indicate changes.
—Harness code that produced these results: Apache-2.0.

If you use these results, please cite:

bibtex

@misc{singh2026smolperfbenchmark,
      title={smolperfbenchmark: On-Device LLM Leaderboard},
      author={Yuvraj Singh},
      year={2026},
      howpublished={\url{https://github.com/YuvrajSingh-mist/smolperfbenchmark}},
}

---

Released freely — support more experiments like it:

![Support me on Ko-fi](https://ko-fi.com/O7W120DR8R) ![GitHub Sponsors](https://github.com/sponsors/YuvrajSingh-mist)

More: <https://huggingface.co/YuvrajSingh9886>