Datasetpublic

YuvrajSingh9886/bonsai-jetson-benchmark-15w

Bonsai Jetson Benchmark — 15W Platform: NVIDIA Jetson Orin Nano Super 8GB · Power mode: 15W Backend: llama.cpp build-jetson · CUDA · -ngl 99 Sweep: prompt ∈ {256, 512, 1024, 2048} tok × gen ∈ {128, 256, 512} tok · 20 reqs/combo Status: Complete — 57 combos (5 models × 12 prompt/gen configs) Key metric: tok/J = output tok/s ÷ VDD_CPU_GPU_CV (W) Models Model Quant Size Bonsai-1.7B Q1_0 (1-bit) ~237 MB Bonsai-4B Q1_0 (1-bit) ~540 MB Bonsai-8B Q1_0… See the full description on the dataset page: https://huggingface.co/datasets/YuvrajSingh9886/bonsai-jetson-benchmark-15w.

sourceHugging Facecc-by-4.0updated 10d agoView on Hugging Face

1likes363downloads

Dataset Card

Bonsai Jetson Benchmark — 15W

Platform: NVIDIA Jetson Orin Nano Super 8GB · Power mode: 15W Backend: llama.cpp build-jetson · CUDA · -ngl 99 Sweep: prompt ∈ {256, 512, 1024, 2048} tok × gen ∈ {128, 256, 512} tok · 20 reqs/combo Status: Complete — 57 combos (5 models × 12 prompt/gen configs) Key metric: tok/J = output tok/s ÷ VDDCPUGPU_CV (W)

Models

Model	Quant	Size
Bonsai-1.7B	Q1\_0 (1-bit)	~237 MB
Bonsai-4B	Q1\_0 (1-bit)	~540 MB
Bonsai-8B	Q1\_0 (1-bit)	~1.07 GB
Ternary-Bonsai-1.7B	Q2\_0 (2-bit ternary)	~300 MB
Ternary-Bonsai-4B	Q2\_0 (2-bit ternary)	~700 MB
Ternary-Bonsai-8B	Q2\_0 (2-bit ternary)	OOM on JetPack R36.4.7

Dataset viewer columns

model · quant · prompt_tokens · gen_tokens · ttft_avg_ms · ttft_p50/p90/p99_ms · itl_avg_ms · itl_p50/p90/p99_ms · tok_s · prefill_tok_s · req_latency_avg/p99_ms · power_w · tok_j · avg_temp_c

Bonsai All-Model Benchmark — Jetson Orin Nano Super 8GB

Date: 2026-05-27 22:13 Backend: CUDA (-ngl 99) Context: 2560 tokens Concurrency: 1 Sweep: prompt in {256,512,1024,2048} gen in {128,256,512} Artifacts: /home/yuvrajsingh/Desktop/benchmark/artifacts/bonsai-all-20260527-0200

Skipped Models (smoke test failed or file not found)

—Bonsai-8B (server failed to start)
—Ternary-Bonsai-4B (server failed to start)
—Ternary-Bonsai-8B (server failed to start)

Full Results

Model	Bits	Prompt (tok)	Gen (tok)	TTFT avg (ms)	ITL avg (ms)	Tok/s	Power (W)	Tok/J
Bonsai-1.7B	1-bit	256	128	210	38.47	25.99	4.49	5.7846
Bonsai-1.7B	1-bit	512	128	370	38.88	25.72	4.49	5.7231
Bonsai-1.7B	1-bit	1024	128	689	39.75	25.16	4.49	5.5993
Bonsai-1.7B	1-bit	2048	128	1353	41.26	24.24	4.49	5.3935
Bonsai-1.7B	1-bit	256	256	210	38.50	25.97	4.49	5.7794
Bonsai-1.7B	1-bit	512	256	370	38.95	25.67	4.49	5.7132
Bonsai-1.7B	1-bit	1024	256	689	39.76	25.15	4.49	5.5976
Bonsai-1.7B	1-bit	2048	256	1354	41.24	24.25	4.49	5.3959
Bonsai-1.7B	1-bit	256	512	210	38.66	25.87	4.49	5.7560
Bonsai-1.7B	1-bit	512	512	371	39.13	25.55	4.49	5.6867
Bonsai-1.7B	1-bit	1024	512	690	39.98	25.01	4.49	5.5659
Bonsai-1.7B	1-bit	2048	512	1354	41.47	24.11	4.49	5.3656
Bonsai-4B	1-bit	256	128	464	70.42	14.20	5.15	2.7558
Bonsai-4B	1-bit	512	128	830	70.92	14.10	5.15	2.7365
Bonsai-4B	1-bit	1024	128	1576	71.88	13.91	5.15	2.7001
Bonsai-4B	1-bit	2048	128	3133	73.89	13.53	5.15	2.6265
Bonsai-4B	1-bit	256	256	465	70.47	14.19	5.15	2.7537
Bonsai-4B	1-bit	512	256	830	70.96	14.09	5.15	2.7349
Bonsai-4B	1-bit	1024	256	1577	71.99	13.89	5.15	2.6956
Bonsai-4B	1-bit	2048	256	3134	73.93	13.53	5.15	2.6250
Bonsai-4B	1-bit	256	512	465	70.74	14.14	5.15	2.7433
Bonsai-4B	1-bit	512	512	830	71.22	14.04	5.15	2.7247
Bonsai-4B	1-bit	1024	512	1577	72.17	13.86	5.15	2.6890
Bonsai-4B	1-bit	2048	512	3134	74.10	13.50	5.15	2.6190
Ternary-Bonsai-1.7B	1.58-bit	256	128	239	26.16	38.23	3.34	11.4583
Ternary-Bonsai-1.7B	1.58-bit	512	128	418	26.53	37.69	3.34	11.2979
Ternary-Bonsai-1.7B	1.58-bit	1024	128	773	27.32	36.61	3.34	10.9718
Ternary-Bonsai-1.7B	1.58-bit	2048	128	1516	28.74	34.79	3.34	10.4282
Ternary-Bonsai-1.7B	1.58-bit	256	256	236	26.07	38.36	3.34	11.4986
Ternary-Bonsai-1.7B	1.58-bit	512	256	414	26.45	37.81	3.34	11.3325
Ternary-Bonsai-1.7B	1.58-bit	1024	256	772	27.20	36.77	3.34	11.0199
Ternary-Bonsai-1.7B	1.58-bit	2048	256	1515	28.73	34.81	3.34	10.4331
Ternary-Bonsai-1.7B	1.58-bit	256	512	235	26.22	38.14	3.34	11.4330
Ternary-Bonsai-1.7B	1.58-bit	512	512	414	26.61	37.58	3.34	11.2638
Ternary-Bonsai-1.7B	1.58-bit	1024	512	772	27.37	36.53	3.34	10.9498
Ternary-Bonsai-1.7B	1.58-bit	2048	512	1516	28.86	34.65	3.34	10.3869

Per-Model Best Tok/J

Model	Bits	Best Tok/J	Prompt (tok)	Gen (tok)	Tok/s	Power (W)
Bonsai-1.7B	1-bit	5.7846	256	128	25.99	4.49
Bonsai-4B	1-bit	2.7558	256	128	14.20	5.15
Ternary-Bonsai-1.7B	1.58-bit	11.4986	256	256	38.36	3.34

Thermal Summary

Model	Avg Power (W)	Avg CPU (C)	Avg GPU (C)	Peak TJ (C)	Throttled
Bonsai-1.7B	4.49	62.4	63.6	65.9	No
Bonsai-4B	5.15	62.3	63.7	66.2	No
Ternary-Bonsai-1.7B	3.34	59.9	61.3	69.2	No

Generated by `benchmark_all_bonsai.sh` on 2026-05-27 22:13:35

License & citation

—Benchmark results and artifacts (aiperf exports, server logs, tegrastats logs, generated reports): CC BY 4.0. Reuse and adaptation allowed, including commercially, provided you credit Yuvraj Singh, link the license, and indicate changes.
—Harness code that produced these results: Apache-2.0.

If you use these results, please cite:

bibtex

@misc{singh2026smolperfbenchmark,
      title={smolperfbenchmark: On-Device LLM Leaderboard},
      author={Yuvraj Singh},
      year={2026},
      howpublished={\url{https://github.com/YuvrajSingh-mist/smolperfbenchmark}},
}

---

Released freely — support more experiments like it:

![Support me on Ko-fi](https://ko-fi.com/O7W120DR8R) ![GitHub Sponsors](https://github.com/sponsors/YuvrajSingh-mist)

More: <https://huggingface.co/YuvrajSingh9886>