CoolFace
Datasetpublic

danielrmarques/bench-marques

bench-marques — local LLM inference measurements Every number here was measured on one machine, with the method written down and the refutations kept. This dataset is the source of truth for the results; the harness that produces them lives at danielrmarques/bench-marques. The machine AMD Ryzen AI MAX+ 395 · Radeon 8060S · 128 GB unified LPDDR5X (~205 GB/s effective, 80% of the 256 GB/s theoretical ceiling) · llama.cpp, Vulkan backend. Every run is on AC power… See the full description on the dataset page: https://huggingface.co/datasets/danielrmarques/bench-marques.

sourceHugging Facemitupdated 17d agoView on Hugging Face
1likes200downloads
Dataset Card

bench-marques — local LLM inference measurements

Every number here was measured on one machine, with the method written down and the refutations kept. This dataset is the source of truth for the results; the harness that produces them lives at danielrmarques/bench-marques.

The machine

AMD Ryzen AI MAX+ 395 · Radeon 8060S · 128 GB unified LPDDR5X (~205 GB/s effective, 80% of the 256 GB/s theoretical ceiling) · llama.cpp, Vulkan backend. Every run is on AC power with zero standby cycles since boot — this machine loses up to 4× to Modern Standby and only a reboot restores it, so that condition is part of the data, not a footnote.

What is in here

filewhat it is
measurements.jsonlone row per measurement — the queryable view
runs/<date>/<model>/run.jsonthe full nested record, exactly as the harness wrote it
runs/<date>/<model>/sweep.jsonone-variable-at-a-time flag sweeps
runs/<date>/<model>/raw/*.txtthe model's raw answers, so any score can be re-derived
python
from datasets import load_dataset
ds = load_dataset("danielrmarques/bench-marques", split="train")
ds.filter(lambda r: r["scenario"] == "decode_vs_ctx")

Reading these numbers honestly

Capability percentages are a floor, not a ranking. Three of the five tasks are scored by anchors that detect whether the required concept appeared. One model scored 7/7 on a task while contradicting its own arithmetic two sections later. Use them to rule out models that never reach the concepts; never to order the ones that do. The raw answers are included precisely so you can check.

Scores are not language-invariant. All three models score lower in Portuguese than in English. In one verified case the same model identified a concurrency race in English and gave a crash-based explanation in Portuguese for the identical question.

Absolute decode differs by scenario. decode_base uses an explanatory prompt; decode_vs_ctx uses a highly predictable one, which inflates draft acceptance on models with a drafter. Compare a model's curve to itself, and compare models only within the same scenario.

Nothing here is compared across sessions or builds. Each run records its llama.cpp build; rows from different builds are different measurements.

What was refuted

Kept deliberately — what does not work is usually worth more than what does.

  • —ROCm backend: dropped. Decode does not fall off with context on the dense model (95% of base at 30k), so the switch would gain nothing and would invalidate the whole Vulkan calibration.
  • —`--slot-save-path`: restores 6,969 tokens of KV in 160 ms, reports success — and the next completion re-prefills all of them. The KV is repopulated, not reused.
  • —PLE offload (-ot per_layer_token_embd=CPU): the flag binds and moves a 26.8 GB tensor; it costs 2%. On unified memory both buffers are the same LPDDR5X at the same bandwidth.
  • —`--spec-draft-p-min 0.6`: costs 15%. Filtering drafts by confidence loses more acceptance than it saves in verification.

License

MIT — Copyright (c) 2026 Daniel Marques.