JacobPEvans/mlx-benchmarks
MLX Benchmarks Structured benchmark results for MLX-quantized and other locally-hosted LLMs on Apple Silicon. Covers throughput, time-to-first-token, tool-calling, code generation, reasoning, knowledge, and math suites. Results are produced by a sweep harness that wires upstream evaluation tools against a local vllm-mlx inference server: EleutherAI/lm-evaluation-harness — coding, reasoning, knowledge, math linusvwe/MLXBench — throughput and time-to-first-token vllm… See the full description on the dataset page: https://huggingface.co/datasets/JacobPEvans/mlx-benchmarks.
MLX Benchmarks
Structured benchmark results for MLX-quantized and other locally-hosted LLMs on Apple Silicon. Covers throughput, time-to-first-token, tool-calling, code generation, reasoning, knowledge, and math suites.
Results are produced by a sweep harness that wires upstream evaluation tools against a local vllm-mlx inference server:
- EleutherAI/lm-evaluation-harness — coding, reasoning, knowledge, math
- linusvwe/MLXBench — throughput and time-to-first-token
- vllm `benchmark_serving` — performance second opinion
- huggingface/lighteval — broader task coverage
All data here is generated on Apple Silicon hardware (MINISFORUM MS-A2 / M4 Max class), stored in flat columnar Parquet for easy querying, and appended to via unique-filename commits so historical shards are never overwritten.
Quickstart
from datasets import load_dataset
ds = load_dataset("JacobPEvans/mlx-benchmarks")
print(ds)
# Example: average throughput per model
import pandas as pd
df = ds["train"].to_pandas()
throughput_rows = df[df.suite == "throughput"]
print(
throughput_rows.groupby("model")["metric_value"]
.mean()
.sort_values(ascending=False)
)Raw Parquet fetch (token-optimal for agents):
curl -sSL \
https://huggingface.co/datasets/JacobPEvans/mlx-benchmarks/resolve/main/data/train-00000-of-00001.parquet \
-o run.parquetSchema
Each input benchmark run produces a JSON envelope (see schema.json in this repo for the authoritative v1 spec). The envelope is exploded row-wise into flat scalar columns — one row per entry in the envelope's results[] array. Skipped runs become a single sentinel row with null metric columns and skipped=true. This mirrors the columnar layout used by the Open LLM Leaderboard contents dataset.
Nested fields from the envelope (tags, errors) are preserved as JSON-serialized strings so no information is lost — rehydrate with json.loads(row["tags_json"]).
Update cadence
New rows are appended on every sweep via a unique-filename commit pattern (data/run-{timestamp}-{sha}-{suite}-{model}.parquet). Historical shards are never overwritten. load_dataset() concatenates all data/*.parquet files into a single train split at load time.
License
Apache 2.0 — same as the underlying upstream evaluation tools.