Datasetpublic

enescingoz/humaneval-apple-silicon

Mac Coding Bench Results v1 — Speed + Code Quality Benchmarks on Apple Silicon Speed and code quality benchmarks for quantized LLMs running locally on Apple Silicon Macs. The dataset pairs inference speed measurements (tokens/sec) with HumanEval+ functional correctness scores for 21 models, across three hardware configurations (M1, M2 Max, M5) totaling 123 benchmark results. Key Highlights Qwen 3.6 35B-A3B achieves 89.6% HumanEval+ pass@1 at 16.7 tok/s — best… See the full description on the dataset page: https://huggingface.co/datasets/enescingoz/humaneval-apple-silicon.

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes8downloads

Dataset Card

Mac Coding Bench Results v1 — Speed + Code Quality Benchmarks on Apple Silicon

Speed and code quality benchmarks for quantized LLMs running locally on Apple Silicon Macs. The dataset pairs inference speed measurements (tokens/sec) with HumanEval+ functional correctness scores for 21 models, across three hardware configurations (M1, M2 Max, M5) totaling 123 benchmark results.

Key Highlights

—Qwen 3.6 35B-A3B achieves 89.6% HumanEval+ pass@1 at 16.7 tok/s — best quality score in the dataset, and fast thanks to MoE architecture
—Qwen 2.5 Coder 7B hits 84.2% at 11.3 tok/s — best quality-to-speed ratio for a dense model
—Phi 4 Mini 3.8B reaches 70.7% at 19.6 tok/s — strong for its size
—Gemma 4 family scores 9-31% on HumanEval+, significantly underperforming Gemma 3 (34-79%)
—21 models evaluated for code quality, 123 total speed benchmark results across 3 chips
—All models quantized to Q4KM (GGUF) or 4-bit (MLX)

Dataset Description

Each row represents one model benchmarked on one hardware configuration. Speed metrics come from llama-bench (GGUF) or mlx_lm.benchmark (MLX). Code quality metrics come from EvalPlus HumanEval+ evaluation (164 problems). Quality scores are available for 21 models on the M5 configuration; the remaining 102 rows have speed-only data across M1, M2 Max, and M5.

Data Fields

Field	Type	Description
`model_name`	string	Human-readable model name
`model_id`	string	Slug identifier for the model
`params`	string	Parameter count (e.g., "7B", "35B")
`quant`	string	Quantization method (Q4KM or 4bit)
`runtime`	string	Inference runtime (llama.cpp or mlx-lm)
`chip`	string	Apple Silicon chip (M1, M2 Max, M5)
`cpu_cores`	int	Number of CPU cores
`gpu_cores`	int	Number of GPU cores
`ram_gb`	int	Total system RAM in GB
`os_version`	string	macOS kernel version
`pp128_toks`	float	Prompt processing speed, 128 tokens (tok/s)
`pp256_toks`	float	Prompt processing speed, 256 tokens (tok/s)
`pp512_toks`	float	Prompt processing speed, 512 tokens (tok/s)
`tg128_toks`	float	Text generation speed, 128 tokens (tok/s)
`tg256_toks`	float	Text generation speed, 256 tokens (tok/s)
`peak_memory_gb`	float	Peak RSS memory usage in GB
`humaneval_plus_pass1`	float	HumanEval+ pass@1 score (0-1), null if not evaluated
`humaneval_base_pass1`	float	HumanEval base pass@1 score (0-1), null if not evaluated
`perplexity`	float	Perplexity score, null if not evaluated
`eval_framework_version`	string	EvalPlus version used
`timestamp`	string	ISO 8601 timestamp of the benchmark run

Hardware Configurations

Chip	CPU Cores	GPU Cores	RAM	Rows
Apple M1	8	7	16 GB	20
Apple M2 Max	12	38	32 GB	39
Apple M5	10	10	32 GB	64

Code quality evaluations (HumanEval+) were run on the M5 configuration only.

Benchmark Methodology

Speed benchmarks:

—GGUF models: llama-bench with flash attention enabled, all layers offloaded to GPU (-ngl 99)
—MLX models: mlx_lm.benchmark
—Prompt processing measured at 128, 256, and 512 input tokens
—Text generation measured at 128 and 256 output tokens

Code quality benchmarks:

—Framework: EvalPlus HumanEval+
—164 problems with 80x test coverage over the original HumanEval test suite
—Greedy decoding (temperature=0)
—Reasoning models evaluated with --no-think flag
—Quantization: Q4KM (GGUF), 4-bit (MLX)

Models with Code Quality Scores

All results on Apple M5, 32 GB, Q4KM quantization.

Model	Params	HumanEval+ pass@1	tg128 (tok/s)
Qwen 3.6 35B-A3B	35B	89.6%	16.7
Qwen 2.5 Coder 32B	32B	87.2%	2.5
Qwen 2.5 Coder 14B	14B	86.6%	5.9
Qwen 2.5 Coder 7B	7B	84.2%	11.3
Phi 4 14B	14B	82.3%	5.3
Devstral Small 24B	24B	81.7%	3.6
Gemma 3 27B	27B	78.7%	3.1
Gemma 3 12B	12B	75.6%	5.7
Mistral Small 3.1 24B	24B	75.6%	3.6
Phi 4 Mini 3.8B	3.8B	70.7%	19.6
Mistral Nemo 12B	12B	64.6%	6.9
Gemma 3 4B	4B	64.6%	16.5
Llama 3.1 8B Instruct	8B	61.0%	10.8
Llama 3.2 3B Instruct	3B	60.4%	24.1
Mistral 7B Instruct v0.3	7B	37.2%	11.5
Gemma 3 1B	1B	34.2%	46.6
Llama 3.2 1B Instruct	1B	32.9%	59.4
Gemma 4 31B	31B	31.1%	5.5
Gemma 4 E4B	4B	14.6%	36.7
Gemma 4 26B-A4B MoE	26B	12.2%	16.2
Gemma 4 E2B	2B	9.2%	29.2

Limitations

—Single quantization level: All GGUF models use Q4KM; only one MLX model included. Results may differ at other quantization levels.
—Limited hardware configs: Three Apple Silicon chips (M1 16GB, M2 Max 32GB, M5 32GB). No M3/M4 Pro/Ultra data.
—Greedy decoding only: All code quality evaluations use temperature=0. Sampling-based pass@k scores would differ.
—Reasoning models: Models like DeepSeek R1 Distill were tested with --no-think, which disables their chain-of-thought reasoning and may understate their capability.
—Quality scores on M5 only: HumanEval+ evaluations were run on a single hardware config. Scores should be hardware-independent, but inference artifacts from quantization on different memory configurations could vary.
—Gemma 4 scores: The Gemma 4 models score unusually low. This may reflect early quantization issues, prompt template incompatibilities, or model behavior at Q4KM precision.

Citation

bibtex

@dataset{mac_coding_bench_v1,
  title   = {Mac Coding Bench Results v1},
  author  = {Enes Cingoz},
  year    = {2026},
  url     = {https://huggingface.co/datasets/enescingoz/humaneval-apple-silicon},
  note    = {Speed and code quality benchmarks for quantized LLMs on Apple Silicon}
}

License

MIT