stamsam/maple-preview-gguf

⚠️ STATUS — READ THIS BEFORE DOWNLOADING

All three packs generate text on GPU: maple-f16.gguf (~385 tok/s, H200), maple-q4_k_m.gguf (GPU-verified), and maple-tq2_0.gguf (~97 tok/s on an RTX 4000 Ada; previously CPU-only), with the Maple-enabled llama.cpp fork at github.com/stamsam/llama.cpp (branch prism, rev 9ee03ee) — all layers GPU-offloaded. Mainline llama.cpp CANNOT run any of these files.** The maple architecture exists only in the stamsam/llama.cpp fork (the PrismML upstream does not have it), and the fork's ternary format is not interchangeable with mainline types. Development artifacts of an ongoing port, shared openly.

Overview

This repository hosts GGUF conversions of [deepgrove/maple-preview](https://huggingface.co/deepgrove/maple-preview) — a 20B-A1B ternary-weight Mixture-of-Experts reasoning LLM by DeepGrove (2026, MIT license) — plus the original BF16 checkpoint (in bf16-checkpoint/). The GGUF pipeline was built on the PrismML llama.cpp fork; the complete Maple runtime (converter + inference graph) lives in the [stamsam/llama.cpp](https://github.com/stamsam/llama.cpp) fork (branch prism).

Model summary

Hosted files

Tensor counts and shapes were verified by post-write inspection (GGUFReader): 291 tensors, architecture key maple, per-tensor types confirmed (f16: F16/F32; tq20: 168x type-35 + 2x Q40 + 121x F32; q4km: 145x Q4K + 25x Q6K + 121x F32).

Status

The Maple compute graph is implemented and verified in [github.com/stamsam/llama.cpp](https://github.com/stamsam/llama.cpp) (branch `prism`). All three packs generate coherent text (tested: "What is 2+2?" → "The answer is 4."): maple-f16.gguf and maple-q4_k_m.gguf with all layers offloaded to an NVIDIA H200, and maple-tq2_0.gguf on an RTX 4000 Ada at ~97 tok/s (CPU: ~5.5 tok/s) and on a Jetson Xavier at ~14.8 tok/s.

What the graph implements (matching the MLX reference deepgrove/maple-preview-2bit-mlx):

• —Hybrid attention — 3:1 SWA-512:GA: sliding-window layers (window 512, partial 64/128-dim RoPE, theta 10000) with full-attention layers every 4th (il % 4 == 3) that carry no RoPE (n_rot = 0).
• —Flash-head QK — per-head RMSNorm (q_norm/k_norm) on the reshaped Q/K before RoPE; k_proj at 512 dims (4 KV heads x 128).
• —MoE — 256 experts, top-8, moe_intermediate 512, clamp-7 SwiGLU (silu(min(gate, +7)) * clip(up, -7, +7)), fp32 softmax + renormalized routing.
• —KV — sliding-window KV cache (window 512) for the SWA layers.

Known limitations:

• —Ternary GPU support is new — tq2_0 (GGML type 35) got its CUDA kernels (dequant, getrows, mmvq vecdot) at fork rev 9ee03ee; big-batch matmuls use the dequant+gemm path (mmq kernels not yet ported).
• —Fork-only — the maple arch lives in stamsam/llama.cpp (branch prism); mainline llama.cpp and the PrismML upstream fork cannot load these files.
• —Early validation — verified for coherent generation on simple prompts; no benchmarks, perplexity, or systematic evals yet.

For reference implementations of the model itself:

• —Apple Silicon: deepgrove/maple-preview-2bit-mlx (official MLX runtime)
• —Elsewhere: the original PyTorch checkpoint (deepgrove/maple-preview) — or the bf16-checkpoint/ copy in this repo

Usage

These files only work with stamsam/llama.cpp (branch prism); mainline llama.cpp cannot load them.

# GPU (f16 or Q4_K_M packs)
llama-cli -m maple-q4_k_m.gguf -p "Your prompt here" -n 512

# Server (OpenAI-compatible API)
llama-server -m maple-q4_k_m.gguf --port 8080

maple-tq2_0.gguf runs the same way but executes on the CPU backend until CUDA kernels for the ternary type land. No Maple-specific CLI flags are required.

Quantization recipe

All steps run CPU-side on an NVIDIA H200 host: the converter and quantizer are CPU processes — no GPU compute is involved, and while CUDA is available to torch, it is unused in these steps.

1. 1.Convert: convert_hf_to_gguf.py (PrismML fork, with the Maple converter in this fork) on the deepgrove/maple-preview checkpoint → maple-f16.gguf (dense reference).
2. 2.Quantize: llama-quantize on maple-f16.gguf:
3. 3.`maple-tq2_0.gguf` — per-tensor type mix mirroring the official maple-preview-2bit-mlx tiering:
4. 4.fork ternary `tq2_0` (type 35, ~2.06 bpw) for attn_q/attn_k/attn_v/attn_output and all ffn_*_exps;
5. 5.Q4_0 for token_embd and output (lm_head);
6. 6.F32/F16 for routers and norms.
7. 7.`maple-q4_k_m.gguf` — uniform Q4_K_M over the entire model.

The maple architecture is registered in the fork's arch table with a dedicated llama_model_maple class (hparams, tensor map, and full compute graph). Note: an earlier attempt used the fork's type-42 Q20 layout, which the quantizer can write but no inference kernel can read — it was superseded by the type-35 `tq20 format used in maple-tq2_0.gguf`.

Caveats

• —Fork-only formats, no interop: the fork's ternary tq2_0 (GGML type 35) and its Q20 variant (type 42) are custom layouts — mainline llama.cpp has different Q20/Q2_K types under the same names and cannot read these files. There is no compatibility layer.
• —CPU-only ternary: maple-tq2_0.gguf has no CUDA kernels yet — expect ~5.5 tok/s CPU throughput on a 96-core host, not H200 speed.
• —Q4 on ternary weights is wasteful: Maple's experts and q/o projections are already ternary in storage; a uniform Q4_K_M re-quantizes them to 4 bits with little precision benefit. maple-q4_k_m.gguf is a quantizer sanity check / compatibility data point, not the recommended format — the tiered maple-tq2_0.gguf mirrors the intended storage layout.
• —Not the MLX format: the tiering philosophy matches the official MLX pack (2-bit ternary for projections+experts at 16 codes/u32, 4-bit for lm_head+embeddings at 8 codes/u32, BF16 for router+norms), but the binary formats differ. Files are not interchangeable between the two runtimes.
• —Development artifacts, not a release: trial/experimental work on a custom fork; formats and tensor layouts may change without notice as the port progresses.

Development log

• —Converted deepgrove/maple-preview (BF16 checkpoint, 20.214 B params / 18,651 tensors) → maple-f16.gguf via the fork's converter (CPU, H200 host).
• —Quantized → maple-tq2_0.gguf (tiered ternary pack: 168x type-35, 2x Q40, 121x F32) and `maple-q4km.gguf` (uniform Q4K_M).
• —Post-write verification: tensor counts and shapes inspected against the source checkpoint (2026-08-04).
• —Graph implemented + verified (2026-08-04): hybrid SWA/GA attention, flash-head QK, 256-expert clamp-7 SwiGLU MoE, fp32 renorm routing — f16 and Q4KM packs generate coherent text on an H200 (~385 tok/s at f16, all layers GPU-offloaded).
• —Ternary GPU kernels (rev 9ee03ee): CUDA dequant + getrows + mmvq vecdot for tq2_0 — verified on an RTX 4000 Ada at ~97 tok/s decode / ~111 tok/s prefill (CPU: 5.5 tok/s).
• —Jetson Xavier verified (2026-08-04): maple-tq2_0.gguf runs CPU-only on the Xavier (aarch64, 8 cores, 14 GB RAM) at 14.8 tok/s decode / 37.5 tok/s prefill, coherent output.
• —Next phase: quality benchmarks vs the MLX reference, CUDA build for the Xavier (sm_72) once its toolkit is set up.

Credits

• —Model: DeepGrove — deepgrove/maple-preview (MIT)
• —Official on-device runtime: deepgrove/maple-preview-2bit-mlx
• —Base tooling: PrismML llama.cpp fork
• —Maple runtime fork + conversion + packaging: stamsam/llama.cpp — MIT