keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF

Qwen3.6-35B-A3B-REAM-192-heretic — APEX-GGUF

⚠️ Preliminary release. Quantized from the bf16 GGUF master of keithnull/Qwen3.6-35B-A3B-REAM-192-heretic. The only validation performed so far is (a) Heretic's first-token KL + refusal-marker eval against its 100-prompt evaluation split during the optimization run, (b) a five-prompt qualitative chat spot-check on the bf16 weights, and (c) basic generation-coherence smoke tests on the resulting GGUFs. No perplexity, KL-divergence, or academic-benchmark numbers have been measured against this APEX recipe yet — the published APEX results are on the un-abliterated Qwen3.5-35B-A3B reference, not this model. Treat this release as a preview while those evaluations are pending. Card will be updated with hard numbers as they land.

APEX (Adaptive Precision for EXpert Models) MoE-aware mixed-precision quantizations of keithnull/Qwen3.6-35B-A3B-REAM-192-heretic. For standard llama.cpp K-quants of the same model, see keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-GGUF.

What is APEX?

https://github.com/mudler/apex-quant

Available quantizations

The "I-" prefix in APEX nomenclature means "imatrix-calibrated" — it changes the scale values stored in K-quant blocks but does not change the tensor formats.

A note on filenames

APEX GGUFs are mixed-precision — every file contains multiple llama.cpp quant types simultaneously (e.g. Q6_K for edge layers, Q5_K for near-edge, IQ4_XS for middle-layer routed experts, Q8_0 for shared experts, etc). HuggingFace's hardware-compatibility widget and various model-listing surfaces auto-detect a file's quantization by scanning the filename for a recognized K-quant or IQ-quant token (Q4_K_M, Q3_K_S, IQ4_XS, etc). Because no single token is strictly correct for a mixed-precision file, the parser would otherwise show "Unknown" / "Cannot determine variant" and the model wouldn't surface on hardware-filtered model lists.

To work around this, each file in this repo has a parser-friendly K-quant suffix appended that matches its dominant middle-layer expert quantization (since middle layers comprise the bulk of routed-expert parameters and most strongly determine the file's effective BPW). For example: APEX I-Compact uses Q4_K-edge + Q3_K-middle experts, so its file is named ...-APEX-ICompact-Q3_K_L.gguf. The Q3_K_L tag isn't exactly the file's contents — the actual tensor types are mixed — but it's the closest single recognizable token for HF's parser, and the BPW lands close to Q3KL's 4.27.

Pay attention to the APEX tier name (Mini / Compact / I-Compact / I-Balanced / I-Quality) for the real description of what's inside each file. Treatt the "Q_#" suffix purely as a UX hint for HF's display layer.

Vision support

These GGUFs are text-only weights, but the underlying REAM-192 architecture preserves the Qwen3.6-VL vision tower. Pairing with the standard Qwen3.6-VL projector enables image input via llama-server's OpenAI-compatible chat-completions endpoint.

The compatible projector is mirrored here for convenience:

Why a REAP-derived projector works with REAM

Both REAM (this model) and REAP (atbender's source) preserve the unmodified Qwen3.6-VL vision encoder, hidden-state dimension, and tokenizer — neither method touches the vision tower or the input-embedding layer. The mmproj's image-token embeddings land in the same hidden-state space the LM was originally trained against. The APEX recipe further keeps shared experts and attention layers at high precision (Q6–Q8) across every tier, so vision-LM alignment survives even at the most aggressive (Mini-IQ2_S) routed-expert quantization.

This is a cross-recipe pairing without any vision-specific fine-tuning. A REAM-native projector would likely improve quality further, but is not yet released.

Validation (2026-05-12, n=1, qualitative)

Single-rater smoke test on a 590×904 JPEG product photo of a Canon PowerShot G5, running APEX Mini-IQ2_S + this projector on Apple M3 Metal:

• —Correctly identified the camera (Canon PowerShot G5) and attached flash (Canon Speedlite 420EX).
• —Read fine print: 5.0 MEGA PIXELS, CANON ZOOM LENS 7.2-28.8mm 1:2.0-3.0.
• —Noted the CENEO watermark across the middle of the image.
• —~40 s end-to-end via an agent harness (pi-mono) on the 10 GB Mini quant.

No quantitative VLM benchmarks (MMMU, MM-Vet, OCR-Bench, etc.) have been run against this pairing yet. The above is a smoke test, not a result. The same probe was previously validated against the REAP-26B-A3B Q4KM native pairing on 2026-05-01 — REAM Mini matches REAP-native on this image.

Usage

llama-server \
  -m Qwen3.6-35B-A3B-REAM-192-heretic-APEX-Mini-IQ2_S.gguf \
  --mmproj mmproj-REAP-26B-F16.gguf \
  -c 32768 -ngl 99 --jinja

Then POST OpenAI-compatible chat completions with an image_url content block (base64 data URI, data:image/jpeg;base64,…). Confirmed working on JPEG and PNG; WebP is rejected by llama-server's mtmd loader at the time of writing — convert to JPEG/PNG first.

For grounding-heavy tasks, the upstream Qwen-VL guidance applies: add --image-min-tokens 1024 if accuracy on text reading or bounding-box-style queries drops.

Refusal and capability metrics (inherited from source bf16)

These numbers are from the Heretic optimization run on the bf16 weights, prior to APEX quantization. APEX-specific refusal/KL deltas (whether Q3-tier or IQ-tier middle layers change the refusal-suppression effectiveness) have not been measured. We do not expect significant change but it is on the validation TODO list.

Quantization recipe

• —Source: Qwen3.6-35B-A3B-REAM-192-heretic-bf16.gguf (51 GB / 16.01 BPW), the bf16 master from the standard GGUF repo.
• —Tooling: APEX configs + goblincore/llama-cpp-turboquant build of llama-quantize and llama-imatrix.
• —Imatrix calibration corpus: bartowski's `calibration_datav3.txt`. APEX's own pipeline references a calibration_v1.2.txt that isn't bundled in the repo; we used the de-facto community standard instead. Quality difference is reportedly small per APEX's own ablations.
• —Imatrix generation: 32 chunks × 2048 ctx (corpus length capped the chunk count), all 40 transformer blocks GPU-resident on H100.
• —Profile-to-config mapping: APEX --profile compact / i-compact / i-balanced / i-quality / mini (the qwen36_heretic_*.txt configs are byte-identical to qwen36_35b_*.txt — no heretic-specific tuning beyond the layer-gradient strategy that's universal across the Qwen3.5/3.6 family).

Quick start

Inference (any APEX tier)

./llama-cli \
  -m Qwen3.6-35B-A3B-REAM-192-heretic-APEX-ICompact-Q3_K_L.gguf \
  -p "Write a Python function that returns the nth Fibonacci number." \
  -n 1000 --temp 0.7 --n-gpu-layers 999

(Use -n 1000 minimum since Qwen3.6's default thinking mode emits long [Start thinking] ... [End thinking] blocks before the answer. Pass /no_think in the prompt to skip thinking for quick sanity checks.)

Server

./llama-server \
  -m Qwen3.6-35B-A3B-REAM-192-heretic-APEX-IBalanced-Q5_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 32768 \
  --n-gpu-layers 999

Preliminary local code-gen smoke test (ICompact only)

A small-scale single-run shakedown of the ICompact-Q3_K_L tier against a real coding workload, run via llama-server from TheTom-llama-cpp-turboquant with KV cache q8_0/turbo3, ngram-simple speculative decode, free-form thinking with --reasoning-budget 1500, and ctx 32768. Date: 2026-05-09.

This is not a benchmark. Sample sizes are tiny (2 standalone tasks + 3 agentic tasks), n=1 per task, single rater. It exists to confirm the recipe survives an agentic workload without obvious breakage, ahead of the larger evals on the validation TODO list.

• —2× standalone TypeScript code-gen (mulberry32 SeededRng + LRUCache<K,V>) | 33/33 vitest tests pass on the produced sources |
• —3× agentic tasks against a ~360-test repo<br>(vitest sanity / Box-Muller nextGaussian + stats test / multi-file readonly name field on post-processing Effect classes + tsc --noEmit) | 3/3 clean diffs; vitest pass; tsc --noEmit clean |
• —Pre-bench probes terminated cleanly (finish_reason=stop, no V-quant attention loop). On the nextGaussian task the agent produced library-grade Box-Muller — caching the second sample of each pair and patching fork() to copy the cache — without being prompted for either detail. The test (10 000 samples; assert |mean| < 0.1 and |stddev − 1| < 0.1) passed first try.

What this can support saying: the I-Compact recipe runs end-to-end through Read/Grep/Edit/Bash + self-verify loops and produces diffs that pass vitest and tsc on the cases tried.

What it cannot support saying: anything about quality vs other APEX tiers (not benched here), vs vanilla REAM Q4KM on agentic loops (no comparison run yet), or about academic-eval scores (still deferred).

Per-task diffs, raw stdout/stderr, and the produced source snapshots are uploaded under `bench/2026-05-09-icompact-smoke/` in this repo.

Lineage

Qwen/Qwen3.6-35B-A3B  (base, 35.11B params, 256 routed experts)
        │
        ▼  REAM (Cerebras / Samsung SAIL Montreal expert merging)
        │  256→192 routed experts
        │
keithnull/Qwen3.6-35B-A3B-REAM-192  (27.05B params, vision tower preserved)
        │
        ▼  Heretic (directional ablation + Optuna kernel optimization)
        │  Custom fork goblincore/heretic@qwen3_5-packed-experts-v2
        │
keithnull/Qwen3.6-35B-A3B-REAM-192-heretic  (bf16 safetensors)
        │
        ▼  llama.cpp convert_hf_to_gguf.py
        │
keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-GGUF  (bf16 master + standard K-quants)
        │
        ▼  APEX mixed-precision recipe + imatrix calibration
        │
keithnull/Qwen3.6-35B-A3B-REAM-192-heretic-APEX-GGUF  (this repo)

Credits

• —@mudler and the LocalAI team for APEX — the MoE-aware mixed-precision quantization recipe, configs, and pipeline that produced these GGUFs.
• —@bartowski1182 for the diverse calibration corpus used for imatrix generation.
• —@p-e-w for Heretic and the underlying abliteration framework.
• —@Sehyo for the per-trial fused-expert ablation approach in PR #207.
• —@grimjim for the Magnitude-Preserving Orthogonal Ablation formulation.
• —Inspired by @llmfan46's prior Qwen3.6 heretic work.

License

Apache 2.0, inherited from the source model.