kingjones777/Ornith-1.5-35B-A3B-Abliterated-ROCmFP4-GGUF

Ornith-1.5-35B-A3B-Abliterated — ROCmFP4 for AMD Strix Halo (gfx1151)

The model here is not our work. The abliteration is [huihui-ai](https://huggingface.co/huihui-ai)'s (Huihui-Ornith-1.5-35B-A3B-abliterated); the BF16 GGUF we quantised from is [PocketAiHub](https://huggingface.co/PocketAiHub)'s; the base model is [ornith-ai](https://huggingface.co/ornith-ai)'s. This repository adds only the ROCmFP4/ROCmFPX quantisation ladder, the dual-backend build, and the measurements. Go star their repos.

Six quantisations of the abliterated Ornith-1.5-35B-A3B MoE, built for AMD Ryzen AI Max+ 395 / Radeon 8060S. Every file runs on BOTH the HIP (ROCm) and Vulkan backends from a single binary — the backend is a runtime `-dev` flag, not a rebuild.

Lead result: on this model the backend choice is worth ~10× more than the quantisation choice. Vulkan is +8.6 to +14.4 % on decode; ROCm is +45 to +67 % on prefill. The three 4-bit tiers sit within 1.5 % of each other. Pick your backend for your workload; the tier is close to a free choice.

Which file should I use?

mmproj-Ornith-1.5-35B-A3B-Abliterated-F16.gguf (vision tower) is inherited from the source repo — pass it with --mmproj for image input.

Quick start — the backend is one flag

# HIP / ROCm — best prefill (long prompts, cold starts, RAG)
llama-server -m Ornith-1.5-35B-A3B-Abliterated-Q4_0_ROCMFP4_FAST.gguf \
  -dev ROCm0 -fa on -ngl 999 -fit off -np 1 \
  -ctk q8_0 -ctv q8_0 -c 32768 -b 8192 -ub 2048 -t 4 --poll 100 --jinja \
  --repeat-penalty 1.05 --repeat-last-n 256 \
  --host 0.0.0.0 --port 8080

# Vulkan — best decode (chat, agentic loops, warm prompt cache)
#   SAME FILE. Only -dev changes.
llama-server -m Ornith-1.5-35B-A3B-Abliterated-Q4_0_ROCMFP4_COHERENT.gguf \
  -dev Vulkan0 -fa on -ngl 999 -fit off -np 1 \
  -ctk q8_0 -ctv q8_0 -c 32768 -b 8192 -ub 2048 -t 4 --poll 100 --jinja \
  --repeat-penalty 1.05 --repeat-last-n 256 \
  --host 0.0.0.0 --port 8080

✅ Verified: this model produced 0 truncated generations in 48 on Vulkan and 0 in 48 on ROCm. The sibling 9B fails this test on Vulkan (8/48); this model does not. See Vulkan stability on gfx1151 below.

⛔ You need a ROCmFPX build with Vulkan enabled — stock llama.cpp will NOT load these files

ROCmFP4/ROCmFPX use ggml tensor types 100–119. Upstream llama.cpp's type table stops at 43, so a stock build rejects these files outright, on any backend:

gguf_init_from_reader: tensor '...' has invalid ggml type 101. should be in [0, 43)

Build one with both backends (this is the exact configuration used for every number on this card):

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
cmake -B build-hipvk -G "Unix Makefiles" \
  -DCMAKE_MAKE_PROGRAM=/usr/bin/gmake \
  -DCMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGGML_NATIVE=ON \
  -DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc \
  -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF
cmake --build build-hipvk -j16 --target llama-server llama-quantize
# verify BOTH backends are present before trusting any of this:
build-hipvk/bin/llama-server --list-devices   # must list ROCm0 AND Vulkan0

Build traps, all hit for real: no Ninja on many Strix boxes (use Unix Makefiles + gmake); glslc is not auto-detected (pass -DVulkan_GLSLC_EXECUTABLE); a stale CMakeCache poisons a generator switch (rm -rf the build dir first); -j32 OOM-killed a process on a 128 GB box — use -j16.

Measured performance — all six tiers, both backends

Ryzen AI Max+ 395 (Strix Halo), Radeon 8060S gfx1151, 128 GB unified, ROCm 7.2.4, Ubuntu 24.04. -c 32768, -ctk/-ctv q8_0, decode tok/s, 4 fixed prompts, temperature 0, top_k 1, 400 tokens, cache_prompt:false, warm-up discarded, median of 3, box idle (0 co-resident seats).

12 of 12 arms loaded. Zero load failures on either backend.

The three findings that matter

1. 1.Backend > tier, by an order of magnitude. Vulkan buys +8.6 % to +14.4 % decode. The three 4-bit tiers differ by ~1.5 %. If you are optimising, change your backend before you change your file.
2. 2.The crossover is a genuine trade, not a free win. ROCm delivers +45 % to +67 % prefill (426 vs 254 tok/s on FAST). Decode-bound work → Vulkan. Prompt-bound work → ROCm.
3. 3.Q6 buys nothing over Q8 on speed. Q6_ROCMFPX_AGENT (7.45 BPW) runs at 42.0 tok/s; Q8_ROCMFPX (8.27 BPW) at 42.3. Choose Q6 for the ~3 GiB footprint, not for throughput.

⚠️ These rankings are model-specific and we say so. On a different model we measured the tier driving a 22 % swing while the backend barely moved. Do not carry this table to another model — measure it there.

Vulkan stability on gfx1151 — ⚠️ verified, and this model is clean

This model was tested directly for silent truncation and passed on every tier, on both backends. 12 reps × 4 workloads per arm, n_predict = 400:

Every tier here — COHERENT, FAST, STRIX_LEAN, Q6_AGENT, Q8_AGENT, Q8 — returned 0 / 48 on Vulkan. The sibling, built from the same recipe on the same box with the same binary and driver, fails the same test. The failure sits at the (9B model × Vulkan backend) intersection: it is not the tier name, not the ROCmFP4 format, and not Vulkan on its own — this model runs Vulkan clean, across the whole ladder.

We still document it here because the two models share a recipe, a build, a driver and a box, and only one of them fails. If you are running a different model on gfx1151, that is the case you should check.

What the sibling does — and why a median-only benchmark misses it

Across the sibling's full six-tier ladder (12 generations per arm, 72 per backend):

It is tier-variable, not tier-exclusive — worst on COHERENT, but it also hit Q8_0_ROCMFPX, so a tier passing once does not make it immune.

It fails silently. The server returns 1 token — empty content — with no error, no crash, a clean exit code and /health still 200. A benchmark reporting only a median cannot see it: it surfaces as an absurd tg = 1000000 tok/s (a 1-token / 0 ms decode) or as a plausible "fast" outlier inside a range.

Other Vulkan failures on the same hardware

What to do about it

• —Validate generation length, not just throughput. Send N completions and count how many reach your n_predict. That check costs minutes and is the only thing that catches this class of bug.
• —If you use speculative decoding, use ROCm. We have not seen spec-decode work on Vulkan on gfx1151.
• —If a Vulkan run aborts with ErrorDeviceLost, that is the driver, not the file — the same GGUF serves on -dev ROCm0. One flag: no re-download, no rebuild.
• —Across every model, tier and backend we measured on this hardware, ROCm produced zero truncated generations (0 / 168). It is the conservative choice; on this model Vulkan is both clean and faster.

Provisional pattern, stated with its sample size. The large Vulkan decode win appeared on the MoE model (this one) and not on the two dense models measured on the same box (≈ −0.5 % and +0.8 %). That is one MoE against two dense — a hypothesis to test on your own model, not a rule to inherit.

Head protection — verified, not assumed

Left to their defaults several ftypes emit a 4-bit `output.weight`, which we have measured degrading a model from 4/5 to 1/5 on held-out checks. Every file here was quantised with an explicit --output-tensor-type and then read back and verified by exact tensor name:

⛔ ftype 105 (`STRIX`) is deliberately not published here — it leaves output.weight and 78 *_shexp tensors at 4 bit. COHERENT (102) is the coherent 4-bit tier.

Source integrity — the abliteration was verified before we built anything

A 35B MoE abliteration can silently ship a de-fused checkpoint whose expert tensors lost their expert dimension — it looks fine and scores badly, and it is easy to misread as "ablation does not survive MoE." We gate on this before quantising:

base_model      ornith-ai/Ornith-1.5-35B-A3B
base_revision   e4dfb35a93d4b6822a811a7676f3488514abe7e2
method          refusal-direction orthogonal weight projection
norm_preserve   true
expert_down     25 physical tensors / 6,400 logical paths   (25 × 256 experts)

Independently verified from the BF16 header:

40/40 blocks · expert_count=256 · expert_used_count=8
120 *_exps + 160 *_shexp tensors
0 expert tensors missing the expert dimension
blk.0.ffn_down_exps.weight = [512, 2048, 256]   ← expert dim intact

The projection was applied across all 256 experts, not to a collapsed checkpoint.

Reproduction block

binary    ROCmFPX @ charlie12345/ROCmFPX, build-hipvk (GGML_HIP=ON GGML_VULKAN=ON GGML_NATIVE=ON,
          gfx1151, Release, CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang, LLAMA_CURL=OFF)
source    PocketAiHub/Ornith-1.5-35B-A3B-Abliterated-GGUF :: Ornith-1.5-35B-A3B-Abliterated-BF16.gguf
          69,376,637,024 bytes, 733 tensors, arch qwen35moe
quantize  llama-quantize --output-tensor-type <q6_K|q8_0> <bf16> <out> <FTYPE> 12
host      Ryzen AI Max+ 395 · Radeon 8060S gfx1151 · 128 GB unified · ROCm 7.2.4 · Ubuntu 24.04 · kernel 7.0.0-30
state     box idle, 0 co-resident seats
method    4 fixed prompts, temp 0, top_k 1, 400 tok, cache_prompt:false, discard rep 1, median of 3

A number without its binary is a rumour. Every figure above was produced by the binary and argv named here.

Perplexity — measured

wikitext-2-raw, 100 chunks, identical for every tier so they are directly comparable to each other. -c 512, ROCm0, -ngl 999 -fa on.

⚠️ Read the error bars before ranking these. The three 4-bit tiers span 0.11 PPL against ±0.138 uncertainty — they are statistically indistinguishable at this sample size. The ordering shown is suggestive, not significant. What is supported: the 6/8-bit tiers sit measurably below the 4-bit group, and even there Q6 vs COHERENT (0.192 apart) is only borderline.

⚠️ This is a 100-chunk measurement, not full wikitext-2. It is valid for comparing these six files to each other. Do not compare these figures against a full-corpus PPL from another card.

⭐ Note Q6 (114) scoring below both Q8 tiers — with overlapping error bars, treat that as "the 6/8-bit tiers are equivalent", not as evidence Q6 is better than Q8.

Verification — every tier, measured

12/12 backend load arms passed. 0 failures.

Coherence — actual generated output, identical in structure across all six tiers (prompt: "Write a Python function to merge two sorted lists. Code only."):

def merge_sorted_lists(list1, list2):
    merged = []
    i = j = 0

    while i < len(list1) and j < len(list2):
        if list1[i] <= list2[j]:
            merged.append(list1[i])

Correct two-pointer merge with proper bounds. Read, not assumed — a fast gibberish generator is a failure we have shipped before.

Tool calling — 3 distinct calls per tier, each verified that the argument round-tripped correctly (city name present in the emitted tool_calls), not merely that some tool call was returned.

⚠️ Vision: the `mmproj` loads on all six tiers — image-grounded accuracy is NOT scored. "Loads" and "works" are different claims and we are only making the first one.

Not yet measured

• —Image-grounded vision accuracy (the mmproj loads; output not scored against ground truth).
• —Context beyond 32768.
• —MTP / speculative decoding — the source ships mtp_included: false.
• —Full-corpus perplexity (the figures above are a 100-chunk run).

License and attribution

• —Base model: `ornith-ai/Ornith-1.5-35B-A3B`, revision e4dfb35a93d4b6822a811a7676f3488514abe7e2.
• —Abliteration: performed by [huihui-ai](https://huggingface.co/huihui-ai) — Huihui-Ornith-1.5-35B-A3B-abliterated. Distributed as GGUF by `PocketAiHub`, whose abliteration-manifest.json and conversion-validation.json made the integrity gate above possible.
• —This repository contributes only the ROCmFP4/ROCmFPX quantisations, the dual-backend measurements, and the verification. All model credit belongs upstream.

Acknowledgements

huihui-ai for the abliteration. PocketAiHub for the BF16 GGUF and — unusually and valuably — for shipping manifests that make source integrity checkable rather than assumed. ornith-ai for Ornith-1.5-35B-A3B. The ROCmFPX project for the FP4/FPX quantisation types and the Strix Halo kernels.