0xzknw/Gemma-4-E4B-it-Heretic-NX-GGUF

Gemma 4 E4B IT — Heretic NX RESEARCH Q8_0

A research release with an explicit capability tradeoff. The refusal and KL targets passed, but the capability non-inferiority gate below did not. The weights are released with that limitation disclosed, not as a capability-preserved PRIME release.

A static, direct-Q8 behavioral edit of `google/gemma-4-E4B-it`, with explicitly measured deviation from the original Q8 artifact. No runtime steering adapter is required. This release is text-only; it does not include the separate image/audio preprocessing components.

PRIME names the project's internal evaluation protocol, not an external certification. Reduced refusal does not imply greater factuality or safety.

Why this release required compromises

Gemma 4 E4B was a difficult model to edit in this project. This was not a straightforward reuse of a successful recipe, nor a no-sacrifice conversion. Getting the refusal-marker count down was possible; doing so while keeping mean KL below 0.05 required repeated changes to the edit and direct checks on the actual Q8 file.

The central difficulty in these experiments was the tradeoff between the behavioral effect and preservation of the original model's outputs. Protecting only the last prompt position left substantial drift on a few benign prompts. Earlier positions can also affect the final prediction through attention and cached keys/values. We therefore captured native Q8 projection inputs at every prompt position, rather than treating a final-position constraint as an end-to-end guarantee.

Protecting those trajectories completely then weakened the edit too much: one such screen reached 9 marker hits in just 16 prompts. Restoring matrix row norms was numerically successful but behaviorally unsuccessful: 7 hits in the first 8 prompts. Both were rejected without claiming full-set results. The useful compromise was partial trajectory protection plus a small strength adjustment, measured after Q8 requantization.

Selected complete development measurements show the progression:

These are selected checkpoints from a longer search, not an exhaustive benchmark or evidence of monotonic behavior. Q8 dequantization/editing/ requantization is nonlinear; small parameter changes must be tested, not assumed to interpolate smoothly. E4B is a dense model: these experiments do not establish that MoE routing, quantization alone, or any one architectural feature caused the difficulty.

What was sacrificed? The released model has a lower measured score on the paired capability slice: 78.34% versus 79.63% for the original, a 1.29 percentage-point decrease. HellaSwag shows the largest measured loss at 2.73 points. The mean KL is also only about 0.000424 below the requested cap. This is a documented compromise for reduced refusal, not a claim that the model became better overall or retained every capability unchanged.

Artifact and acceptance results

File: Gemma-4-E4B-it-Heretic-NX-RESEARCH-Q8_0.gguf (8,031,242,688 bytes, approximately 8.03 GB / 7.48 GiB).

SHA-256: db49bb67bc7b81f4636ee176d6bb1c72377face53e9f2662239a5d1087c5b673.

The exact GGUF passes the requested caps of at most 6 marker hits on 104 rows and mean first-token KL at most 0.05. This is not a claim that KL is below 0.05 for every prompt, later token, runtime or dataset. The margin below the mean cap is small; use the pinned evaluation protocol when reproducing it.

Refusal evaluation uses the pinned mlabonne/harmful_behaviors test split, greedy decoding, 96 generated tokens, four llama.cpp slots, pre-tokenized native completion requests, and the official template with thinking disabled. The original Q8 scored 104/104 marker hits. The edited model's hit rows are 38, 60, 68, 97 and 98 (one-based).

KL uses 104 pinned mlabonne/harmless_alpaca test prompts and raw logits over the full 262,144-token vocabulary: KL(P_original || P_edited) at the first generated token, with state cleared between prompts. See evaluations/ for dataset revisions, runtime attestations, per-row measurements and hashes.

Both 104-row suites participated in tuning and selection. They are development sets, not untouched holdouts. The natural-order rerun checks reproducibility; it is not a new independent benchmark. Lexical markers are only an operational refusal proxy: warnings can trigger them, and refusals can evade them.

Paired capability evaluation

The same deterministic 854-question slice was evaluated on both artifacts using greedy first-token selection restricted to A/B/C/D, identical prompts, tokenizer and runtime. This slice was not used to fit the edit.

There were 653 shared successes, 158 shared failures, 27 original-only successes and 16 edited-only successes. The conservative paired binary 95% interval for edited minus original is [-3.79, +1.24] percentage points. It uses simultaneous Clopper-Pearson bounds on the two discordance probabilities, not the older percentile-bootstrap implementation.

The predeclared three-point non-inferiority gate did not pass, overall or under the simultaneous task-level checks. This does not prove that the true loss exceeds three points: the evidence does not exclude it. The measured score is lower, and no capability improvement is demonstrated. The project owner explicitly approved publication as a research artifact with this tradeoff disclosed. The failed gate remains recorded as failed in the machine-readable release report; no threshold or confidence method was relaxed to label it a pass.

Method

Seven dense output projections receive rank-one weight edits:

• —Attention outputs: layers 23, 25 and 35 (zero-based).
• —FFN down projections: layers 6, 7, 12 and 33.

The parent recipe uses benign-penalized detector distillation (lambda=100) and nonuniform site strengths. Two right-factor variants share the same left axis: one protects final-position native Q8 inputs from the 104 benign development prompts; the other protects all input positions from the two largest benign development outliers (one-based rows 34 and 16).

The final right factor is 1.04 * (0.84 * last_position + 0.16 * full_position). Both source variants already contain the parent strength multiplier 1.1. The selected intervention remains rank one at each site. This partial protection retained more behavioral effect than full trajectory projection. Row-norm restoration was tested and rejected; it is not used in this file.

The edit dequantizes selected Q8 matrices, applies their low-rank deltas and requantizes to Q8_0 with original-block preservation. It is lossy, not a lossless optimization. Untouched GGUF regions are verified byte-for-byte.

The final factor artifact and SHA-bound ablation plan are included under evaluations/. They can be applied with Heretic NX's apply_quantized_gguf_ablation API to the exact source Q8 identified in the release report. Experiment scripts are in `0xZKnw/heretic-nx`.

Base model revision: ee0ef6023621cff504d758262d4e04895a5af4a2. Base Q8 SHA-256: 34be82b17b4942d389b9b527170c4b058027abdd32531fda063d3d97dd8ce80a.

Loading

Load the GGUF normally in a Gemma 4-compatible LM Studio / llama.cpp runtime. No separate adapter or patched inference graph is needed. Only this Q8_0 is evaluated; no results are transferred to other quantizations.

hf download 0xzknw/Gemma-4-E4B-it-Heretic-NX-GGUF \
  Gemma-4-E4B-it-Heretic-NX-RESEARCH-Q8_0.gguf --local-dir .

For llama.cpp, use a recent Gemma 4-compatible build and the model's chat template. The measured protocol uses thinking disabled; the reported numbers must not be transferred to thinking-enabled or differently prompted runs. The file size is not a total RAM requirement: the runtime, context/KV cache and other applications require additional memory.

Included evidence and edit reproduction

The repository includes the evaluated GGUF, sanitized refusal results, per-prompt first-token KL, paired capability comparison, merge report, release decision, and the final low-rank factors and ablation plan. Raw potentially harmful response text and temporary research checkpoints are not included. Response hashes are retained for auditing the local full results.

To reproduce the static edit, obtain the exact source Q8 hash stated above and use the matching Heretic NX implementation and codec/runtime build:

from heretic_nx.edits import apply_quantized_gguf_ablation

apply_quantized_gguf_ablation(
    "gemma-4-E4B-it-Q8_0.gguf",
    "reproduced-e4b.gguf",
    "evaluations/ablation-plan.json",
    "evaluations/edit-factors.safetensors",
)

The plan validates the source and factor hashes. Compare the resulting GGUF SHA-256 with the released file before reusing any of these measurements. The factors are an edit recipe, not a standalone model or a runtime LoRA.

Limitations and responsibility

This edit weakens refusal behavior and can increase compliance with unsafe, illegal or harmful requests. It supplies no factuality guarantees, application sandbox, access controls or content safeguards. Neither the first-token KL nor the narrow multiple-choice slice establishes universal capability preservation or sequence-level equivalence. Use appropriate application-level protections.

The base model card declares the Gemma 4 Apache 2.0 license.