kkxao/Qwen-Image-2.1-Text-Encoder-Heretic

[!IMPORTANT] This repo is the HF `transformers` checkpoint (bf16, 4 shards) — it does not load in ComfyUI. ComfyUI text encoders use a different key layout (the model.language_model. prefix is dropped when repacking), so pointing ComfyUI at these shards will not work. For ComfyUI use one of these instead — all keep the vision tower, which 2.1 needs for editing: | Repo | File | Loader | |---|---|---| | `…-NVFP4` | qwen3vl_8b_nvfp4_heretic.safetensors | CLIPLoader, type qwen_image | | `…-W4A8` | qwen3vl_8b_w4a8_heretic.safetensors | CLIPLoader, type qwen_image | | `…-GGUF` | qwen3vl_8b_heretic-Q4_K_M.gguf + mmproj-…-f16.gguf | CLIPLoaderGGUF (ComfyUI-GGUF node) | You also need a ComfyUI new enough to know QwenImage21: 0.34.2 does not, 0.36.0 does. If TextEncodeQwenImage21 is missing from your node list, that is why. Use this repo for transformers / diffusers / vLLM, or as the base for your own quantization.

Qwen-Image-2.1 Text Encoder — Heretic (Abliterated)

Not affiliated with, or endorsed by, Alibaba / Qwen. Community derivative (refusal-ablated) of `Qwen/Qwen3-VL-8B-Instruct` — the model Qwen-Image-2.1 uses, unmodified, as its text encoder. Qwen releases that model under Apache-2.0, so this derivative is redistributed under Apache-2.0 (see LICENSE and NOTICE).

The text encoder of `Qwen/Qwen-Image-2.1` (a Qwen3-VL-8B-Instruct) with refusal behaviour removed via Heretic directional ablation.

Drop-in replacement for the stock text encoder. Weights are bf16, same shapes, same parameter count — nothing else was changed.

Results

Measured by Heretic on mlabonne/harmful_behaviors (test split) for refusals and mlabonne/harmless_alpaca for KL divergence — i.e. lower refusals and lower distribution shift on benign inputs.

Independent verification

Refusal rate and general capability were re-checked with a separate script (different code, different refusal keyword set) rather than trusting the optimiser's own numbers:

• —Refusals: 0/20 on held-out harmful prompts
• —Benign questions: 4/4 correct and coherent (e.g. "What is the capital of France?" → "The capital of France is Paris…")

★ Search budget matters — measured, not assumed

Heretic's documented defaults are n_trials = 200, n_startup_trials = 60. A first run with 100/20 was done for comparison:

Doubling the search budget nearly halved the refusal rate and cut KL divergence by a third. 100 trials is not enough for this model. If you are abliterating something similar, use the documented defaults.

Pareto front

The optimiser returns a Pareto front; this release uses the knee point, not the extreme:

Reproduction

uvx --from "git+https://github.com/p-e-w/heretic@3521f8648a0dccf6e12a92666862632235fac7e6" heretic \
  --model <path to Qwen-Image-2.1/text_encoder + processor, flattened> \
  --dtypes bfloat16 --device-map auto \
  --max-memory '{"0":"14GiB","1":"14GiB"}' \
  --offload-outputs-to-cpu --max-batch-size 32 \
  --n-trials 200 --n-startup-trials 60 \
  --study-checkpoint-dir <ckpt> \
  --trial-index 1 --model-action save \
  --save-directory <out> --export-strategy MERGE

Hardware: 2× RTX 5070 Ti (16 GB each), ~48 min for 200 trials (~14.5 s/trial).

Gotchas worth knowing

• —Pin the commit. git+…/heretic without a revision is a moving target; the commit above reports v2.0.0.dev0. The PyPI release heretic-llm==1.4.0 is older and rejects --trial-index / --model-action / --save-directory.
• —`--trial-index` is the index into the sorted Pareto front, not the Optuna trial id. Passing a trial id silently falls back to the interactive menu.
• —`--checkpoint-action continue` replaces the entire settings object with the one stored in the checkpoint (main.py:404-407), discarding your CLI flags. To export a different trial afterwards you must patch the settings stored in the study journal, not the command line.
• —Finishing a run opens an interactive TUI; with stdin=/dev/null it raises EOFError. Pass a valid --trial-index to avoid it.

Usage

Standard transformers:

from transformers.models.qwen3_vl import Qwen3VLForConditionalGeneration
model = Qwen3VLForConditionalGeneration.from_pretrained(
    "pottokao/Qwen-Image-2.1-Text-Encoder-Heretic", dtype="bfloat16", device_map="auto")

For ComfyUI, note that Comfy-Org's repack strips the model.language_model. prefix (model.language_model.layers.N.… → model.layers.N.…). Weights quantized straight from this HF layout will not load in ComfyUI until the keys are remapped.

Notes

• —Only the text encoder is modified. The DiT and VAE of Qwen-Image-2.1 are untouched.
• —Ablation targets o_proj and down_proj (Heretic's defaults for this model).
• —Quantizing this model behaves the same as quantizing the original: NVFP4 round-trip error measured 9.52 % on ablated layers vs 9.51 % on untouched layers vs 9.44 % on the stock encoder — ablation does not make the weights harder to quantize, so the same recipe applies.