kkxao/Qwen-Image-2.1-PE-T2I-Heretic

Qwen-Image-2.1-PE-T2I — Heretic (Abliterated)

Not affiliated with, or endorsed by, Alibaba / Qwen. Community derivative of `Qwen/Qwen-Image-2.1-PE-T2I`, redistributed under the Qwen Research License (copy included as LICENSE, per §3.a). Non-commercial use only (§1.i / §2.a); commercial use needs a separate licence from Qwen (model-business@notice.qwencloud.com, §2.b).

The prompt rewriter for Qwen-Image-2.1 — a fine-tuned Qwen3.5-VL 9B that turns a brief request in any language into a detailed English prompt plus a recommended aspect ratio — with refusal behaviour removed via Heretic directional ablation.

bf16, same shapes and parameter count as the source.

`system_prompt.txt` is included and the model is useless without it. It is a 10 KB document that defines the entire output contract — structure, register, length, and the JSON shape. Heretic's export does not carry it; it is copied in here deliberately.

[!IMPORTANT] This repo holds the original weights — bf16, HF `transformers` format, 4 shards. Use it with transformers / diffusers, or as the base for your own quantization. It is not a drop-in file for ComfyUI or vLLM. Use one of the quantized builds: | Format | Runtime | Status | |---|---|---| | [GGUF](https://huggingface.co/pottokao/Qwen-Image-2.1-PE-T2I-Heretic-GGUF) (Q4KM, 5.49 GB) | llama.cpp, and ComfyUI via an LLM/GGUF node | available | | [NVFP4](https://huggingface.co/pottokao/Qwen-Image-2.1-PE-T2I-Heretic-NVFP4) (mixed precision, 7.76 GB) | vLLM | available | Note on ComfyUI: this model is a prompt rewriter — an LLM, not a text encoder. It does not load into CLIPLoader. In ComfyUI it is consumed through an LLM node reading the GGUF, to rewrite your prompt before it reaches the DiT. If you are looking for the Qwen-Image-2.1 text encoder for ComfyUI, that is a different model — `…-Text-Encoder-Heretic` and its NVFP4 / W4A8 / GGUF builds.

Results

Measured on mlabonne/harmful_behaviors (refusals) and mlabonne/harmless_alpaca (KL).

Pick your own operating point

This is one point on a Pareto front, not "the" answer. Across 500 trials the front came out monotonic — every refusal you remove costs KL divergence, which is the damage to everything the model was already good at:

For a prompt rewriter, KL is not an abstraction — it is exactly the length discipline, the structured JSON compliance and the spatial-description ability you are paying for. If you mainly want the model to stop refusing image briefs rather than chat instructions, a higher-refusal / lower-KL point may serve you better. All of these are reproducible from the same study (see below).

A statistics caveat, stated plainly

Refusals are measured on n = 100. At p ≈ 0.07 the binomial standard deviation is σ ≈ 2.6, so 7/100 and 14/100 are about 2.7σ apart — real but not dramatic, and 7 vs 4 would be noise. Do not read small differences in the refusal column as meaningful. KL is a continuous measure and far more reliable for comparing points.

The rewriting still works

Ablation is only useful if the model still does its job — and for this model the job is structured: it must emit one valid JSON line with a rewritten prompt and an aspect ratio. That is the first thing a bad ablation breaks.

It was also checked in a real pipeline rather than in isolation: dropped in as the brief writer for a 50-photographer style system, it obeyed a 10-field JSON schema it was never trained on, and wrote 1200–1500 character briefs where a general 26B model wrote ~650. The length discipline is in the weights, not in the prompt — feeding the same system_prompt.txt to another model reproduced the register and the JSON format but only about half the length.

One honest finding from that test: the same baked-in habits that give it length discipline also make it less steerable. Asked to write in the manner of specific photographers, it overrode style instructions it was given — a pale high-key palette came back warm, a deadpan brief came back sensual. That is a property of the original fine-tune, not of the ablation, but it is worth knowing before you wire it into a pipeline that expects to direct it.

⚠️ A warning about measuring this model

Before running Heretic I wrote five probes of my own — image-prompt requests with mildly sensitive subject matter — and all five were rewritten without complaint. I concluded the model was barely gated and that ablation was not worth doing.

Heretic's baseline on the standard evaluation set then measured 98/100 refusals.

The two measurements are not in conflict: this model is relaxed about edgy image subjects and near-total in refusing direct chat-style harmful instructions. But a small hand-written probe set asked a different question than I thought it was asking, and pointed to exactly the wrong conclusion.

If a standard evaluation set exists for what you are measuring, use it. A handful of probes you invented will feel informative and may be inverted.

Reproduction

uvx --from "git+https://github.com/p-e-w/heretic@3521f8648a0dccf6e12a92666862632235fac7e6" heretic \
  --model <path to Qwen-Image-2.1-PE-T2I> \
  --dtypes bfloat16 --device-map auto \
  --max-memory '{"0":"14GiB","1":"14GiB"}' \
  --offload-outputs-to-cpu --max-batch-size 16 \
  --n-trials 500 --n-startup-trials 60 \
  --study-checkpoint-dir <ckpt> \
  --trial-index 0 --model-action save \
  --save-directory <out> --export-strategy MERGE

Hardware: 2× RTX 5070 Ti (16 GB each). ~56 min per 100 trials.

On search budget. 200 trials looked converged; 300 added nothing at the useful end; 500 broke through to 7/100 at trial 385, and a follow-up run that widened the ablation-strength bound (max_weight, whose optimum had been clipped at the ceiling) reached 3/100 @ KL 0.036 at trial 270 — dominating the 7/100 point on both axes. Total search across three range strategies: 1,600 trials (800 default + 300 widened-layers + 500 widened-strength). This repo now ships that trial-270 build after a 60-trial plateau. A stagnation window is not convergence — with a stochastic sampler on a noisy objective, plateaus and breakthroughs alternate. If you repeat this, budget more trials than the point where it looks finished.

Pin the commit. git+…/heretic without a revision is a moving target; the commit above reports v2.0.0.dev0. The PyPI release heretic-llm==1.4.0 is older and rejects --trial-index / --model-action / --save-directory.

--trial-index is the index into the sorted Pareto front, not the Optuna trial id. Index 0 is the lowest-refusal point.

Resuming. --checkpoint-action continue replaces the whole settings object with the one stored in the checkpoint (main.py:404-407), so command-line --n-trials / --trial-index are silently ignored on a resume. Patch the checkpoint's stored settings instead.

Usage

import json, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

mid = "pottokao/Qwen-Image-2.1-PE-T2I-Heretic"
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16, device_map="auto").eval()

system_prompt = open("system_prompt.txt").read().strip()   # shipped in this repo
text = tok.apply_chat_template(
    [{"role": "system", "content": system_prompt},
     {"role": "user", "content": "夕陽下的燈塔"}],
    tokenize=False, add_generation_prompt=True, enable_thinking=True)

out = model.generate(**tok(text, return_tensors="pt").to(model.device),
                     max_new_tokens=2048, do_sample=True, temperature=1.0, top_p=0.95)
gen = tok.decode(out[0], skip_special_tokens=True)
result = json.loads(gen.partition("</think>")[2].strip())
# {"rewritten_prompt": "...", "wh_ratio": "1:1"}

The rest of the pipeline

Together these cover the whole front end of Qwen-Image-2.1: the rewriter that reads your request, and the text encoder that conditions the image.