catplusplus/Qwen21_Text_Encoder_Heretic

Qwen3-VL-8B Heretic Text & Vision Encoder (Prompt Adherence & Geometric Alignment Edition) 🌺✨

This repository provides an optimized, abliterated checkpoint of the Qwen3-VL-8B text and vision encoder from [Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1), processed with Norm-Preserving Biprojected Abliteration.

The primary purpose of this model is maximum instruction following and prompt adherence: it banishes geometric representation deflection ("internal blush" / hesitation vectors) that otherwise causes safety-tuned VLMs to corrupt diffusion conditioning on dynamic poses, human figures, athletic wear, and complex scenes.

🔬 The Core Problem: Why VLM Safety Alignment Degrades Diffusion Conditioning

In text-generation tasks, safety alignment mechanisms steer models to emit refusal text (e.g. "I cannot fulfill this request..."). However, modern multimodal diffusion architectures like Qwen-Image-2.1 do not generate text tokens:

$$\text{DiT Conditioning} \longleftarrow \mathbf{h}_L = \text{TextEncoder}(\text{tokens})[-1]$$

The diffusion transformer taps the raw pre-RMSNorm residual hidden states $\mathbf{h}_L$ directly from the text encoder to drive cross-attention.

The "Internal Blush" / Hesitation Deflection Phenomenon

When prompts describe human subjects, dynamic physical actions, athletic attire (e.g. swimwear, volleyball, gymnastics), or expressive emotions, safety-tuning vectors inside the language model activate even on completely benign, non-refusal prompts.

Because the model cannot output a refusal string, these alignment vectors manifest as a geometric rotation of the latent representation:

$$\mathbf{h}{\text{sensitive}} = \mathbf{h}{\text{clean}} + \mathbf{v}_{\text{refusal}}$$

This hidden deflection rotates the conditioning signal by over 60% relative norm away from the prompt's intended semantic visual trajectory!

Visual Consequences in Image Generation

When cross-attention layers in the DiT receive a representation deflected into the refusal/modesty subspace, the model displays hesitation artifacts:

1. 1.Modesty Hallucinations & Clothing Confusion: Spontaneous addition of mismatched cloth, awkward white ruffles, or extra fabric covering swimwear or sportswear.
2. 2.Anatomical Occlusion: The model avoids rendering human limbs or athletic poses, awkwardly hiding arms behind character backs or contorting torsos.
3. 3.Subject & Prop Merging: Equipment or background elements get fused into characters (e.g. sports balls bizarrely merged onto heads as hair ornaments).
4. 4.Action Damping: Dynamic verbs ("jumping to spike the ball") are subdued into passive, static standing postures.

📊 Quantitative Measurement of Representation Deflection

Using contrastive prompt pairs across benign and sensitive subjects, we measured the layer-by-layer cosine similarity and relative deflection norm across all 37 positions (input embeddings + 36 decoder layers) of Qwen3-VL-8B:

Between Layer 20 and Layer 36, representation deflection accelerates rapidly, culminating in a 60.3% vector distortion. By surgically neutralizing this direction, the text encoder reflects the exact intended prompt semantics.

🛠️ Methodology: Norm-Preserving Biprojected Abliteration

To eliminate hesitation deflection without degrading general language comprehension, we applied Norm-Preserving Biprojected Abliteration (create_heretic_text_encoder.py):

1. 1.Refusal Subspace Extraction: Difference-of-means vectors were extracted across contrastive prompt sets: $$\mathbf{r}l = \boldsymbol{\mu}{\text{sensitive}}^{(l)} - \boldsymbol{\mu}_{\text{benign}}^{(l)}$$
2. 2.Benign Subspace Orthogonalization: The general semantic direction was stripped from the refusal vector: $$\mathbf{v}l = \mathbf{r}l - \text{proj}{\mathbf{u}{\text{benign}}}(\mathbf{r}_l)$$
3. 3.Norm-Preserving Rank-1 Projection: Across 54 linear projection matrices (self_attn.o_proj and mlp.down_proj in layers 9–35, centered at layer 26 with Gaussian falloff $\lambda \in [0.10, 1.00]$): $$W{\text{norm}} = \text{normalize}(W, p=2, \text{dim}=1)$$ $$W' = \text{normalize}\Big(W{\text{norm}} - \lambda \mathbf{v}l (\mathbf{v}l^T W{\text{norm}})\Big) \cdot \|W\|{\text{row}}$$

Because exact row norms ($\|W\|_{\text{row}}$) are strictly preserved, the network's overall activation scales and general reasoning capabilities remain completely intact.

🤝 Pairing with Quantized DiT (nunchaku-qwen-image-2.1)

This text encoder is specifically engineered to be paired with [`nunchaku-qwen-image-2.1`](https://huggingface.co/models/nunchaku-qwen-image-2.1) for consumer GPU setups:

• —Resident DiT + Streamed Text Encoder:
• —DiT (best_quality_fp4.safetensors): 4.08 GB resident VRAM.
• —VAE (AutoencoderKLQwenImage21): 0.64 GB resident VRAM.
• —Qwen3-VL-8B ViT Vision Encoder: 1.07 GB resident VRAM.
• —Qwen3-VL-8B Language Model: Streamed layer-by-layer through a static 368 MB GPU buffer over PCIe at ~28.7 GB/s via stream_encoder.py.
• —Total VRAM Footprint: ~6.17 GB active VRAM, leaving ~9.5 GB free headroom on a single 16 GB GPU (such as RTX 5060 Ti or RTX 4080)!
• —Inference Speed: Multimodal prompt encoding completes in 1.06s (saving 16s vs CPU), and 25-step image generation runs in ~20s.

🚀 Quickstart Usage

1. Installation

pip install diffusers transformers accelerate torch sentencepiece

2. Loading with Diffusers

import torch
from diffusers import QwenImage21Pipeline
from transformers import Qwen3VLForConditionalGeneration, Qwen3VLProcessor

# 1. Load Heretic text encoder and processor
text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(
    "models/Qwen21_Text_Encoder_Heretic",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
)
processor = Qwen3VLProcessor.from_pretrained("models/Qwen21_Text_Encoder_Heretic")

# 2. Assemble into pipeline
pipe = QwenImage21Pipeline.from_pretrained(
    "Qwen/Qwen-Image-2.1",
    text_encoder=text_encoder,
    processor=processor,
    torch_dtype=torch.bfloat16,
)
pipe.enable_sequential_cpu_offload(gpu_id=0)

# 3. Generate with precise prompt adherence
image = pipe(
    prompt="Two cute anime girls in colorful bikinis playing beach volleyball on a sunny tropical beach, dynamic action pose, jumping to spike the ball, sharp focus",
    height=1024,
    width=1024,
    num_inference_steps=25,
    true_cfg_scale=1.0,
).images[0]

image.save("beach_volleyball.png")

3. High-Throughput Server Usage

Run the bundled ImageEditServer with NVFP4 DiT and Heretic text encoder:

# Start server on port 4500 (uses Heretic text encoder by default)
./extras/imagegen_qwen21_nvfp4.sh 4500

📦 Packaged Sources (extras/)

• —create_heretic_text_encoder.py: Complete script used to measure refusal vectors and perform norm-preserving biprojected abliteration.
• —stream_encoder.py: Zero-quality-loss layerwise weight streaming engine for Qwen3-VL-8B.
• —test_heretic_beach_volleyball.py: Empirical verification script comparing stock vs Heretic encoders.
• —QwenImage21NVFP4Backend.py: Diffusers + Nunchaku backend supporting custom text encoder overrides.
• —ImageEditServer.py & imagegen_qwen21_nvfp4.sh: Resident image generation server.

📜 Citation & Credits

• —Qwen-Image-2.1 & Qwen3-VL: Qwen Team, Alibaba Cloud.
• —Abliteration Principles: Arditi et al. (Refusal in Language Models Is Mediated by a Single Direction).
• —Heretic LLM: Heretic project (Directional Abliteration Toolkit).
• —Abliteration & Diffusion Conditioning Optimization: Oleg K. / Nikola Seeker Project.