baki0115/gaze-controlnet-qwen-image-edit

GazeCtrl — a gaze-conditioned ControlNet for Qwen-Image-Edit-2509

Turn a person in a photo to face a target gaze direction (dx, dy, dz) — head and body, over the full 360° including facing away — while keeping the same identity.

This is a ControlNet adapter, not a standalone model. It does nothing without the frozen Qwen-Image-Edit-2509 backbone and its VAE. The loading code is in the repository below; these weights are not diffusers-loadable as-is.

• —Code, training recipe, evaluation: https://github.com/VicsonPeng/GazeCtrl_training-based
• —Labels used to train it: https://huggingface.co/datasets/baki0115/GazeCtrl_dataset

How it works

        target gaze (dx,dy,dz)
                 │
                 ▼
   flat RGB field   R=(dx+1)/2  G=(dy+1)/2  B=(dz+1)/2      512×512
                 │
              VAE encode
                 │
noisy latent ─► InstantX ControlNet Union (5 blocks) ── 5 residuals ─┐
                                                                     ▼
source image ──────────────────────► Qwen-Image-Edit-2509 fp8 (frozen) ──► output
   (in-context reference token — this is what carries identity)

The gaze vector is the control signal: no skeleton, no pose-prediction stage, and no text prompt (txt_emb is a zero tensor, so every bit of control arrives through the ControlNet). Identity is carried by the source image entering the backbone as an in-context reference, not by the ControlNet.

Convention — dx > 0 looks image-right, dy > 0 looks up, dz > 0 looks toward the camera. yaw = atan2(dx, dz), so ±180° means facing away.

Files

3.54 GB each, bf16, 1768 M parameters.

Usage

import torch
from safetensors.torch import load_file
from qwen_models import (QwenBackbone, QwenControlNet, load_qwen_vae, vae_encode, vae_decode,
                         patchify, unpatchify, compute_rope_freqs_3d, compute_text_rope_freqs)

dev = "cuda"
vae = load_qwen_vae(VAE_PATH, device=dev)
bb  = QwenBackbone(QWEN_IMAGE_EDIT_2509_FP8_PATH, device=dev)   # frozen
cn  = QwenControlNet(); cn.load_state_dict(load_file("cn_step085000.safetensors"))
cn.to(dev, torch.bfloat16).eval()

The full denoising loop — flow matching, 16 steps, cn_scale=1.0 to match training — is demo/app.py in the code repository. cn_scale is the knob that trades obedience against fidelity; values away from 1.0 change the reported metrics.

Training

Evaluation

Held-out = 8 identities that were never trained on, cn_scale=1.0, 16 steps, seed 0.

Limitations — please read these

• —The generalisation gap is large: 19° on training identities against ~56° on held-out ones. The held-out split is small (8 identities) and carries only automatic labels while 48 % of training labels are human, so some of that gap is label noise — but not all of it.
• —Training longer stopped helping. 70k → 90k improved the training-set metric and did nothing for held-out performance.
• —Identity preservation is marginal. id_sim ≈ 0.28 sits on ArcFace's own same-person threshold, and no loss term currently targets identity. Expect drift, worst at large rotations.
• —`gaze_err` is a proxy. It scores head pose (6DRepNet) against the requested gaze vector. Head pose and gaze differ by construction — the two estimators that produced the labels disagree by a median 26.7° on the same frame — so a floor of that order is expected.
• —Back-facing outputs cannot be scored for identity — ArcFace needs a face, hence the separate face_det column.
• —Trained on 512×512 pad-to-square inputs, single person, roughly upright. Crowds, heavy occlusion and unusual framing are out of distribution.
• —Label semantics are not uniform across directions — see the dataset card; this is the clearest known defect in the training signal.

Licence

Apache-2.0 for these weights. The base model (Qwen/Qwen-Image-Edit-2509) and the InstantX ControlNet Union it was initialised from keep their own licences; this adapter is useless without the former, so its terms govern any deployment.