sona-forge/sd15-ipadapter-fp16

Sona Forge — SD 1.5 IP-Adapter UNet (ONNX FP16)

A single fused ONNX FP16 graph combining the SD 1.5 UNet with IP-Adapter image-conditioning weights baked into the cross-attention layers. Used by the Sona Forge Android app for identity-preserving avatar generation. Pair with `sona-forge/clip-vit-h-14-image-fp16`.

Revision 1.1.0 (2026-05-01) adds 13 optional ControlNet residual inputs (12 down-block residuals + 1 mid-block residual) so the same UNet drives both Phase 6 (IP-Adapter only — pass zero-filled residuals or rely on the residual-aware export's pass-through-when-empty semantics) and Phase 7 (IP-Adapter + ControlNet Canny — pass the residuals from `sona-forge/sd15-controlnet-canny-fp16`).

ONNX shape

Down-block residual canonical shapes (SD 1.5): [batch, 320, 64, 64] ×3, [batch, 320, 32, 32], [batch, 640, 32, 32] ×2, [batch, 640, 16, 16], [batch, 1280, 16, 16] ×2, [batch, 1280, 8, 8] ×3.

How it was made

Pinned conversion environment:

Conversion sequence:

1. 1.Load runwayml/stable-diffusion-v1-5 UNet at FP16.
2. 2.Download h94/IP-Adapter's models/ip-adapter_sd15.bin checkpoint (image-projection MLP + cross-attn K/V).
3. 3.Apply weights via unet._load_ip_adapter_weights([state_dict]) (the diffusers 0.27.2 internal — public unet.load_ip_adapter() doesn't exist on UNet2DConditionModel in this version).
4. 4.Set attn_processor.scale = [1.0, ...] on each IPAdapterAttnProcessor / IPAdapterAttnProcessor2_0.
5. 5.Wrap the UNet so added_cond_kwargs={"image_embeds": [image_embeds]} is positional and down_block_additional_residuals / mid_block_additional_residual flow through to the UNet forward call. Then torch.onnx.export at opset 17 with FP16 dummy inputs at canonical SD 1.5 shapes.

Re-running the conversion from the same pinned environment produces byte-identical output (same sha256). Conversion artefacts include a spike report with full TracerWarning output, validation metrics, and round-trip checks.

Files

No external-data sidecar — graph + weights fit under the 2 GB protobuf single-file limit.

The 1.1.0 export is a strict superset of the 1.0.0 input signature: zero-filled residual inputs reproduce the 1.0.0 numerical output (verified during the Phase 7 spike).

Licence

The fused ONNX is a composite of two upstream artefacts:

• —SD 1.5 base UNet — CreativeML OpenRAIL-M (use-based restrictions; permits redistribution + modification).
• —IP-Adapter weights — Apache-2.0.

The composite is distributed under the most restrictive of these terms — CreativeML OpenRAIL-M.

Memory footprint

ORT CPU EP promotes FP16 to FP32 at session load (Phase 6 spike measured ~3.5 GB resident just for this UNet). On Android, NNAPI / XNNPack execute FP16 natively and the on-device working set is closer to the FP16 disk size + activation buffers. Sona Forge gates this pack to Tier B+ devices (≥ 7 GB total RAM).

Usage

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

# CFG batch=2; uncond at index 0, cond at index 1.
sample = np.random.randn(2, 4, 64, 64).astype(np.float16)
timestep = np.array([999.0, 999.0], dtype=np.float16)
encoder_hidden_states = np.random.randn(2, 77, 768).astype(np.float16)

# image_embeds: zeros for uncond branch, scaled CLIP embeds for cond branch.
clip_emb = np.random.randn(1, 1024).astype(np.float16)  # one reference image
ip_scale = 0.7
image_embeds = np.stack([
    np.zeros_like(clip_emb),
    clip_emb * ip_scale,
]).astype(np.float16)  # shape (2, 1, 1024)

noise_pred = session.run(None, {
    "sample": sample,
    "timestep": timestep,
    "encoder_hidden_states": encoder_hidden_states,
    "image_embeds": image_embeds,
})[0]

Provenance

• —Original SD 1.5 weights: `runwayml/stable-diffusion-v1-5`. (The official repo was removed from HF in early 2024; community mirrors persist.)
• —Original IP-Adapter checkpoint: `h94/IP-Adapter/models/ip-adapter_sd15.bin`.