CoolFace
Modelpublic

gguf-org/pig-clip

sourceHugging Facemitupdated 1d agoView on Hugging Face
4likes4.3kdownloads
Model Card

🐷pig architecture gguf clip

  • —natively trained multipurpose clip for gk engine
  • —low vram alternative/substitute of t5xxl, umt5xxl, etc.
  • —80-90% smaller, more efficient and cost effective (hopefully)

rationale

since LLM in diffusion model basically serves as text encoder, which means its primary goal/function is delivering a prompt to the generator precisely, no talk, no deep reasoning, no long response, make it as simple as possible

limitation/cutoff

we stop the training when similarity reaches .95 or over 20k cycles and similarity no less than .75

adapterstockpig-clipdelta
t50.78720.7913+0.0040
t5 (20k)0.79130.8080+0.0167
umt50.75330.7634+0.0100
qwen30.96490.9671+0.0022
qwen3vl0.92960.9378+0.0082
qwen3vl 8b0.93850.9431+0.0046
qwen3vl (layerwise)0.94750.9504+0.0029
llada0.95020.9651+0.0149

how it works

simply replace --t5xxl t5xxl.gguf with --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf

examples

test it with pixart

bash
ggk diffuser engine -- --diffusion-model pixart-nvfp4.gguf --vae pig_pixart_vae_fp16-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --diffusion-fa -v -o out.png

test it with sd-lite

bash
ggk diffuser engine -- --diffusion-model model.gguf --vae vae.gguf --clip_l clip_l.gguf --clip_g clip_g.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_t5_adapter-f16.gguf -p "close-up portrait of dog" --steps 8 --cfg-scale 1 --sampling-method euler --clip-on-cpu --diffusion-fa -v -o out.png

umt5 adapter

test it with wan2.1 or wan2.2

bash
ggk diffuser engine -- -M vid_gen --diffusion-model wan2.1_t2v_1.3b-q4_0.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_umt5_adapter-f16.gguf -p "a pig moving quickly in a beautiful winter scenery nature trees sunset tracking camera" --cfg-scale 6.0 --sampling-method euler -v -n "blurry ugly bad" -W 480 -H 480 --diffusion-fa --offload-to-cpu --video-frames 14 -o out.avi

qwen3vl-layerwise adapter

test it with krea2

bash
ggk diffuser engine -- --diffusion-model krea2_turbo-q2_k.gguf --vae pig_wan_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_layerwise_adapter-f16.gguf -p "a lovely pig holding a sign says Hi" --cfg-scale 1.00 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3-4b adapter

test it with x-image

bash
ggk diffuser engine -- --diffusion-model  x_image-nvfp4.gguf --vae pig_flux_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3_4b_adapter-f16.gguf -p "cute anime style girl with pinky messy long hair blue eyes wearing a maid outfit with a long black gold leaf pattern dress and a white apron, it is a postcard held by a hand in front of a beautiful realistic city at sunset and there is cursive writing that says PIG" --cfg-scale 1.0 --steps 8 --offload-to-cpu --diffusion-fa -v -o out.png

qwen3vl-4b adapter

test it with mageflow

bash
ggk diffuser engine -- --diffusion-model mageflow-edit-turbo-nvfp4.gguf --vae pig_mageflow_vae_fp32-f16.gguf --llm pig_clip-nvfp4.gguf --llm-adapter pig_qwen3vl_4b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-4b-it-f16.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

qwen3vl-8b adapter

test it with qwen-image2.1

bash
ggk diffuser engine -- --diffusion-model qwen-image-2.1-nvfp4.gguf --vae pig_qwen_image_2.1_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_8b_adapter-f16.gguf -p "an anime pig holding a sign says GGUF" -W 512 -H 512 --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

bash
ggk diffuser engine -- --diffusion-model qwen-image-2.1-nvfp4.gguf --vae pig_qwen_image_2.1_vae_fp32-f16.gguf --llm pig_clip-f16.gguf --llm-adapter pig_qwen3vl_8b_adapter-f16.gguf --llm_vision mmproj-qwen3vl-8b-it-q8_0.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 8 --sampling-method euler --diffusion-fa -v -o out.png

llada adapter

test it with llada

bash
ggk diffuser engine -- --diffusion-model LLaDA-image-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-q8_0.gguf --llm-adapter pig_llada_adapter-q8_0.gguf -p "a pig in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

edit image with vision clip (optional)

bash
ggk diffuser engine -- --diffusion-model LLaDA-image-turbo-nvfp4.gguf --vae pig_flux2_vae_fp32-f16.gguf --llm pig_clip-q4_k_m.gguf --llm-adapter pig_llada_adapter-f16.gguf --llm_vision pig_llada_sigvq-q4_0.gguf --ref-image sheep.png -p "a sheep in sunglasses" --cfg-scale 1.00 --steps 4 --sampling-method euler --diffusion-fa -v -o out.png

screenshot

Reference

pig engine - the new gguf compute kernels (gk)

gguf trainer - how to train your own adapter