aozihaoz/Cosmos3-Super-Text2Image

Cosmos3-Super-Text2Image

A Gradio demo for **nvidia/Cosmos3-Super-Text2Image** — NVIDIA's 64B-parameter omnimodal world model for Physical AI — running on a single Blackwell GPU via NVFP4 weight-only quantization (torchao).

Status: awaiting GPU

This Space is currently a static holding page. The full Gradio app lives in `app.py` and is written for ZeroGPU (import spaces, @spaces.GPU, module-scope .to("cuda")). Once GPU hardware is attached, flip the frontmatter to:

sdk: gradio
sdk_version: 6.22.0
app_file: app.py
python_version: "3.12"
startup_duration_timeout: 1h

Why quantization is required

BF16 does not fit on any single ZeroGPU slice, and the model has no Inference Provider, so the demo has to host it. NVFP4 weight-only quantization via torchao brings the transformer within budget.

NVIDIA officially tests this checkpoint only at BF16 — FP4, FP8 and FP16 are not officially supported. Expect some quality drift versus the full-precision recipe.

Two load-bearing patches

Both are applied in app.py before the model loads:

1. 1.ZeroGPU weight packing vs. tensor subclasses. ZeroGPU's empty_fake calls empty_like + set_ on each parameter to build the pinned-CPU mirror it streams from. That is meaningless for NVFP4Tensor, which wraps several inner storages. The patch recurses via transform_subclass so each inner tensor is packed individually.

1. 1.NVFP4 dispatch dtype mismatch. Cosmos3's time_proj emits fp32 sinusoidals; vanilla F.linear upcasts the weight, but the NVFP4 handlers require input.dtype == weight.orig_dtype. The patch wraps the matmul-family handlers to cast non-NVFP4 float inputs to the weight's orig_dtype.

Generation settings

NVIDIA's recommended ranges, exposed in Advanced settings:

• —num_inference_steps — 35–50
• —guidance_scale — 4.0–6.0
• —flow_shift — 3.0–10.0
• —Resolutions — 256p / 480p / 720p at 16:9, 4:3, 1:1, 3:4, 9:16 (plus 1024x1024, which is what NVIDIA's own diffusers snippet uses)

Prompting

For best quality NVIDIA recommends upsampling prompts into a structured JSON format (subjects, lighting, aesthetics, cinematography, quadrant scan) — see the prompt upsampling guide. The app detects JSON input and forwards it to the pipeline as-is; plain text also works.

Notes

• —The pipeline is loaded with enable_safety_checker=False. NVIDIA's reference snippet enables cosmos_guardrail, but the guardrail models add both startup time and VRAM on top of an already tight 96 GB budget. Re-enable it if you need it.
• —The app launches with mcp_server=True, so the Space also serves as an MCP server with generate exposed as a tool.

Credit: the NVFP4-on-ZeroGPU approach and both patches follow multimodalart/Cosmos3-Super-Text2Image.

License

The model is released under OpenMDW1.1.