ubr-physical-ai/Cosmos3-Edge-NF4-bnb

Cosmos3-Edge — NF4 (bitsandbytes)

nvidia/Cosmos3-Edge quantised to 4-bit NormalFloat (NF4) with double quantisation, language model only. 2.3 GB on disk, 2.6 GB peak in use, against 5.1 GB at bf16 — and no measurable accuracy cost on the benchmark it was validated on.

This is a research artefact from a quantisation comparison, not a deployment build. Read §"What this is not" before reaching for it.

Recipe

BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    llm_int8_skip_modules=["model.visual", "model.projector"],   # see the trap below
)

169 `Linear4bit` modules — 28 decoder layers x 6 projections plus lm_head — with the vision tower and the multimodal projector held at FP16. That matches, layer for layer, what the AWQ INT4 export of the same model quantises, so the two are comparable as quantisers rather than as different layer selections.

On `lm_head`: quantising it or not makes no measurable difference. Adding "lm_head" to the skip list gives 168 modules instead of 169 and leaves the output head at FP16. Both recipes were run on the same 24-item grounded benchmark on a Jetson Orin Nano, same prompt and scorer: identical F1 (0.40), identical target count (3/12), identical schema validity (0.125). The only difference was peak resident memory, 5.86 GiB with the head at FP16 against 6.17 GiB with it quantised — i.e. skipping it costs ~0.3 GiB and buys nothing. Measurement credit: the team running the board, who also confirmed the 168/169 counts in their own audit. Use whichever suits your memory budget. This repo ships the 169-module version.

A trap worth knowing. llm_int8_skip_modules patterns are matched anchored at the start of the full module path (or as an exact suffix). A bare "visual" does not match model.visual.encoder.layers.0... and silently skips nothing. Our first attempt did exactly that, quantised the vision tower as well (333 modules instead of 169), and the model emitted degenerate repetition on 98 % of items. Check the count.

Measured

986 distance + left/right items and 456 mcq items from the PhysicalAI Spatial-Intelligence-Warehouse validation split. Same host, same harness, same prompts, same parser as the bf16 reference. Greedy decode, enable_thinking=False. Significance by exact paired McNemar on the same items.

No task shows a difference this benchmark can detect. The distance calibration constant barely moves either: k = 1.155 at bf16, 1.163 at NF4 (split-half fit, cross-validated).

Throughput and memory, on a DGX B300 — not an edge figure: 59 tok/s at 2.6 GB peak, against 76 tok/s at 5.1 GB for bf16. The speed drop is bitsandbytes' dequantisation overhead on a small model in a Python loop, not a property of 4-bit arithmetic.

What this is validated for, and what it is not

Validated: competence. Every task above has a one-word or one-number answer. The result says 4-bit NF4 does not damage what this model can work out about a scene.

Not validated: contract adherence. A separate grounded benchmark on real hardware — structured JSON output with required fields, enums and nesting — scored poorly for this model family regardless of quantisation, and the failure there was schema adherence rather than numerics: well-formed JSON that omits required fields and emits out-of-enum values. Nothing here predicts that this model will emit your schema correctly. If you need structured output, constrain the decoder; do not assume the weights will do it.

The chance levels are worth carrying too: left/right is a two-way choice (50 %), and mcq offers between 3 and 13 regions per item, so its chance level is 14.2 %, not 25 %. A 17.5 % mcq score is about 3 points over chance, not 5.

What this is not

• —Not a TensorRT or Jetson build. These weights load only through bitsandbytes on CUDA. For an engine on Orin, use the AWQ INT4 ONNX export at `ubr-physical-ai/Cosmos3-Edge-INT4-AWQ`, which has a confirmed board bring-up.
• —Not quantised in a portable format. NF4 is re-created from the recipe in config.json; it is not a packed format another runtime can consume.
• —Not calibrated. NF4 applies one fixed rule and never looks at data. That is the point of this artefact — it is the crude baseline against which a calibrated method is measured.
• —Not a safety-certified component, and not evidence about any deployed system. A bench result.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

mid = "ubr-physical-ai/Cosmos3-Edge-NF4-bnb"
model = AutoModelForImageTextToText.from_pretrained(mid, device_map="cuda")  # config carries the 4-bit recipe
proc  = AutoProcessor.from_pretrained(mid)

Requires bitsandbytes and a CUDA device. transformers >= 5.16 for the cosmos3_edge model type.

Licence and notices of origin

Derived from `nvidia/Cosmos3-Edge`, distributed under the OpenMDW 1.1 licence. These quantised weights are a derivative of the Model Materials, not an output of the model, so OpenMDW's distribution terms apply to them: this repository retains the licence reference above and attributes origin to NVIDIA. Consult the base model card for NVIDIA's own safety, bias, privacy and explainability statements, which apply to this derivative.

Quantised by Team UBR Stack for the NVIDIA / OpenHackathons / Oracle Open Models Codefest 2026.