IntellAgents/Nemotron-Labs-Diffusion-VLM-8B-TextOnly-ONNX-Nsight

VLM-8B language-only inference copies

These are the language-side weights extracted from nvidia/Nemotron-Labs-Diffusion-VLM-8B revision adca93d16471c1e07d594ae444d23e1876f6b365. The image encoder and multimodal projector are omitted entirely, not quantized to zero. There are no vision weight files, and the text driver never creates a vision session. This is a text-only inference derivative, not a full training checkpoint or a drop-in Transformers model.

The language graphs are byte-for-byte identical to the corresponding components in the full VLM ONNX release. No second lossy quantization pass was applied. FP16 embeddings and output head are preserved in every variant. Only the 238 language attention/MLP matrices are quantized: symmetric per-output-channel INT8 or signed INT4 groups of 128, both round-to-nearest standard ONNX QDQ. Norms, activations and KV caches remain floating point. The checkpoint has no separate learned four-layer sampler.

NVIDIA's card says the original VLM language decoder weights match its 8B text model. We have not independently proved weight equality with that separate repository. This package nevertheless extracts this VLM's actual weights, and the same process can later extract your updated language weights from a fine-tuned full VLM. If fine-tuning freezes the language backbone and only changes vision/projector weights, those training changes will not carry into a text-only derivative.

Storage and precision

Removing the FP16 vision path saves 856,944,640 bytes (about 0.86 GB) per selected full-VLM setup. The text backbone remains the large majority of the model. These are storage figures, not VRAM guarantees: a backend may dequantize weight-only QDQ to FP16. INT4 remains experimental and can materially change logits and text.

Use

Install a CUDA-compatible PyTorch wheel and pip install -r requirements.txt on a suitable machine. Download the shared embedding_fp16.onnx / embedding.fp16.data, one decoder ONNX/data pair, tokenizer/config files, runtime_vlm.py and generate_text.py. Keep graph/data pairs together.

python generate_text.py --folder . --variant int8 --prompt "Explain why plants need sunlight." --max-new-tokens 64

Variants are fp16, int8, int4. Generation budgets must be positive multiples of 32. Batch size is one. No image processor is imported, no image weights are loaded, and images are not supported. Thinking is off by default; --thinking enables the upstream template and is outside the exact-generation probes. The original VLM's unseeded 32-token confidence-threshold diffusion loop is retained. Prefill and commits are causal; within-block denoising is bidirectional. Denoising KV outputs are discarded.

For Deep Learning Designer, open embedding_fp16.onnx and the chosen decoder graph separately. Decoder inputs are inputs_embeds [B,Q,4096] FP16, position_ids [B,Q] INT64, additive attention_mask [B,1,Q,P+Q] FP16, and 34 pairs of past_key_i/past_value_i [B,8,P,128] FP16. Outputs are logits [B,Q,131073] and 34 present-KV pairs [B,8,P+Q,128]. The external host still owns generation and KV lifecycle. FP16 uses opset 18; quantized graphs use opset 21; IR is 10. TensorRT 10.13.3.9 parsed these same graphs, but engine builds and Designer GUI execution are not verified.

reference_text.json and validation_text_*.json report text-only checks against the full source VLM invoked without images, including a two-block 64-token case and the packaged CLI. FP16 and INT8 match both reference token sequences exactly. INT4 runs and answers both prompts, but changes the outputs substantially; it is an experimental size/quality tradeoff. The raw reports include the original model's extra newlines/thinking markers rather than hiding them. This is a small correctness suite, not a quality benchmark. The FP16 oracle is the original BF16 checkpoint loaded as FP16; native BF16 parity is not claimed. text_extraction_manifest.json verifies that no vision/projector initializers survived. SHA256SUMS.json records the release inventory and hashes. repeatability_report.json confirms that rerunning the saved export process reproduces all four weight data files byte-for-byte.

Repeat after fine-tuning

Keep the full-precision full VLM as your training/master checkpoint. Do not fine-tune these RTN ONNX copies and do not use them to reconstruct removed vision weights. For LoRA/PEFT, merge the adapters into the full base model first, then save a complete safetensors checkpoint with config, tokenizer and required custom code. Preserve the updated language weights; this process never substitutes weights from the original model.

The included export_text_only.py accepts a local merged checkpoint directory and a new empty output directory. It supports the current 34-layer VLM architecture, sharded or single-file safetensors, and rejects several incompatible configuration changes. It does not download, mutate the source, or publish anything. Supply the real training checkpoint identifier and revision for provenance:

python export_text_only.py --source /content/my_finetuned_vlm --output /content/my_text_export --checkpoint-id my-account/my-finetuned-vlm --revision IMMUTABLE_TRAINING_CHECKPOINT_ID

This exports FP16 directly from that checkpoint, then independently makes INT8 and INT4 from FP16. Embedding/head stay FP16. Export scripts default to Transformers 5.0.0; changes to architecture, vocabulary, masks, RoPE, custom model arithmetic or sampling require re-auditing, not simply changing a model ID. No learned sampler is added automatically.

Validate the new checkpoint; the original model's reports cannot certify fine-tuned weights:

python collect_text_reference.py --source /content/my_finetuned_vlm --golden /content/my_text_goldens --report /content/my_text_export/reference_text.json
python validate_text_only.py --folder /content/my_text_export --golden /content/my_text_goldens --variant fp16
python validate_text_only.py --folder /content/my_text_export --golden /content/my_text_goldens --variant int8
python validate_text_only.py --folder /content/my_text_export --golden /content/my_text_goldens --variant int4

Review the reports, especially exact-token flags, add representative prompts for your task, and test in the intended runtime before publishing. The validator reports generation differences rather than treating every difference as an execution failure. Reference collection loads the full VLM on a suitable remote GPU. Training-specific image/task evaluations should be completed before removing vision from the deployment copy. Unknown non-vision tensors (including a newly supplied learned sampler or unmerged adapters) cause extraction to stop; do not bypass that check without adapting and validating the architecture.

NVIDIA's original card is included as NVIDIA_MODEL_CARD.md. Its terms identify NVIDIA Source Code License (NSCLv1), while its badge links elsewhere; consult NVIDIA for definitive redistribution/commercial terms. No relicensing is intended. This is an unofficial experimental conversion.