jarvis-pet/Phi-4-multimodal-instruct-int8-onnx

Phi-4-multimodal-instruct — ONNX INT8 decoder + GPTQ INT4 vision

An INT8 language decoder paired with a GPTQ-calibrated INT4 vision tower, rebuilt from `microsoft/Phi-4-multimodal-instruct-onnx` for `onnxruntime-genai` on the CUDA execution provider. 5.6 B parameters: a 3.8 B Phi-4-mini language backbone, a ~0.45 B SigLIP vision encoder, a ~0.46 B conformer speech encoder, and the modality LoRA adapters.

This is the byte-exact build. Its sibling `jarvis-pet/Phi-4-multimodal-instruct-gptq-int4-onnx` is 1.99 GiB smaller and buys a bigger context window plus both modalities in one process on a 12 GB card, but it drops the occasional character in short strings. This one does not: pick it when byte-exact short-field extraction matters more than VRAM headroom.

Why it is not simply the published artifact. Microsoft's GPU release quantizes both towers with INT4 RTN, and both are damaged by it. On the vision tower, image_features measured against FP16 have a relative L2 error of 0.94–1.09 — error equal to signal — so OCR either degenerates into repetition or confabulates. The INT4 decoder separately breaks EOS, so page transcription never terminates. int4 was the whole problem, in both components; this build replaces the tower with data-aware GPTQ and the decoder with 8-bit.

The name states the DECODER's precision, matching the convention already used for this account's OpenVINO uploads (where -int4-ov likewise names the language model while other submodels sit at INT8). The vision tower here is GPTQ INT4, as the table below records.

Released under the MIT License, inherited from the base model `microsoft/Phi-4-multimodal-instruct`. This is a derivative quantization only; all model rights and terms follow the base model.

Files

19 files, 6.78 GiB total:

The adapter files are vestigial here, deliberately. The locally built decoder bakes the LoRAs in (896 lora_* initializers, 0 adapter inputs — Phi4MMModel.make_layer makes vision the baked default), so adapter_filename has been removed from the vision and speech sections of genai_config.json. Leaving it in makes the runtime bind tensors the graph does not have. Losing the speech LoRA was expected to cost audio quality — measured, it does not (see below).

Audio input — 16 kHz required, 8 kHz optional

The speech encoder consumes 80-bin log-mel features at a 25 ms window and a 10 ms hop. speech_processor.json declares AudioDecoderEx.target_sample_rates: [8000, 16000], and Phi4AudioEmbed carries a complete STFT/log-mel parameter set for each rate:

Both paths emit the same feature geometry — 80 mel bins at 100 frames/second — which is what the encoder is built for; audio_compression_rate: 8 then decimates that 8× along time. What differs is bandwidth, not shape.

• —Supply 16 kHz mono PCM16 WAV. This is the primary path and what every audio measurement below used.
• —8 kHz is natively supported, not merely tolerated — a first-class second parameter set intended for telephony-band input. Nyquist is 4 kHz there, so the 4–8 kHz band is absent from the features; expect that to cost fricative and sibilant detail.
• —Any other rate is resampled by `AudioDecoderEx`, not rejected. That path is untested here, so do not rely on its fidelity — resample to 16 kHz yourself if you care about the transcript. This differs from an OpenVINO serve of the same model, where nothing resamples and a wrong-rate waveform transcribes confidently wrong.

Measured results

Measured on an RTX 3060 12 GB (Windows 11, driver 610.88, no CUDA toolkit — pip wheels only), onnxruntime-genai-cuda 0.15.2 + onnxruntime-gpu 1.29.0, at `max_length = 4096`:

Against the published INT4-RTN artifact on the same invoice: finish_reason length → `stop`, identical-line repeats 10 → 1, gold fields 5/6 → 6/6, Subtotal wrong → exact, routing number MISS → HIT, and severe character doubling → none.

Cost: 12 GB serves one modality per process

Weights are 7.4 GB, and ORT's BFC arena grows and never releases, so:

• —`max_length = 8192` does not fit a 12 GB card at all. 4096 works, for OCR-only or audio-only.
• —An audio request after a page request dies with AllocateRawInternal Failed to allocate memory for requested buffer of size 1338028032 — the vision activation buffer — with ~1.4 GB nominally free. A serve that must do both in one process needs a bigger card.

Both limits are what the INT4-decoder sibling exists to lift: at 4.77 GB of weights it holds 8192 and takes audio after a page in the same process, on the same 12 GB card.

How the quantization was produced

• —Decoder — INT8. Upstream's build_text() hardcodes precision = "int4" and ignores --precision, so create_model was called directly with precision="int8" and exclude_embeds=true, filename=phi-4-mm-text.onnx to match what genai_config.json names.
• —Vision tower — GPTQ INT4, block 128. build_vision() runs the nbits quantizer unconditionally at INT4 RTN, so the FP16 tower was exported with that step patched out and torch.onnx.export(..., dynamo=False) (torch ≥ 2.9's dynamo exporter fails to capture the SigLIP tower and silently rewrites opset 14 → 18), then quantized with MatMulNBitsQuantizer(bits=4, block_size=128, is_symmetric=True) under GPTQWeightOnlyQuantConfig(percdamp=0.1, actorder=False, mse=False, perchannel=True).
• —GPTQ is Python-API-only: --quant_method gptq from the CLI builds the config with no calibration_data_reader, so INC iterates None and dies with TypeError: 'NoneType' object is not iterable.

Fidelity of the vision tower against FP16, five page fixtures, CUDA EP:

Cosine 0.95 passes every end-to-end probe and scored identically to an INT8 tower while being 166 MB smaller, which is why the INT4 GPTQ tower is used here rather than an INT8 one. Note finer groups made GPTQ worse, the opposite of the usual expectation.

Gotchas worth knowing

• —`device_id` must be 0. ORT's CUDA EP creates its cudnn handle before cudaSetDevice, so any non-zero device_id dies at model load with CUDNN failure 2007: CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH. To serve on a different physical card, mask it with CUDA_VISIBLE_DEVICES=<n> (plus CUDA_DEVICE_ORDER=PCI_BUS_ID) and keep device_id=0.
• —`max_length` reserves the whole KV cache up front — genai_config.json sets past_present_share_buffer: true. The artifact declares context_length: 131072, which on this geometry (32 layers × 8 KV heads × 128 head size × 2 × 2 B) would ask ~17 GB.
• —ORT's arena never releases. Peak VRAM is set by the widest activation the process has ever allocated, not by the current request — which is why modality order matters here.
• —`nvidia_awq` cannot quantize either half of this model. NVAWQWeightOnlyQuantConfig synthesises input_ids/attention_mask/position_ids/past-KV, but the vision tower takes pixel_values/image_attention_mask/image_sizes and the decoder — built exclude_embeds=true — takes inputs_embeds with no input_ids. Structural, not a tuning problem.

Usage

import onnxruntime as ort
import onnxruntime_genai as og

# The pip CUDA wheels place their DLLs outside the process search path.
ort.preload_dlls()

cfg = og.Config("Phi-4-multimodal-instruct-int8-onnx")
cfg.clear_providers()
cfg.append_provider("cuda")
cfg.set_provider_option("cuda", "device_id", "0")   # must be 0 — see gotchas
model = og.Model(cfg)

tokenizer = og.Tokenizer(model)
params = og.GeneratorParams(model)
params.set_search_options(max_length=4096, do_sample=False)   # 8192 does not fit 12 GB here

gen = og.Generator(model, params)
gen.append_tokens(tokenizer.encode(
    "<|user|><|image_1|>\nTranscribe all text in this image exactly.<|end|><|assistant|>"))

Images and audio are passed as og.Images.open_bytes(...) / og.Audios.open_bytes(...) to model.create_multimodal_processor(), and the prompt must carry the inline markers this architecture expects — <|image_1|>, <|audio_1|>, numbered from 1 independently per modality. Use one modality per process on a 12 GB card, per the arena limit above.

Acknowledgements

Base model © Microsoft — `microsoft/Phi-4-multimodal-instruct` and its ONNX release `microsoft/Phi-4-multimodal-instruct-onnx`, whose embedding and speech graphs, adapters and tokenizer ship here unchanged. Quantization tooling: ONNX Runtime, its bundled Intel Neural Compressor GPTQ implementation, and onnxruntime-genai's model builder.