amaan784/Llama3-LLaVA-NeXT-8B-AWQ-W4A16-substation

Llama-3-LLaVA-NeXT-8B-AWQ-W4A16 (substation-domain calibrated)

A domain-calibrated INT4 quantization of llava-hf/llama3-llava-next-8b-hf for industrial visual inspection (substation, pump impeller, turbine blade, motor thermal). The Llama-3-8B text tower is W4A16-quantized via AWQ/GPTQ; the LLaVA-NeXT vision tower (CLIP-style) and multi_modal_projector are kept in FP16.

Built by HPML Team 23 (Columbia, Spring 2026). Companion to the AssetOpsBench-VLM-22 benchmark.

Headline result on AssetOpsBench-VLM-22

Single L4 (24 GiB), vLLM 0.19, sequential single-batch traffic, 22 scenarios. The GPU column is the per-run vram_used_mib metric from results/hpml_metrics.csv; vLLM can reuse freed model-weight memory for the KV-cache pool, so this is not the packed checkpoint size.

The reliability finding

In the final 22-scenario benchmark sweep, the generic-calibrated variant looks faster on the logged valid rows (3157 ms mean) but has a runaway-generation failure mode: two L1g transformer scenarios timed out around the token cap (~122 s), dropping the judged denominator from 44 to 40.

The domain-calibrated variant -- this model -- did not trigger that failure (max logged E2E = 8.8 s; judge denominator 44/44). For inspection workloads where bounded latency matters, domain calibration is the safer choice even though its logged mean is slightly higher.

This is the project's main contribution: aggregate latency hides reliability differences that domain-aligned calibration data fixes.

Quantization recipe

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=[
        "re:.*lm_head",
        "re:.*vision_tower.*",
        "re:.*multi_modal_projector.*",
    ],
)
oneshot(
    model, tokenizer,
    dataset=domain_calibration_set,    # 128 hand-authored substation prompts
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=128,
    sequential_targets=["LlamaDecoderLayer"],
)
dispatch_model(model)                            # consolidate accelerate offload
remove_hook_from_module(model, recurse=True)     # strip remaining offload hooks
model.save_pretrained(out, save_compressed=True) # pack-quantized format

The CLIP-style vision tower and multi_modal_projector stay FP16. INT4 on the vision tower is well-documented to degrade LLaVA-family output quality.

Calibration data

128 hand-authored substation-inspection text prompts spanning equipment identification, defect descriptions, condition assessments, and gauge readings, intentionally aligned with the test scenario distribution. The generic counterpart (not this model) used 128 samples from HuggingFaceH4/ultrachat_200k.

Usage

vLLM (recommended)

vllm serve amaan784/Llama3-LLaVA-NeXT-8B-AWQ-W4A16-substation \
  --quantization compressed-tensors \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85 \
  --limit-mm-per-prompt image=1

The quantization_config.ignore list in config.json must use regex form (re:.*vision_tower.*, re:.*multi_modal_projector.*); expanded q/k/v names will trigger KeyError: qkv_proj.weight on vLLM 0.19.

transformers

from transformers import LlavaNextForConditionalGeneration, AutoProcessor

model = LlavaNextForConditionalGeneration.from_pretrained(
    "amaan784/Llama3-LLaVA-NeXT-8B-AWQ-W4A16-substation",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "amaan784/Llama3-LLaVA-NeXT-8B-AWQ-W4A16-substation"
)

Pinned stack

Validated only on the following exact versions:

• —vLLM 0.19.0
• —llmcompressor 0.10.0.2
• —compressed-tensors 0.14.0.1
• —transformers 4.57.6 (with one module_map save-path patch -- see companion repo)
• —torch 2.10.0+cu129

Six integration bugs surfaced and were fixed during this work; the most important are listed in the companion repo's report (LlavaNext module_map[image_newline] KeyError on save, accelerate hook import location change at >=1.0, llmcompressor 0.3 fake-quant output without save_compressed=True, vLLM 0.19 vision-tower ignore-list regex requirement).

Compression numbers

• —16 GB FP16 weights -> 6.0 GB packed INT4 on disk (2.7x).
• —15.54 GiB -> 5.9 GiB weight VRAM at runtime (2.6x), confirmed from vLLM gpu_model_runner startup logs.
• —Freed 9.6 GiB of weight VRAM becomes KV-cache pool: 21K -> 100K concurrent token capacity (4.7x), a real serving-side win for batched throughput (not measured here, single-batch only).

Limitations

• —Validated only on a 22-scenario rubric-graded inspection benchmark. Accuracy on general VQA / TextVQA / MMBench is not measured.
• —Vision tower stays FP16; VRAM win is from the LLM tower only.
• —Single-batch sequential traffic. Throughput at concurrency not measured.
• —Calibration data is substation-themed. Out-of-domain prompts may degrade more than the generic-calibrated counterpart.
• —Llama-3.2-11B-Vision was the original target; substituted with Llama-3-LLaVA-NeXT-8B for L4 (24 GiB) VRAM fit.

License

This model is a derivative of `llava-hf/llama3-llava-next-8b-hf`, which incorporates Meta Llama 3 weights. Distribution is governed by the [Llama 3 Community License](https://llama.meta.com/llama3/license).

By downloading or using these weights you agree to the upstream license terms (acceptable use, attribution, MAU restriction). The quantization recipe and surrounding code are released under Apache-2.0 in the companion repo.

Citation

@misc{llama3-llava-next-awq-substation,
  title  = {Llama-3-LLaVA-NeXT-8B-AWQ-W4A16 (Substation-Domain Calibrated)},
  author = {HPML Team 23 (Columbia University)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/amaan784/Llama3-LLaVA-NeXT-8B-AWQ-W4A16-substation}}
}