Datasetpublic

yakdoli/rocm-vlm-ocr-awq-configs

ROCm VLM/OCR AWQ+BF16 Serving Config Bundle A4 @ 200 DPI coverage verified (max_model_len >= 8192, visual tokens ~3800 + text ~4096). Model Profiles Profile Model Quant max_model_len Concurrent VRAM vllm-qwen3-vl-awq.json cyankiwi/Qwen3-VL-8B-Instruct-AWQ-4bit AWQ W4A16 8192 0.30 vllm-bizonai-bf16.json ONTHEIT/BizOnAI-OCR BF16 8192 0.26 sglang-qwen3-vl-awq.json cyankiwi/Qwen3-VL-8B-Instruct-AWQ-4bit AWQ W4A16 8192 0.30 A4 @ 200 DPI… See the full description on the dataset page: https://huggingface.co/datasets/yakdoli/rocm-vlm-ocr-awq-configs.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes226downloads

Dataset Card

ROCm VLM/OCR AWQ+BF16 Serving Config Bundle

A4 @ 200 DPI coverage verified (maxmodellen >= 8192, visual tokens ~3800 + text ~4096).

Model Profiles

Profile	Model	Quant	max_model_len	Concurrent VRAM
`vllm-qwen3-vl-awq.json`	cyankiwi/Qwen3-VL-8B-Instruct-AWQ-4bit	AWQ W4A16	8192	0.30
`vllm-bizonai-bf16.json`	ONTHEIT/BizOnAI-OCR	BF16	8192	0.26
`sglang-qwen3-vl-awq.json`	cyankiwi/Qwen3-VL-8B-Instruct-AWQ-4bit	AWQ W4A16	8192	0.30

A4 @ 200 DPI Token Budget

—A4 = 210mm x 297mm @ 200dpi = 1654px x 2339px = ~3.87M pixels
—Qwen3-VL 32x spatial compression: ~3800 visual tokens
—Text tokens for output: ~4096
—Total required context: ~8000 → safe maxmodellen = 8192

BizOnAI AWQ Status

—AWQ conversion is currently blocked on ROCm:
—autoawq: qwen3_vl model type not supported
—llm-compressor: sequential pipeline trace fails on VLM forward
—amd-quark: GPU kernel crash during AWQ activation caching
—BizOnAI is served as BF16 with conservative VRAM allocation (0.26)
—When AWQ conversion becomes available on ROCm, switch to AWQ profile

Notes

—Qwen3-VL AWQ requires float16 (not bfloat16) with VLLM_USE_TRITON_AWQ=1
—For multi-model concurrency, cap gpu_memory_utilization conservatively
—max-num-seqs 1 recommended for OCR workload
—limit-mm-per-prompt.image 4 prevents runaway image token allocation