Modelpublic

Advantech-EIOT/Qwen3.5-35B-A3B-GGUF

sourceHugging Faceapache-2.0updated 11d agoView on Hugging Face

0likes443downloads

Model Card

Qwen3.5-35B-A3B GGUF

Q4KM and IQ3KT quantizations of the official [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) checkpoint, with model-matched BF16 and Q4K_M multimodal projectors. Published by Advantech-EIOT; not an official Qwen quantization release.

Files and checksums

—Q4_K_M/Qwen3.5-35B-A3B-Q4_K_M.gguf: 21,166,758,496 bytes; SHA-256 55d8ff760b118f2d6c0a2b55f3096afedd9a78e8a916531566336769d5eba645.
—IQ3_KT/Qwen3.5-35B-A3B-IQ3_KT.gguf: 13,968,681,056 bytes; SHA-256 74d0e9848bc8424d89df92786b920b0e1001171a5ab2827ea58195d830bab0f1.
—mmproj/mmproj-Qwen3.5-35B-A3B-BF16.gguf: 902,822,400 bytes; SHA-256 79104874d7ab7b680e6e6abe48b03a040abc63cd73d0e778ea01759118d74f2e.
—mmproj/mmproj-Qwen3.5-35B-A3B-Q4_K_M.gguf: 514,318,464 bytes; SHA-256 2dc7a331047bbf8f590d48cfd03c94e3b0b056c1e9fe20d873666798eb6ac726.

Machine-readable tensor types, sizes and provenance: manifest.json.

Usage and compatibility

The tested runtime is a custom llama.cpp b8779-lineage build with Qwen3.5 VLM and IQ3KT support. The upstream tutorial anchor is `75f3bc94e649616162981c322e8e6b88ca5491e8`; this is not a claim that pristine upstream b8779 or arbitrary stock llama.cpp builds support these files. The custom runtime is not bundled here. Verify your runtime's architecture, projector and IQ3KT tensor support before use.

With that compatible build, from the downloaded repository root:

bash

GGML_CUDA_DISABLE_GRAPHS=1 /path/to/compatible/llama-server \
  -m Q4_K_M/Qwen3.5-35B-A3B-Q4_K_M.gguf \
  --mmproj mmproj/mmproj-Qwen3.5-35B-A3B-BF16.gguf \
  --host 127.0.0.1 --port 18779 \
  -c 2048 -b 256 -ub 128 -ngl 16 --fit off \
  -t 12 -tb 12 --parallel 1 --jinja --no-warmup

API: http://127.0.0.1:18779/v1/chat/completions. These are the tested partial-offload settings, not performance recommendations for every device. Respect occupied ports and available RAM/VRAM.

Select the IQ3_KT/ main model or the mmproj/ Q4KM projector for the other tested pairings. Never use a projector from another model size. Exact preset is IQ3_KT, not IQ3KT. Its preset/file_type 152 and tensor enum 154 are different identifiers.

The Q4KM projector uses native mixed precision (Q4K, Q50, Q8_0, F16, F32) due to vision tensor shapes; not all weights are 4-bit. No language-model imatrix was applied to the projector.

Provenance

—Official source revision: 59d61f3ce65a6d9863b86d2e96597125219dc754.
—Both main quantizations came from the same official BF16 conversion and the same accepted, model-specific VLM importance matrix. No third-party low-bit requantization.
—Q4KM: custom b8779 quantizer. IQ3KT: ikllama.cpp quantizer; inference remained on the custom b8779 runtime.
—Formal calibration: 768 samples; 511 entries; DAT SHA-256 4aa719b5d666a88d0a5f73fb4819e6ba6eb2108f4e3ae93d07099ae14be4cb7f.

The calibration-only collector retained all prompt/image/answer tokens through the final FFN. Expanded mixed768 data achieved full coverage of 120 expert tensors and 30,720 expert rows with no absent expert rows; incomplete earlier DATs were not used. Routing and weights were not altered to manufacture coverage.

Functional validation and limitations

Both main quants were tested with both projectors: four pairings, 16 text/caption/OCR/spatial requests on the original custom runtime. All returned normally without token truncation. Text sanity and the book-title OCR case passed; captions described the main visible scene.

These are functional smoke checks, not an accuracy benchmark. Cases came from existing calibration data, so this is not independent generalization evidence. Spatial answers matched the dataset label, but the couch-edge image is visually ambiguous; agreement is not proof of correctness. No additional BF16-main quality benchmark or reference-logit comparison was performed.

License

Apache-2.0, inherited from the official source. See LICENSE and the linked base-model card for upstream information.