WayBob/Qwen3VL-8B-4bit-GGUF-Jetson-Deployment

.
├── README.md                           # Model card and documentation
├── qwen3vl_8b_disaster_merged/         # Merged Hugging Face checkpoint (BF16)
│   ├── config.json
│   ├── generation_config.json
│   ├── model-00001-of-00004.safetensors
│   ├── model-00002-of-00004.safetensors
│   ├── model-00003-of-00004.safetensors
│   ├── model-00004-of-00004.safetensors
│   ├── model.safetensors.index.json
│   ├── tokenizer.json
│   ├── tokenizer_config.json
│   ├── special_tokens_map.json
│   ├── chat_template.jinja
│   ├── preprocessor_config.json
│   └── vocab.json
├── gguf_16bit_4bit/                    # GGUF files for llama.cpp
│   ├── disaster-8b-f16.gguf            # F16 GGUF
│   └── disaster-8b-q4km.gguf           # Q4_K_M GGUF (recommended for Jetson)
└── merge_lora/                         # Merge configuration
    └── qwen3_vl_8b_xview2lora.yaml     # LLaMA-Factory merge config

import torch
from PIL import Image
from transformers import Qwen3VLForConditionalGeneration, AutoProcessor

model = Qwen3VLForConditionalGeneration.from_pretrained(
    "WayBob/Qwen3VL-8B-4bit-GGUF-Jetson-Deployment",
    subfolder="qwen3vl_8b_disaster_merged",
    torch_dtype="auto",
    device_map="auto"
)

processor = AutoProcessor.from_pretrained(
    "WayBob/Qwen3VL-8B-4bit-GGUF-Jetson-Deployment",
    subfolder="qwen3vl_8b_disaster_merged"
)

image = Image.open("disaster_image.jpg").convert("RGB")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "What type of disaster occurred in this image?"}
        ]
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0
)

generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

print(output_text[0])

# Clone and build llama.cpp on Jetson
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Download the quantized model from this repository
huggingface-cli download WayBob/Qwen3VL-8B-4bit-GGUF-Jetson-Deployment \
  --include "gguf_16bit_4bit/disaster-8b-q4km.gguf" \
  --local-dir models/

# Download the Qwen3-VL vision projector
huggingface-cli download Qwen/Qwen3-VL-8B-Instruct-GGUF \
  --include "*mmproj*Q8*" \
  --local-dir models/

# Launch the server
./build/bin/llama-server \
  -m models/disaster-8b-q4km.gguf \
  --mmproj models/qwen3-vl-8b/mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf \
  -ngl 99 --fit off -c 8192 \
  -ctk q8_0 -ctv q8_0 \
  --host 0.0.0.0 --port 8080

import base64
import requests

with open("disaster_image.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

response = requests.post(
    "http://localhost:8080/v1/chat/completions",
    json={
        "messages": [
            {
                "role": "system",
                "content": (
                    "You are a disaster recognition expert. "
                    "When analyzing disaster images, first identify the disaster type, "
                    "then explain the key visual evidence supporting your classification. "
                    "Respond in the same language as the user."
                )
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {"url": f"data:image/jpeg;base64,{img_b64}"}
                    },
                    {
                        "type": "text",
                        "text": "What type of disaster occurred in this image?"
                    }
                ]
            }
        ],
        "temperature": 0
    },
    timeout=300
)

print(response.json()["choices"][0]["message"]["content"])

# Build llama.cpp on Jetson
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Download model and vision projector
huggingface-cli download WayBob/Qwen3VL-8B-4bit-GGUF-Jetson-Deployment \
  --include "gguf_16bit_4bit/disaster-8b-q4km.gguf" \
  --local-dir models/

huggingface-cli download Qwen/Qwen3-VL-8B-Instruct-GGUF \
  --include "*mmproj*Q8*" \
  --local-dir models/

# Launch server
./build/bin/llama-server \
  -m models/gguf_16bit_4bit/disaster-8b-q4km.gguf \
  --mmproj models/mmproj-Qwen3VL-8B-Instruct-Q8_0.gguf \
  -ngl 99 \
  --fit off \
  -c 8192 \
  -ctk q8_0 -ctv q8_0 \
  --host 0.0.0.0 \
  --port 8080

QLoRA Fine-tuning
    ↓
LoRA Adapter
    ↓  LLaMA-Factory export
Merged HF Model (BF16)
    ↓  llama.cpp convert_hf_to_gguf.py
GGUF F16
    ↓  llama-quantize
GGUF Q4_K_M
    ↓
Deployment on Jetson with llama.cpp

model_name_or_path: Qwen/Qwen3-VL-8B-Instruct
adapter_name_or_path: WayBob/Qwen3VL-8B-QLora-4bit-xView2-Disaster-Recognition
template: qwen3_vl_nothink
trust_remote_code: true
export_dir: <replace-with-your-LLaMA-Factory-path>/output/qwen3vl_8b_disaster_merged
export_size: 5
export_device: cpu
export_legacy_format: false

python -m pip install gguf

git clone https://github.com/ggml-org/llama.cpp <replace-with-your-llama.cpp-path>

# Convert merged HF checkpoint to GGUF
python3 <replace-with-your-llama.cpp-path>/convert_hf_to_gguf.py \
  <replace-with-your-LLaMA-Factory-path>/output/qwen3vl_8b_disaster_merged \
  --outfile <replace-with-your-LLaMA-Factory-path>/output/gguf_16bit_4bit/disaster-8b-f16.gguf

# Build quantization tool
cd <replace-with-your-llama.cpp-path>
cmake -B build
cmake --build build --target llama-quantize -j$(nproc)

# Quantize to Q4_K_M
./build/bin/llama-quantize \
  <replace-with-your-LLaMA-Factory-path>/output/gguf_16bit_4bit/disaster-8b-f16.gguf \
  <replace-with-your-LLaMA-Factory-path>/output/gguf_16bit_4bit/disaster-8b-q4km.gguf \
  Q4_K_M

Q: What type of disaster occurred in this image?
A: This is a fire disaster. Key visual evidence includes charred and blackened terrain,
   scorched vegetation, and widespread burn patterns across the landscape.

Q: この画像ではどのような種類の災害が発生しましたか？
A: 火災災害が発生しました。地表が黒く焼けており、植生や建物に焼失の痕跡が見られます。

Q: 当前图片发生了什么灾害呢？
A: 当前图片发生了风灾灾害。可以看到大量树木倒伏、建筑受损，以及明显的风灾破坏痕迹。

You are a disaster recognition expert. When analyzing disaster images, first identify the disaster type, then explain the key visual evidence supporting your classification. Respond in the same language as the user.

@misc{wang2026qwen3vl_jetson_disaster,
  title={Qwen3VL-8B-4bit-GGUF-Jetson-Deployment: Merged and Quantized Vision-Language Model for Disaster Type Classification},
  author={WayBob},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/WayBob/Qwen3VL-8B-4bit-GGUF-Jetson-Deployment}
}

@misc{wang2026qwen3vl_disaster_lora,
  title={Qwen3VL-8B-QLora-4bit-xView2-Disaster-Recognition},
  author={WayBob},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/WayBob/Qwen3VL-8B-QLora-4bit-xView2-Disaster-Recognition}
}

@misc{waybob2026disaster_dataset,
  title={Disaster Recognition RemoteSense Dataset (EN/CN/JA)},
  author={WayBob},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/datasets/WayBob/Disaster_Recognition_RemoteSense_EN_CN_JA}
}

@inproceedings{xview2,
  title={xBD: A Dataset for Assessing Building Damage from Satellite Imagery},
  author={Gupta, Ritwik and Hosfelt, Richard and Sajeev, Sandra and Patel, Nirav and Goodman, Bryce and Doshi, Jigar and Heim, Eric and Choset, Howie and Gaston, Matthew},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year={2019}
}