WayBob/Qwen3vl-8b-qlora-4bit-Japanese-photo-conversation

Qwen3VL-8B QLora 4-bit - Japanese Photo Conversation

<div align="center">

🎨 Vision-Language Model | 📸 Photo Description | 🇯🇵 Japanese Specialized

   

Built with Qwen3-VL | Fine-tuning: 4-bit QLoRA (84MB adapter) | Framework: LLaMA-Factory | Hardware: 2×RTX 4090 24GB (Multi-node)

</div>

A Japanese-specialized vision-language model fine-tuned from Qwen/Qwen3-VL-8B-Instruct using 4-bit QLoRA on multi-node distributed infrastructure.

Model Description

This model adapts the general-purpose Qwen3-VL-8B into a specialized Japanese photo description system. Through LoRA fine-tuning on 11,808 Japanese photo-conversation pairs, it learns to produce concise, objective descriptions while significantly reducing hallucinations common in the base model.

Key Improvements Over Base Model

• —Eliminated location hallucinations - Base model frequently guessed specific place names incorrectly
• —Fixed infinite generation loops - Base model got stuck repeating text in some cases
• —Concise objective descriptions - Matches human annotation style instead of encyclopedia-like responses
• —No more OCR over-reliance - Focuses on visual understanding rather than text reading
• —Consistent output format - Predictable response length and structure

| Output Consistency | Variable length/format | Consistent format |

Quick Start

What is This Model?

This is a LoRA adapter (not a full model). You need to:

1. 1.Load the base model: Qwen/Qwen3-VL-8B-Instruct
2. 2.Apply this LoRA adapter on top of it

Advantage: Only ~84MB download instead of ~8.7GB full model!

Installation

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .

Usage

from llamafactory.chat import ChatModel

# Initialize model with LoRA adapter
chat_model = ChatModel(args={
    "model_name_or_path": "Qwen/Qwen3-VL-8B-Instruct",
    "adapter_name_or_path": "WayBob/Qwen3vl-8b-qlora-4bit-Japanese-photo-conversation",
    "template": "qwen3_vl_nothink",
    "quantization_bit": 4,
    "trust_remote_code": True,
    "flash_attn": "fa2",  # Optional: enable flash attention for faster inference
    "infer_backend": "huggingface",
})

# Ask questions about images
messages = [{"role": "user", "content": "<image>\nこの画像には何が写っていますか？"}]
responses = chat_model.chat(messages=messages, images=["your_image.jpg"])
print(responses[0].response_text)

See inference_example.py for a complete working example with multiple question types.

Hardware Requirements

Recommended GPU: RTX 3090 / 4090 / A100 or equivalent with 12GB+ VRAM

Training Details

Base Model

• —Source: Qwen/Qwen3-VL-8B-Instruct
• —Parameters: 8.7 billion
• —Architecture: Qwen3-VL (Vision-Language)
• —Context Length: 262,144 tokens
• —Vision Encoder: ViT-based with spatial merge

Training Data

Primary Dataset: WayBob/Japanese_Photo_conversation_cleaned

This dataset is cleaned and organized from:

• —llm-jp/japanese-photos-conversation
• —ThePioneer/japanese-photos

We thank the original authors for their excellent work.

Training Configuration

Hardware & Infrastructure:

• —GPUs: 2× NVIDIA RTX 4090 24GB
• —Distributed Setup: Multi-node DDP (Distributed Data Parallel)
• —Master Node: 192.168.11.8
• —Worker Node: 192.168.11.10
• —1 GPU per node
• —Framework: LLaMA-Factory

Hyperparameters:

method: qlora_4bit
quantization: 4-bit (BitsAndBytes)
lora_rank: 8
lora_alpha: 16
lora_target: all (all linear layers)
lora_dropout: 0.05
learning_rate: 1.0e-4
batch_size: 4 per device × 2 accumulation = 8 effective
epochs: 3
optimizer: AdamW
lr_scheduler: cosine
warmup_ratio: 0.1
precision: bfloat16

Training Duration: ~2.85 hours (10,253 seconds)

Memory Usage: ~10-12GB VRAM per GPU

Training Results

Training Loss Curve:

[image]

Validation Loss Curve:

[image]

Performance Evaluation

We conducted comprehensive comparisons between the base model and fine-tuned model on both training and validation sets.

Example 1: Coastal Scene (Training Set)

[image]

Question: この画像はどのような場所を映していますか？

Result: HALLUCINATION ELIMINATED - Model provides objective description without guessing specific locations.

Example 2: Indoor Building (Training Set)

[image]

Question: この画像はどのような場所で撮影されたものですか？

Result: CRITICAL BUG FIXED - Eliminates infinite loop, generates coherent response.

Example 3: Night Cityscape (Training Set)

[image]

Question: この画像に写っているのは昼間の景色ですか、夜の景色ですか？

Result: ACCURATE & CONCISE - Correctly answers without misidentification or unnecessary explanation.

For detailed comparison with all 6 examples (3 training + 3 validation), see model_comparison_report.md.

Quantitative Improvements Summary

Training Reproduction

Single-Node Training

llamafactory-cli train qwen3vl_8b_japanese_4bit.yaml

Multi-Node Distributed Training (Used in this model)

This model was trained using 2 nodes with the provided scripts.

On Master Node (192.168.11.8):

bash train_qwen3vl_8b_4bit_master.sh

On Worker Node (192.168.11.10):

bash train_qwen3vl_8b_4bit_worker.sh

The scripts are included in this repository with the exact configuration used for training.

Key environment variables for multi-node setup:

export FORCE_TORCHRUN=1
export NNODES=2
export NODE_RANK=0  # 0 for master, 1 for worker
export MASTER_ADDR=192.168.11.8
export MASTER_PORT=29500
export CUDA_VISIBLE_DEVICES=0

Model Files

Model Weights & Config

• —adapter_config.json - LoRA adapter configuration
• —adapter_model.safetensors - LoRA adapter weights (~84MB)
• —training_args.bin - Training arguments

Training Results

• —training_loss.png - Training loss curve
• —training_eval_loss.png - Validation loss curve
• —trainer_log.jsonl - Detailed training logs
• —all_results.json - Final training metrics
• —train_results.json - Training statistics
• —eval_results.json - Evaluation statistics

Training Configuration

• —qwen3vl_8b_japanese_4bit.yaml - LLaMA-Factory training configuration
• —train_qwen3vl_8b_4bit_master.sh - Multi-node training script (master node)
• —train_qwen3vl_8b_4bit_worker.sh - Multi-node training script (worker node)

Documentation & Examples

• —model_comparison_report.md - Detailed comparison report with 6 examples
• —inference_example.py - Simple inference example using LLaMA-Factory
• —sample_images/ - Sample images used in comparison report (6 images)

Limitations

• —Language: Primarily trained on Japanese; performance on other languages not guaranteed
• —Domain: Specialized for photo description; may not generalize to other vision tasks (e.g., OCR, diagram analysis)
• —Quantization: Designed for 4-bit quantization; full precision inference not tested
• —Output Style: Trained to produce concise descriptions; may not provide detailed analysis when needed
• —Context: 2048 token cutoff during training

Intended Use Cases

Recommended:

• —Japanese photo description and captioning
• —Visual question answering in Japanese
• —Content moderation (image understanding)
• —Accessibility applications (image-to-text for visually impaired)
• —Dataset annotation assistance

Not Recommended:

• —Medical image diagnosis
• —Fine-grained object detection (use specialized models)
• —OCR tasks (use dedicated OCR models)
• —Video understanding (trained on static images only)

Citation

@misc{qwen3vl-8b-qlora-japanese-photo,
  author = {WayBob},
  title = {Qwen3VL-8B QLora 4-bit Japanese Photo Conversation},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/WayBob/Qwen3vl-8b-qlora-4bit-Japanese-photo-conversation}
}

Acknowledgements

Base Model:

• —Qwen3-VL-8B-Instruct - Alibaba Cloud Qwen Team
• —Licensed under Apache 2.0

Training Datasets:

• —llm-jp/japanese-photos-conversation - Original Japanese photo conversation dataset
• —ThePioneer/japanese-photos - Japanese photo collection
• —WayBob/Japanese_Photo_conversation_cleaned - Cleaned and organized version

Training Framework:

• —LLaMA-Factory by hiyouga

Method:

• —QLoRA: Efficient Finetuning of Quantized LLMs by Dettmers et al.

Infrastructure:

• —Local multi-node setup with 2× NVIDIA RTX 4090 24GB GPUs

License

This model is licensed under Creative Commons Attribution 2.0 (CC-BY-2.0).

Key License Terms

• —Share: You can copy and redistribute the material in any medium or format
• —Adapt: You can remix, transform, and build upon the material for any purpose, even commercially
• —Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made
• —No Additional Restrictions: You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits

Full License: See CC-BY-2.0 License for complete terms.

Contact

• —HuggingFace: WayBob
• —Model Repository: Qwen3vl-8b-qlora-4bit-Japanese-photo-conversation
• —Dataset Repository: Japanese_Photo_conversation_cleaned