cyankiwi/Ministral-3-3B-Instruct-2512-AWQ-4bit

Ministral-3-3B-Instruct-2512 AWQ - INT4

Model Details

Quantization Details

• —Quantization Method: AWQ
• —Bits: 4
• —Group Size: 32
• —Calibration Dataset: 5CD-AI/LLaVA-CoT-o1-Instruct
• —Quantization Tool: llm-compressor

Memory Usage

Evaluations

• —Evaluation Context Length: 16384

Inference

Prerequisite

pip install -U vllm

Basic Usage

vllm serve cyankiwi/Ministral-3-3B-Instruct-2512-AWQ-4bit --tokenizer_mode mistral --config_format mistral --load_format mistral --enable-auto-tool-choice --tool-call-parser mistral

Additional Information

Changelog

• —v1.0.0 - Initial quantized release

Authors

• —Name: Ton Cao
• —Contacts: ton@cyan.kiwi

Ministral 3 3B Instruct 2512 BF16

The smallest model in the Ministral 3 family, Ministral 3 3B is a powerful, efficient tiny language model with vision capabilities.

This model is the instruct post-trained version, fine-tuned for instruction tasks, making it ideal for chat and instruction based use cases.

The Ministral 3 family is designed for edge deployment, capable of running on a wide range of hardware. Ministral 3 3B can even be deployed locally, capable of fitting in 16GB of VRAM in BF16, and less than 8GB of RAM/VRAM when quantized.

We provide a no-loss FP8 version here, you can find other formats and quantizations in the Ministral 3 - Additional Checkpoints collection.

Key Features

Ministral 3 3B consists of two main architectural components:

• —3.4B Language Model
• —0.4B Vision Encoder

The Ministral 3 3B Instruct model offers the following capabilities:

• —Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
• —Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
• —System Prompt: Maintains strong adherence and support for system prompts.
• —Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
• —Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
• —Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
• —Large Context Window: Supports a 256k context window.

Use Cases

Ideal for lightweight, real-time applications on edge or low-resource devices, such as:

• —Image captioning
• —Text classification
• —Real-time efficient translation
• —Data extraction
• —Short content generation
• —Fine-tuning and specialization
• —And more...

Bringing advanced AI capabilities to edge and distributed environments for embedded systems.

Ministral 3 Family

Other formats available here.

Benchmark Results

We compare Ministral 3 to similar sized models.

Reasoning

Instruct

Base

License

This model is licensed under the Apache 2.0 License.

You must not use this model in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.