JongYeop/Mistral-7B-Instruct-v0.2-FP4-W4A4

Mistral-7B-Instruct-v0.2-FP4-W4A4

Model Description

This is an NVFP4 (NVIDIA FP4) quantized version of mistralai/Mistral-7B-Instruct-v0.2 using the compressed-tensors quantization method.

• —Base Model: mistralai/Mistral-7B-Instruct-v0.2
• —Quantization Method: compressed-tensors
• —Quantization Type: NVFP4 W4A4 (4-bit Weight and Activation)
• —Model Size: ~4.2GB (compared to ~14GB for BF16)
• —Compression Ratio: ~3.3x

Quantization Configuration

This model uses NVFP4 (NVIDIA FP4) quantization with grouped quantization for both weights and activations:

Weights

• —Precision: NVFP4 (4-bit floating point)
• —Strategy: Tensor-group (grouped quantization)
• —Group Size: 16
• —Symmetric: Yes
• —Dynamic: No (static quantization)
• —Observer: MinMax

Activations

• —Precision: NVFP4 (4-bit floating point)
• —Strategy: Tensor-group (grouped quantization)
• —Group Size: 16
• —Symmetric: Yes
• —Dynamic: Local (dynamic quantization with local calibration)
• —Observer: MinMax

Other Details

• —Format: nvfp4-pack-quantized (packed 4-bit format)
• —KV Cache: Not quantized
• —Ignored Layers: lm_head
• —Target Layers: Linear layers
• —Quantization Version: 0.11.0

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "JongYeop/Mistral-7B-Instruct-v0.2-FP4-W4A4"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

# Generate text
messages = [
    {"role": "user", "content": "What is machine learning?"}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

Model Architecture

• —Architecture: MistralForCausalLM
• —Hidden Size: 4096
• —Intermediate Size: 14336
• —Number of Layers: 32
• —Number of Attention Heads: 32
• —Number of KV Heads: 8
• —Vocabulary Size: 32000
• —Max Position Embeddings: 32768

Intended Use

This quantized model is intended for efficient inference with significantly reduced memory footprint while maintaining reasonable performance. It is suitable for:

• —Resource-constrained environments
• —Edge deployment
• —Applications requiring minimal memory usage
• —High throughput scenarios
• —GPU inference with FP4 support

Limitations

• —FP4 quantization may result in more accuracy loss compared to FP8 or INT8 quantization
• —Best performance is achieved on hardware with native FP4 support (e.g., NVIDIA H100, Ada Lovelace, Blackwell GPUs)
• —Dynamic activation quantization may introduce additional runtime overhead
• —Grouped quantization requires compatible inference engines

Performance Notes

• —Memory Usage: ~3.3x reduction compared to BF16
• —Speed: Requires hardware with FP4 tensor core support for optimal performance
• —Accuracy: May experience some degradation compared to higher precision formats

Citation

If you use this model, please cite the original Mistral paper and the compressed-tensors library.

License

Same as the base model: Apache 2.0