prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8

Q3.5-9B-DS-v4-Flash-v2.0-fp8

Q3.5-9B-DS-v4-Flash-v2.0-fp8 is an efficient 8-bit floating point (FP8_DYNAMIC) quantized checkpoint of prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0 (derived from Qwen/Qwen3.5-9B). Quantized via LLM Compressor into the compressed-tensors format, this release cuts VRAM usage by ~50% while preserving high reasoning fidelity across math, science, code, and long-context analysis. This model is an experimental research release. Due to distillation and abliterated alignment techniques, it may exhibit unexpected behaviors or unfiltered reasoning artifacts.

Model Overview

Quantization Details

Quantization was performed using LLM Compressor with dynamic per-tensor activation scaling applied to standard Linear projections. Sensitive layers—including input embeddings, the LM head, linear attention mechanisms, and vision blocks—were excluded from quantization to maintain mathematical reasoning integrity and prevent output degradation.

Quantization Recipe

default_stage:
  default_modifiers:
    QuantizationModifier:
      targets: [Linear]
      ignore:
        - 're:.*lm_head'
        - 're:.*embed_tokens$'
        - 're:.*visual.*'
        - 're:.*model.visual.*'
        - 're:.*linear_attn.*'
      scheme: FP8_DYNAMIC
      bypass_divisibility_checks: false
      requires_calibration_data: false

Key Highlights

• —High-Throughput FP8: Reduces memory footprint to ~9–10 GB, enabling fast local execution on consumer GPUs (RTX 3090/4090, L40S, A100, H100).
• —DeepSeek V4 Flash Traces: Fine-tuned on ~3K long-context DeepSeek V4 Flash reasoning traces covering complex mathematics, technical coding, and analytical benchmarks.
• —Preserved Sensitivity: Unquantized attention projection exceptions (linear_attn) and heads prevent numerical drift during extended multi-step generation.
• —Native vLLM & Transformers Compatibility: Direct drop-in support via the compressed-tensors standard.

Quick Start & Inference

1. High-Throughput Serving with vLLM (Recommended)

Install vLLM:

pip install vllm

Launch an OpenAI-compatible API server:

vllm serve prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code

Run inference via Python:

from vllm import LLM, SamplingParams

llm = LLM(
    model="prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8",
    trust_remote_code=True,
    max_model_len=32768
)

sampling_params = SamplingParams(
    temperature=0.6,
    top_p=0.95,
    max_tokens=2048
)

messages = [
    {"role": "user", "content": "Solve the following problem step-by-step: Let f(x) = x^3 - 3x + 1. Find the number of distinct real roots in the interval [-2, 2]."}
]

outputs = llm.chat(messages, sampling_params)
print(outputs[0].outputs[0].text)

2. Transformers & Compressed-Tensors

pip install transformers compressed-tensors accelerate torch

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "prithivMLmods/Q3.5-9B-DS-v4-Flash-v2.0-fp8"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True
)

messages = [
    {
        "role": "user",
        "content": "Explain how multi-head latent attention reduces KV cache overhead in large language models."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.95,
        do_sample=True
    )

print(
    tokenizer.decode(
        outputs[0][inputs.shape[-1]:],
        skip_special_tokens=True
    )
)

Model Files & Quantizations

Intended Use

• —Reasoning Research: High-efficiency research into long-context reasoning chains and distillation behavior.
• —Mathematical & Scientific Problem Solving: Structured multi-step derivations with lower compute latency.
• —Constrained VRAM Deployments: Single-GPU local or edge deployment setups requiring 32K context windows without requiring 24GB+ FP16 allocations.

Limitations & Risks

• —Experimental Output: The underlying model utilizes abliteration and multi-stage distillation; refusal behaviors may be significantly minimized.
• —Quantization Artifacts: Although FP8 dynamic scaling retains high fidelity, small numerical discrepancies can occasionally manifest in long reasoning chains.
• —Hardware Support: FP8 native hardware speedups require NVIDIA Ada Lovelace, Hopper, or newer architectures. Older architectures fall back to emulated or dequantized kernels.

Acknowledgements

• —[Qwen Team](https://huggingface.co/Qwen): For the foundational Qwen/Qwen3.5-9B architecture.
• —[vLLM Project](https://github.com/vllm-project/llm-compressor): For llm-compressor and the compressed-tensors specification.
• —[DeepSeek AI](https://deepseek.com): For foundational inspiration and distillation trace topologies.