JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16

<p align="center"> <img src="logo.png" alt="AIOne-Agent" width="160"/> </p>

<h1 align="center">AIOne-Agent-52B-A36B-it-NVFP4A16</h1>

<p align="center"> <b>NVFP4A16 weight-only quantization of AIOne-Agent-52B-A36B-it for memory-efficient deployment on NVIDIA Blackwell GPUs with vLLM.</b> </p>

Model Description

AIOne-Agent-52B-A36B-it-NVFP4A16 is a 4-bit weight-only quantization of JDONE-Research/AIOne-Agent-52B-A36B-it, a Korean multimodal Mixture-of-Experts model based on Gemma 4 31B IT (text backbone ~52 B, ~36 B active per token). The text-decoder dense Linear weights are compressed to NVFP4 (4-bit floating point), while activations stay in BF16 — so the model preserves the language and reasoning quality of the source model while drastically reducing VRAM footprint.

Vision tower preserved. The SigLIP-style vision tower, the multimodal embedding projection, and the LM head are kept in BF16 to keep image and video understanding bit-exact with the unquantized model.

The name follows the Gemma 4 convention (google/gemma-4-26B-A4B-it) with an -NVFP4A16 suffix to denote 4-bit weight / 16-bit activation quantization: 52B is the text-backbone parameter count, A36B is the per-token active parameter count, and the vision encoder (0.57 B) is reported separately.

Key Capabilities

• —Korean reasoning and instruction following.
• —Image understanding (caption, VQA, document understanding).
• —Video understanding (frame-by-frame reasoning).
• —Long-context document QA in Korean.
• —Bilingual: Korean (primary) + English.

All capabilities are inherited from AIOne-Agent-52B-A36B-it; only the storage precision of the dense text-decoder weights changes.

Quick Start

Transformers

Note. When loading the compressed model with transformers you may want to set TORCHDYNAMO_DISABLE=1 (or raise torch._dynamo.config.cache_size_limit) because the FP4 unpack kernel is torch.compile-decorated and Gemma 4's many distinct Linear shapes can otherwise trip the default recompile limit. vLLM does not need this — it uses native NVFP4 kernels.

import os
os.environ["TORCHDYNAMO_DISABLE"] = "1"

import torch
from transformers import AutoProcessor, Gemma4ForConditionalGeneration

MODEL_ID = "JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16"

model = Gemma4ForConditionalGeneration.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(MODEL_ID)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "file:///path/to/image.jpg"},
            {"type": "text", "text": "이 사진에 무엇이 보이나요? 한국어로 답해주세요."},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

generated = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(
    processor.tokenizer.decode(
        generated[0, inputs.input_ids.shape[1]:], skip_special_tokens=True
    )
)

vLLM (recommended for serving)

vllm serve JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16 \
    --quantization compressed-tensors \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len 32768 \
    --tensor-parallel-size 1

model: JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16
quantization: compressed-tensors  # NVFP4A16 weights
kv_cache_dtype: fp8_e4m3          # FP8 KV cache (optional)
gpu_memory_utilization: 0.9
max_model_len: 32768
tensor_parallel_size: 1
enable_prefix_caching: true
enable_chunked_prefill: true

Quantization Recipe

This model was quantized with llm-compressor using the compressed-tensors nvfp4-pack-quantized format.

default_stage:
  default_modifiers:
    GPTQModifier:
      targets: [Linear]
      ignore:
        - lm_head
        - "re:.*\\.experts\\..*"
        - "re:.*\\.router\\..*"
        - "re:.*vision_tower.*"
        - "re:.*embed_vision.*"
        - "re:.*multi_modal_projector.*"
      scheme: NVFP4A16
      block_size: 128
      dampening_frac: 0.01
      actorder: static
      offload_hessians: false
      sequential_targets: [Gemma4TextDecoderLayer]

What is quantized vs. preserved

Model Specs

Hardware Recommendations

NVFP4 weight-only quantization is designed for NVIDIA Blackwell-class GPUs (sm120, e.g. RTX PRO 6000, B100/B200) and is supported by recent vLLM builds with --quantization compressed-tensors. The model also runs on Hopper (H100/H200) at BF16 dequantized speed.

Files in This Repository

Intended Use

• —Korean enterprise agent backend (long-context tool use, RAG, multi-turn reasoning).
• —Image and video understanding with Korean output.
• —Document QA in Korean.

Out-of-Scope Use

• —Sole-source decision-making with legal consequences.
• —Automated use of force or coercive control based purely on this model's output.
• —Any media analysis that infringes on personal privacy, image rights, or applicable data-protection laws.

License

This model is released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0) license.

• —Commercial use, redistribution, and modification are permitted, subject to the conditions of the Apache 2.0 License (including preservation of copyright notices).
• —Provided "as is" without warranties or conditions of any kind.

Citation

@misc{aione_agent_52b_a36b_it_nvfp4,
  title        = {AIOne-Agent-52B-A36B-it-NVFP4A16: NVFP4 Weight-Only Quantization of a Korean Sparse-MoE Multimodal Model},
  author       = {JDONE Research},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/JDONE-Research/AIOne-Agent-52B-A36B-it-NVFP4A16}},
  note         = {Quantized with llm-compressor}
}