Frosty40/Qwen-AgentWorld-35B-A3B-NVFP4

Qwen-AgentWorld-35B-A3B-NVFP4

NVFP4-quantized Qwen-AgentWorld-35B-A3B for vLLM. Uses FlashInfer Cutlass NVFP4 GEMM kernels, compressed-tensors quantization, and the Qwen3_5MoeForConditionalGeneration architecture.

Benchmarks (NVIDIA GB10, 131 GB VRAM, vLLM v0.23.0)

In Testing

temperature=0.6, max 1024 tokens, enforce_eager.

Important: The model outputs <think>...</think> blocks (Qwen3.5 thinking format). At temperature=0 the greedy sampling may stall in the thinking phase. Use temperature>=0.6 (the model's default) for reliable answer extraction. For agent workloads, use --max-tokens >= 1024 to accommodate think+answer.

Quick Start

pip install vllm flashinfer

vllm serve ./nvfp4_model \
    --enforce-eager \
    --max-model-len 4096 \
    --port 8000

Query:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"./nvfp4_model","messages":[{"role":"user","content":"Hello!"}],"max_tokens":128,"temperature":0}'

Credits

• —Qwen-AgentWorld-35B-A3B — base model by the Qwen team
• —vLLM — serving engine and NVFP4 quantization support
• —FlashInfer — NVFP4 GEMM kernels (FlashInferCutlassNvFp4LinearKernel)
• —llm-compressor (Neural Magic) — compressed-tensors NVFP4 compression pipeline
• —NVIDIA — GB10 hardware, CUDA, NVFP4 format
• —FlashAttention — FLASH_ATTN attention backend