onprem-ai/Apertus-v1.5-70B-FP8

Apertus v1.5 70B -- FP8

Quantized checkpoints for Apertus 1.5 70B.

Full-precision FP8 quantization of swiss-ai/Apertus-v1.5-70B using the FP8DYNAMIC scheme on all linear projections (excluding lmhead, embeddings, and vision/audio tokenizers which remain in BF16).

Use this checkpoint when maximum quality is critical and GPU memory allows -- 71 GiB leaves approximately 14 GiB for KV cache, enough for roughly 60k context with fp8 KV cache on a 96 GiB card.

Thanks to: Swiss AI, blancsw, Anunay-Yadav, Oleg, Cyrilvallez, AryanAhadinia, robmsmt, and Neural Magic. See Attribution for details.

Usage (vLLM)

Until Apertus 1.5 support is available in a native vLLM release, use the `ghcr.io/swiss-ai/vllm_apertus_1.5_release` image. The build recipe and patched Dockerfile are available at github.com/swiss-ai/model-launch/tree/main/images/vllm_apertus_1.5.

# Download the checkpoint
huggingface-cli download onprem-ai/Apertus-v1.5-70B-FP8 \
  --local-dir ./apertus-v1.5-70b-fp8

# Serve it
docker run --gpus all -v ./apertus-v1.5-70b-fp8:/model \
  ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64 \
  --model /model \
  --served-model-name apertus-v1.5-70b \
  --host 0.0.0.0 --port 8080 \
  --dtype auto \
  --chat-template /model/chat_template.jinja \
  --tool-call-parser apertus --enable-auto-tool-choice \
  --reasoning-parser apertus

Performance

Measured on NVIDIA RTX PRO 6000 Blackwell (96 GiB VRAM) using llmapibenchmark.

Note: The NVFP4 checkpoint achieves ~50% higher decode throughput (31 TPS) due to 33% smaller weights reducing memory bandwidth pressure. For throughput-constrained deployments, prefer the NVFP4 checkpoint.

Quality

Benchmarked via lm-evaluation-harness (MMLU, 5-shot, 100 samples) on NVIDIA RTX PRO 6000 Blackwell. FP8_DYNAMIC produces no statistically significant quality loss -- a well-established result across dozens of open models.

Multimodal Support

All vision and audio capabilities from the original checkpoint are preserved. The quantization respects the ignore list: vision tokenizer, audio tokenizer, and embedding layers remain unquantized.

Quantization Process

Single-pass calibration-free quantization using llm-compressor 0.12.0:

scheme: FP8_DYNAMIC
targets: Linear
ignore: lm_head, model.language_model.embed_tokens, model.embed_tokens,
        re:.*vision_tokenizer.*, re:.*audio_tokenizer.*

Duration: ~4 minutes on RTX PRO 6000 Blackwell.

Limitations

• —vLLM only. SGLang does not support compressed-tensors checkpoints.
• —Requires vLLM with Apertus1p5 arch support. Use the ghcr.io/swiss-ai/vllm_apertus_1.5_release image or build from upstream PR #50496.
• —Requires 96 GiB GPU for full multimodal. For text-only inference with short context, the 71 GiB weights may fit on 80 GiB cards when multimodal is disabled.

Supported Hardware

Attribution

• —Original model: swiss-ai/Apertus-v1.5-70B by Swiss AI (including Oleg Lavrovsky (loleg), Cyrilvallez), Apache 2.0
• —GPU-native multimodal refactor: blancsw at Infomaniak -- foundational architecture work, Emu3.5 vision encoder, WavTokenizer audio encoder, apertus tool parser
• —Upstream PR: vllm-project/vllm #50496 by Anunay-Yadav -- GPU-native Apertus 1.5 multimodal encoders for upstream vLLM
• —Reasoning parser: AryanAhadinia
• —Double BOS-token fix: robmsmt
• —Quantization tooling: llm-compressor by Neural Magic
• —Benchmarking: llmapibenchmark by onprem-ai