thanet-s/Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10

Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10

This is a DGX Spark optimized quant for vLLM: a GB10-oriented compressed-tensors quantization of llmfan46/Ornith-1.0-35B-uncensored-heretic for vLLM inference.

The source model is llmfan46/Ornith-1.0-35B-uncensored-heretic.

Quantization

• —Dense attention and linear-attention projections: FP8 W8A8.
• —Routed MoE experts and shared expert projections: NVFP4 W4A4.
• —Preserved in BF16: embeddings, lm_head, router gates, visual modules, norms, Conv1D, A_log, dt_bias, and non-target tensors.
• —Visual encoder: copied from the source checkpoint and preserved in BF16 for multimodal/image support.
• —Calibration data: HuggingFaceH4/ultrachat_200k, train_sft.
• —Calibration samples: 512.
• —Calibration sequence length: 2048.
• —Pipeline: sequential, CPU offload, all MoE experts calibrated.

The quantization groups use mutually exclusive explicit target regexes.

Output

• —Format: mixed-precision.
• —Quantization method: compressed-tensors.
• —Config groups: group_0, group_1.
• —Weight layout: sharded safetensors with model.safetensors.index.json.
• —Safetensors shards: 5.
• —Indexed tensors: 124376.
• —Max position embeddings: 262144.
• —MTP weights: not present in the inspected source checkpoint.

DGX Spark vLLM

This model is intended for the GB10 patched vLLM path used by demon-zombie/Qwen3.5-122B-A10B-NVFP4-FP8Dense-GB10.

Example:

docker run --gpus all -p 8000:8000 --ipc host \
  -v /opt/vllm-cache:/root/.cache/huggingface \
  -e CUBLASLT_WORKSPACE_SIZE=33554432 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:False \
  -e RUNAI_STREAMER_MEMORY_LIMIT=4294967296 \
  vllm/vllm-openai:v0.22.1 \
  --model thanet-s/Ornith-1.0-35B-uncensored-heretic-nvfp4-fp8dense-gb10 \
  --served-model-name Ornith-1.0-35B-uncensored-heretic \
  --kernel-config '{"moe_backend": "flashinfer_b12x"}' \
  --load-format runai_streamer \
  --gpu-memory-utilization 0.50 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4176 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

On DGX Spark / GB10, the vLLM image may still need the same CUTLASS DSL and flashinfer SM12x patch files described in the reference model card above.

DGX Spark benchmark

Measured on NVIDIA DGX Spark / GB10 with vllm/vllm-openai:v0.22.1, the GB10 CUTLASS DSL and flashinfer SM12x patches, --gpu-memory-utilization 0.50, --kv-cache-dtype fp8, --kernel-config {"moe_backend": "flashinfer_b12x"}, and --max-num-batched-tokens 4176.

• —vLLM log throughput during a warm single request: Avg generation throughput: 60.8 tokens/s.
• —Client-side measurement for the same warm request: 1024 completion tokens in 16.905s, about 60.58 tok/s.
• —First measured request after server readiness: 768 completion tokens in 13.676s, about 56.15 tok/s; vLLM log showed Avg generation throughput: 53.5 tokens/s.
• —Model load memory reported by vLLM: 21.04 GiB.
• —Max model length reported by vLLM: 262144 tokens.
• —GPU KV cache size at --gpu-memory-utilization 0.50: 3,344,321 tokens.
• —Maximum concurrency reported by vLLM for 262144-token requests: 12.76x.

These figures are from one local DGX Spark run and may change with prompt shape, sampling settings, vLLM version, patch versions, and CUDA graph/cache warmup state.

Status

Local DGX Spark conversion validation:

• —Source: llmfan46/Ornith-1.0-35B-uncensored-heretic.
• —Conversion completed with the same NVFP4-FP8Dense GB10 recipe used for the non-uncensored Ornith 35B build.
• —verify-output.py passed after vLLM config patching and visual tensor merge.
• —Visual tensors preserved: 333 model.visual.* tensors copied from the source checkpoint.
• —vLLM serving smoke test passed on DGX Spark / GB10 after upload.