Modelpublic

azampatti/Ornith-1.0-35B-int4-AutoRound-SAR

sourceHugging Faceupdated 2mo agoView on Hugging Face

2likes59downloads

Model Card

Ornith-1.0-35B-A3B-int4-AutoRound-SAR

A 4-bit AutoRound quantized version of deepreinforce-ai/Ornith-1.0-35B, optimized for efficient inference on consumer hardware including the NVIDIA DGX Spark (GB10). Includes MTP Support

Model Details

Property	Value
Base Model	deepreinforce-ai/Ornith-1.0-35B
Architecture	Qwen3.5-MoE (Mixture of Experts)	MTP Heads Available
Quantization	int4 AutoRound (W4A16)
Bits	4-bit
Group Size	128
Symmetric	Yes
Calibration Dataset	NVIDIA OpenCodeInstruct
Quantization Tool	spark-auto-round v0.14.3
Context Length	262144 tokens
Model Size	~20 GB (quantized)

Architecture

—Type: Qwen3.5-MoE with Vision (multimodal)
—Hidden Size: 2,048
—Num Layers: 40
—Num Experts: 256
—Experts Per Token: 8
—Attention Heads: 16 (full attention every 4th layer, linear attention otherwise)
—Vocab Size: 248,320

Quantization Details

This model was quantized using AutoRound, an advanced quantization technique from Intel that uses signed gradient descent to jointly optimize weight rounding and clipping ranges.

Key Settings

--batch_size 8
--iters 1000
--nsamples 512
--seqlen 2048
--dataset opencode-instruct
--group_size 128

Quantization Quality

Metric	Value
Peak RAM	111.31 GB
Peak VRAM	28.00 GB
Layers Passed	22/40 (55%)
Layers Warning	18/40 (45%)

The model retains strong quality with all layers meeting minimum thresholds. Layers 22-39 show slightly elevated sensitivity (cosine similarity 0.986-0.990) which is typical for deeper MoE layers.

Note: Shared expert gates are preserved in FP16 to maintain MoE routing accuracy.

Usage with Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "whpthomas/Ornith-1.0-35B-int4-AutoRound"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
)

# Generate text
messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Usage with vLLM

For optimal performance, use vLLM with the recommended serving configuration:

Installation

bash

pip install vllm

Basic Serving

bash

vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \
  --served-model-name ornith-1.0-35b \
  --max-model-len 196608 \
  --gpu-memory-utilization 0.55 \
  --load-format auto \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --speculative-config '{{"method":"mtp","num_speculative_tokens":3}}'

Recommended Production Configuration

For DGX Spark (GB10) with optimal performance:

bash

vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \
  --served-model-name ornith-1.0-35b \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.55 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --optimization-level 3 \
  --performance-mode throughput \
  --load-format instanttensor \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --default-chat-template-kwargs '{"preserve_thinking":true}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --generation-config auto \
  --speculative-config '{{"method":"mtp","num_speculative_tokens":3}}'
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":-1,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Environment Variables (DGX Spark)

bash

export TORCH_MATMUL_PRECISION=high
export NVIDIA_FORWARD_COMPAT=1
export NVIDIA_DISABLE_REQUIRE=1
export CUDA_DEVICE_MAX_CONNECTIONS=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

Tool Calling

This model supports tool calling with the Qwen3 coder parser. When using vLLM with --enable-auto-tool-choice --tool-call-parser qwen3_coder, the model can invoke tools and return structured function calls.

Reasoning

The model supports extended thinking with the Qwen3 reasoning parser. Use --reasoning-parser qwen3 and set preserve_thinking: true in chat template kwargs to enable reasoning traces.

Performance Characteristics

Based on testing with the Qwen3.5-MoE architecture on DGX Spark:

Metric	Value
Throughput	~60-65 t/s
Throughput (MTP)	~80-91 t/s
Latency (Time to First Token)	~100-200ms

Performance varies based on context length, batch size, and hardware configuration.

Citation

If you use this quantized model, please cite both the base model and the quantization tool:

bibtex

@misc{ornith2025,
  title={Ornith-1.0-35B},
  author={DeepReinforce AI},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B}
}

@misc{sparkautaround2025,
  title={Spark Auto Round},
  author={whpthomas},
  year={2025},
  publisher={GitHub},
  url={https://github.com/whpthomas/spark-auto-round}
}

Acknowledgments

—Base Model: deepreinforce-ai/Ornith-1.0-35B - The original bf16 model
—Quantization: Intel auto-round - The underlying quantization framework
—Spark Auto Round: spark-auto-round - GB10-optimized quantization wrapper
—vLLM Serving: spark-vllm-docker - Community vLLM Docker configuration for DGX Spark

License

The quantized model inherits the license of the base model. Please refer to the original model card for license details.

Model Type

This is an int4 AutoRound quantized model. It requires:

—A CUDA-capable GPU
—vLLM (recommended) or transformers with auto-round support
—Sufficient GPU memory (~20GB+ depending on context length)