gigascake/Ornith-397B-EXL3-HQ-35bpw

Ornith-1.0-397B EXL3 --hq 3.5bpw (512 Experts, Mixed Precision)

EXL3 mixed-precision quantization with 512 MoE Experts fully preserved (no pruning) and Attention at 8bpw for enhanced multi-language token quality.

Base model: deepreinforce-ai/Ornith-1.0-397B (BF16, 794GB)

Quantization Specification

Mixed Precision Bit Allocation (--hq)

Model Scale

Serving (TabbyAPI + exllamav3)

This model uses the EXL3 format (exllamav3 only). SGLang, vLLM, and Transformers cannot load it.

Requirements

• —GPU: 2x 96GB VRAM (or equivalent total)
• —Python: 3.12+
• —exllamav3: 0.0.43+
• —TabbyAPI: latest

Quick Start

python main.py \
  --host 0.0.0.0 --port 28000 \
  --model-dir /path/to/models \
  --model-name Ornith-397B-EXL3-HQ-35bpw \
  --backend exllamav3 \
  --cache-mode 8,8 \
  --max-seq-len 262144 \
  --tensor-parallel true \
  --reasoning true \
  --reasoning-start-token "<think>" \
  --reasoning-end-token "</think>" \
  --tool-format qwen3_coder \
  --config config.yml

config.yml

model:
  model_dir: /path/to/models
  model_name: Ornith-397B-EXL3-HQ-35bpw
  backend: exllamav3
  max_seq_len: 262144
  cache_size: 262144
  cache_mode: 8,8
  gpu_split_auto: true
  tensor_parallel: true
  reasoning: true
  reasoning_start_token: "<think>"
  reasoning_end_token: "</think>"
  tool_format: qwen3_coder

sampling:
  default_temperature: 0.3
  default_top_p: 0.8
  default_top_k: 20

VRAM Usage (2x RTX PRO 6000 Blackwell 96GB)

GPU 0: ~90 GB / 96 GB
GPU 1: ~88 GB / 96 GB
Total: ~178 GB (model weights + KV cache + CUDA)

API Usage

from openai import OpenAI

client = OpenAI(base_url="http://localhost:28000/v1", api_key="YOUR_KEY")

response = client.chat.completions.create(
    model="Ornith-397B-EXL3-HQ-35bpw",
    messages=[{"role": "user", "content": "Write a Python REST API with Flask."}],
    temperature=0.3,
    max_tokens=2048,
)

msg = response.choices[0].message
print("Answer:", msg.content)
print("Reasoning:", getattr(msg, "reasoning_content", None))

Korean Language Patch (korean_fix.py)

Problem

Quantized MoE models using BPE tokenizers may produce U+FFFD (replacement character) for certain Korean words (실행, 결과, 선택, 아키텍처, etc.). This is caused by multi-byte Korean sub-tokens being structurally vulnerable to quantization noise — even at 8bpw attention precision.

Root cause: BPE byte-level sub-tokens for Korean syllables produce invalid UTF-8 sequences when the model predicts slightly wrong token IDs. This is a tokenizer-level issue, not a model quality issue.

Solution: Post-correction Module

korean_fix.py intercepts API responses and replaces U+FFFD patterns with context-appropriate Korean words.

Installation:

1. 1.Copy korean_fix.py to your TabbyAPI directory:

cp korean_fix.py /path/to/tabbyAPI/

1. 1.Patch TabbyAPI's exllamav3 backend to apply the fix. Add to backends/exllamav3/model.py:

# At the top of the file, after existing imports:
try:
    from korean_fix import fix_korean_ufffd
    def _fix_text(text):
        return fix_korean_ufffd(text) if text else text
except ImportError:
    def _fix_text(text):
        return text

# In the generation loop, before yielding chunks:
chunk = _fix_text(chunk)  # Apply Korean fix to each streaming chunk

# In handle_finish_chunk:
"full_text": _fix_text(full_text),  # Fix the complete response

1. 1.Restart TabbyAPI.

How It Works

Covered Words

실행 (execution)     결과 (result)       선택 (selection)    설계 (design)
적용 (application)   구현 (implementation) 확인 (check)       평가 (evaluation)
아키텍처 (architecture) 다운로드 (download)  페이지 (page)

Test Results (with korean_fix)

Agentic Coding Benchmark

Hardware: 2x RTX PRO 6000 Blackwell (SM120, 96GB x2) Engine: TabbyAPI + exllamav3, tensor-parallel Date: 2026-07-06

Summary

Detailed Results

Code Generation

Bug Fixing

Algorithm

Tool Calling

Multi-step & SQL

Inference Speed

Quantization Verification

Static Analysis

Conversion Details

Limitations

• —exllamav3 only: Cannot load with SGLang, vLLM, or Transformers. TabbyAPI required.
• —Inference speed: ~35-49 tok/s (slower than BF16 on high-end GPUs).
• —Text-only serving: Original is a VLM, but exllamav3 does not support vision inference. Vision encoder weights are not included.
• —Reasoning model: All responses include a <think>...</think> reasoning block. Final answer in content, reasoning in reasoning_content.
• —Korean U+FFFD: Without korean_fix.py patch, ~0.5% of Korean characters produce U+FFFD. Apply the patch described above for clean Korean output.

File Structure

Ornith-397B-EXL3-HQ-35bpw/
├── config.json
├── quantization_config.json
├── model.safetensors.index.json
├── model-00001-of-00060.safetensors
├── ...
├── model-00060-of-00060.safetensors
├── tokenizer.json
├── tokenizer_config.json
├── chat_template.jinja
├── generation_config.json
├── preprocessor_config.json
├── processor_config.json
├── video_preprocessor_config.json
├── vocab.json
└── README.md

Original Model

This is an EXL3 quantization of deepreinforce-ai/Ornith-1.0-397B.

Ornith-1.0-397B is a 397B MoE model (post-trained on Qwen 3.5) specialized for agentic coding, achieving SOTA on Terminal-Bench 2.1, SWE-Bench, and NL2Repo.

Citation

@misc{ornith_397b,
    title = {{Ornith-1.0-397B}: Agentic Coding, Open to All},
    url = {https://deep-reinforce.com/ornith_1_0.html},
    author = {{DeepReinforce Team}},
    year = {2026}
}

License

MIT (same as original model)