CoolFace
Modelpublic

saintlits/Ornstein3.6-27B-MTP-NSC-ACE-SABER-oQ6-mtp

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face
0likes9downloads
Model Card

Ornstein3.6-27B-MTP-NSC-ACE-SABER (oQ6-mtp)

Quantized variant of [Ornstein3.6-27B-MTP-NSC-ACE-SABER](https://huggingface.co/GestaltLabs/Ornstein3.6-27B-MTP-NSC-ACE-SABER)

Support this work on Ko-fi

Quantization Details

This is an oMLX Q6 quantized variant of Ornstein3.6-27B-MTP-NSC-ACE-SABER, optimized for local inference with minimal quality loss.

ParameterValue
Quantization methodoMLX (affine)
Base precisionQ6 (group_size=64)
Special layersQ8 for LM head, layer 49-53/60-63 attention/projection layers
FormatFull safetensors (5 shards)
Approx. size~23 GB total
MTP supportYes (verified)

Why Q6?

The Q6 quantization provides an excellent balance between model quality and memory footprint. Compared to Q8, it reduces VRAM usage by ~30% with negligible quality degradation. The selective Q8 treatment of critical layers (LM head, key projection matrices) helps preserve generation quality where it matters most.

How to Use

Transformers (PyTorch)

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "saintlits/Ornstein3.6-27B-MTP-NSC-ACE-SABER-oQ6-mtp",
    torch_dtype="auto",  # Will auto-detect bfloat16
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(
    "saintlits/Ornstein3.6-27B-MTP-NSC-ACE-SABER-oQ6-mtp"
)

# The model includes a chat_template.jinja for proper conversation formatting
messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=512)

llama.cpp (GGUF path)

For GGUF quantization, use the official GGUF repo: GestaltLabs/Ornstein3.6-27B-MTP-NSC-ACE-SABER-GGUF

bash
# Use a llama.cpp build with Qwen3.5/Qwen3.6 MTP support
llama-cli -m model.gguf --spec-type mtp -p "Hello!"

LM Studio

This model is compatible with LM Studio. Simply download the safetensors files and load them directly.

Intended Use

This is an experimental agentic/tool-calling checkpoint intended for research, local evaluation, and downstream experimentation. Validate behavior for your own task distribution before production use.

Training Notes

NSC-ACE (Neural Steering Committee for Agentic Co-Evolution) generates multiple internally steered rollouts for the same prompt and rewards convergence in tool-call structure across independently steered latent modes.

SABER (Compliance-first calibration) optimized for compliance first, then KLD and PPL retention. The selected checkpoint reports 92.54% compliance proxy (968/1046 eval prompts) with only 0.67% HarmBench ASR.

Ornstein SFT merged last using premium reasoning refinement (1 epoch, 50 steps, rank 32/32, dropout 0.05).

Related Repositories

License

Apache 2.0 (same as the original Ornstein3.6-27B-MTP-NSC-ACE-SABER and Qwen3.6-27B-MTP)

Acknowledgements