TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors

Behemoth-X-123B-v2.1 — Quantized (compressed-tensors for vLLM)

This repository provides quantized runtime builds of [TheDrummer/Behemoth-X-123B-v2.1](https://huggingface.co/TheDrummer/Behemoth-X-123B-v2.1), repackaged for vLLM using the compressed-tensors format.

TL;DR - Quantized with branch W4A16 (INT4 weights / A16 activations). - Serve with vLLM using --quantization compressed-tensors. - Calibration uses a chat-formatted dataset; 512 samples at 2048 tokens max seq length. :contentReference[oaicite:0]{index=0}

Revisions & Branches

The `main` branch is a landing page (model card + links). All runnable artifacts live under per-revision branches.

• —main — placeholder / landing page
• —W4A16 — 4-bit weights / 16-bit activations builds and runtime assets

Quick links

• —main: https://huggingface.co/TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors/tree/main
• —W4A16: https://huggingface.co/TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors/tree/W4A16

What’s inside (per revision)

• —Sharded quantized weights (*.safetensors) + index (model.safetensors.index.json)
• —config.json with compressed-tensors metadata (weight_format, quantization, quantization_config, etc.)
• —Tokenizer artifacts (tokenizer.json, tokenizer.model, merges/vocab if applicable)
• —Optional: chat_template.jinja (inherits the parent finetune’s chat style)

Exact files can vary by branch; see Files and versions for each revision.

Quantization recipe (based on the provided script; parameterized for W4A16)

• —Method: AWQ via llm-compressor (one-shot), targeting Linear layers, with lm_head ignored during quantization. :contentReference[oaicite:1]{index=1}
• —Calibration dataset: neuralmagic/LLM_compression_calibration (split: train), which contains message-structured conversations rendered with tokenizer.apply_chat_template. :contentReference[oaicite:2]{index=2}
• —Calibration samples: 512 (num_calibration_samples=512). :contentReference[oaicite:3]{index=3}
• —Max calibration sequence length: 2048 (max_seq_length=2048). :contentReference[oaicite:4]{index=4}
• —Group size: 128 groups for weight quantization. :contentReference[oaicite:5]{index=5}
• —Export: saved with save_compressed=True so vLLM reads compressed-tensors metadata. :contentReference[oaicite:6]{index=6}

The attached script shows an INT8 configuration; this release applies the same pipeline with num_bits=4 for the W4A16 branch (INT4 weights, A16 activations). All other knobs remain as above (dataset, samples, sequence length, targets, ignore list, group size). :contentReference[oaicite:7]{index=7}

Quickstart — vLLM (compressed-tensors)

Install vLLM (recent version recommended):

pip install vllm

Serve (adjust to your hardware):

CUDAVISIBLEDEVICES=0,1,2,3,4,5,6,7 \ vllm serve TheHouseOfTheDude/Behemoth-X-123B-v2.1_Compressed-Tensors \ --quantization compressed-tensors \ --tensor-parallel-size 8 \ --max-model-len 32768 \ --gpu-memory-utilization 0.70 \ --dtype bfloat16

Query (OpenAI-compatible Chat Completions):

curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "TheHouseOfTheDude/Behemoth-X-123B-v2.1Compressed-Tensors", "messages": [ {"role":"system","content":"You are Behemoth-X — helpful, precise, and safe."}, {"role":"user","content":"Give three robust strategies for long-context retrieval."} ], "maxtokens": 512, "temperature": 0.7, "top_p": 0.95 }'

Note: compressed-tensors is a vLLM runtime format. Loading directly with vanilla 🤗 Transformers is not supported. For Transformers, use a compatible quant (e.g., GPTQ/AWQ export) or full-precision weights.

Prompting / Chat Template

This package follows the parent finetune’s chat conventions. If a chat_template.jinja is present, apply_chat_template will pick it up automatically.

Guidelines:

• —Keep a concise system message to set behavior/tone.
• —Structure user prompts clearly; enumerate steps for multi-part tasks.

Intended use

• —General instruction-following assistants
• —Long-form drafting & summarization
• —RAG/agent pipelines (pair with a retriever/tool layer)

Always review the parent/base model’s license and evaluate on your domain before production use.

Lineage

• —Finetuned parent: https://huggingface.co/TheDrummer/Behemoth-X-123B-v2.1
• —This repo: Quantized child of the finetune (compressed-tensors for vLLM)

Hardware tips (rule-of-thumb)

• —123B-class models benefit from multi-GPU (tensor parallel) for throughput.
• —Long contexts are KV-cache heavy — tune --max-model-len and batch size.
• —Use BF16 on GPUs with native support; otherwise FP16.
• —Enable P2P/NVLink where possible; consider CUDA Graphs if stable.

Changelog

• —v2.1 (current) — Initial compressed-tensors release; branch W4A16 published; model card marked Quantized.