dystrio/Llama-3.1-8B-Instruct-sculpt-throughput

dystrio/Llama-3.1-8B-Instruct-sculpt-throughput

26% smaller, +27% faster prefill, drop-in replacement. No custom kernels. No runtime changes.

Dystrio Sculpt structurally compresses transformer models, producing dense models that load with standard transformers — no custom code, no new ops, no deployment friction.

This is the Throughput tier of Llama 3.1 8B Instruct.

Quick Start

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("dystrio/Llama-3.1-8B-Instruct-sculpt-throughput", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("dystrio/Llama-3.1-8B-Instruct-sculpt-throughput")

inputs = tokenizer("The future of AI inference is", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Benchmark Results

All tiers compiled from Llama 3.1 8B Instruct on A100 80GB, bf16:

Key Metrics (this model)

All Sculpt Tiers

What is Dystrio Sculpt?

Dystrio Sculpt compiles transformer models into smaller, faster variants. Output models:

• —Are dense (not sparse) — standard architecture, fewer parameters
• —Load with standard HuggingFace Transformers — no custom code needed
• —Require no custom kernels and no runtime changes
• —Work as a one-step compile before deployment
• —Stack with quantization (AWQ, GPTQ, GGUF) for compound savings

Compatibility

• —✅ HuggingFace Transformers
• —✅ vLLM
• —✅ TGI (Text Generation Inference)
• —✅ llama.cpp / GGUF conversion
• —✅ AWQ / GPTQ quantization
• —✅ Any framework that loads standard safetensors

Benchmark Environment

• —GPU: NVIDIA A100-SXM4-80GB
• —dtype: bf16
• —Torch: 2.10.0+cu128
• —Transformers: 5.3.0
• —Deterministic: True
• —Single-GPU, standard HuggingFace Transformers, no custom kernels.

Metric Definitions

• —PPL ratio: WikiText-103 perplexity relative to baseline. <1.0 = quality improved.
• —Prefill TPS: Tokens per second during prompt encoding (higher = faster).
• —TTFT p95: Time to first token at 95th percentile (lower = faster).
• —Decode TPS: Tokens per second during generation (higher = faster).
• —Weights (GB): Model parameter memory (deterministic, runtime-independent).

Citation

@misc{dystrio_sculpt_2026,
  title={Dystrio Sculpt: Structural Compilation for Transformer LLMs},
  author={Dystrio},
  year={2026},
  url={https://huggingface.co/dystrio}
}

Downstream Benchmarks (lm-eval)

Evaluated with lm-eval-harness on A100-80GB, bf16, zero-shot.