dystrio/Mistral-7B-v0.1-sculpt-balanced

What is Dystrio Sculpt?

Dystrio Sculpt produces dense compiled variants of existing models that:

• —reduce memory footprint
• —improve prefill throughput
• —remain runtime-agnostic
• —require no custom kernels
• —load with standard HuggingFace Transformers

Key Results

Compared to mistralai/Mistral-7B-v0.1 baseline on an A100 80GB:

• —Weights memory: -11% (Conservative) / -23% (Balanced)
• —RAG latency (TTFT p95): -7% / -14%
• —Decode throughput: ~flat
• —No runtime changes: no custom kernels, no new ops, standard transformers loading

Notes: TTFT includes prefill + first decode step. “Weights memory” is computed from parameter sizes (GiB) and is workload-independent.

Benchmark Results

Benchmark Environment

• —GPU: NVIDIA A100-SXM4-80GB
• —dtype: bf16
• —Torch: 2.10.0+cu128
• —Transformers: 5.2.0
• —Deterministic: False
• —Seed: 0
• —Single-GPU, Hugging Face Transformers, no custom kernels.

Metric Definitions

• —TTFT incl. prefill: Wall time from prompt submission to first generated token (prefill forward + first decode step). Per-prompt request-level measurement.
• —First decode step: Wall time of the first decode forward call only (post-prefill). Per-prompt request-level measurement.
• —Prefill/Decode TPS: Throughput from batched microbenchmark iterations (not request-level; used for throughput comparison only).
• —Weights (GiB): Model parameter memory only (sum of numel * element_size for all parameters). Deterministic and runtime-independent.
• —Post-load (GiB): torch.cuda.memory_allocated() immediately after model.eval() + torch.cuda.empty_cache(). Captures weights + framework overhead before any inference.
• —End-of-bench (GiB): torch.cuda.memory_allocated() at end of benchmark workload. Includes KV-cache and activations still held.
• —Peak (GiB): torch.cuda.max_memory_allocated() during benchmark. High-water mark for planning GPU headroom.