ayesmaj/syntropic-live-gpu-compression

Syntropic Live GPU Compression Demo

A premium, locally-runnable dashboard that benchmarks KV-cache compression on your own GPU. Built for in-room client demos: every number on the screen is measured live, never hardcoded.

1. What this demo does

• —Detects your local compute backend (CUDA → Apple MPS → CPU).
• —Loads a real transformer model (GPT-2 or DistilGPT-2) onto that backend.
• —Generates N tokens with use_cache=True and captures the KV cache.
• —Runs three Syntropic compression modes against the captured KV tensors: Conservative (4-bit), Standard (2-bit), Aggressive (2-bit + per-tensor metadata).
• —Reports measured compression ratios, reconstructed-tensor cosine similarity, KV cache memory in MB, and peak GPU memory.
• —Translates the measured ratio into an estimated annual cost delta for a configurable KV-cache-limited workload (GPUs, $/hr, hours, days).
• —Exports JSON, CSV, and a plain-text client summary.

2. What this demo proves

• —That an orthogonal rotation + low-bit quantization yields measurable compression of real KV tensors on your hardware.
• —That the reconstructed KV tensors retain high cosine similarity to the FP16 originals at all three modes.
• —That on KV-cache-limited workloads, the measured ratio translates into a proportional reduction in GPUs and cost.

3. What this demo does not prove

• —It does not re-decode text from compressed KV cache, so it does not prove output identity. The dashboard surfaces this explicitly.
• —It does not benchmark throughput / latency in a serving stack (vLLM, TGI, etc.) — that is a separate measurement.
• —The annual-savings figure is an estimate. Real infrastructure savings depend on model size, batch shape, context length, traffic pattern, and the serving architecture.

4. Installation

# Clone or copy this folder, then:
python -m venv .venv

Windows

.venv\Scripts\activate

macOS / Linux

source .venv/bin/activate

Install dependencies

pip install -r requirements.txt

5. Windows setup (NVIDIA / CUDA)

The default torch wheel installed by pip may be CPU-only depending on your pip index. If you have an NVIDIA RTX card, install the CUDA build from the official PyTorch site:

<https://pytorch.org/get-started/locally/>

Example for CUDA 12.4 (adjust to your driver):

pip install torch --index-url https://download.pytorch.org/whl/cu124

Verify after install:

python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda)"

If it prints True <version> you're good.

6. macOS setup (Apple Silicon)

pip install -r requirements.txt

PyTorch on macOS ships with MPS support by default. Verify:

python -c "import torch; print(torch.backends.mps.is_available())"

7. Running the app

streamlit run app.py

Streamlit will open http://localhost:8501 in your browser. The hardware card at the top shows the detected backend. Configure the run in the sidebar and click Run Live GPU Test.

8. GPU troubleshooting

Verify GPU detection independently:

python -c "from engine.gpu import detect_device; print(detect_device())"

9. Client demo instructions

A field-tested walkthrough for an in-room client demo:

1. 1.Open the app before the call so the model is pre-cached.
2. 2.Show the Hardware card first — point out the GPU name and CUDA version. This builds trust that it's a real local run.
3. 3.Click Run Live GPU Test. Narrate the progress: "we're loading the model on your GPU, generating 100 tokens, capturing the KV cache, running three compression passes."
4. 4.When the dashboard appears, walk through the KPI row left to right: ratio → reduction → similarity → device → savings.
5. 5.Open the Generated output vs. compression analysis panel and explicitly call out the honesty note: "we are not re-decoding from compressed KV — we are measuring tensor reconstruction similarity." Technical buyers respect this. Bluffing here costs you the deal.
6. 6.Open the Business impact panel and tune the GPU count / $/hr to match the client's stack.
7. 7.Click Download Client Summary at the bottom. Send the .txt to the client during the call.
8. 8.Optional: open the H100 reference card and explain the 26.5× reference number is not measured on the laptop.

10. Project structure

syntropic-demo-app/
├── app.py                  # Streamlit dashboard
├── requirements.txt
├── README.md
└── engine/
    ├── __init__.py         # public re-exports
    ├── gpu.py              # device detection, memory snapshots
    ├── compression.py      # rotate-quantize-reconstruct + mode catalogue
    ├── metrics.py          # cost translation
    ├── demo_runner.py      # end-to-end orchestrator
    └── export.py           # JSON / CSV / plain-text exports

11. Honesty policy

This is a client-facing demo. Marketing language is deliberately avoided. The codebase enforces the following rules:

• —No hardcoded ratios, GPU names, or quality numbers anywhere.
• —Output is described as "FP16 baseline" — never as "compressed-regenerated" unless the app actually re-decodes from compressed KV.
• —Quality is described as "KV reconstruction similarity," never as "zero quality loss."
• —Cost figures are described as "estimated," tied to the user's inputs, and qualified by a note about workload dependence.
• —The H100 26.5× number is shown only on a card explicitly labeled as a reference, not measured on the local machine.