reyden009/speculative-decoding-lab

1---2license: mit3tags:4  - speculative-decoding5  - llm-inference6  - benchmark7  - quantization8  - gguf9  - llama.cpp10---11 12# Speculative Decoding Lab13 14Systematic, reproducible empirical evaluation of **modern speculative decoding**15(EAGLE-3, DFlash, DSpark vs. vanilla and model-free) on an **RTX 4060 Ti 16 GB**16with **Gemma 4 12B** and **Qwen3-8B** GGUF-quantized models: acceptance by17cognitive domain, quantization effects, and hardware break-even maps.18 19> Paper: [PDF](paper/paper.pdf) · [arXiv-ready source bundle](paper/arxiv-bundle.tar.gz) · [source (markdown)](paper/paper.md) · [paper page](https://huggingface.co/spaces/reyden009/speculative-decoding-paper)20 21## Hardware used22 23The study was developed and benchmarked on this machine, so all reported24numbers (speedups, acceptance, break-even) are tied to these specs. The code25itself is portable: it only needs llama.cpp (CUDA build) and enough VRAM for26the target model, and can be re-run on any other hardware.27 28| Component | Detail |29| --- | --- |30| GPU | NVIDIA RTX 4060 Ti **16 GB** (Ada Lovelace, compute 8.9) |31| CPU | Intel i5-12400 (6C/12T, 4.4 GHz) |32| RAM / Disk | 31 GB / ~185 GB free |33| OS | Ubuntu 24.04 · driver 595.84 |34 35Environment principle: **everything user-space, no sudo** (uv + CUDA runfile in `$HOME`).36 37## Stack38 39- **Python 3.12 + uv** — `pyproject.toml` + `uv.lock` (numpy, ruff, pyright, cmake, ninja).40- **CUDA toolkit 12.8.1** in `~/cuda` (runfile `--toolkitpath`). *Note: the `nvcc` binary is not on PyPI (the wheels only ship ptxas).*41- **llama.cpp master 0713275 (v22)** (CUDA build) in `~/llama.cpp` (built from source; no CUDA binaries are published for Linux).42- **GGUF models** in `models/` (not versioned; downloaded with `curl -C -`).43 44## Setup (once)45 46```bash47uv sync                        # create .venv with all dependencies48source scripts/spec-env.sh     # PATH + CUDA_HOME + env vars (idempotent)49```50 51## Quality (required before committing)52 53```bash54uv run ruff check .            # lint55uv run ruff format --check .   # formatting56uv run pyright                 # types57```58 59## Execution regime60 61Overnight sessions with **pause/resume**: `scripts/bench_accept.py --resume`62(incremental JSONL per prompt; server-based runner with α/τ), downloads with63`scripts/dl.py`, runs in tmux/nohup. Measurement campaigns are launched with64`scripts/run_final_chain.sh`, `scripts/run_curves_chain.sh` and65`scripts/run_ksweep_chain.sh`; final analysis with `scripts/analyze_final.py`.66 67## Structure68 69```70scripts/       environment, downloads, benchmark runner and analysis (bench_accept.py, analyze_final.py)71experiments/   runs/ (raw, not versioned) and prompts/ (JSONL subsamples)72models/        GGUFs (not versioned)73```74 75## For agents76 77Read [`AGENTS.md`](AGENTS.md) before working in this repo (conventions, git78rules, sessions and handoff).79 80## License81 82MIT — see [`LICENSE`](LICENSE).83