tifkin/NextTokenPrediction

NextTokenPrediction

Start locally

uv sync

cd frontend
npm install
cd ..

./start.sh

Then open http://localhost:7860.

For frontend development with hot reload:

./start.sh --dev

Then open http://localhost:5173. API calls are proxied to the FastAPI server on port 7860.

Optional GGUF models

The model picker also supports Phi-4-mini-reasoning-Q2_K.gguf and Qwen3-0.6B-Q8_0.gguf through llama-cpp-python. The GGUF extra uses the upstream pre-built CPU wheel for llama-cpp-python.

uv sync --extra gguf

./start.sh

Phi-4-mini-reasoning-Q2_K.gguf and Qwen3-0.6B-Q8_0.gguf are downloaded into models/ the first time the default SmolLM2 model is loaded. Selecting either GGUF model directly also downloads it if it is not already present.

Set PHI4_GGUF_PREFETCH=0 to skip the background Phi download while keeping Phi in the model picker. Selecting Phi still downloads it on demand:

PHI4_GGUF_PREFETCH=0 ./start.sh

To keep the GGUF file elsewhere, set PHI4_GGUF_PATH before starting the app:

PHI4_GGUF_PATH=/absolute/path/to/Phi-4-mini-reasoning-Q2_K.gguf ./start.sh

To use a pre-downloaded Qwen file elsewhere, set QWEN3_GGUF_PATH:

QWEN3_GGUF_PATH=/absolute/path/to/Qwen3-0.6B-Q8_0.gguf ./start.sh

GGUF_N_CTX and GGUF_N_GPU_LAYERS configure both GGUF models. The Phi starter prompt uses the model's raw chat format, and the Qwen starter prompt enables /think, so the token panel can split context tokens from reasoning output tokens.

The Docker Space build sets PHI4_GGUF_PREFETCH=0 and runs python -m next_token_prediction.download_models, so only SmolLM2 and Qwen3 are cached in the image. Phi remains available in the picker and downloads only when selected. To build a Docker image with Phi cached too, pass --build-arg PHI4_GGUF_PREFETCH=1.

Test

uv sync
uv run pytest

cd frontend
npm install
npm test
npm run build
npm run test:e2e

Start with Docker

docker build -t next-token-prediction .
docker run --rm -p 7860:7860 next-token-prediction

Then open http://localhost:7860.