jpanasuk/tabby-tavern-stack

Tabby Tavern UNIFIED

Taproot + Basecamp + Tabby Tavern on one Compose project and one Docker network.

Default GPU engine is Qwen3.5-9B (llama.cpp CUDA). TabbyAPI EXL3 and Ollama are real, and they sit behind --profile alternate-inference so a 12 GB card does not load three engines at once.

This is a private-lab / portfolio stack — production-shaped, not a multi-tenant SaaS product and not a hosted inference endpoint.

Lab baseline hardware: NVIDIA GeForce RTX 4070 (Linux + Docker Compose + NVIDIA Container Toolkit, WSL2 Ubuntu).

Release v3.0.0 UNIFIED (Aug 2026) matches the live lab after Hermes reconciled Taproot and Basecamp into this tree: 15 default services on tabby-tavern_ai-network, Qwen healthy, MCPO /docs 200, character cards in cards/.

Weights are not included. You download EXL3 and/or GGUF models yourself. Secrets are not included. Copy examples and generate your own keys. Character cards are included. Import the PNGs under cards/ into SillyTavern.

Service ports (compose defaults)

Default path is Qwen → Open WebUI. Inside Docker use the service name, not localhost.

Host bookmark bar (Chrome / Firefox)

These are the host homes for every UNIFIED add-on with a web UI. Import `bookmarks.html` into Chrome (Bookmarks → Import bookmarks and settings → Bookmarks HTML file) or Firefox. The in-stack browser is already seeded: http://127.0.0.1:3010

Mongo (:27017) and Postgres (:5432) have no HTML home. TabbyAPI (:5000) and Ollama (:11435) stay behind --profile alternate-inference.

Open WebUI on the unified stack:

ENABLE_OLLAMA_API=false
ENABLE_OPENAI_API=true
OPENAI_API_BASE_URL=http://qwen:8080/v1
ENABLE_RAG_WEB_SEARCH=true
RAG_WEB_SEARCH_ENGINE=searxng
SEARXNG_QUERY_URL=http://searxng:8080/search?q=<query>

All services share the bridge network ai-network.

System requirements

Host

• —OS: Linux native, or Windows WSL2 with Docker Desktop / Docker Engine
• —GPU: NVIDIA GPU with recent drivers (lab used RTX 4070)
• —Docker: Docker Engine + Compose plugin (docker compose version)
• —NVIDIA Container Toolkit (required for GPU passthrough into containers)
• —Disk: room for Docker images plus model weights (EXL3 8B class is often tens of GB; GGUF varies by quant)
• —RAM / VRAM: 8B-class EXL3 fits a 12 GB class card with headroom when KV/cache is tuned; larger models need more VRAM

One-time NVIDIA Container Toolkit check

nvidia-smi

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If nvidia-smi works on the host but fails in Docker, fix the toolkit / Docker daemon GPU runtime before bringing the stack up.

Optional host tools

pipx install "huggingface_hub[cli]"
# or: python3 -m venv .venv && . .venv/bin/activate && pip install -U "huggingface_hub[cli]"

Cold start (stranger path)

Prefer cloning GitHub for day-to-day work. This HF repo is the public model card + sanitized mirror of the same layout.

0) Clone

git clone https://github.com/jpanasuk-netizen/tabby-tavern-stack.git
cd tabby-tavern-stack

Alternate (HF mirror — same public layout, may lag GitHub):

git lfs install   # only if you later pull large assets; weights still not shipped
git clone https://huggingface.co/jpanasuk/tabby-tavern-stack
cd tabby-tavern-stack

1) Build the TabbyAPI image used by compose

Compose expects this exact local tag:

docker build -f Dockerfile.tabby -t local/tabbyapi:exl3-fixed .

Dockerfile.tabby is the WSL2-tested image: ghcr.io/theroyallab/tabbyapi:latest plus build deps and a libcuda.so symlink so ExLlamaV3/Triton can link inside WSL2.

2A) Download an EXL3 model for TabbyAPI

Weights go under tabby_models/ (gitignored). Compose mounts:

./tabby_models  →  /app/models   (inside tabbyapi)

TabbyAPI model_name is a directory name under that mount.

mkdir -p tabby_models

huggingface-cli download turboderp/Llama-3.1-8B-Instruct-exl3 \
  --revision 6.0bpw \
  --local-dir tabby_models/Llama-3.1-8B-Instruct-exl3-6.0bpw

ls -la tabby_models/Llama-3.1-8B-Instruct-exl3-6.0bpw | head

EXL2 will not load. Current TabbyAPI dropped the ExLlamaV2 backend. Use EXL3 quants only.

2B) Configure TabbyAPI (required before first launch)

cp -n tabby_config/config.example.yml tabby_config/config.yml

Generate real keys and paste them in:

python3 -c "import secrets; print('admin_key:', secrets.token_hex(32)); print('api_key:  ', secrets.token_hex(32))"

Set model_name to the folder you downloaded. For a 12 GB card the lab uses:

model:
  model_dir: models
  model_name: Llama-3.1-8B-Instruct-exl3-6.0bpw
  max_seq_len: 8192
  cache_size: 8192
  cache_8bit: true

Compose always mounts:

./tabby_config/config.yml  →  /app/config.yml

Put the same api_key in:

• —open-webui OPENAI_API_KEY
• —mcpo/config.json TABBYAPI_KEY
• —SillyTavern API settings

2C) Ollama / GGUF path (secondary backend)

After the stack is up:

docker exec -it tabby-tavern-ollama-1 ollama pull llama3.1:8b
docker exec -it tabby-tavern-ollama-1 ollama list
docker exec -it tabby-tavern-ollama-1 ollama run llama3.1:8b "Say hello in one sentence."

You can run TabbyAPI (EXL3) and Ollama (GGUF) together; on a 12 GB card load one heavy model at a time unless you know your headroom. Lab measurement: ~8 GB VRAM for Llama-3.1-8B EXL3 6.0bpw + 8-bit cache, ~4 GB left for a small GGUF.

3) Frontend / search secrets (placeholders only)

SillyTavern (sillytavern_config/config.yaml):

• —Default listen port 8000
• —browserLaunch.enabled: false (no browser inside Docker)
• —basicAuthMode: true — ST refuses to start on 0.0.0.0 without auth
• —Replace:

basicAuthUser:
  username: "YOUR_ST_USERNAME_HERE"
  password: "YOUR_ST_PASSWORD_HERE"

SearXNG (searxng_config/settings.yml):

use_default_settings: true
server:
  secret_key: "YOUR_SEARXNG_SECRET_KEY_HERE"
  image_proxy: true
search:
  formats:
    - html
    - json

JSON format is required for MCPO searxng_search and Open WebUI RAG.

4) Character cards (SillyTavern)

This release ships Tavern character cards under cards/:

Use the cards:

1. 1.Start SillyTavern → http://localhost:8000
2. 2.Characters → Import → pick cards/default_Seraphina.png
3. 3.Copy cards/Seraphina/ into SillyTavern's character expressions folder if the importer does not pull sprites automatically (sillytavern_data/default-user/characters/Seraphina/)

Do not commit chats, secrets.json, or live API keys from a running data directory.

5) Launch

docker compose up -d
docker compose ps

Or:

chmod +x start-stack.sh load-model.sh
./start-stack.sh

6) Health checks (every service)

docker compose ps
docker compose logs --tail=80 tabbyapi
docker compose logs --tail=40 ollama
docker compose logs --tail=40 sillytavern
docker compose logs --tail=40 open-webui
docker compose logs --tail=40 searxng
docker compose logs --tail=40 mcpo

curl -sS -o /dev/null -w "sillytavern  %{http_code}\n" http://127.0.0.1:8000/ || true
curl -sS -o /dev/null -w "open-webui   %{http_code}\n" http://127.0.0.1:3000/ || true
curl -sS -o /dev/null -w "tabbyapi     %{http_code}\n" http://127.0.0.1:5000/ || true
curl -sS -o /dev/null -w "ollama       %{http_code}\n" http://127.0.0.1:11435/ || true
curl -sS -o /dev/null -w "searxng      %{http_code}\n" http://127.0.0.1:8080/ || true
curl -sS -o /dev/null -w "mcpo         %{http_code}\n" http://127.0.0.1:8001/docs || true

docker exec -it tabby-tavern-tabbyapi-1 nvidia-smi || true

Browser targets:

• —SillyTavern → http://localhost:8000
• —Open WebUI → http://localhost:3000
• —TabbyAPI → http://localhost:5000
• —Ollama (host) → http://localhost:11435
• —SearXNG → http://localhost:8080
• —MCPO Swagger → http://localhost:8001/docs
• —Firefox (in-stack) → http://localhost:3010
• —AnythingLLM → http://localhost:3002
• —LibreChat → http://localhost:3080
• —Lobe Chat → http://localhost:3210
• —n8n → http://localhost:5678
• —Qdrant dashboard → http://localhost:6333/dashboard
• —Chroma → http://localhost:8005
• —Meilisearch → http://localhost:7700
• —code-server → http://localhost:8443

7) Point SillyTavern at TabbyAPI

In SillyTavern API settings:

• —API type: OpenAI-compatible / TabbyAPI
• —Endpoint: http://tabbyapi:5000/v1 from another container, or http://127.0.0.1:5000/v1 from the host
• —API key: the api_key in tabby_config/config.yml

If ST cannot reach TabbyAPI, check:

1. 1.Both containers on ai-network
2. 2.Keys match
3. 3.TabbyAPI finished loading the EXL3 model (docker compose logs -f tabbyapi)

8) Switch / reload TabbyAPI model

# Edit model_name in tabby_config/config.yml to another folder under tabby_models/
docker compose restart tabbyapi
docker compose logs -f tabbyapi

MCPO (MCP → OpenAPI)

MCPO (ghcr.io/open-webui/mcpo:main) launches MCP servers as subprocesses and exposes their tools as OpenAPI HTTP endpoints.

mcp-servers/server.py is a FastMCP server (not raw JSON-RPC) with 7 tools:

Config notes:

• —mcpo/config.json must call /app/.venv/bin/python3 (MCPO image venv has mcp)
• —Env: TABBYAPI_URL, OLLAMA_URL, TABBYAPI_KEY (placeholder in public tree)
• —Tools are served under /host-master/ (server name in config)
• —Empty "mcpServers": {} crashes MCPO — keep at least one entry
• —Set --api-key in compose (public tree uses REPLACE_WITH_YOUR_MCPO_API_KEY)

A second server, mcp-servers/tavern_mcp.py, is the lab connectivity toolkit (status, self_check, wire, models, chat). Wire it only if you also ship discover.py next to it.

GPU / compose tuning (what the lab actually ships)

If you OOM: lower max_seq_len / cache_size, use a smaller bpw EXL3, or stop Ollama models while TabbyAPI holds a large model.

Repository structure

tabby-tavern-stack/
├── docker-compose.yml              # lab orchestration (6 services)
├── docker-compose.starter.yml      # optional coding-tools overlay
├── Dockerfile / Dockerfile.tabby   # TabbyAPI image (WSL2 libcuda fix)
├── start-stack.sh
├── load-model.sh
├── cards/                          # SillyTavern character cards (PNG)
├── mcpo/config.json                # MCPO server map (placeholders)
├── mcp-servers/server.py           # FastMCP stack tools
├── tabby_config/config.example.yml
├── sillytavern_config/config.yaml
├── searxng_config/settings.yml
├── tabby_models/                   # EXL3 weights (gitignored contents)
├── SECURITY.md
├── DEVLOG.md
├── LICENSE
└── docs/                           # sell-sheet extras

Compose service names (authoritative):

tabbyapi · sillytavern · ollama · open-webui · searxng · mcpo

Environment & secrets guidance

Private-lab defaults are intentional. This is not hardened multi-tenant hosting.

Before any LAN/WAN exposure:

1. 1.Replace every placeholder credential
2. 2.Prefer binding host ports to 127.0.0.1
3. 3.Put a reverse proxy + TLS in front if you leave the machine
4. 4.Read `SECURITY.md`

Older public revisions of this mirror contained lab convenience keys (including a TabbyAPI key in mcpo/config.json). Treat any key you ever saw in a public file as burned and rotate it.

Common failure modes

Measured lab results (defensible only)

Numbers from checked-in sample telemetry in `local_grid_suite` (benchmarks/sample_hardware_runs.json). Single-box lab runs — not a product SLA.

→ ~27× decode uplift on that run series after GPU routing / tuning.

Warm stream suite on qwen3:8b: ~76 tok/s (400-token runs).

WSL2 EXL3 chat (Llama-3.1-8B-Instruct 6.0bpw, RTX 4070): ~53 tok/s processing in the verified 24/24 check pass.

Re-measure on your hardware. Do not advertise these as guaranteed throughput.

Engineering notes

See `DEVLOG.md` for the build log:

• —Compose consolidation (TabbyAPI + ST + Open WebUI + Ollama + SearXNG + MCPO)
• —EXL3 adoption; EXL2 dropped upstream
• —GPU env tuning (shm_size, flash-attn / KV cache, cache_8bit)
• —Container-to-TabbyAPI auth / whitelist fixes
• —WSL2 NVIDIA toolkit + libcuda.so linker fix
• —MCPO FastMCP server (7 tools verified)
• —Character cards shipped in cards/

Optional coding starter overlay

docker-compose.starter.yml is an optional overlay (browser IDE, vector DBs, n8n, extra chat UIs). It is not required for the core six-service lab. It expects the core stack network ai-network to already exist. Change every placeholder password before up.

docker compose up -d
docker compose -f docker-compose.starter.yml up -d

WSL2 field notes — fresh-install lessons (Aug 2026)

A from-scratch rebuild on a clean WSL2 Ubuntu environment (same RTX 4070). Additive to the cold-start guide.

EXL2 vs EXL3 — TabbyAPI dropped EXL2 support

ValueError: Models quantized with 'exl2' require the exllamav2 backend, which is no longer supported. Please use an exl3 or unquantized model.

Use EXL3 quants only (turboderp/Llama-3.1-8B-Instruct-exl3 branches 2.0–8.0 bpw).

NVIDIA Container Toolkit repo URL

The deb .../noble main line is wrong for this repo (apt: Malformed entry (Component)). NVIDIA uses a flat structure:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor --yes -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

echo 'deb [arch=amd64 signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

libcuda.so not found inside TabbyAPI on WSL2

Even with torch.cuda.is_available() == True, model load can fail with cannot find -lcuda. The patched Dockerfile in this repo creates:

RUN ln -sf /usr/local/cuda-12.8/compat/libcuda.so /usr/lib/x86_64-linux-gnu/libcuda.so && ldconfig

Triton “not supported, roll back to CPU” warnings are cosmetic — exllamav3 uses its own CUDA kernels.

Port conflict when host Ollama is already running

Compose maps 11435:11434. Open WebUI still uses http://ollama:11434 internally.

Verified working state

After the v2.0.0 tree: 24/24 checks passing on the lab box — six containers, EXL3 chat, Ollama llama3.1:8b, Open WebUI dual backend, SillyTavern with cards, SearXNG JSON, MCPO 7 tools, ~8 GB / 12 GB VRAM.

Related spine

Author

Jeremy Panasuk — enterprise data / platform background; Aug 2024–present independent local-AI systems year (private Docker LLM lab, decode telemetry, multi-agent prototypes).

• —GitHub: @jpanasuk-netizen
• —Hugging Face: jpanasuk
• —LinkedIn: jeremy-p-34203322

License

MIT