Apppublic

mohitCodepy/quickbites-support-bot

sourceHugging Facemitupdated 5mo agoView on Hugging Face

0likes

App README

QuickBites Support Bot

Prod evaluation: 1990 / 2200 (90%) across all 22 graded simulator scenarios. Score sheet: docs/prod_score.json.

Customer-support bot for the Techjays QuickBites take-home. The service reads the customer's message, looks up the relevant order / customer / restaurant / rider context from app.db, extracts structured signals (LLM with a deterministic keyword fallback), runs a rule-based policy engine, and returns the structured simulator actions (issue_refund, file_complaint, escalate_to_human, flag_abuse, close) along with a friendly customer-facing reply.

The LLM classifies the message and matches item names. It does not decide refund amounts, complaint targets, or escalation -- those are owned by policy.py and driven from structured evidence in app.db. This keeps the bot predictable, auditable, and resistant to prompt injection.

Submission links

—Hosted URL: TODO — fill in once deployed.
—Video walkthrough: TODO — Loom link.
—Design doc: docs/DESIGN.md
—Prod score sheet: docs/prod_score.json
—Threshold calibration: docs/calibration.md
—Single-file dev guide (HTML): docs/developer-guide.html

Setup

powershell

python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
Copy-Item .env.example .env

Fill .env (see .env.example for every variable). Minimal:

env

ANTHROPIC_API_KEY=sk-ant-...
USE_LLM_EXTRACTION=auto       # on when API key is set
USE_CLAUDE=false              # prose rewriting; off by default
CANDIDATE_TOKEN=your-token

The bot runs fully offline with USE_LLM_EXTRACTION=false -- the keyword classifier takes over. Unit tests and the eval harness always use the deterministic path.

Run

powershell

# HTTP service (deploy this URL for the reviewers)
uvicorn quickbites_bot.api:app --host 0.0.0.0 --port 8000

# Single reply from the CLI
python -m quickbites_bot.cli reply "Order 564, Garlic Bread was missing"

# One dev simulator session
python -m quickbites_bot.cli run-session --mode dev --scenario-id 101

# Production evaluation (only after dev looks sane)
python -m quickbites_bot.cli run-session --mode prod

# Fetch prod score
python -m quickbites_bot.cli summary

HTTP endpoints:

—GET /health
—POST /bot/reply - {session_id, customer_message} -> {bot_message, actions}

Internal debug routes (/bot/state/{id}, /bot/reset/{id}, /simulator/run, /simulator/summary) are disabled unless ENABLE_INTERNAL_ROUTES=true.

Tests and evals

powershell

# Unit + branch + fixture-eval tests (46 tests, no network)
python -m unittest discover -s tests

# Just the fixture eval harness, with a human-readable report
python -m quickbites_bot.evals

# Regenerate threshold calibration after a DB snapshot change
python -m quickbites_bot.calibrate

quickbites_bot/evals.py replays each JSON fixture under tests/evals/fixtures/ through the bot and diffs the action list. Matchers are loose on prose and strict on graded fields (action type, targettype, method, amount or amountmin / amount_max). Add a new fixture file to cover a new scenario -- no code changes needed.

Layout

quickbites_bot/
  api.py         FastAPI service (public /bot/reply + gated debug routes)
  bot.py         SupportBot facade: session state + orchestration
  calibrate.py   Generates docs/calibration.md from app.db
  claude.py      Optional prose rewriter (Sonnet); skipped on abuse
  cli.py         CLI entrypoints for one-off replies and simulator runs
  config.py      .env loading, Settings dataclass, tristate flags
  db.py          SQLite access, typed CustomerStats/RestaurantStats/RiderStats
  evals.py       Fixture-based eval harness
  extraction.py  LLM signal extraction + keyword fallback + sieve
  models.py      ConversationState, Decision, RiskAssessment
  nlp.py         Deterministic keyword classifier (reference + fallback)
  policy.py      Rule-based PolicyEngine: risk tiers, issue handlers,
                 cap/method helpers
  simulator.py   Simulator wire protocol
tests/
  test_extraction.py
  test_policy_branches.py
  test_policy_regressions.py
  test_evals.py
  evals/fixtures/*.json
docs/
  ASSIGNMENT.md
  DESIGN.md
  SIMULATOR_API.md
  calibration.md    (generated)