dcblackhouse/treasurytakehome

<!-- The YAML block above configures a Hugging Face Space (sdk: docker). Normal Markdown viewers ignore it; Hugging Face reads it when this repo is deployed as a Space. See the Deployment section below. -->

TTB Label Verification Prototype

 

An AI-assisted tool that helps TTB compliance agents verify alcohol beverage label artwork against the expected application data. Given a label image, it checks the three things agents check by eye today:

1. 1.Brand name matches the application
2. 2.Alcohol content (ABV) matches
3. 3.Government health warning is present and correct (exact, all-caps)

It returns a clear pass/fail verdict per field in a few seconds, supports batch uploads, and runs entirely locally — no external API, no API key, no data leaving the machine.

This is a standalone proof-of-concept, not integrated with COLA. The full problem brief (stakeholder interviews, requirements, evaluation criteria) lives in `docs/PROJECT_BRIEF.md`. Agent-facing guidance is in `AGENTS.md`. Status: runs locally, tested end-to-end. The backend pipeline and the frontend both build and run. pytest covers the matcher (dependency-free) and the full OCR pipeline against generated sample labels — 16 tests, all passing. On this machine a single label verifies in ~120–320 ms, well within the 5-second budget. Public deployment is the one remaining manual step (see Deployment).

Why this architecture

The brief contains several hard constraints that pushed the design toward a fully local, open-source pipeline:

The guiding principle: OCR is the only fuzzy step; the pass/fail decision is plain, testable code. A compliance tool has to explain why it rejected a label ("warning used title case"), and you can't audit a black box.

How it works

                 ┌──────────────┐
   label image   │  Frontend    │   upload one image or a batch
  + expected ───▶│  (static UI) │──────────────┐
    field data   └──────────────┘              │ multipart POST
                                               ▼
                                   ┌────────────────────────┐
                                   │  FastAPI  /verify       │
                                   │                         │
                                   │  1. preprocess (OpenCV) │  deskew, de-glare,
                                   │  2. OCR (Tesseract)     │  contrast, grayscale
                                   │  3. extract fields      │  parse text → fields
                                   │  4. match               │  brand / ABV / warning
                                   └───────────┬─────────────┘
                                               │ JSON verdict
                                               ▼
                                   ┌────────────────────────┐
                                   │ per-field pass/fail +   │
                                   │ reason + confidence     │
                                   └────────────────────────┘

The pipeline is four stages, each independently testable:

1. 1.Preprocess (preprocess.py, OpenCV) — normalize the image: grayscale, contrast/threshold, deskew, reduce glare. This is also where the stretch goal of tolerating imperfect photos (angles, lighting) is addressed.
2. 2.OCR (ocr.py, Tesseract by default; optional PaddleOCR for higher accuracy) — extract text from the label. The engine is isolated behind one run_ocr() interface (Paddle is tried first when installed, then Tesseract), so it can be swapped without touching verdict logic.
3. 3.Extract (extract.py) — turn raw OCR text into candidate fields: brand name, ABV (e.g. 45% Alc./Vol.), class/type, net contents, and the warning block.
4. 4.Match (matching.py) — compare extracted fields against the expected application data and produce a per-field verdict. This is the heart of the tool, and it deliberately applies opposite tolerances per field:

Project layout

.
├── README.md                 # this file
├── AGENTS.md                 # requirements & constraints for AI coding agents
├── docs/
│   └── PROJECT_BRIEF.md       # the original take-home brief
├── backend/                   # Python · FastAPI
│   ├── app/
│   │   ├── main.py            # FastAPI app + /verify and /verify/batch routes
│   │   ├── preprocess.py      # OpenCV image normalization
│   │   ├── ocr.py             # OCR engine wrapper (Tesseract / optional PaddleOCR)
│   │   ├── extract.py         # raw OCR text → structured label fields
│   │   ├── matching.py        # brand (fuzzy) / ABV / warning (strict) verdicts
│   │   ├── models.py          # request/response schemas (Pydantic)
│   │   └── sample_labels.py   # render synthetic test labels (PIL)
│   ├── tests/                 # test_matching.py (unit) + test_pipeline.py (e2e)
│   ├── requirements.txt
│   └── requirements-paddle.txt  # optional higher-accuracy OCR engine
├── samples/                   # generated example labels (compliant + rejections)
└── frontend/                  # static upload + results UI (Vite)

API

POST /verify

Verify a single label.

Request (multipart/form-data):

Response (application/json):

{
  "overall": "fail",
  "checks": [
    { "name": "brand_name", "passed": true,  "expected": "Old Tom Distillery", "found": "OLD TOM DISTILLERY", "score": 1.0 },
    { "name": "abv",        "passed": true,  "expected": "45%",                "found": "45.0%" },
    { "name": "warning",    "passed": false, "reason": "'GOVERNMENT WARNING:' is not in all caps" }
  ],
  "elapsed_ms": 235,
  "filename": "label.png"
}

POST /verify/batch

Accepts multiple images (the "big importer dumps 300 applications at once" case) and returns an array of per-label results. Images run concurrently on a thread pool to stay within the latency budget.

Running it

Quick start (one command)

./run.sh          # or: make

This installs everything it needs (Python venv + backend deps, frontend packages, sample labels — and Tesseract via Homebrew on macOS if it's missing), then starts the API on :8000 and the UI on :3000. Open <http://localhost:3000> and press Ctrl+C to stop both. It's idempotent, so re-running just relaunches.

The steps below are the equivalent manual setup if you'd rather run the pieces separately.

Prerequisites

• —Python 3.11+
• —Node 18+ (for the frontend)
• —The Tesseract OCR binary:
• —macOS: brew install tesseract
• —Debian/Ubuntu: sudo apt-get install tesseract-ocr
• —No API keys, no cloud accounts — everything runs locally.

Backend

cd backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt        # FastAPI, OpenCV, pytesseract, RapidFuzz, Pillow
uvicorn app.main:app --reload          # serves http://localhost:8000

For higher OCR accuracy on real-world photos, optionally also pip install -r requirements-paddle.txt — the wrapper uses PaddleOCR when present and otherwise falls back to Tesseract. Either way it runs fully offline after install (PaddleOCR downloads its weights once, then caches them).

Frontend

cd frontend
npm install
npm run dev                            # serves http://localhost:3000

Open http://localhost:3000, drop a label image (or several), enter the expected brand name and ABV, and read the color-coded result.

Try it with the sample labels

The repo ships generated example labels in `samples/` — one compliant label plus three rejection cases (title-case warning, missing warning, wrong ABV). Regenerate them anytime with:

cd backend && python -m app.sample_labels      # writes ../samples/*.png

Upload samples/compliant.png with brand Old Tom Distillery and ABV 45% for a full pass; the others demonstrate each rejection reason.

Tests

cd backend
pytest                                 # 16 tests: matcher unit tests + full-pipeline e2e

test_matching.py runs with no third-party dependencies (the matcher falls back to stdlib difflib). test_pipeline.py renders sample labels and runs the real OCR pipeline end-to-end; it auto-skips if no OCR engine is installed.

CI runs this same suite (with Tesseract installed) plus a frontend build on every pull request and push to main — see `.github/workflows/ci.yml`. To make it a hard gate, enable branch protection on main requiring the backend-tests and frontend-build checks.

Deployment

The whole app — API and built frontend — ships as a single Docker image (see `Dockerfile`). A multi-stage build compiles the frontend and serves it from the FastAPI backend, with Tesseract installed. Because the frontend is served by the backend, it's one origin and one URL — no CORS or API base to configure in production. Nothing depends on a hosted ML service.

Run the whole thing locally with Docker

docker compose up --build        # → http://localhost:7860

Deploy to Hugging Face Spaces (free)

Hugging Face builds the Dockerfile directly (it does not run Compose — the image is identical either way) and reads the Space settings from the YAML front-matter at the top of this README (sdk: docker, app_port: 7860).

1. 1.Create a new Space → SDK: Docker → Blank.
2. 2.Add the Space as a git remote and push (or point the Space at this GitHub repo in the Space's settings):

   git remote add space https://huggingface.co/spaces/<user>/<space-name>
   git push space main

1. 1.The Space builds the image and serves the app on port 7860 — you get a public URL. No API key, nothing else to configure.

Keep the Space in sync automatically (GitHub Action)

`.github/workflows/deploy-hf-space.yml` mirrors this repo to your Space on every push to main, so the Space rebuilds itself. One-time setup in Settings → Secrets and variables → Actions:

• —Secret `HF_TOKEN` — a Hugging Face access token with write scope.
• —Variable `HF_USERNAME` — your Hugging Face username or org.
• —Variable `HF_SPACE` — the Space name.

Create the Space once (above); until those are set the workflow skips itself (no failed runs), then keeps the Space current on every push thereafter.

Other hosts

The same image runs on Google Cloud Run, Render, Fly.io, Koyeb, or any VM — the container listens on $PORT when the platform sets one (default 7860). For a split deploy instead, the static frontend/dist/ bundle can go on any static host while the backend container runs elsewhere.

Design decisions & trade-offs

• —Local OCR over a hosted vision API. Trades some raw extraction accuracy for zero network egress, zero per-label cost, and no PII leaving the box — the right call given the firewall and data-handling constraints in the brief. The OCR layer is isolated behind one interface, so a future engine swap (or even a local vision-language model) is a one-file change.
• —Deterministic verdicts, not model judgment. The pass/fail decision is ordinary Python so it is strict, reproducible, and explainable — essential for the exact-warning requirement and for an auditable compliance workflow.
• —Asymmetric matching by design. Lenient for brand, strict for the warning. A single "fuzzy-match everything" approach would either reject valid brands or accept doctored warnings.
• —Stateless / no storage. Simplest thing that satisfies the no-sensitive-data posture; nothing to secure, retain, or purge for the prototype.

Limitations & roadmap

• —Imperfect photos (angles, glare, lighting) are partially handled by the OpenCV preprocessing stage; heavy distortion may still require a re-shoot, as it does for agents today. Improving this is the main stretch goal.
• —Field extraction currently targets the common distilled-spirits/wine/beer label fields in the brief; unusual layouts may need extraction tuning.
• —Not COLA-integrated, no auth, no audit log — out of scope for a prototype, but the obvious next steps for a production path.

See `docs/PROJECT_BRIEF.md` for the full requirements and `AGENTS.md` for the constraints this design is built against.