Samarth1812/lumera-backend

Luméra — AI Skincare Analysis Platform

A full-stack web application that analyses your skin from a photo. It classifies your skin type (combination, dry, normal, oily, sensitive), detects up to 8 skin concerns (acne, blackheads, dark circles, eye bags, hyperpigmentation, lip hyperpigmentation, redness, texture), recommends personalised skincare products, generates morning and night routines, tracks your progress over time with an interactive calendar, and lets you chat with an AI skincare consultant. Built as a personal project from scratch.

Live App: https://lumera-wheat.vercel.app Backend API: https://samarth1812-lumera-backend.hf.space/api/health

Table of Contents

1. 1.Tech Stack
2. 2.Architecture Overview
3. 3.Project Structure
4. 4.Local Development Setup
5. 5.Environment Variables
6. 6.Database — Neon PostgreSQL
7. 7.Image Storage — Cloudinary
8. 8.ML Models — Git LFS
9. 9.Backend Deployment — Hugging Face Spaces
10. 10.Why We Migrated Away From Render
11. 11.Frontend Deployment — Vercel
12. 12.Making Changes and Redeploying
13. 13.Keep HF Space Awake — UptimeRobot
14. 14.Email OTP Authentication
15. 15.ML Models Deep Dive
16. 16.Face Detection Pipeline
17. 17.Concern Detection Architecture
18. 18.API Reference
19. 19.Database Schema
20. 20.Frontend Pages
21. 21.Features Deep Dive
22. 22.Training the Models
23. 23.Complete Bug Fix History
24. 24.Known Limitations
25. 25.Roadmap

Tech Stack

Frontend

Backend

Data & Storage

Architecture Overview

User Browser (https://lumera-wheat.vercel.app)
     │
     │  React + TypeScript + Tailwind
     │  Axios attaches JWT to every request
     │
     ▼
Vercel CDN — static React build
     │  also hosts /api/send-email.js — Node.js serverless function
     │  for Gmail SMTP relay (HF Spaces blocks outbound SMTP)
     │
     │  HTTPS API calls to Hugging Face Spaces
     │
     ▼
HF Spaces Docker Container (https://samarth1812-lumera-backend.hf.space)
     │  Gunicorn → Flask → SQLAlchemy
     │  1 worker, 300s timeout, 16 GB RAM
     │  Models baked into Docker image via Git LFS — no runtime download
     │
     ├──► Neon PostgreSQL
     │    users, analyses, skin_concerns,
     │    routines, routine_steps, product_recommendations
     │
     ├──► Cloudinary
     │    Original + compressed uploaded images
     │    Served directly to frontend via https:// URL
     │
     ├──► Groq API (llama-3.1-8b-instant)
     │    Recommendations, routines, chatbot responses
     │
     └──► Vercel /api/send-email (HTTPS POST, port 443)
          Gmail SMTP relay for OTP emails
          HF Spaces → Vercel → Gmail → user inbox

Key design decisions:

• —Images are compressed to max 1024px JPEG before ML processing — prevents timeouts on large phone photos
• —ML models load in a background thread at startup — server is live immediately, models ready within 2–3 minutes
• —Models are baked directly into the Docker image via Git LFS — zero re-download on every deploy (unlike the previous Render setup)
• —Cloudinary URLs stored in DB instead of base64 — keeps database lean
• —Normalised face crop (300×300 padded) stored as base64 in DB for instant Results page display without re-fetching
• —OTP codes stored in the users table with expiry timestamp — no separate OTP table needed
• —Email sending is always fire-and-forget in a daemon thread — Flask endpoints return immediately without waiting for SMTP
• —HF Spaces blocks outbound SMTP (ports 465/587) — routed through a Vercel serverless function over HTTPS (port 443) instead

Project Structure

lumera/
├── .gitignore
├── .gitattributes                     # Git LFS tracking rules — *.keras and *.h5 tracked via LFS
├── PROJECT.md                         # This file — full project documentation
├── README.md                          # HF Spaces config (YAML frontmatter only)
├── Dockerfile                         # Root-level Dockerfile for HF Spaces Docker SDK
├── render.yaml                        # Legacy Render config — kept for reference, no longer used
├── frontend/
│   ├── vercel.json                    # SPA routing fix — rewrites all paths to index.html
│   ├── .env.development               # VITE_API_URL=http://localhost:3001/api
│   ├── .env.production                # VITE_API_URL=https://samarth1812-lumera-backend.hf.space/api
│   ├── api/
│   │   ├── send-email.js              # Vercel serverless function — Gmail SMTP relay for OTP emails
│   │   └── package.json              # { "type": "commonjs" } — overrides frontend ESM for this folder
│   ├── public/
│   │   └── favicon.svg                # Purple gradient L icon
│   ├── src/
│   │   ├── api/
│   │   │   └── axios.ts               # Axios instance — reads VITE_API_URL env var
│   │   ├── components/
│   │   │   ├── PageShell.tsx          # Shared bg: #f5f3ff + dot-grid SVG + purple accent circles
│   │   │   ├── Navbar.tsx             # Responsive navbar, logo routes to dashboard/home by auth state
│   │   │   └── ProtectedRoute.tsx     # Synchronous JWT guard — no async delay, no login flash
│   │   ├── pages/
│   │   │   ├── Home.tsx               # Landing page
│   │   │   ├── Login.tsx              # Two-tab login: password OR email OTP (passwordless)
│   │   │   ├── Signup.tsx             # Registration form — redirects to VerifyOtp on submit
│   │   │   ├── VerifyOtp.tsx          # Shared 6-digit OTP entry — handles verify/login/reset purposes
│   │   │   ├── ForgotPassword.tsx     # Email entry for password reset flow
│   │   │   ├── ResetPassword.tsx      # New password entry after OTP verified — strength meter included
│   │   │   ├── Dashboard.tsx          # Scan history grid, quick actions
│   │   │   ├── Upload.tsx             # Camera/file upload with photo guide + image compression
│   │   │   ├── Results.tsx            # Concerns · Products · Routine tabs
│   │   │   ├── Progress.tsx           # Calendar + day panel
│   │   │   ├── Chatbot.tsx            # AI skincare consultant
│   │   │   ├── Routines.tsx           # Morning/night routine manager
│   │   │   └── WeeklyReport.tsx       # Bar chart + PDF download
│   │   ├── types/index.ts
│   │   └── App.tsx                    # Routes — includes /verify-otp, /forgot-password, /reset-password
│   └── package.json
│
└── backend/
    ├── app.py                         # Flask factory — CORS, blueprints, background model loading
    ├── config.py                      # Reads DATABASE_URL, CLOUDINARY_*, GROQ_API_KEY from env
    │                                  # Includes pool_pre_ping + pool_recycle for Neon idle reconnection
    ├── models.py                      # SQLAlchemy ORM: User (+ OTP fields), Analysis, SkinConcern, etc.
    ├── download_models.py             # Legacy — downloaded .keras files from Google Drive on Render
    │                                  # No longer called at startup — models are baked into image
    ├── skin_concern_detector.py       # SkinConcernDetector — hybrid ML ensemble + CV signals
    ├── requirements.txt
    ├── routes/
    │   ├── auth.py                    # /register /login /logout /me
    │   │                              # + /verify-otp /resend-otp /send-login-otp
    │   │                              # + /forgot-password /reset-password
    │   ├── analysis.py                # /upload (compress→ML→Cloudinary) /history /result/:id
    │   ├── chatbot.py                 # /chat — Groq with last 5 scan context
    │   ├── routines.py                # CRUD + /activate
    │   ├── products.py                # /recommend — Groq + OBF images
    │   └── report.py                  # /summary (JSON) + /weekly (PDF)
    ├── services/
    │   └── ml_service.py              # SkinAnalyzer: face detection, two-crop, CNN inference
    ├── utils/
    │   ├── helpers.py                 # allowed_file() — validates PNG/JPG/JPEG/WEBP
    │   └── email_service.py           # Dual-mode OTP emailer:
    │                                  #   local dev → Gmail SMTP directly (port 465, works on Mac)
    │                                  #   production → POST to Vercel /api/send-email (HTTPS)
    │                                  #   always fire-and-forget in a daemon thread
    └── ml_model/
        ├── best_model_v2.keras        # Skin type CNN v2 — tracked via Git LFS
        ├── concern_model_v3.keras     # Concern CNN v3 — tracked via Git LFS
        ├── concern_model_v2.keras     # Concern CNN v2 — tracked via Git LFS
        ├── concern_model.keras        # Concern CNN v1 — tracked via Git LFS
        ├── class_indices.json         # Skin type class order (small, committed normally)
        ├── concern_class_indices*.json
        └── train_*.py                 # Training scripts (run locally, not on HF Spaces)

Local Development Setup

Prerequisites

• —Python 3.12+
• —Node.js 18+
• —Git + Git LFS installed (git lfs install)
• —A free Groq API key — https://console.groq.com
• —A free Cloudinary account — https://cloudinary.com
• —The .keras model files — automatically available after git clone if LFS is installed
• —A Gmail account with an App Password for OTP emails (local dev uses SMTP directly)

Backend

cd lumera/backend

# Create and activate virtual environment
python -m venv venv
source venv/bin/activate      # Mac/Linux
# venv\Scripts\activate       # Windows

# Install all dependencies
pip install -r requirements.txt

# Create .env file (see Environment Variables section)
touch .env
# Paste your credentials into it

# Start backend
python app.py
# Runs on http://localhost:3001

Expected startup output:

✓ Database tables created
⏳ ML models loading in background...
🚀 Starting backend on http://localhost:3001
 * Running on http://127.0.0.1:3001
...
✓ Skin type model ready — classes: ['Combination', 'Dry', 'Normal', 'Oily', 'Sensitive']
✓ Concern model ready: concern_model_v3.keras (weight=1.0)
✓ Concern model ready: concern_model_v2.keras (weight=0.8)
✓ Concern model ready: concern_model.keras (weight=0.5)
✅ All models loaded and ready

Frontend

cd lumera/frontend
npm install
npm run dev
# Runs on http://localhost:5173 (or 5174 if 5173 is taken)

The frontend reads VITE_API_URL from .env.development which points to http://localhost:3001/api. Both CORS origins (5173 and 5174) are whitelisted in app.py.

Verify Everything Works

# Backend health check
curl http://localhost:3001/api/health
# Expected: {"status": "ok", "message": "Backend is running"}

# Verify skin type model
python3 -c "
from services.ml_service import get_analyzer
a = get_analyzer()
print('Model loaded:', a.model is not None)
print('Classes:', a.skin_types)
"

# Verify concern ensemble
python3 -c "
from skin_concern_detector import SkinConcernDetector
d = SkinConcernDetector()
for model, weight, name in d._load_ensemble():
    print(f'{name}  weight={weight}')
"

# Test OTP email locally (should print OTP to console if GMAIL_USER not set,
# or send a real email if GMAIL_USER + GMAIL_APP_PASSWORD are set in .env)
curl -X POST http://localhost:3001/api/auth/send-login-otp \
  -H "Content-Type: application/json" \
  -d '{"email": "your@email.com"}'

Environment Variables

backend/.env — local only, never commit

SECRET_KEY=lumera-super-secret-key-min-32-chars-long-12345
JWT_SECRET_KEY=lumera-super-secret-key-min-32-chars-long-12345

# Leave DATABASE_URL absent locally — Flask uses SQLite automatically
# DATABASE_URL=postgresql://...   ← only set this on HF Spaces

GROQ_API_KEY=gsk_your_actual_key_here

CLOUDINARY_CLOUD_NAME=your_cloud_name
CLOUDINARY_API_KEY=your_api_key
CLOUDINARY_API_SECRET=your_api_secret

# OTP email — local dev uses Gmail SMTP directly
# Leave VERCEL_EMAIL_URL unset locally — its absence triggers SMTP mode
GMAIL_USER=noreplylumera@gmail.com
GMAIL_APP_PASSWORD=xxxx xxxx xxxx xxxx   # App Password, NOT your Gmail login password

frontend/.env.development — local frontend

VITE_API_URL=http://localhost:3001/api

frontend/.env.production — committed, no secrets

VITE_API_URL=https://samarth1812-lumera-backend.hf.space/api

HF Spaces Secrets — set in Space dashboard

Go to: https://huggingface.co/spaces/Samarth1812/lumera-backend → Settings → Variables and secrets

Vercel Environment Variables — set in Vercel dashboard

Go to: vercel.com → Lumera project → Settings → Environment Variables

Note: No MODEL_ID_* variables are needed. Models are baked into the Docker image via Git LFS and are available on disk at container startup — no runtime download.

Important: DATABASE_URL must NOT be set in your local .env. Without it, config.py falls back to sqlite:///lumera.db — local dev uses SQLite, production uses Neon. They never share data or interfere with each other.

Important: VERCEL_EMAIL_URL must NOT be set in your local .env. Its absence is what triggers SMTP mode in email_service.py — local dev sends Gmail SMTP directly, production routes through Vercel.

Database — Neon PostgreSQL

Provider: neon.tech — free tier, no credit card, 0.5 GB storage.

How it connects: config.py reads DATABASE_URL from the environment:

SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL', 'sqlite:///lumera.db')

On HF Spaces the Neon URL is present. Locally it falls back to SQLite.

Connection pool settings (added to fix Neon idle SSL drops):

SQLALCHEMY_ENGINE_OPTIONS = {
    'pool_pre_ping':  True,    # test connection before use — silently reconnects if Neon closed it
    'pool_recycle':   300,     # recycle connections after 5 min — prevents stale SSL sessions
    'pool_size':      3,
    'max_overflow':   2,
    'connect_args':   {'connect_timeout': 10},
}

Without pool_pre_ping, the second request after an idle period always failed with psycopg2.OperationalError: SSL connection has been closed unexpectedly.

Tables created automatically by SQLAlchemy's db.create_all() on every startup:

• —users — email, username, bcrypt password hash (VARCHAR 512 — scrypt hashes are long), OTP fields
• —analyses — skin type, confidence, Cloudinary image URL, normalized face base64, skin concerns JSON
• —skin_concerns — per-concern scores, severity, AI notes, annotated zone images (base64)
• —routines + routine_steps — AI-generated morning/night routines with steps
• —product_recommendations — product data

One manual migration needed when switching from SQLite to Neon — the password_hash column needs widening to 512 chars:

python -c "
from app import create_app
from models import db
app = create_app()
with app.app_context():
    db.session.execute(db.text('ALTER TABLE users ALTER COLUMN password_hash TYPE VARCHAR(512)'))
    db.session.commit()
    print('Done')
"

OTP columns migration — run once on existing Neon databases to add the auth fields:

ALTER TABLE users
  ADD COLUMN IF NOT EXISTS is_verified    BOOLEAN   NOT NULL DEFAULT FALSE,
  ADD COLUMN IF NOT EXISTS otp_code       VARCHAR(6),
  ADD COLUMN IF NOT EXISTS otp_expires_at TIMESTAMP,
  ADD COLUMN IF NOT EXISTS otp_purpose    VARCHAR(20);

-- To keep existing users logged in without re-verifying:
UPDATE users SET is_verified = TRUE WHERE is_verified = FALSE;

Neon connection string format:

postgresql://username:password@ep-xxxx.ap-southeast-1.aws.neon.tech/neondb?sslmode=require

The ?sslmode=require is mandatory — Neon rejects unencrypted connections.

Image Storage — Cloudinary

Provider: cloudinary.com — free tier: 25 GB storage, 25 GB bandwidth/month.

Upload flow in `routes/analysis.py`:

User uploads image (any size, any format)
        │
        ▼
Saved temporarily to backend/uploads/ on the container's ephemeral disk
        │
        ▼
_compress_image() — resize to max 1024px, save as JPEG quality 85
  → typically reduces a 5MB phone photo to ~300KB
  → consistent input size for ML pipeline
        │
        ▼
analyze_skin() — ML inference on compressed local file
        │
        ▼
cloudinary.uploader.upload() — compressed file → Cloudinary folder lumera/uploads/
        │
        ▼
Local temp file deleted
        │
        ▼
Cloudinary secure_url stored in analyses.image_path column

Frontend rendering: The AuthImage component checks if image_path starts with https:// — if so, renders it directly as an <img> tag. No proxy needed, no JWT required for Cloudinary URLs.

ML Models — Git LFS

The .keras model files are 25–35 MB each and cannot be committed to GitHub as regular files (100 MB file limit). They are tracked via Git LFS and pushed directly to the Hugging Face Spaces repository.

How Git LFS tracking works

The .gitattributes file at the repo root declares LFS tracking:

*.keras filter=lfs diff=lfs merge=lfs -text
*.h5    filter=lfs diff=lfs merge=lfs -text

Both .keras and .h5 extensions are tracked. The .h5 rule was added after HF rejected a push containing skin_type_model.h5 as a regular binary. Any binary model file in either format is automatically stored in LFS.

How models get into the Docker image

When you push to HF Spaces, the HF git server resolves LFS pointers and provides the actual binary files to the Docker build context. The COPY backend/ . instruction in the Dockerfile then copies the entire backend/ folder — including ml_model/*.keras — into the container image at build time. The models are frozen into the image layer and are immediately available on disk when the container starts.

This is fundamentally different from the previous Render approach (see Why We Migrated Away From Render) where models were downloaded from Google Drive at runtime on every deploy.

Model files and sizes

Pushing model updates

If you retrain a model and want to deploy the new version:

# Models are already LFS-tracked — just add and commit normally
git add backend/ml_model/concern_model_v3.keras
git commit -m "update concern model v3 with new training run"
git push huggingface main --force

HF will rebuild the Docker image with the new model baked in. The --force is needed because HF rewrites history differently from GitHub.

Important: Always push to huggingface remote for backend changes and to origin for GitHub sync. They are two separate remotes:

git remote -v
# origin        https://github.com/Samarthsalvade/lumera.git
# huggingface   https://Samarth1812:hf_token@huggingface.co/spaces/Samarth1812/lumera-backend

Backend Deployment — Hugging Face Spaces

Provider: huggingface.co/spaces — free tier Docker Spaces.

Space URL: https://huggingface.co/spaces/Samarth1812/lumera-backend API base URL: https://samarth1812-lumera-backend.hf.space

How HF Spaces Docker works

HF Spaces accepts a Dockerfile at the root of the pushed repository and builds it on their infrastructure. The resulting container runs on HF's servers. Key constraints of the free tier:

• —16 GB RAM — this is the critical advantage over Render (512 MB). TensorFlow at ~400 MB plus the three concern models at ~36 MB fits easily.
• —Port 7860 — HF Spaces always proxies port 7860. The Dockerfile must EXPOSE 7860 and Gunicorn must bind to 0.0.0.0:7860.
• —Persistent container — unlike Render's free tier which sleeps after 15 minutes of inactivity (triggering a cold start), HF Spaces containers stay running as long as UptimeRobot pings them. The Space does sleep after ~48h of total inactivity but this is prevented by UptimeRobot.
• —Public Space required — the free tier requires the Space to be public. The API is accessible to anyone who knows the URL, but JWT authentication on all endpoints is the actual security layer.
• —Ephemeral filesystem — like Render, the container filesystem resets on redeploy. This is why Cloudinary is used for image storage and Neon for the database — both are external services that survive container restarts.
• —SMTP blocked — HF Spaces blocks outbound connections on ports 465 and 587. Gmail SMTP cannot be used directly from HF. OTP emails are routed through a Vercel serverless function over HTTPS instead (see Email OTP Authentication).

Root-level Dockerfile

The Dockerfile lives at the repo root (not inside backend/), because HF looks for it at root. It copies from backend/ explicitly:

FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
    libglib2.0-0 \
    libsm6 \
    libxext6 \
    libxrender-dev \
    libgomp1 \
    libgl1 \
    && rm -rf /var/lib/apt/lists/*

COPY backend/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY backend/ .

RUN mkdir -p ml_model uploads && chmod -R 777 ml_model uploads

ENV PORT=7860
EXPOSE 7860

CMD ["gunicorn", "--workers", "1", "--timeout", "300", "--bind", "0.0.0.0:7860", "app:create_app()"]

Key notes:

• —libgl1 is used instead of libgl1-mesa-glx — the latter does not exist in Debian trixie (the base image used by python:3.12-slim as of early 2026). Using the old package name causes the build to fail at the apt-get install step with "Package has no installation candidate."
• —git and git-lfs are NOT installed at runtime — they are only needed during the build phase (when HF's build system resolves LFS pointers). Installing them in the runtime layer wastes image space.
• —COPY backend/ . copies the entire backend folder into /app, including ml_model/*.keras which were resolved from LFS by HF's build system before Docker even runs.

Root-level README.md for HF Spaces config

HF Spaces reads configuration from a YAML frontmatter block in README.md at the repo root. Without this, the Space shows a CONFIG_ERROR. The file must use exact --- fences:

---
title: Lumera Backend
emoji: 🌿
colorFrom: purple
colorTo: pink
sdk: docker
pinned: false
---

The sdk: docker line is what tells HF to use the Dockerfile rather than treating the repo as a Gradio or Streamlit app. Without sdk: docker, HF will attempt to run it as a Gradio app and fail immediately.

Important: The repo root also contains PROJECT.md (this file, the full project documentation). README.md at root is intentionally kept minimal — just the HF config frontmatter — because HF reads README.md for its Space card display.

Gunicorn configuration

gunicorn --workers 1 --timeout 300 --bind 0.0.0.0:7860 app:create_app()

Why `--workers 1`: TensorFlow alone uses ~400 MB RAM. Even with 16 GB available, running multiple workers means multiple copies of TensorFlow and all three concern models in memory simultaneously. One worker is the correct choice given that inference is CPU-bound and requests are short.

Why `--timeout 300`: The first upload after a deploy triggers TensorFlow to JIT-compile the model graphs — this can take 30–60 seconds on cold CPU. 300 seconds gives enough headroom.

Deploying backend changes

git add backend/<changed_files>
git commit -m "describe your change"
git push huggingface main --force

The --force is required because HF Spaces and GitHub have diverged histories (due to the initial force-push during migration). This only applies to the huggingface remote — never force-push to origin.

Watch the build at: https://huggingface.co/spaces/Samarth1812/lumera-backend → Logs tab

Build times:

• —First build or requirements change: 8–12 minutes (pip installing TensorFlow etc.)
• —Code-only changes: 1–2 minutes (Docker layer cache skips the pip install layer)
• —Model updates: 3–5 minutes (LFS resolution + COPY layer invalidation)

Checking Space status via API

curl -s https://huggingface.co/api/spaces/Samarth1812/lumera-backend | python3 -m json.tool | grep "stage"

Possible stages:

• —CONFIG_ERROR — README.md missing or has bad YAML frontmatter
• —NO_APP_FILE — HF cannot find a runnable app at root (Dockerfile missing or wrong SDK)
• —BUILDING — Docker build in progress
• —RUNNING — container is up and accepting requests
• —FAILED — build or runtime error — check Logs tab

Why We Migrated Away From Render

The original backend was deployed on Render's free tier web service. After sustained use, two critical problems emerged that made Render unsuitable for this project:

Problem 1 — 512 MB RAM limit caused OOM kills

TensorFlow 2.21 requires approximately 400 MB of RAM just to load the runtime and JIT-compile the model graphs. The three concern models add another ~36 MB. Serving a single inference request pushed the process to ~450–480 MB — dangerously close to the 512 MB free tier limit. Any additional memory pressure (a slightly larger image, a concurrent request, OS overhead) caused the process to be OOM-killed by the Linux kernel. The Gunicorn worker would die mid-request, the user's upload would hang until the 300-second timeout, and the app would appear frozen.

Problem 2 — Models re-downloaded from Google Drive on every deploy

Because Render's filesystem is ephemeral and resets on every deploy, the .keras model files (~130 MB total) had to be downloaded from Google Drive at container startup via download_models.py. This download ran in a background thread so the server could start immediately, but:

• —The download added 3–4 minutes to every deploy before the ML pipeline was ready
• —During this window, uploads fell back to a feature-based analysis (skin brightness, texture variance) rather than CNN inference
• —Google Drive occasionally throttled large file downloads, causing the models to download as 0-byte files and making the fallback permanent until the next restart
• —The MODEL_ID_* environment variables for each model's Google Drive file ID had to be maintained in the Render dashboard

The solution — Hugging Face Spaces with Docker

Hugging Face Spaces Docker tier resolves both problems:

The migration required:

1. 1.Adding a root-level Dockerfile that copies from backend/ (since the full repo is pushed, not just the backend subfolder)
2. 2.Adding a root-level README.md with HF Spaces YAML frontmatter (sdk: docker)
3. 3.Removing the download_models() call from app.py's background thread (models are already on disk)
4. 4.Updating app.py CORS origins to include https://samarth1812-lumera-backend.hf.space
5. 5.Updating frontend/.env.production to point to the new HF URL
6. 6.Removing backend/services/blaze_face_short_range.tflite from git history (HF rejected binary files not tracked via LFS)
7. 7.Purging the .tflite file from all git history using git filter-branch (the file existed in old commits even after removal)
8. 8.Replacing libgl1-mesa-glx with libgl1 in the Dockerfile (the former package does not exist in Debian trixie)

Migration issues encountered and resolved

`libgl1-mesa-glx` not found: The python:3.12-slim base image uses Debian trixie as of early 2026. The libgl1-mesa-glx package was replaced by libgl1 in trixie. The Docker build failed at the apt-get install step with "Package has no installation candidate." Fixed by replacing libgl1-mesa-glx with libgl1.

`.tflite` binary file rejection: HF Spaces rejects binary files that are not tracked via LFS. The blaze_face_short_range.tflite file was committed as a regular binary in an old commit. Even after git rm --cached and a new commit, the file still existed in the git history and HF's pre-receive hook rejected the push. Fixed by running git filter-branch to rewrite all history and remove the file from every commit, followed by git reflog expire and git gc --prune=now --aggressive.

README YAML not parsed: The cat > README.md << 'EOF' ... EOF heredoc command was accidentally written into the file as literal text (the shell command became the file contents). HF's YAML parser could not find the --- fences and showed CONFIG_ERROR. Fixed by using printf instead: printf -- '---\ntitle: ...\n---\n' > README.md.

Full repo pushed instead of just backend: The first attempts to use git subtree split --prefix backend to push only the backend folder as root failed due to the repo being in a Luméra/ directory (Unicode in path). The workaround was to keep the full repo on HF but place the Dockerfile and README at the repo root, with the Dockerfile using COPY backend/requirements.txt . and COPY backend/ . to copy from the backend subfolder.

HF auth requires token, not password: git push huggingface main failed with "Invalid username or password." HF Spaces git remotes require a personal access token with Write scope, not your HF account password. Fixed by updating the remote URL to include the token: https://Samarth1812:hf_token@huggingface.co/spaces/Samarth1812/lumera-backend.

`NO_APP_FILE` after config error resolved: After fixing the README YAML, the stage changed from CONFIG_ERROR to NO_APP_FILE. HF was looking for an app.py or Dockerfile at the repo root but found them inside backend/. Fixed by adding a root-level Dockerfile.

`skin_type_model.h5` rejected by HF: After adding the OTP auth feature, a push to HF was rejected because backend/ml_model/skin_type_model.h5 existed in the git history as a regular binary (not tracked by LFS). git filter-branch failed due to unstaged changes. Fixed using git filter-repo --path backend/ml_model/skin_type_model.h5 --invert-paths --force which fully purged the file from all history. The *.h5 pattern was then added to .gitattributes for LFS tracking.

Frontend Deployment — Vercel

Provider: vercel.com — free tier, unlimited deployments.

Vercel project settings:

• —Root Directory: frontend
• —Framework Preset: Vite
• —Build Command: npm run build
• —Output Directory: dist
• —Install Command: npm install

Environment variables in Vercel dashboard:

VITE_API_URL        = https://samarth1812-lumera-backend.hf.space/api
GMAIL_USER          = noreplylumera@gmail.com
GMAIL_APP_PASSWORD  = (16-char Gmail App Password)
EMAIL_SECRET        = (shared secret string, same as HF Spaces EMAIL_SECRET)

VITE_API_URL must be set in the Vercel dashboard in addition to being in frontend/.env.production. Vite bakes the API URL at build time — if the dashboard variable differs from the file, the dashboard value wins.

`frontend/vercel.json` — required for React Router:

{
  "rewrites": [
    { "source": "/(.*)", "destination": "/index.html" }
  ]
}

Without this, refreshing /dashboard returns a Vercel 404 because Vercel tries to find a static dashboard.html file.

`frontend/api/` directory — Vercel serverless functions for email sending. Vercel auto-detects any .js file in api/ at the project root and deploys it as a serverless function. The frontend/api/package.json containing { "type": "commonjs" } overrides the parent frontend/package.json's "type": "module" setting — without this, Node.js treats the function as an ES module and require() throws ReferenceError.

CORS configuration in `app.py`:

CORS(app, resources={
    r"/*": {
        "origins": [
            "http://localhost:5173",
            "http://localhost:5174",
            "https://lumera-wheat.vercel.app",
            "https://samarth1812-lumera-backend.hf.space",
        ],
    }
})

Making Changes and Redeploying

Backend changes (Flask, ML, routes)

git add backend/<changed_files>
git commit -m "describe your change"
git push huggingface main --force   # deploys to HF Spaces
git push origin main                # syncs to GitHub (may need --force once)

Frontend changes (React, TypeScript, Vercel functions)

git add frontend/<changed_files>
git commit -m "describe your change"
git push origin main                # triggers Vercel auto-deploy

Both changed simultaneously

git add .
git commit -m "describe your change"
git push huggingface main --force
git push origin main

Exception — environment variables / secrets: New secrets must be added manually in the HF Spaces dashboard or Vercel dashboard. Pushing to git does not update secrets.

Keep HF Space Awake — UptimeRobot

HF Spaces free tier containers sleep after ~48 hours of inactivity. The first request after sleep triggers a cold start (~5–10 seconds). Set up a free external ping to prevent this:

1. 1.Go to uptimerobot.com → sign up free
2. 2.Click Add New Monitor
3. 3.Monitor Type: HTTP(s)
4. 4.Friendly Name: Lumera Backend
5. 5.URL: https://samarth1812-lumera-backend.hf.space/api/health
6. 6.Monitoring Interval: 5 minutes
7. 7.Click Create Monitor

The health endpoint returns {"status": "ok", "message": "Backend is running"} and is extremely lightweight — no database query, no ML inference. This keeps the server permanently warm at zero cost.

Email OTP Authentication

Luméra uses a full email-verified OTP authentication system. No third-party auth service is used — all OTP logic runs in Flask with codes stored in the users table.

Authentication flows

Signup flow:

1. 1.User submits username, email, password
2. 2.Backend creates an unverified user row (is_verified=False) and generates a 6-digit OTP
3. 3.OTP stored in users.otp_code with 10-minute expiry in users.otp_expires_at and otp_purpose='verify'
4. 4.Email sent asynchronously via send_otp_email() in a daemon thread
5. 5.Frontend navigates to /verify-otp with purpose='verify'
6. 6.User enters code → backend verifies → sets is_verified=True, clears OTP fields, issues JWT
7. 7.User lands on dashboard

Login — password tab:

• —Standard email + password check
• —If account exists but is_verified=False: blocks login, re-sends OTP, returns requires_verify: true
• —Frontend detects this flag and redirects to /verify-otp

Login — email code tab (passwordless):

1. 1.User enters email only
2. 2.Backend looks up email, silently returns 200 whether or not email exists (avoids leaking account existence)
3. 3.If found and verified: generates OTP with otp_purpose='login', sends email
4. 4.Frontend navigates to /verify-otp with purpose='login'
5. 5.User enters code → backend issues JWT → user lands on dashboard

Password reset flow:

1. 1.User enters email on /forgot-password
2. 2.Backend generates OTP with otp_purpose='reset', sends email (always returns 200)
3. 3.Frontend navigates to /verify-otp with purpose='reset'
4. 4.User enters code → backend returns a short-lived (15-min) reset_token JWT with { reset: true } claim
5. 5.Frontend navigates to /reset-password with resetToken in router state
6. 6.User enters new password → backend verifies the reset claim → updates password hash

OTP fields on the User model

is_verified    = db.Column(db.Boolean,  default=False, nullable=False)
otp_code       = db.Column(db.String(6),  nullable=True)   # 6-digit numeric string
otp_expires_at = db.Column(db.DateTime,   nullable=True)   # UTC, 10 minutes from generation
otp_purpose    = db.Column(db.String(20), nullable=True)   # 'verify' | 'login' | 'reset'

A single set of OTP columns handles all three purposes — only one OTP is ever active per user at a time. Generating a new OTP always overwrites the previous one.

Email transport — dual-mode

HF Spaces blocks outbound SMTP (ports 465 and 587). Gmail SMTP hangs silently and times out after 2–3 minutes, making auth endpoints appear frozen. The solution is a dual-mode transport in utils/email_service.py:

VERCEL_EMAIL_URL not set (local dev)
    → Gmail SMTP directly on port 465
    → Works on Mac/Linux developer machines
    → Credentials: GMAIL_USER + GMAIL_APP_PASSWORD in backend/.env

VERCEL_EMAIL_URL set (production on HF Spaces)
    → HTTP POST to https://lumera-wheat.vercel.app/api/send-email
    → Vercel serverless function uses nodemailer to send via Gmail SMTP
    → HF → Vercel over HTTPS (port 443, always open) → Gmail → inbox
    → Authenticated with shared EMAIL_SECRET to prevent abuse

Email sending is always fire-and-forget — send_otp_email() spawns a daemon thread and the Flask endpoint returns immediately. The OTP is already committed to the database before the thread starts, so there is no race condition.

Gmail App Password setup

The Gmail account noreplylumera@gmail.com uses an App Password (not the regular Gmail login password) for SMTP authentication:

1. 1.Sign into the Gmail account → myaccount.google.com
2. 2.Security → 2-Step Verification → turn ON (required before App Passwords appear)
3. 3.Security → App passwords → name: "Lumera" → Generate
4. 4.Copy the 16-character code (shown once — store it immediately)
5. 5.Add to both backend/.env (local) and Vercel dashboard (production)

Vercel email function

frontend/api/send-email.js is a Node.js serverless function deployed automatically with the frontend. It:

• —Accepts POST with { to, username, code, purpose, secret }
• —Validates secret against process.env.EMAIL_SECRET — returns 401 if mismatch
• —Uses nodemailer with Gmail SMTP to send a branded HTML email
• —Returns { ok: true } on success, error JSON on failure

The function file must be .js (not .cjs) with a sibling frontend/api/package.json containing { "type": "commonjs" } to override the parent Vite project's "type": "module" ESM setting.

Frontend auth pages

All five auth pages match the existing design system exactly — PageShell background, bg-white/90 backdrop-blur-sm card, bg-gradient-to-r from-purple-600 to-indigo-600 gradient headings, text-base font size, Lucide React icons (no emojis):

The VerifyOtp page handles all three OTP purposes with a single component — title, subtitle, button label, and back-link all adapt based on the purpose value passed via React Router location.state.

Resend OTP

/api/auth/resend-otp accepts { email, purpose } and regenerates + resends the code. The frontend VerifyOtp page shows a 60-second cooldown timer after each resend to prevent spam. The endpoint supports all three purposes: verify, login, reset.

ML Models Deep Dive

Luméra uses two independent CNN pipelines: one for skin type classification and one for concern detection.

Skin Type Classifier

Takes a 224×224 pixel tight crop of the detected face and outputs one of five classes: Combination, Dry, Normal, Oily, Sensitive. Tries best_model_v2.keras first, falls back to best_model.keras.

v1 — best_model.keras — 85.09% val accuracy

Architecture: MobileNetV2 (ImageNet, include_top=False) → GAP → BatchNorm → Dense(256, ReLU) → Dropout(0.4) → Dense(128, ReLU) → Dropout(0.3) → Dense(5, softmax)

Preprocessing note: MobileNetV2 expects [-1, +1] input. The v1 model has preprocess_input rescaling baked in as Multiply, TrueDivide, Subtract layers. Feed raw [0,1] floats — never call preprocess_input() manually on v1 inputs or predictions will be garbage.

v2 — best_model_v2.keras — improved architecture

Addresses overconfident predictions and poor separation of similar skin types.

Problem 1 — Overconfident predictions. v1 predicted "Normal: 97%" for ambiguous faces. Cross-entropy with hard one-hot targets allows the model to reduce loss without bound.

Fix — Label smoothing (ε=0.10). Trains on soft targets [0.02, 0.02, 0.92, 0.02, 0.02]. Applied only to training labels — validation uses hard one-hot for honest metrics.

Problem 2 — Missing texture information. MobileNetV2's final feature map is 7×7 — too coarse for pore texture or subtle shine patterns.

Fix — Multi-scale feature fusion:

• —Fine: block_6_expand_relu → 28×28×192 — 8px receptive field, captures pore/texture detail
• —Coarse: final MobileNetV2 block → 7×7×1280 with Squeeze-and-Excitation recalibration

Squeeze-and-Excitation learns per-channel importance weights to suppress illumination-detector channels that fire on bright pixels regardless of skin type.

Training datasets:

Concern Detection — Three Versions, One Ensemble

Three model versions run together as a weighted ensemble on every inference:

concern_model_v3.keras  →  weight 1.0  (full-face, bbox-aware training)
concern_model_v2.keras  →  weight 0.8  (per-concern branches, F1 monitoring)
concern_model.keras     →  weight 0.5  (legacy — lower weight, training mismatch)

Weighted average → per-class calibration → final concern scores

v1 — concern_model.keras — 95.69% val accuracy

Uses sigmoid (not softmax) — a face can have multiple concurrent concerns.

Architecture: MobileNetV2 → GAP → BN → Dense(256, ReLU) → Drop(0.4) → Dense(128, ReLU) → Drop(0.3) → Dense(6, sigmoid)

Key limitation: Trained on tight zone crops (dark circles filling the entire 224×224 frame), inferred on full faces. Severe train/test distribution mismatch.

v2 — concern_model_v2.keras

Per-concern branches: Each concern gets its own Dense(32) → Drop(0.15) → Dense(1, sigmoid) head. No cross-concern weight entanglement.

F1 monitoring: Uses macro binary F1 (not accuracy) — prevents high accuracy from predicting all-zeros.

v3 — concern_model_v3.keras

Directly solves the train/test distribution mismatch by training on full-face 224×224 images (same as inference).

Bbox-aware soft labels: Each training image gets a soft float 0.40–0.95 based on how well the annotated concern bbox overlaps with the expected anatomical zone:

bbox IoU with expected zone:
  > 0.50  →  label = 0.95  (excellent localisation)
  > 0.15  →  label = 0.75  (good localisation)
  > 0.00  →  label = 0.55  (concern present, unusual position)
  = 0.00  →  label = 0.40  (no overlap — train weakly)
Non-Roboflow images (no bboxes):
  →  label = 0.85  (concern present, no spatial info)

Expected anatomical zones:

Training datasets for v3:

Staged class counts (capped at 2,000):

Face Detection Pipeline

Input image (any resolution, any format)
        │
        ▼ _compress_image()
Resized to max 1024px JPEG — consistent input, prevents timeouts
        │
        ▼
OpenCV Haar Cascade (primary detector)
  ├─ 6 parameter combinations (scaleFactor, minNeighbors, minSize)
  ├─ Runs on both original grayscale AND histogram-equalised version
  ├─ Scores candidates:
  │     score = (face_area / total_area) × 2.5
  │           + eye_bonus (0.35 for 2 eyes, 0.12 for 1, 0.0 for none)
  │           − (horizontal_distance_from_centre / half_width) × 0.5
  └─ Selects highest-scoring candidate
        │
        ├──► DISPLAY CROP (stored in DB as base64)
        │    Padding: 50% L/R, 70% top, 45% bottom
        │    Square-padded with RGB(245,245,245) grey
        │    Resized to 300×300 Lanczos
        │    Stored in analyses.normalized_image_b64
        │
        └──► ANALYSIS CROP (ML inference only, never stored)
             Tight bbox, no padding
             Resized to 224×224 Lanczos
             RGB uint8 numpy array
             → skin type CNN + concern ensemble + CV signals

If Haar fails → MediaPipe BlazeFace (disabled on HF Spaces — protobuf conflict with TF 2.21)
If both fail  → face_found=False, user sees error message

Why two crops? Zone coordinates in skin_concern_detector.py are calibrated to the tight 224×224 analysis crop. Using the padded display crop would cause zone coordinates to point to wrong face regions (e.g. under-eye zone pointing to the forehead).

Face detection confidence:

conf = min(0.55 + eyes × 0.20 + (face_area / total_area) × 5.0, 0.99)

Concern Detection Architecture

The SkinConcernDetector class implements a 4-layer hybrid system.

Layer 1 — Anatomical Zones

12 named zones as (y_start, y_end, x_start, x_end) fractions of the 224×224 analysis crop:

ZONES = {
    'forehead':        (0.04, 0.28, 0.22, 0.78),
    'left_cheek':      (0.38, 0.72, 0.04, 0.38),
    'right_cheek':     (0.38, 0.72, 0.62, 0.96),
    'nose':            (0.32, 0.65, 0.36, 0.64),
    'chin':            (0.72, 0.92, 0.28, 0.72),
    'left_eye':        (0.18, 0.36, 0.10, 0.42),
    'right_eye':       (0.18, 0.36, 0.58, 0.90),
    'under_left_eye':  (0.32, 0.44, 0.10, 0.42),
    'under_right_eye': (0.32, 0.44, 0.58, 0.90),
    'lip':             (0.65, 0.82, 0.30, 0.70),
    't_zone':          (0.04, 0.72, 0.30, 0.70),
    'face_centre':     (0.15, 0.85, 0.15, 0.85),
}

Skin tone factor computed from LAB L-channel mean — adjusts thresholds to prevent false positives on darker skin tones.

Layer 2 — Ensemble ML + CV Signals

ML ensemble: Weighted average of all loaded models, then per-class calibration:

calibrated = max(0.0, (raw_prob - baseline) / scale)

CV-only signals (always run regardless of ML models):

eye_bags — brightness delta between under-eye and eye socket in LAB L-channel. Gate lowered to 3.0 delta units (was 8.0). Three-component score: brightness delta + row variance + row-to-row std.

lip_hyperpigmentation — L-channel comparison of lip vs face_centre, plus purple/blue HSV hue ratio. Scaled by skin tone factor.

Layer 3 — Cross-Concern Calibration

• —acne > 0.35 → add 0.06 to redness (inflamed acne involves redness)
• —CV-only signals use display gate 0.08; ML signals use 0.15
• —All scores clamped to [0.0, 1.0]

Layer 4 — Zone Annotation Images

Per-concern 224×224 annotated images with semi-transparent fills (22% opacity), coloured borders (2px), white inner highlight (25%), severity-coloured label pills. Upscaled to 300×300 and stored as base64 in skin_concerns.annotated_image_b64.

Severity Thresholds

API Reference

All endpoints except /api/auth/register, /api/auth/login, /api/auth/verify-otp, /api/auth/resend-otp, /api/auth/send-login-otp, and /api/auth/forgot-password require Authorization: Bearer <token>.

Auth — /api/auth

POST `/verify-otp` body: { "email": "...", "otp": "123456", "purpose": "verify|login|reset" }

• —verify → activates account, returns full JWT + user object
• —login → returns full JWT + user object (passwordless login)
• —reset → returns 15-minute reset_token JWT with { reset: true } claim

Analysis — /api/analysis

POST `/upload` accepts multipart/form-data, field name image. Max 16MB before compression. Formats: PNG, JPG, JPEG, WEBP. Returns HTTP 201 on success, HTTP 200 with success: false if no face detected.

Products — /api/products

POST `/recommend` — Groq generates product names, backend enriches each with Open Beauty Facts image URL.

Request: { "skin_type": "Normal", "concerns": [{"concern_type": "dark_circles", "severity": "moderate"}], "count": 5 }

Routines — /api/routines

POST `/generate` — AI routine based on scan. GET `/` — list all. DELETE `/:id` — delete. POST `/:id/activate` — set as active (deactivates others of same type).

Chatbot — /api/chatbot

POST `/chat` — Groq with last 5 scan context injected into system prompt. Frontend sends full conversation history with each request — backend is stateless.

Request: { "message": "What routine should I follow?", "history": [...] }
Response: { "reply": "For your skin type..." }

Report — /api/report

GET `/summary` — 7-day JSON summary with total scans, average confidence, dominant skin type, recurring concerns.

GET `/weekly` — PDF file streamed from BytesIO buffer (no temp files written to disk).

Database Schema

users

analyses

skin_concerns

routines

routine_steps

Frontend Pages

Home (/) — Landing page for unauthenticated users

Feature cards explaining Upload → Analyse → Recommend workflow. Stats strip (95% accuracy, 9 concerns, AI routines). PageShell background: #f5f3ff + dot-grid SVG + two blurred purple accent circles.

Login (/login)

Two-tab card: "Password" tab (standard email + password with Forgot Password link) and "Email Code" tab (passwordless — sends OTP to inbox, redirects to VerifyOtp). On unverified-account login, backend returns requires_verify: true and frontend silently redirects to VerifyOtp. Uses LogIn and Mail Lucide icons.

Signup (/signup)

Registration form with username, email, password, confirm password. On success, redirects to /verify-otp — does NOT issue a JWT immediately. Unverified accounts cannot access any protected route.

VerifyOtp (/verify-otp)

Shared 6-digit OTP entry page. Receives { email, purpose } via React Router location.state. Six individual digit input boxes with auto-focus-advance, backspace-retreat, and paste support. 60-second resend cooldown. All copy adapts based on purpose: verify / login / reset. Uses ShieldCheck, KeyRound, Mail Lucide icons.

ForgotPassword (/forgot-password)

Single email field. Always navigates to /verify-otp after submit regardless of whether the email exists (prevents account enumeration). Uses KeyRound Lucide icon.

ResetPassword (/reset-password)

Receives resetToken via React Router location.state — shows "Invalid session" if missing. Password + confirm fields with show/hide toggle (Eye/EyeOff icons). 4-segment strength bar (red → yellow → green). Sends PATCH with Authorization: Bearer {resetToken} header.

Dashboard (/dashboard)

Welcome banner with username + live stats. Scan history grid. AuthImage component checks if image_path starts with https:// — if yes, renders directly from Cloudinary. If legacy local path, fetches via Axios with JWT.

Upload (/upload)

Two-column layout: upload/camera form + photo guide. Custom inline SVG illustration showing full-face ✓ vs zoomed ✗. Camera mode with front/rear toggle. Oval guide with aspectRatio: 3/4.2 to encourage stepping back for full face. Amber warning banner explaining why full-face photos are required.

Results (/results/:id)

Three tabs: Concerns (annotated zone images + severity badges + AI notes), Products (horizontal slider with OBF real images, brand initial tile fallback), Routine (AI generation, saves to DB).

Progress (/progress)

Interactive calendar with skin type colour coding per day. Fixed-height day panel with internal scroll — doesn't expand the page regardless of scan count. Skin type distribution bars. Full scan history table.

Chatbot (/chatbot)

Conversation interface. Initial greeting with 4 suggested quick questions. Gradient purple user bubbles, grey assistant bubbles. Typing indicator (3 bouncing dots) while waiting for Groq response.

Routines (/routines)

Accordion cards grouped by Morning and Night. Expanding shows steps with step number circles, product type, instruction, timing, key ingredient. Set Active and Delete per routine.

Weekly Report (/report)

Interactive bar chart (past 7 days, bars coloured by skin type). Click bar → scan detail modal. Recurring concerns section with annotated zone thumbnails. PDF download triggers blob fetch from /api/report/weekly.

Features Deep Dive

Image Compression Pipeline

All uploaded images go through _compress_image() before ML processing:

def _compress_image(filepath, max_dimension=1024, quality=85):
    with PILImage.open(filepath) as img:
        img = img.convert('RGB')
        if max(img.size) > max_dimension:
            ratio = max_dimension / max(img.size)
            img = img.resize((int(img.width * ratio), int(img.height * ratio)), PILImage.LANCZOS)
        new_filepath = os.path.splitext(filepath)[0] + '_compressed.jpg'
        img.save(new_filepath, 'JPEG', quality=quality, optimize=True)
        return new_filepath

A 10MB phone photo becomes ~300KB. ML accuracy is unchanged — face detection and CNN both operate on 224×224 crops regardless of input resolution.

Background Model Loading

Models load in a background thread so the server starts immediately:

thread = threading.Thread(target=_load_models_background, daemon=True)
thread.start()
print("⏳ ML models loading in background...")

On HF Spaces, the models are already on disk (baked into the Docker image via Git LFS). The background thread only needs to run TensorFlow's JIT compilation, which takes ~2–3 minutes. During this window uploads use feature-based analysis as fallback.

Note: download_models.py still exists in the repo but _load_models_background() in app.py no longer calls it. The download logic was removed when migrating to HF Spaces. The file is kept for reference in case a future deployment target requires runtime model download.

Two-Phase Groq Recommendations

1. 1.Initial recommendations generated from skin type + confidence only
2. 2.After concern detection, regenerated with full context (skin type + all concerns + severities)
3. 3.Second set overwrites first in the database

Ensures concern-specific advice: "use eye cream with caffeine" instead of generic skin type advice when dark circles are detected. Falls back to static dict if Groq is unavailable.

Dynamic Products with Real Images

resp = requests.get('https://world.openbeautyfacts.org/cgi/search.pl', params={
    'search_terms': f"{brand} {product_name}",
    'json': 1, 'page_size': 5,
    'fields': 'product_name,brands,image_front_url,image_url',
}, timeout=4)

4-second timeout prevents blocking the UI. OBF has strong coverage of CeraVe, The Ordinary, La Roche-Posay, Neutrogena, Kiehl's. Falls back to branded initial tile for unrecognised brands.

Training the Models

Skin Type Model v2

cd lumera/backend
source venv/bin/activate

kaggle datasets download -d shakyadissanayake/oily-dry-and-normal-skin-types-dataset -p dataset_downloads/ds1 --unzip
kaggle datasets download -d ritikasinghkatoch/normaldryoily-skin-type -p dataset_downloads/ds2 --unzip
kaggle datasets download -d killa92/facial-skin-analysis-and-type-classification -p dataset_downloads/ds3 --unzip

python ml_model/merge_datasets.py
python ml_model/train_model.py   # saves to ml_model/best_model_v2.keras

Concern Model v3

# Set ROBOFLOW_API_KEY in backend/.env first
python download_concern_datasets.py

python ml_model/train_concern_model_v3.py          # dry run
python ml_model/train_concern_model_v3.py --train  # actually train

Critical implementation notes:

• —Label path: Path(str(img_path).replace('/images/', '/labels/')).with_suffix('.txt') — NOT Path.stem (truncates multi-dot Roboflow filenames like 125_jpg.rf.abc123.jpg)
• —ds_rf1 uses YOLOv8-OBB polygon format — parser handles both standard (4 coords) and polygon (8+ even coords)
• —Phase 3 must use make_callbacks(reduce_lr=False) — ReduceLROnPlateau incompatible with CosineDecay
• —compile=False at inference — custom BinaryF1 metric not registered with Keras serialisation

Complete Bug Fix History

Known Limitations

HF Spaces free tier sleeping. Without UptimeRobot, the Space sleeps after ~48 hours of inactivity. First request after sleep takes ~10 seconds. With UptimeRobot pinging every 5 minutes, this never happens in practice.

Single Gunicorn worker. Even with 16 GB RAM, one worker means two simultaneous uploads queue — one waits for the other. This is acceptable at 10–50 scans/day. For higher traffic, increase --workers and adjust accordingly.

Ephemeral container filesystem. Like Render, HF Spaces containers reset on redeploy. Uploaded images are stored in Cloudinary (not local disk) and the database is in Neon — both survive redeploys. The only thing lost on redeploy is any temp file in /tmp which is intentional.

Sensitive skin — 80 training images. vs ~3,000 for Normal. Model frequently misclassifies as Normal or Dry. 10× class weight partially compensates.

Redness — 399 training images. Lowest concern class. Only fires on severe redness. Mild redness usually missed. No CV-only fallback for redness.

v3 soft labels mostly 0.85. Roboflow polygon annotations tend to cover the full face — IoU with specific anatomical zones is low, most samples fall back to the 0.85 fixed label. Meaningful spatial differentiation requires tightly-annotated datasets.

Open Beauty Facts coverage. Strong for CeraVe, The Ordinary, La Roche-Posay, Neutrogena, Kiehl's. Limited for Indian and Asian brands — falls back to brand initial tile.

Calibration demographics. CV signal thresholds tuned on Indian male skin, medium tone, indoor lighting. Edge cases possible for very dark/light skin, heavy facial hair, unusual lighting conditions.

MediaPipe disabled on HF Spaces. mediapipe==0.10.14 has a protobuf<5 requirement that conflicts with TensorFlow 2.21's protobuf>=6. Haar cascade alone handles all face detection on production. This was also the case on Render.

Gmail OTP send rate. Gmail free SMTP allows ~500 emails/day. Sufficient for personal/indie use. For higher traffic, migrate frontend/api/send-email.js to use a transactional email API (Resend, Postmark, SendGrid) by changing the nodemailer transport.

OTP email delivery to spam. Emails sent from a personal Gmail account (noreplylumera@gmail.com) via a Vercel relay may occasionally land in spam, especially for first-time recipients. Adding the sender to contacts resolves this. A custom domain with SPF/DKIM records would eliminate it.

Roadmap

• —[ ] Sensitive skin data — 400+ labelled images to reach parity
• —[ ] Redness data — 1,000+ rosacea images
• —[ ] eye_bags ML class — dedicated training images, 7th ML class in v4
• —[ ] Amazon PA API — official product images instead of OBF
• —[ ] Streaming chatbot — SSE for word-by-word Groq responses
• —[ ] Scan journal — free-text notes per scan (diet, products, sleep, stress)
• —[ ] Streak tracking — gamification with scan consistency badges
• —[ ] Before/after comparison — side-by-side scan view
• —[ ] Weekly email digest — Flask-Mail + scheduled job
• —[ ] PWA — manifest.json + service worker for mobile install
• —[ ] v3 tighter annotations — datasets with tight per-concern bboxes for meaningful soft label spread
• —[ ] Upgrade HF Spaces — paid tier for persistent storage and guaranteed uptime SLA
• —[ ] Confidence intervals — uncertainty range on predictions
• —[ ] Multi-language support — especially for Indian skin tone annotations
• —[ ] Skin type v3 — full-face + spatial annotation if localised datasets become available
• —[ ] Custom domain for Gmail sender — add SPF/DKIM records to eliminate OTP spam risk
• —[ ] OAuth login — Google/GitHub sign-in as alternative to email+password