ridwanalfarezi/inez-normalizer

Indonesian Slang Normalizer

A high-performance full-stack application for normalizing Indonesian slang to formal Indonesian. It features a hybrid normalization pipeline, user authentication, and per-user custom wordlists.

Project Overview

This project consists of a FastAPI backend and a modern React frontend (built with Vite, TailwindCSS, and shadcn/ui). The normalization process uses a hybrid approach:

1. 1.Lookup Table: Fast matching for 1000+ common slang abbreviations.
2. 2.mT5 Model: Deep learning-based inference (supports local or HuggingFace models).
3. 3.Guardrails: Validation to prevent hallucinations and character repetition.
4. 4.Levenshtein Fallback: Distance-based correction against a formal Indonesian vocabulary.

Project Structure

inez/
├── app/                # FastAPI backend source code
│   ├── routers/        # API route handlers (auth, user)
│   ├── model.py        # Model loading and inference logic
│   ├── lookup.py       # Slang lookup table implementation
│   └── ...
├── frontend/           # React + Vite + TailwindCSS frontend
│   ├── src/            # Frontend source (components, hooks)
│   └── ...
├── local_model/        # (Optional) Directory for local model files
├── gdrive_download.py  # Script to download model from Google Drive
├── main.py             # Backend entry point
├── requirements.txt    # Python dependencies
└── .env                # Environment configuration (not committed)

Features

• —Hybrid Normalization: Combines speed (lookup) with intelligence (mT5).
• —Modern UI: Clean, responsive React dashboard with authentication.
• —User Persistence: JWT-based auth with personal normalization history.
• —Global Slang Dictionary: Admin users can define custom slang→formal mappings that take the highest pipeline priority and apply to all users. Non-admins have read-only access.
• —Evaluation Metrics: Calculates true Levenshtein Character Error Rate (CER) and smoothed BLEU-4 scores for each normalization.
• —Local Inference: Optimized for running models locally without cloud dependencies.

Quick Start

1. Prerequisites

• —Python 3.9+
• —Node.js (for frontend development)
• —Bun or NPM (Bun is used in the codebase)

2. Backend Setup

# Create and activate virtual environment
python -m venv .venv
.venv\Scripts\activate  # Windows
source .venv/bin/activate  # Linux/Mac

# Install dependencies
pip install -r requirements.txt

# Setup environment variables
copy .env.example .env

3. Model Setup (Optional but Recommended)

You can use the HuggingFace model or download a local version for better performance.

To use the local model:

1. 1.Set GDRIVE_MODEL_FOLDER_ID in your .env.
2. 2.Run the download script:

    python gdrive_download.py

1. 1.Update MODEL_PATH=./local_model in .env.

4. Running the Application

Development Mode (Backend + Frontend):

1. 1.Start the backend:

    python main.py

1. 1.Start the frontend (in a separate terminal):

    cd frontend
    bun install
    bun run dev

Production Mode (Single Server):

1. 1.Build the frontend:

    cd frontend
    bun install
    bun run build

1. 1.Run the backend (it will serve the built frontend at /ui):

    cd ..
    python main.py

Environment Configuration

API Documentation

Once the backend is running, you can access:

• —Swagger UI: http://127.0.0.1:8000/docs
• —ReDoc: http://127.0.0.1:8000/redoc
• —Frontend UI: http://127.0.0.1:8000/ui (after building)

Testing

Run the test suite using pytest:

PYTHONPATH=. pytest

License

This project is licensed under the MIT License.

Deployment (Hugging Face Spaces)

Live demo: https://huggingface.co/spaces/ridwanalfarezi/inez-normalizer

This project can be deployed to Hugging Face Spaces. Two common approaches are documented below: using the project's Dockerfile (recommended for full backend + frontend) or deploying a lightweight Gradio/Streamlit wrapper.

Option A — Docker (recommended)

1. 1.Create a new Space on Hugging Face and choose Docker as the SDK.
2. 2.Push this repository to the Space (you can connect the Space to a Git repository or push directly to the Space Git remote).
3. 3.In the Space settings, set the following Secrets / Environment Variables:
4. 4.SECRET_KEY: a long random string for JWT signing
5. 5.PORT: 7860 (Spaces route traffic on port 7860)
6. 6.DATABASE_URL: sqlite:////app/data/slang_normalizer.db (when using Persistent Storage)
7. 7.HF_API_TOKEN (optional, only if you use upload_model.py from the Space)
8. 8.(Optional) Enable Persistent Storage in the Space settings if you want the SQLite database and uploads to survive restarts. When enabled, the persistent path is available at /app/data inside the container.
9. 9.The included Dockerfile builds the frontend and backend together; the container will run the app entrypoint. Make sure PORT is set to 7860 so FastAPI binds to the port expected by Spaces (the app reads PORT via main.py).

Notes:

• —The app uses PORT and HOST environment variables (see .env.example). On Spaces set PORT=7860.
• —For persistent SQLite storage use the DATABASE_URL value above. The URI uses four slashes for an absolute path on Linux containers: sqlite:////app/data/slang_normalizer.db.

Note: this repository does not use Gradio. The recommended deployment path for Hugging Face Spaces is the Docker-based approach described above.

Connecting to an existing Space

If you already have a Space (for example the live demo linked above), you can push updates by adding the Space Git remote and pushing to it:

# Example: add HF Spaces remote (replace <username> and <space> with your values)
git remote add hf https://huggingface.co/spaces/ridwanalfarezi/inez-normalizer
git push hf main

After pushing, update the Space settings (Secrets, Persistent Storage) as needed and redeploy.