Piyush23890/Sign_Language_Decoder

🤟 SignBridge — Indian Sign Language Smart Communication System

Real-time ISL gesture → Text → Speech, with live English ↔ Hindi translation. Single-click desktop app — no Python required for end users.

Table of Contents

1. 1.What is SignBridge?
2. 2.Features
3. 3.System Architecture
4. 4.Project Structure
5. 5.Quick Start (End Users)
6. 6.Developer Setup
7. 7.Data Collection
8. 8.Training Models
9. 9.Running the App
10. 10.Building the .exe
11. 11.Controls & Keyboard Shortcuts
12. 12.Tech Stack
13. 13.Troubleshooting

What is SignBridge?

SignBridge converts Indian Sign Language (ISL) hand gestures captured via a standard webcam into readable English text and spoken audio — in real time.

It bridges the communication gap between India's ~6.3 million hearing-impaired ISL users and the general public, requiring no specialist hardware beyond a laptop camera.

Features

System Architecture

Webcam Frame (OpenCV)
        │
        ▼
MediaPipe Hands
  21 landmarks × 3 coords × 2 hands = 126 features
        │
        ▼
  Motion Score  =  ‖keypoints_t − keypoints_{t-1}‖
        │
   ┌────┴────────────────────┐
   │ motion < 0.10           │ motion ≥ 0.10
   ▼                         ▼
Random Forest            ONNX LSTM
(static A–Z)         (dynamic words)
   │                         │
   └──────────┬──────────────┘
              ▼
     SentenceBuilder
  (auto-space · backspace · clear)
              │
              ▼
    Deep-Translator  (EN ↔ HI)
              │
        ┌─────┴─────┐
        ▼           ▼
  WebSocket      Video Feed
  (SocketIO)     (MJPEG stream)
        │
        ▼
   Browser UI
  (HTML + JS)
        │
   Web Speech API
  (TTS + STT)

Project Structure

SignBridge/
│
├── app.py                      ← Main Flask application (run this)
├── sentence_builder.py         ← SentenceBuilder class (also importable)
│
├── templates/
│   └── index.html              ← UI template (Jinja2)
│
├── static/
│   ├── style.css               ← Dark glassmorphism styles
│   └── script.js               ← WebSocket + STT/TTS + clipboard JS
│
├── dataset/                    ← Static landmark CSVs (created by collection)
│   ├── A/data.csv
│   ├── B/data.csv
│   └── ...Z/data.csv
│
├── dynamic_dataset/            ← Dynamic .npy sequences
│   ├── hello/
│   │   ├── 0.npy ... N.npy
│   └── thank_you/
│       ├── 0.npy ... N.npy
│
├── isl_alphabet_model.pkl      ← Trained static RF model  (Git LFS)
├── dynamic_sign_model.h5       ← Trained LSTM model       (Git LFS)
├── dynamic_sign_model.onnx     ← ONNX version for runtime (Git LFS)
│
├── hand_landmarks_dataset.py   ← Collect static data
├── collect_dynamic_data.py     ← Collect dynamic data
├── merge_dataset.py            ← Merge per-letter CSVs → final_dataset.csv
├── train_model.py              ← Train Random Forest
├── train_dynamic_model.py      ← Train LSTM
├── convert_to_onnx.py          ← Convert .h5 → .onnx
├── run_setup_wizard.py         ← All-in-one first-time setup
├── live_predict.py             ← Standalone webcam prediction demo
│
├── test_sentence_builder.py    ← Unit tests
├── requirements.txt
├── SignBridge.spec             ← PyInstaller build spec
├── .gitignore
└── .gitattributes              ← Git LFS config

Quick Start (End Users)

1. 1.Download SignBridge.exe from the dist/ folder (or the GitHub Release).
2. 2.Double-click SignBridge.exe.
3. 3.Your browser opens automatically at http://127.0.0.1:5000.
4. 4.Show your hand to the webcam and start signing ✋.

No Python, no installation required.

Developer Setup

1 — Clone the repository

git lfs install
git clone https://github.com/HetviPandav123/sign-language-smart-communication.git
cd sign-language-smart-communication
git lfs pull        # download model files tracked via LFS

2 — Create a virtual environment

python -m venv venv

# Windows
venv\Scripts\activate

# Linux / macOS
source venv/bin/activate

3 — Install dependencies

pip install -r requirements.txt

GPU users: Replace tensorflow with tensorflow-gpu in requirements.txt CPU-only machines: Use tensorflow-cpu instead

Data Collection

Option A — Automated wizard (recommended for first-time setup)

python run_setup_wizard.py

The wizard guides you through:

• —Recording 50 samples per letter (A–Z) with on-screen prompts
• —Recording 30 gesture sequences each for "Hello" and "Thank You"
• —Training both models automatically after collection

Option B — Manual collection

Static signs (A–Z):

# Collect 200 samples for letter A
python hand_landmarks_dataset.py --sign A --samples 200

# Repeat for B through Z
python hand_landmarks_dataset.py --sign B --samples 200
# ...

Dynamic gestures:

python collect_dynamic_data.py --action hello     --samples 200
python collect_dynamic_data.py --action thank_you --samples 200

Tips for good data: - Use consistent lighting (avoid backlighting) - Vary hand distance (30–80 cm from camera) - Slightly vary the angle between samples for robustness

Training Models

Train static Random Forest model

python train_model.py

Output: isl_alphabet_model.pkl + label_map.pkl

Train dynamic LSTM model

python train_dynamic_model.py

Output: dynamic_sign_model.h5

Convert LSTM → ONNX (required for app.py)

python convert_to_onnx.py

Output: dynamic_sign_model.onnx

Merge static CSVs (optional — for inspection)

python merge_dataset.py

Output: final_dataset.csv

Running the App

python app.py

The browser opens automatically at http://127.0.0.1:5000.

If the browser doesn't open, navigate there manually.

Building the .exe

Requires both models to be trained and present first.

pip install pyinstaller
pyinstaller SignBridge.spec

The executable is written to dist/SignBridge.exe.

The spec file excludes TensorFlow from the bundle (it is mocked at runtime) which keeps the .exe size manageable. Only onnxruntime is bundled for inference.

Controls & Keyboard Shortcuts

In-app buttons

Keyboard shortcuts

Tech Stack

Troubleshooting

"Paging file too small" on Windows

Caused by Flask debug mode + memory-mapped files. Fix: app.py already sets debug=False. If you still see this, restart your PC to clear the paging file.

Webcam not detected

# Check which index works (try 0, 1, 2)
python -c "import cv2; cap=cv2.VideoCapture(0); print(cap.isOpened())"

Change cv2.VideoCapture(0) in app.py to the correct index.

MediaPipe import error in .exe

The SignBridge.spec already includes collect_all('mediapipe') and sys._MEIPASS path resolution. Rebuild with the provided spec — do not use pyinstaller app.py directly.

Static sign flickering

Each letter is locked after STATIC_FRAMES=5 stable frames and won't repeat until the hand moves away. Increase STATIC_FRAMES in app.py for stricter locking.

Poor recognition accuracy

• —Collect more samples per sign (200+ recommended)
• —Vary lighting conditions during collection
• —Ensure both hands are visible for two-handed signs
• —Re-train with the new data

Translation not working

Requires an internet connection. deep-translator uses Google Translate API. If offline, the raw English sentence is shown as fallback.

Author

Hetvi Pandav BE – Artificial Intelligence & Machine Learning

⭐ If SignBridge helped you, star the repo!