paew/FaceLip_model_v2

VocaLip Thai — Real-time Lip Reading

Real-time webcam inference for the VocaLip Thai 44-word deployment checkpoint.

Files

• —app.py — real-time webcam inference
• —requirements.txt — dependencies
• —03_vocalip_44words_deployment.pt — deployment checkpoint

Model input contract

The model was fine-tuned in Notebook 03 on cached mouth clips:

• —16 frames
• —112 × 112
• —3 channels
• —cached channel order preserved (BGR)
• —ImageNet normalization:
• —mean [0.485, 0.456, 0.406]
• —std [0.229, 0.224, 0.225]

The checkpoint contains 44 Thai word classes.

Real-time behavior

The webcam sends frames to the Gradio streaming event. The app keeps a sliding window of the latest 16 mouth crops and runs inference on the window.

The current Top-1 prediction is shown immediately. A word is added to the running transcript only after repeated agreement to reduce flicker.

Important preprocessing note

The uploaded training notebooks document the cached-frame format and model preprocessing, but they do not contain the complete original mouth_cache_v3 generation implementation. Therefore this deployment uses MediaPipe Face Mesh to extract the mouth ROI at runtime.

MediaPipe receives a temporary RGB copy for landmark detection. The actual crop is taken from the original BGR frame, so the model input remains BGR.

For production/research claims of exact preprocessing parity, replace extract_mouth_bgr() with the original cache-generation ROI code when it is available.

Scope

This is a closed-vocabulary 44-word lip-reading model. It cannot recognize arbitrary Thai words that were not among the 44 training classes.

The final test metrics stored in the deployment checkpoint are:

• —Accuracy: 20.85%
• —Macro F1: 0.1798
• —Top-3: 38.22%

These are the checkpoint's test metrics, not a guarantee of real-time webcam accuracy.