ntsrigaud/maestro-lstm-hybrid

twostreamattnfinetunehybrid10classfinetune_20260606T150031Z

A real-time hand gesture classifier trained on a Hybrid Jester+IPN gesture dataset (Jester dynamic classes + IPN pointing classes).

This model is part of the Maestro pipeline that enables touchless control of presentation and meeting software through hand gestures captured from a standard webcam using MediaPipe for landmark extraction.

Model Description

• —Architecture: EnhancedTwoStreamLSTM (BiLSTM h=96×2, MHA 8 heads, proj=96, mean+max pool, MLP gate)
• —Parameters: 1,208,554
• —Input: (batch, 16, 147) — 16-frame sliding window at 30 FPS ≈ 533 ms
• —Output: Softmax logits over 10 gesture classes
• —Inference latency: < 1 ms per call (CPU, single sample)
• —Feature schema: feature-schema-v5

Architecture

EnhancedTwoStreamLSTM splits the 147-dim feature vector into two parallel streams and processes them through a BiLSTM + self-attention + MLP-gate pipeline:

Input (B, T=32, 147)
    │
    ├─ Stream A — Pose/Shape (73 dims)
    │   Linear+LN+GELU → 96
    │   2-layer BiLSTM (h=96) → (B, T, 192)
    │   LayerNorm → Self-MHA (8 heads) + residual + post-LN
    │   mean+max pool → pool_LN → ctx_a (B, 192)
    │
    ├─ Stream B — Motion/Dynamics (74 dims)
    │   (identical structure) → ctx_b (B, 192)
    │
    ├─ MLP cross-stream gate
    │   gate_a = Sigmoid(
    │     Linear(96→192)(
    │       Tanh(Linear(192→96)(ctx_b))))
    │   ctx_a  = LN(ctx_a × gate_a + ctx_a)
    │   gate_b = Sigmoid(
    │     Linear(96→192)(
    │       Tanh(Linear(192→96)(ctx_a))))
    │   ctx_b  = LN(ctx_b × gate_b + ctx_b)
    │
    └─ cat(ctx_a, ctx_b) → (384,)
       LN → Linear(384→192) → GELU → Dropout → Linear(192→10)

Design rationale:

• —BiLSTMs encode temporal order via their recurrent cell state — no positional encoding needed.
• —Mean+Max pooling captures both sustained gesture shape (mean) and transient click events (max).
• —The 2-layer MLP gate provides non-linear cross-modal recalibration at ~37 K params (vs ~263 K for full MHA cross-attention with a degenerate mean-pooled query).

Gesture Classes

Gesture Usage In Presentation System

Feature Schema (feature-schema-v5)

How to Use

import torch
from huggingface_hub import hf_hub_download
from maestro.infrastructure.model.checkpoint_loader import load_inference_artifact

# Download the artifact (cached after first call)
local_path = hf_hub_download(
    repo_id="ntsrigaud/maestro-lstm-hybrid",
    filename="two_stream_attn_finetune_hybrid_10class_finetune_20260606T150031Z_inference.pt",
)

# Load the artifact (includes model, class labels, and feature schema)
artifact = load_inference_artifact(
    artifact_path=local_path,
    device=torch.device("cpu"),
)
artifact.model.eval()

# Build a 147-dim feature vector using LandmarkFeatureTransformer
# and fill a 32-frame SlidingWindowSequenceBuffer, then:
with torch.no_grad():
    # tensor shape: (batch=1, T=32, F=147)
    window_tensor = torch.tensor(window_np, dtype=torch.float32).unsqueeze(0)
    logits = artifact.model(window_tensor)
    pred_class = artifact.class_labels[logits.argmax(dim=1).item()]

Training Dataset

• —Source: Hybrid merge of Jester and IPN-Hand windows: Jester provides unknown/swiping/zoom/stopsign classes; IPN-Hand provides pointone and point_two
• —Used classes: 10 (9 active gestures + unknown background)
• —Dataset split: 70% train / 15% val / 15% test (stratified by class)
• —Augmentation: temporal scale ±20%, spatial jitter σ=0.005

Training Strategy

Two-phase transfer learning pipeline:

• —Phase 1 (pretraining): backbone pretrained on Jester (27-class) to learn generic gesture dynamics.
• —Phase 2 (fine-tuning): head replaced and model adapted on Hybrid Jester+IPN 10-gesture vocabulary.
• —Stage A — frozen backbone (10 epoch(s)): classification head trained alone; AdamW with lr=1e-3 for fast convergence.
• —Stage B — full model (up to 80 epoch(s)): all layers jointly fine-tuned.
• —Backbone LR warmup: 5-epoch linear backbone LR ramp (0 → 0.1× head LR) — eliminates the loss spike from cold AdamW state.
• —Class weighting: inverse-frequency weights balance rare and common gestures.
• —Max samples per class: capped to reduce majority-class dominance.
• —Regularisation: label smoothing, dropout, ReduceLROnPlateau scheduler, early stopping.
• —Continual-learning retention terms (replay, distillation, GPM) were not applied — the model trains directly on the target vocabulary.

Training Configuration

Evaluation Results (Test Set)

Per-Class Recall

Limitations and Risks

• —Trained on IPN Hand subjects only. Performance may degrade with unusual hand sizes, skin tones, or lighting conditions not represented in training data.
• —The unknown class represents background/transition frames. At runtime, predictions are filtered through per-class confidence thresholds defined in production_hybrid.yaml.
• —Requires mediapipe>=0.10.14 for landmark extraction at inference time.
• —Not intended for safety-critical or accessibility-critical applications.
• —Performance was measured on a held-out test split from the same dataset; real-world generalisation may differ.

Environmental Impact

Training was performed on CPU/MPS. Estimated training time: ~10 minutes. Estimated CO₂ equivalent: negligible (<0.001 kg CO₂eq).

Generated by the Maestro training pipeline on 2026-06-06.