Modelpublic

Skunk0/openglasses-fingerspelling-ctc

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes3downloads

Model Card

Fingerspelling CTC (Core ML)

On-device American Sign Language fingerspelling recognizer used by the OpenGlasses accessibility tier. A CTC sequence model over MediaPipe Holistic landmarks, converted to a fixed-window Core ML program:

—Input: features (1, 768, 1629) float32 — up to 768 frames × 543 landmarks × xyz (canonical order: face 0–467, left hand 468–488, pose 489–521, right hand 522–542), per-window standardised, NaN→0, zero-padded — plus mask (1, 768) float32 validity.
—Output: (1, 384, 62) CTC logits (stride 2; read the first ⌈T/2⌉ rows; class 0 is the blank, classes 1–59 map through vocab.txt, the last two are auxiliary tokens).
—Greedy decode scores 20.8% mean CER (median 10.8%) on a 300-sequence held-out set of the training corpus's rerun benchmark.

Files

File	Purpose
`Fingerspelling2P.mlpackage`	the Core ML model (fp16, iOS 17+)
`vocab.txt`	CTC charset sidecar (`<blank>` + 59 symbols, one per line)
`holistic_landmarker.task`	MediaPipe holistic landmark extractor consumed alongside the model

License & attribution

—Model weights and conversion: Apache-2.0 (see LICENSE).
—Trained on the Google ASL Fingerspelling corpus (Google / Deaf Professional Arts Network), CC-BY 4.0.
—holistic_landmarker.task is a MediaPipe model asset (Google), Apache-2.0.