Modelpublic

luciayen/ksl-word-video-multimodal

sourceHugging Facecc-by-nc-sa-4.0updated 2mo agoView on Hugging Face

0likes

Model Card

KSL Word-Video Multimodal Experiments

This repository contains trained checkpoints and evaluation artifacts for Kenyan Sign Language word-level video experiments.

Included Experiments

—E1 RGB VideoMAE
—E2 RGB MViT V2 Small
—E3 Pose BiLSTM
—E4 RGB + Pose MViT V2 Small + BiLSTM late fusion
—E5 Pose TransSLR
—E6 Pose Region-Aware ST-GCN + Temporal Transformer

Files

—ksl_word_video_multimodal_results.json: full machine-readable results
—ksl_word_video_multimodal_results.csv: results table
—ksl_word_video_multimodal_results.md: markdown summary
—model_checkpoints/: trained PyTorch checkpoints

Results

KSL Word-Level Video Multimodal Results

—Dataset: laurettekazenga/ksl-video-dataset
—Classes: 30
—Split counts: {'train': 1487, 'test': 450, 'val': 300}

Results

experiment	architecture	modality	params	latency_ms	val_top1	val_top5	val_macro_f1	val_weighted_f1	test_top1	test_top5	test_macro_f1	test_weighted_f1	checkpoint	drive_checkpoint
E2rgbmvitv2s	MViT V2 Small RGB classifier	rgb	34253214	35.5923	0.58	0.87	0.55766	0.55766	0.54	0.908889	0.504951	0.504951	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E2rgbmvitv2s_best.pt	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E2rgbmvitv2s_best.pt
E5posetransslr	TransSLR: Linear 225-to-512 + temporal BN + 4-layer Transformer encoder + MLP	pose	8701726	1.70427	0.736667	0.943333	0.749118	0.749118	0.3	0.608889	0.321461	0.321461	/content/kslwordvideomultimodalwork/runs/E5posetransslr_best.pt	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E5posetransslr_best.pt
E4rgbposemvitbilstmlatefusion	MViT V2 Small RGB encoder + BiLSTM pose encoder + late-fusion MLP	fusion	37591966	38.6892	0.473333	0.853333	0.451084	0.451084	0.326667	0.786667	0.318178	0.318178	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E4rgbposemvitbilstmlatefusion_best.pt	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E4rgbposemvitbilstmlatefusion_best.pt
E3posebilstm	2-layer BiLSTM over 64x225 MediaPipe landmarks	pose	2828062	3.63519	0.553333	0.9	0.542375	0.542375	0.304444	0.762222	0.299352	0.299352	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E3posebilstm_best.pt	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E3posebilstm_best.pt
E6poseregionstgcntransformer	Region-aware body/left-hand/right-hand ST-GCN encoders + temporal Transformer	pose	7372222	3.28754	0.503333	0.78	0.434383	0.434383	0.237778	0.626667	0.214148	0.214148	/content/kslwordvideomultimodalwork/runs/E6poseregionstgcntransformer_best.pt	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E6poseregionstgcntransformer_best.pt
E1rgbvideomae	VideoMAE base fine-tuned on RGB clips	rgb	86259486	24.1565	0.303333	0.663333	0.250683	0.250683	0.211111	0.582222	0.172256	0.172256	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E1rgbvideomae_best.pt	/content/drive/MyDrive/NSL/kslwordvideomultimodal/modelcheckpoints/E1rgbvideomae_best.pt

Saved Model Checkpoints

—E5_pose_transslr: /content/drive/MyDrive/NSL/ksl_word_video_multimodal/model_checkpoints/E5_pose_transslr_best.pt
—E6_pose_region_stgcn_transformer: /content/drive/MyDrive/NSL/ksl_word_video_multimodal/model_checkpoints/E6_pose_region_stgcn_transformer_best.pt
—E3_pose_bilstm: /content/drive/MyDrive/NSL/ksl_word_video_multimodal/model_checkpoints/E3_pose_bilstm_best.pt
—E2_rgb_mvit_v2_s: /content/drive/MyDrive/NSL/ksl_word_video_multimodal/model_checkpoints/E2_rgb_mvit_v2_s_best.pt
—E1_rgb_videomae: /content/drive/MyDrive/NSL/ksl_word_video_multimodal/model_checkpoints/E1_rgb_videomae_best.pt
—E4_rgb_pose_mvit_bilstm_late_fusion: /content/drive/MyDrive/NSL/ksl_word_video_multimodal/model_checkpoints/E4_rgb_pose_mvit_bilstm_late_fusion_best.pt

What to report in the paper

—Top-1 accuracy, Top-5 accuracy, macro-F1, and weighted-F1 on validation and test.
—Parameter count, FLOPs if available, and single-sample latency.
—Per-class classification report and confusion matrix for each model.
—Signer overlap audit and duplicate leakage audit.
—Same split, same label map, same preprocessing across all experiments.