AMFORGE/sam-mm-har-checkpoints

SAM-MM-HAR

SAM-MM-HAR is a compact multimodal model for Human Activity Recognition (HAR) on privacy-preserving, non-RGB sensors. It fuses depth, skeleton, inertial (IMU), thermal, infrared and mmWave-radar signals to classify 40 everyday activities, and is designed to run on-device within a strict edge budget.

Built by AMEFORGE Lab (Amega Mike) on a proprietary sparse Transformer architecture, and developed for the CUHK-X Multimodal Human Activity Challenge (Small Model Track, co-located with UbiComp 2026).

⚠️ The architecture internals are proprietary and intentionally not disclosed. This card describes the model's behaviour, interface and results — not its design.

Highlights

The model handles missing modalities gracefully — any available subset of sensors works at inference; absent modalities are simply skipped.

Why non-RGB / privacy-preserving?

The model never sees a colour camera image. It perceives a scene only through physical measurements — depth maps, skeleton keypoints, inertial motion, heat and radar reflections. This preserves visual privacy (no identifiable faces or footage) while retaining enough signal to recognise the activity, which is the deployment reality for healthcare, elderly-care and smart-home monitoring.

Modalities & encoders

A temporal world-model component models the dynamics of human motion across frames, which is the key contributor to recognising movement-based activities.

The 40 activity classes

Daily-living activities centred on autonomy, self-care and household tasks (e.g. 0_Wash_face, 1_Brush_teeth, 36_Walk, 37_Take_medicine, 39_Take_body_temperature, plus cooking, cleaning, exercise and desk activities). See class_mapping.csv from the CUHK-X dataset for the full list.

Results

Cross-subject evaluation (held-out subjects never seen in training).

The CPU-only latency is the notable figure here: the model runs in near real-time on a plain processor with no GPU, which is what makes on-device / edge deployment realistic (e.g. a small home box, a single-board computer).

Strengths: large-motion activities (walking, jumping, squats) are recognised with high accuracy thanks to the temporal world-model. Known limitations: fine seated/hand activities (using a phone, taking medicine, typing) are harder to disambiguate, as they differ only in subtle hand-level detail. Radar is frequently empty and contributes little.

Intended use

• —Privacy-preserving activity monitoring for elderly care and healthcare (fall/inactivity detection, autonomy tracking) on low-power edge devices.
• —Research on multimodal sensor fusion for HAR without RGB.
• —A reusable feature/representation backbone for related sensor-fusion tasks (robotics perception, ambient smart-home understanding).

Out of scope / cautions

• —Not a medical device; predictions must not be used for clinical decisions.
• —Trained on a specific lab sensor configuration; real-world deployment on different sensors requires re-training or fine-tuning on the target rig.
• —Cross-subject generalization is inherently limited by the number of training subjects; expect a drop on populations dissimilar to the training set.

Usage

import torch
from huggingface_hub import hf_hub_download

ckpt = hf_hub_download("AMFORGE/sam-mm-har", "best.pt")
# Use inference.py from the repo, which inlines the architecture and
# the exact preprocessing for each modality:
#   python inference.py --checkpoint best.pt --clip path/to/clip_folder
#   python inference.py --checkpoint best.pt --data <root> --test-csv test.csv --out submission.csv

Each test clip is a folder with per-modality sub-folders (Depth_Color/, IR/, Thermal/, Skeleton/, IMU/, Radar/). The model aggregates per-clip and outputs one activity id (0–39).

Training setup

• —From scratch (no pretrained weights), single GPU.
• —Input: 12 frames/clip at 96×96; skeleton COCO-17; IMU resampled to fixed length.
• —Cross-subject split; label smoothing; data augmentation (flip, brightness/contrast, spatial shift, noise).
• —Checkpoints pushed to the Hub every 500 steps with auto-resume.

Citation

@misc{sam_mm_har_2026,
  title  = {SAM-MM-HAR: Compact Multimodal Human Activity Recognition
            on Privacy-Preserving Sensors},
  author = {Amega Mike},
  year   = {2026},
  note   = {AMEFORGE Lab. Built on a proprietary sparse Transformer architecture.
            Developed for the CUHK-X Multimodal Human Activity Challenge (UbiComp 2026).}
}

Dataset: CUHK-X Small Model Track — CUHK AIoT Lab.

Architecture internals are proprietary and not disclosed. © AMEFORGE Lab 2026.