hyunseop/vision-cardio-rppg

Vision Cardio — rPPG Heart-Rate Model (PhysNet, UBFC fine-tuned)

Camera-based remote photoplethysmography (rPPG) model that estimates heart rate from short front-camera face clips. PhysNet-style 3D-CNN that outputs an rPPG waveform; HR is read off as the FFT peak in the 0.7–3.0 Hz band.

Powers the on-device VISION CARDIO iOS app (offline, Korean UI, wellness-only): front camera → HR, with running HR-phase zones (1–5) and weights set/rest recovery coaching measured against a calibrated baseline HR. Built and running on iPhone (iOS 16+); Core ML runs on GPU/CPU.

Files

Performance

Trained on synthetic SCAMPS, then fine-tuned on real UBFC-rPPG faces with a strict by-participant split (no subject leakage).

Fine-tune split: 40 train / 10 val subjects; best-by-val checkpoint (early-stopped — the tiny set overfits after a couple of epochs).

I/O contract

input  "clip"     : (1, 3, T=128, H=112, W=112) float, RGB, [0,1] normalized
output "waveform" : (1, 128)  predicted rPPG pulse
HR = FFT peak in [0.7, 3.0] Hz, fs = clip_frames / clip_seconds = 128 / 20 = 6.4 Hz

Core ML metadata carries clip_frames, clip_size, clip_seconds, fs, hr_band_hz so the app stays in sync automatically.

Usage (PyTorch)

import torch
from ml.physnet import PhysNet, hr_from_wave   # repo: PFSV/ByeongYeok_RnD_NLP (vision_cardio)
ck = torch.load("rppg_physnet_ubfc.pt", map_location="cpu")
m = PhysNet(width=ck["width"]); m.load_state_dict(ck["state_dict"]); m.eval()
wave = m(clip)[0].numpy()           # clip: (1,3,128,112,112) in [0,1]
bpm = hr_from_wave(wave, fs=6.4)

Caveats

• —Wellness/research only — not a medical device, no diagnostic claims.
• —Fine-tuned on a small real set (UBFC); generalization to arbitrary lighting / skin tone / motion is not guaranteed. Use confidence gating before trusting an estimate.

Code, pipeline & iOS app: `PFSV/ByeongYeok_RnD_NLP`, branch `vision-cardio-ubfc-app` → `TUTORIAL_DRILL/neuro-nlp/vision_cardio`.