myned-ai/audio2face-mediapipe-arkit-teacher

audio2face-mediapipe-arkit-teacher

<p align="center"> <img src="./assets/mpvisual.gif" alt="source video · MediaPipe landmarks · MediaPipe mesh" width="640" /> <br/> <em>Left: source video frame (face-cropped). Middle: MediaPipe FaceLandmarker's 478 landmark points. Right: an illustrative subset of <code>mpbs</code> — the 52-channel ARKit blendshape vector shipped in this dataset — as horizontal bars updating per frame.</em> </p>

14,703 emotional-speech clips, each annotated with a 52-channel ARKit blendshape sequence extracted by MediaPipe FaceLandmarker from the source video (or from audio-driven synthesis where no video exists), plus per-frame face-detection confidence and prosody features (pitch + energy).

Reference-only dataset — the original audio/video is not shipped. Each row contains a clip_id and audio_path_hint that let you join with source media you download yourself from CREMA-D, RAVDESS, MEAD, or HDTF.

Released by myned-ai as a complementary teacher signal to its sibling dataset `audio2face-emotion-arkit-teacher` (which provides NVIDIA Audio2Face-3D + LAM_Audio2Expression labels). This dataset's value-add is that the blendshape labels are derived from real video of human actors performing the emotion, rather than synthesized by an audio→face model. For corpora that ship video (MEAD, HDTF, and CREMA-D's video portion), MP saw the actual face; the labels reflect what a vision-based tracker thinks the face looked like. That's a fundamentally different supervision signal from audio-derived NIM/LAM blendshapes.

Why "MediaPipe ARKit"

MediaPipe FaceLandmarker outputs 52 channels with names that mirror the Apple ARKit blendshape vocabulary. That makes its output directly compatible with any rig built for ARKit-52 — including the rest of the audio2face ecosystem.

What's in it

The 52 columns of mp_bs are in canonical ARKit-52 order: browDownLeft, browDownRight, browInnerUp, browOuterUpLeft, browOuterUpRight, cheekPuff, cheekSquintLeft, …, tongueOut.

Gotcha — MediaPipe's native output is NOT in this order. MP's face_blendshapes list prepends a _neutral category at index 0, which shifts every ARKit-named slot by +1, and MP does not emit tongueOut at all (only 52 categories: _neutral + 51 ARKit names). During this dataset's build we remap by name to canonical ARKit-52 order and fill tongueOut with 0.0 for every frame. If you re-run MediaPipe yourself on the same source media, you'll need to perform that same name-based remap before your output can be compared row-wise against mp_bs.

Canonical emotion ids (FROZEN as of 2026-05-24)

0 neutral   1 happy   2 sad   3 surprised   4 angry   5 fear   6 disgust

• —RAVDESS's calm class is folded into neutral (192 clips, too few for a separate class)
• —MEAD's contempt class is dropped (confusable with disgust)
• —HDTF's clips are uniformly labelled neutral (in-the-wild news/political/YouTube speech — no emotion annotations)

Splits

Stratified by (source × emotion_label) so each split keeps the same per-class proportions across all four corpora.

Per-source × class:

Quick start

from datasets import load_dataset
import numpy as np

ds = load_dataset("myned-ai/audio2face-mediapipe-arkit-teacher")

row = ds["train"][0]
print(row["clip_id"], row["source"], row["emotion_label"])

mp_bs = np.array(row["mp_bs"], dtype=np.float32)        # (T, 52)
face_conf = np.array(row["face_conf"], dtype=np.float32) # (T,)
print(f"T = {mp_bs.shape[0]} frames @ 30 fps")
print(f"mean face_conf = {face_conf.mean():.3f}")

# Filter low-confidence frames before training
mask = face_conf > 0.5
mp_bs_clean = mp_bs[mask]

Joining with audio

The dataset does NOT ship audio or video. To use it for training, download each source corpus separately and join via audio_path_hint. For video corpora (MEAD, HDTF) extract audio with ffmpeg.

import os, soundfile as sf

CORPUS_ROOTS = {
    "cremad":  "/path/to/CREMA-D",   # audio at AudioWAV/<clip_id>.wav
    "ravdess": "/path/to/RAVDESS",   # audio at Actor_xx/<id7>.wav
    "mead":    "/path/to/MEAD",      # video at M0xx/video/front/<emo>/level_<n>/<clip>.mp4
    "hdtf":    "/path/to/HDTF",      # video at <speaker>/<segment_range>.mp4
}

src_path = os.path.join(CORPUS_ROOTS[row["source"]], row["audio_path_hint"])
if src_path.endswith(".wav"):
    audio, sr = sf.read(src_path)
else:
    # ffmpeg -i <mp4> -vn -ar 16000 -ac 1 <out.wav>
    ...

See examples/join_with_audio.py for a runnable loader.

How mp_bs was generated

For audio corpora (CREMA-D audio-only portion, RAVDESS):

1. 1.The original audio was driven through a synthesis pipeline that produced a talking-face video frame stream.
2. 2.MediaPipe FaceLandmarker (face_landmarker_v2_with_blendshapes.task) was run on each frame at 30 fps.
3. 3.Output blendshapes were aligned to the audio at exactly T = round(audio_duration_s × 30) frames.
4. 4.face_conf is MP's per-frame face-detection score.

For video corpora (MEAD, HDTF, CREMA-D video subset):

1. 1.The source video was decoded to RGB frames at 30 fps.
2. 2.MediaPipe FaceLandmarker was run on each frame directly — the labels reflect what the camera saw of the actor's actual face during the take.
3. 3.face_conf is MP's detection score on each video frame (drops near scene cuts, occlusions, profile views).

⚠️ Known limitations of MediaPipe FaceLandmarker

• —eyeSquint over-fires during blinks (Google MediaPipe issue #5329). When you train against MP eye-region labels, the model can over-emit eyeSquintLeft/eyeSquintRight on every blink frame. Mitigation: mask the squint loss on frames where eyeBlinkLeft > 0.3 OR eyeBlinkRight > 0.3.
• —Profile shots / occlusions drop face_conf to ~0; filter ≥ 0.5 for training stability.
• —MP and Apple ARKit-on-iPhone share names, not calibration. Google designed MP's face_blendshapes head to emit the ARKit-52 vocabulary so MP output drops into ARKit-rigged pipelines (e.g. iOS RealityKit). But MP and Apple's TrueDepth tracker are separately trained models — the per-channel value of, say, eyeSquintLeft = 0.5 is not guaranteed to mean the same facial state as eyeSquintLeft = 0.5 from an iPhone capture. We've measured one specific calibration drift in MP (the eyeSquint over-fire above), and other channels likely have their own. Use this dataset for relative expressive supervision — train a model to match the distribution shipped here. Don't treat it as ground truth for iPhone-ARKit playback.

Source corpora

You need to download each corpus separately from its official source. Licensing varies — combining all four limits this dataset to non-commercial use (MEAD and HDTF are research-only).

Citations:

@article{cao2014cremad,
  title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset},
  author={Cao, Houwei and Cooper, David G and Keutmann, Michael K and Gur, Ruben C and Nenkova, Ani and Verma, Ragini},
  journal={IEEE Transactions on Affective Computing},
  year={2014}, volume={5}, number={4}, pages={377--390},
}

@inproceedings{wang2020mead,
  title={MEAD: A large-scale audio-visual dataset for emotional talking-face generation},
  author={Wang, Kaisiyuan and Wu, Qianyi and Song, Linsen and Yang, Zhuoqian and Wu, Wayne and Qian, Chen and He, Ran and Qiao, Yu and Loy, Chen Change},
  booktitle={ECCV},
  year={2020}
}

@misc{livingstone2018ravdess,
  title={The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)},
  author={Livingstone, Steven R and Russo, Frank A},
  year={2018}, publisher={Zenodo}, doi={10.5281/zenodo.1188976},
}

@inproceedings{zhang2021hdtf,
  title={Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset},
  author={Zhang, Zhimeng and Li, Lincheng and Ding, Yu and Fan, Changjie},
  booktitle={CVPR},
  year={2021}
}

@misc{mediapipe2023,
  title={MediaPipe Face Landmarker},
  author={Google},
  year={2023},
  url={https://developers.google.com/mediapipe/solutions/vision/face_landmarker},
}

License

This dataset (the parquet files and the schema) is released under CC-BY-NC-4.0 to reflect the most-restrictive license among its constituent corpora (MEAD, RAVDESS, HDTF are non-commercial). The build scripts (build_dataset.py, parse_metadata.py, examples/) are released under Apache-2.0.

Relation to the sibling dataset

If you're building a student model and need complementary teacher signal:

• —`myned-ai/audio2face-emotion-arkit-teacher` — NIM (synthesized from audio) + LAM (synthesized) + 26-D emotion vector. ~14k clips from CREMA-D, RAVDESS, TESS, JL Corpus. Apache-2.0.
• —This dataset — MediaPipe (extracted from real video for video corpora, from synthesized video for audio-only corpora). ~14.7k clips from CREMA-D, RAVDESS, MEAD, HDTF. CC-BY-NC-4.0.

Both ship reference-only, on the same 30 fps timeline and the same 52-channel ARKit ordering. You can join them on (source, clip_id) for the 8,882 clips that appear in both (CREMA-D + RAVDESS), giving you NIM + LAM + MP teacher signal on the same audio.