THULab/audio2face-emotion-arkit-teacher

Audio2Face Emotion ARKit Teacher Labels (TsFile)

Apache TsFile version of `myned-ai/audio2face-emotion-arkit-teacher`.

Overview

14,082 emotional-speech clips, each annotated with two parallel 52-channel ARKit blendshape sequences (NVIDIA Audio2Face-3D-v2.3.1-James and LAM_Audio2Expression) plus a 26-dimensional NVIDIA Audio2Emotion conditioning vector. Released by myned-ai to support teacher-student distillation research: condensing heavy GPU-bound audio2face teacher models into small student networks that run in real time on CPU.

The original dataset is reference-only — the audio itself is not shipped. Each row carries a clip_id and audio_path_hint to join with source audio downloaded separately from CREMA-D, RAVDESS, TESS, or JL Corpus.

• —Clips: 14,082 (train 12,675 / validation 709 / test 698), stratified by (source, emotion_label).
• —Blendshape rate: 30 fps; both teacher sequences and the emotion_26d conditioning vector are retained in full.
• —Splits: the source train / validation / test files are converted into three separate .tsfile files.

Schema (TsFile structure)

The source is one row per clip with variable-length (T, 52) blendshape arrays; TsFile stores scalar fields, so each clip is expanded into one row per frame (the natural time axis at 30 fps) with the 52 channels flattened into scalar columns.

• —Time (INT64, milliseconds) — round(frame_index * 1000 / 30), restarts per clip; frame_index is kept as a FIELD for exact frame reconstruction.
• —clip_id (TAG) — the device dimension (one device per clip, globally unique); query one clip with WHERE clip_id = '...'.
• —frame_index (FIELD, INT64) — frame position within the clip.
• —Scalar metadata (FIELD): source, actor_id, emotion_label, emotion_label_native, intensity, audio_path_hint (points at audio that is not shipped by either the source or this repository), audio_sr, num_frames.
• —Flattened vectors (single-precision FLOAT):
• —nim_bs_0..nim_bs_51 — ARKit-52 blendshapes from NVIDIA Audio2Face-3D-v2.3.1-James.
• —lam_bs_0..lam_bs_51 — ARKit-52 blendshapes from LAM_Audio2Expression (identity 2).
• —emotion_26d_0..emotion_26d_25 — NVIDIA Audio2Emotion v2.2 conditioning vector (constant per clip).

No source column is dropped. The 52 ARKit channels follow the standard ARKit order (browDownLeft, browDownRight, browInnerUp, ..., tongueOut).

Usage

Read the .tsfile files with the Apache TsFile Java or Python SDK.

Source & license

• —Original dataset: https://huggingface.co/datasets/myned-ai/audio2face-emotion-arkit-teacher
• —Author / publisher: myned-ai
• —License: apache-2.0 (source audio, if downloaded separately, is governed by each corpus's own license)