diddmstjr/ave-speech-lipemg-processed

AVE Speech — preprocessed for lip–EMG fusion

Code: diddmstjr07/silent-speech-viseme-emg — model definition, training, evaluation, and the analyses behind these numbers.

Related releases: checkpoints · AVE preprocessed · Confusable-100

A derivative of the AVE Speech corpus (Zhou et al., IEEE THMS 2025), preprocessed into the exact form used to train the lip–EMG fusion models in the companion work. This is not new recorded data; it is the original corpus with the preprocessing pipeline already applied, released so the experiments are reproducible without re-deriving it.

Please cite and prefer the original corpus. If you want raw AVE Speech, get it from the authors at the link above.

What is in it

2015 sentence-level utterances from 10 speakers, each stored as a directory:

s{subject}_ss{session}_{index}/
  lip.npy    (T, 96, 96)  uint8    mouth ROI, grayscale
  emg.npy    (2000, 6)    float32  filtered surface EMG
manifest.json

manifest.json gives, per utterance: name, text (romanized pinyin), relative paths to lip.npy / emg.npy, frame and sample counts, and subject / session / idx for constructing speaker-independent splits.

Preprocessing applied

Video. The original AVE video is already a mouth close-up, so the landmark-based crop used by standard lip-reading pipelines is unnecessary and failure-prone on it. We substitute a deterministic center square crop and resize to 96×96 grayscale. Caveat: we did not verify quantitatively that this matches the crop geometry the pretrained front-end expects — it may contribute to a weaker visual baseline.

EMG. DC removal, 4th-order Butterworth band-pass 20–110 Hz, 60 Hz notch. Kept at the native 1 kHz. No z-normalization — that is applied at training time, so the stored signal remains close to the filtered original.

Targets. Romanized pinyin from the corpus transcription, space-separated syllables. Upper-case them before tokenizing with an English SentencePiece vocabulary; lower-case maps to unknown tokens. Tone is not encoded, so the task is toneless syllable transcription.

Nothing else is changed. Audio is not included (it was used only to derive syllable boundaries via forced alignment, and is not needed to reproduce the fusion experiments).

Splits used in the companion work

Speaker-independent: subjects 1–8 train, 9–10 test (404 test utterances). No face and no electrode placement is shared across the split. Construct it from the subject field in the manifest.

Loading

import json, numpy as np
items = json.load(open("manifest.json"))
it = items[0]
lip = np.load(it["lip"])   # (T, 96, 96) uint8
emg = np.load(it["emg"])   # (2000, 6)   float32
print(it["text"])          # e.g. "wo e le"

Important limitations

• —Closed set. The corpus is 100 Mandarin sentences repeated by every speaker, so a model can succeed by recognizing which memorized sentence it sees. Systems trained on it reach near-saturation sentence accuracy. No result on this data is an open-vocabulary claim.
• —Ten speakers. Two held-out talkers cannot separate speaker-independent generalization from two individuals' idiosyncrasies.
• —Derivative, not source. Preprocessing choices above are baked in. If they matter to you, start from the original corpus.

License and attribution

CC-BY-NC-SA-4.0, inherited from AVE Speech. Non-commercial use only; attribution required; derivatives must carry the same license.

This dataset is modified from the original: video cropped and resized, EMG filtered, audio removed, manifest added.

@article{avespeech2025,
  author  = {Zhou, Dongliang and Zhang, Yakun and Wu, Jinghan and Zhang, Xingyu
             and Xie, Liang and Yin, Erwei},
  title   = {{AVE} Speech: A Comprehensive Multi-Modal Dataset for Speech Recognition
             Integrating Audio, Visual, and Electromyographic Signals},
  journal = {IEEE Transactions on Human-Machine Systems},
  year    = {2025},
  doi     = {10.1109/THMS.2025.3585165}
}