Datasetpublic

levossadtchi/RuLips-1k

RuLips-1k: recipe for a 1,100-hour Russian lip-reading corpus RuLips-1k is a recipe dataset: it contains everything needed to rebuild a 1,134-hour corpus of Russian talking faces (423,047 clips, 224×224, 25 fps, one speaker per clip, transcript with word timestamps, 478 MediaPipe face landmarks per frame, head-pose angles), but no video. Videos are re-downloaded from their original hosts (RuTube, YouTube) and cropped by the included scripts, which reproduce our crops… See the full description on the dataset page: https://huggingface.co/datasets/levossadtchi/RuLips-1k.

sourceHugging Facecc-by-nc-4.0updated 15d agoView on Hugging Face

0likes417downloads

Dataset Card

RuLips-1k: recipe for a 1,100-hour Russian lip-reading corpus

RuLips-1k is a recipe dataset: it contains everything needed to rebuild a 1,134-hour corpus of Russian talking faces (423,047 clips, 224×224, 25 fps, one speaker per clip, transcript with word timestamps, 478 MediaPipe face landmarks per frame, head-pose angles), but no video. Videos are re-downloaded from their original hosts (RuTube, YouTube) and cropped by the included scripts, which reproduce our crops pixel-for-pixel (verified on 352 clips: IoU of face boxes 1.000 median, max deviation 1.6 px).

	RuTube part	MultiVSR part (Russian subset, re-cut)	total
clips	70,737	352,310	423,047
hours	422.3	712.2	1,134.4
videos	2,033	7,197	9,230
speakers (channels / videos)	933 channels	7,197 videos
transcript	Whisper large-v3-turbo, per clip, word timestamps	MultiVSR authors' text
audio in rebuilt clip	yes (16 kHz mono)	no
split	speaker-disjoint by channel	MultiVSR official test (whole videos)

Split sizes (hours): train 1,103.2 · val 12.5 · test 18.7 (RuTube 6.1 / MultiVSR 12.6).

What is in the repository

manifest.jsonl          one line per clip — where it comes from, its boundaries, text, quality
splits.json             {id: "train" | "val" | "test"}
landmarks/NNNNN.tar     478 MediaPipe landmarks per frame + head pose, one .npz per clip (91 shards, 86 GB)
landmarks/NNNNN.idx     byte offsets of members → read one clip with an HTTP range request
windows/multivsr/       per-frame crop windows for MultiVSR clips (npz per clip, sharded)
recipe/                 build.py, verify.py, faces.py, rutube.py, YuNet weights, requirements
docs/quality.md         quality report with figures
docs/collection.md      how the corpus was collected and filtered

manifest.jsonl (per clip)

field	meaning
`id`	`<source>_<video>_<start_frame>-<end_frame>`, stable
`source`, `video`, `url`	`rutube` / `multivsr`; video id; link to the source video
`channel`, `speaker_id`	RuTube channel name and id (`rutube:<author_id>`); for MultiVSR `youtube:<video>`
`src_start_s`, `src_end_s`, `src_start_frame`, `src_end_frame`	clip boundaries in the source (25 fps timeline)
`fetch_span`	RuTube only: the [t0, t1] window that was downloaded (HLS) — download this, not the whole video
`track_t0_s`, `bbox_first`, `bbox_mid`	RuTube only: start of the face track inside `fetch_span`, detector boxes of its first and middle frame (source pixels) — used to identify the track and to verify your rebuild
`dur_s`, `fps`, `frames`	clip length
`text`, `text_norm`, `words`, `segments`, `asr_source`	transcript; `words` = [{w, s, e}] in clip seconds
`text_prev`, `asr_disagree`	RuTube: the earlier span-level transcript and its character error rate vs `text` (≥0.5 flags dubious clips)
`sync_conf`, `offset_frames`	RuTube: SyncNet confidence and audio offset found at collection
`subs_frac`, `mouth_rel`	RuTube: fraction of frames with burned-in subtitles; mouth box (relative)
`bbox_scale`, `align_iou`, `aligned`, `mv_track`, `mv_split`	MultiVSR: agreement of our re-cut with the authors' track, their track id and split label
`face_coverage`, `pose_az_med/iqr`, `pose_zen_med/iqr`, `pose_roll_med/iqr`, `mouth_open_med`	from landmarks: share of frames with a face, head azimuth / zenith / roll (degrees, 0 = facing camera), mouth opening (fraction of inter-ocular distance)
`landmarks_shard`, `landmarks_member`	where the clip's landmarks live
`split`	train / val / test

landmarks (per clip, .npz inside a shard)

key	shape	meaning
`valid`	[T] bool	face found in this frame
`lm`	[T, 478, 3] int16	landmarks as deltas between consecutive valid frames, quantised to 1/8 px of the 224-px clip (`x = cumsum(deltas)/1792`, see `recipe/verify.py:unpack`); x, y in fractions of the frame, z as MediaPipe gives it
`angles`	[T, 3] float32	azimuth, zenith, roll in degrees (0 = frontal)
`mouth_open`	[T] float32	mouth opening / inter-ocular distance

Rebuilding the videos

bash

pip install -r recipe/requirements.txt          # opencv, numpy, scipy, yt-dlp; ffmpeg on PATH
python recipe/build.py --manifest manifest.jsonl --out clips --split test
python recipe/verify.py --manifest manifest.jsonl --clips clips --landmarks landmarks

build.py downloads only the needed windows of each RuTube video (HLS segments, 720p, converted to 25 fps), re-detects faces with the same detector and parameters (YuNet, stride 3, half-resolution, median-13 smoothing, crop window 2.8× the face half-size, see recipe/faces.py), identifies the clip's track by track_t0_s and bbox_first, and cuts the clip. MultiVSR clips are cut from the yt-dlp download using the per-frame windows in windows/multivsr. verify.py checks frame counts and that the face found in your clip matches the stored landmarks (IoU ≥ 0.8).

Throughput measured on a 16-core machine: download ≈ 7 MB/min of source per minute of video, face detection 5.5× real time. The full RuTube part (529 h of spans) is ≈ 140 machine-hours.

Some videos will have disappeared from the hosts since collection (September 2026); build_log.jsonl lists them.

How it was collected (short)

RuTube: channels were selected for talking-head content; videos were scanned for speech, only the promising windows were downloaded; faces were detected (YuNet) and tracked, tracks were cut at scene changes, filtered by size and length; audio-visual synchrony (SyncNet) selected the speaking face; Whisper large-v3-turbo produced the transcript, first per window (kept as text_prev), then per clip with word timestamps. MultiVSR: the Russian subset of MultiVSR (YouTube) was re-cut from the authors' face tracks with the same crop convention; text and split come from the authors. Details, filters and known defects: docs/collection.md.

Known limitations

—All RuTube clips but 13 have per-clip transcripts (asr_source); the 13 keep the window-level transcript because the per-clip pass failed on them.
—asr_disagree ≥ 0.5 (6,292 clips, 8.9%) marks clips where the two transcripts differ a lot: music, several speakers, or a hallucinated line. Filter them if you need clean text.
—Speaker identity for RuTube is the channel: a channel may host several people.
—Rebuilt RuTube clips match ours frame for frame (frames is the exact frame count of the original clip, equal to the number of landmark rows); the audio track is aligned within one frame of the original (the collection-time SyncNet shift is applied).
—MultiVSR test/val labels are per track; we assign whole videos (a video with any test track is test), so our MultiVSR test is 12.7 h instead of the authors' 10.3 h.

Integrity audit

Every published file was checked against the manifest on 9 September 2026 (shard completeness, LFS sizes vs. actual, index ↔ tar ↔ manifest, random members read from every shard with frame counts verified, windows, splits, card numbers, recipe): docs/audit-2026-09-09.md. The audit script lives in the project repository (shared/scripts/release/audit_release.py).

License and terms

The metadata, landmarks and scripts are released under CC BY-NC 4.0. The videos remain the property of their authors on RuTube and YouTube and are not redistributed; use the recipe under the terms of those platforms and for research purposes.

Citation

@misc{rulips1k2026,
  title  = {RuLips-1k: recipe for a 1,100-hour Russian lip-reading corpus},
  author = {Ossadtchi, Lev},
  year   = {2026},
  url    = {https://huggingface.co/datasets/levossadtchi/RuLips-1k}
}