sprited/dancing-stick-figures

Dancing Stick Figures — v0.2

A small, fully-labelled synthetic video dataset for learning (and teaching) video diffusion on one consumer GPU.

1,340 clips · 6 s @ 20 fps · 128×128 RGBA · 482,400 frames · 134 text prompts × 10 seeds × 3 cameras · every frame carries the 3D skeleton, camera and G-buffer (depth, normals, part segmentation) that produced it.

<p align="center"><img src="figs/datasetcontactsheet.png" width="900"></p>

Think of it as an MNIST for video generation: small enough that a 64² video diffusion model trains from scratch in a few hours on a 24 GB card, structured enough that you can measure what the model got wrong (missing arm? detached leg? wrong colour?) instead of eyeballing it. It is also a clean playground for the "image model first, then video" curriculum used by Seedance-class systems.

Start here →  **one route, end to end, sized for a 16 GB T4:** look at the data → train a 64² image model → warm-start a video model from it → watch the GIF → measure visible topology and motion. The measured optimization phases take about 25 and 22 minutes; validation and diffusion sampling add time afterward. Same five commands in the [code repo README](https://github.com/sprited-ai/dancing-stick-figures#the-route-colab-t4-1-h--rtx-4090-20-min). **Sibling dataset:** [sprited/dancing-chibi-figures](https://huggingface.co/datasets/sprited/dancing-chibi-figures) — the same motions and cameras rendered as a volumetric chibi character (paired by `clipid`). v0.2 uses an in-domain seed split. Captions remain the raw motion prompts. Seeds 0--7 train, seed 8 validates, and seed 9 tests; every split contains all 134 prompts. Nine of the original 143 prompts were removed after visual QA because their motions do not visibly perform the requested action (see the curation note below). Feedback and issues are welcome.

Which config?

Quick start

from datasets import load_dataset
ds = load_dataset("sprited/dancing-stick-figures", "frames", split="validation")   # 128 px frames + labels ("mini" = 64 px, 0.79 GB)
row = ds[0]
row["color"]            # PIL RGBA image (transparent background, colour-coded bones)
row["text"]             # "A person does the running man dance."
import numpy as np
xy  = np.frombuffer(row["joint_xy"],  np.float32).reshape(27, 2)   # normalised [0,1] image coords
xyz = np.frombuffer(row["joint_xyz"], np.float32).reshape(27, 3)   # metres, figure frame (x left, y up, z fwd)
vis = np.frombuffer(row["joint_visible"], np.uint8)                # 1 = joint visible in this camera

Motion (one row per clip, raw generator output):

mo = load_dataset("sprited/dancing-stick-figures", "motion", split="validation")[0]
T = mo["n_frames"]
P   = np.frombuffer(mo["posed_joints"],   np.float32).reshape(T, 27, 3)     # world, metres
Rl  = np.frombuffer(mo["local_rot_mats"], np.float32).reshape(T, 27, 3, 3)  # per-joint local rotations
fc  = np.frombuffer(mo["foot_contacts"],  bool).reshape(T, 4)

Training a baseline (code: <https://github.com/sprited-ai/dancing-stick-figures>, MIT):

python -m train.cache --data frames --out cache            # uint8 memmap of all frames
python -m train.video_ddpm   --cache cache --size 64 --frames 8 --batch 16 --fast --compile   # UNet, ~13 GB
python -m train.video_dit_fm --cache cache --size 64 --frames 8 --batch 16 --patch 2 --fast --compile   # DiT-FM

What is in a frame

<p align="center"><img src="figs/datasetlabelsrow.png" width="1000"></p>

color (RGBA over white) · seg (bone id per pixel) · depth16 · camera-space normals · `joint_xy` overlay (green = visible, red = occluded)

`frames` config — one row per rendered frame (482,400 rows).

Binary columns are raw little-endian arrays: np.frombuffer(row[col], dtype).reshape(shape).

Skeleton (cskel27, index order): Hips, Spine, Spine1, Spine2, Spine3, Neck, Head, RightShoulder, RightArm, RightForeArm, RightHand, RightHandEnd, RightHandThumb1, LeftShoulder, LeftArm, LeftForeArm, LeftHand, LeftHandEnd, LeftHandThumb1, RightUpLeg, RightLeg, RightFoot, RightToeBase, LeftUpLeg, LeftLeg, LeftFoot, LeftToeBase.

Colour code (figure's own left/right): head, neck, torso, clavicles black; left upper arm ■ #E84030, left forearm+hand ■ #FF9628; right upper arm ■ #286EE6, right forearm+hand ■ #50C8F0; left thigh ■ #C832A0, left shin+foot ■ #FF78C8; right thigh ■ #1E965A, right shin+foot ■ #78DC5A (generator/render.py:PALETTE, keyed by the bone's child joint). Colours are anti-aliased (4× supersampled); if you need hard-edged colour, rebuild it from seg + the palette.

`motion` config — one row per clip (1,340 rows, 327 MB). Raw output of the motion generator (NVIDIA ARDY): posed_joints f32[T,27,3] (world, m), local_rot_mats / global_rot_mats f32[T,27,3,3], root_positions, smooth_root_pos f32[T,3], global_root_heading f32[T,2], foot_contacts bool[T,4], plus frame0_basis f32[3,3] (rows = figure-frame x/left, y/up, z/forward in world coordinates; figure_joints = (posed_joints − Hips) @ basis.T, before bone_scale). Same clip_id (minus the camera suffix), split, group as frames.

Splits

Split is by ARDY generation seed: seeds 0--7 train, seed 8 validation, and seed 9 test. All 134 prompts and all camera families occur in every split, while the underlying source-motion realizations remain disjoint. This is an in-domain generation split; it does not claim zero-shot generalisation to unseen prompt vocabulary.

Prompt curation (v0.2). Nine of the 143 generated prompts are excluded from the release because their ARDY motions do not visibly perform the requested action in this rendered domain: near-static failures (sways to slow music), motion that cannot render (shakes their head no — the head is a filled circle), unrecognisable or absent actions (yoga warrior pose, yoga tree pose, sit up, push up, standing long jump — the figure glides without an airborne phase), interactions with objects that do not exist in the render (climbs over a low wall), and lies down on the floor. The exclusion list with per-prompt reasons ships as prompts/v02_excluded.txt in the code repository. Stillness prompts whose stillness is semantically correct (the idle group, stands- prompts, balances on one leg*) are kept even when the frozen QA flag fires.

How it was made

prompt → NVIDIA ARDY text-to-motion (6 s, 20 fps, seeds 0–9) → 27-joint skeleton in a canonical figure frame, per-clip body jitter (bone lengths ±8 %, stroke, scale) → 3 orthographic cameras per clip (70 % from a canonical set of yaws, 30 % uniform) → z-buffered capsule rasteriser writes colour / depth / normal / segmentation in one pass → parquet. Everything is deterministic from clip_id; the generator is in the repo (generator/).

Motion prompts: 134 hand-written English sentences in 6 groups (dance 33, gesture 29, locomotion 21, transitions 13, idle 10, sport 28); acrobatics and moonwalk were removed during generation QA, and nine further prompts were removed by the v0.2 release curation above (ARDY did not render them faithfully).

Historical v0.1 baselines and the structural evaluator

The results below were produced on the original prompt-disjoint v0.1 partition. They document the released checkpoints, but they are not the official v0.2 seed-split baseline; a matched v0.2 reference run is reported separately.

Because every bone has its own colour, a rendered frame can be parsed: count colour segments per limb, check they touch their parent, and measure colour purity. The v0.1 files called this rule-based checker oracle v0; the current paper uses the more descriptive name structural evaluator. It reports:

• —tvr — topology violation rate: fraction of the eight limb colours whose visible mask has a connected-component count other than one (missing, detached/fragmented, or duplicated colour regions)
• —lie — limb-identity/adjacency error: fraction of the eight expected torso→proximal or proximal→distal colour adjacencies that are absent (a full left/right chain swap preserves this graph and is not detected by v0)
• —cpe — colour purity error: fraction of foreground pixels with an undefined colour
• —clean — fraction of frames with lie = tvr = 0

Real frames do not score 0: an occluded arm looks like a missing arm to a pixel parser. So every number is reported next to the score of real validation frames at the same resolution (the floor); a model at the floor makes these kinds of errors no more often than the data itself. The structural evaluator is blind to geometry such as proportions and joint angles; a validated learned rig estimator remains future work.

Unconditional image models, 512 samples each, 50 sampling steps (oracle v0):

<p align="center"><img src="figs/img64128final_grids.png" width="1000"></p>

Video (64², unconditional, UNet 46 M). Two 8-frame models (from scratch, 85k steps; warm-started from the image model, 61k) and one autoregressive model (--ar_ctx 8: 8 context + 8 new frames per chunk, 10 fps, rolls out to any length; 60k steps). Oracle vs real clips of the same length: per-frame anatomy within ~0.03 of real, temporal jitter 1.2–1.3× real, FVD ~80–100 above the real-vs-real floor. Full table, checkpoints and GIFs: sprited/dancing-stick-figures-baselines. Warm-starting from the image model reaches the same loss ~2.5× sooner but converges to the same quality. 5.6-second rollout:

<p align="center"><img src="https://huggingface.co/sprited/dancing-stick-figures-baselines/resolve/main/unetar64rollout.gif" width="512"></p>

DiT-track (Seedance-style two-stage, interim) and class-conditional checkpoints are in the same model repo.

Intended use / limitations

• —Teaching and prototyping video/image diffusion, motion-conditioned generation, pose estimation from renders, I2V, and structural evaluation. Not a human-motion dataset: it is stick figures with a single body preset (jittered).
• —Motion realism is bounded by the generator (ARDY); some prompts are only loosely followed. Use qa_flags and inspect motion samples when prompt semantics are central to an experiment.
• —134 prompts is small for text conditioning; captions are the raw motion prompts. Dense templated captions (camera, body, root motion; dynamic + static) remain future work.
• —The real-reference TVR is non-zero because occlusion hides coloured limbs at these resolutions (about 14% at 64²).

Versioning

• —v0.1 (2026-08-18) — initial public release: 1,430 clips, frames + motion configs, oracle v0, image baselines.
• —v0.2 (2026-08-25) — seed-disjoint in-domain train/validation/test partitions, visual-QA prompt curation (143 → 134 prompts), public-motion reconstruction and verification, seeded instructor render variants, prompt-conditioned reference models, and an image-to-video Colab lesson. A learned rig estimator remains future work until its generated-video scores are validated.

License and attribution

• —Data (this dataset): CC0-1.0. Motion was generated with ARDY's 20-fps Core model; the original generation record did not retain the checkpoint revision. ARDY's source code is Apache-2.0; its released checkpoints are governed by the NVIDIA Open Model Agreement, which states that NVIDIA claims no ownership of generated outputs. The rendering, labels, and skeleton conventions are ours.
• —Code (generator, trainers, oracle): MIT, <https://github.com/sprited-ai/dancing-stick-figures>.

If you use it:

@misc{dancingstickfigures2026,
  title  = {Dancing Stick Figures: A Synthetic Video Dataset, Renderer, and Diagnostic Evaluation Suite},
  author = {Cho, Jin Hyuk},
  year   = {2026},
  url    = {https://huggingface.co/datasets/sprited/dancing-stick-figures}
}

Made by Sprited. Questions → open a discussion on this repo.