sprited/dancing-chibi-figures

Dancing Chibi Figures — v0.1

One template Q-version (chibi) character, animated by 1,430 motion clips, rendered with exact labels and motion-grounded captions — and paired frame-for-frame with [Dancing Stick Figures](https://huggingface.co/datasets/sprited/dancing-stick-figures).

1,423 clips · 6 s @ 20 fps · 128×128 RGBA · 514,800 frames · 143 text prompts × 10 seeds × 3 cameras · every frame carries the 3D skeleton, camera, depth, camera-space normals, part segmentation, motion events and five levels of caption.

<p align="center"><img src="figs/contact_sheet.png" width="900"></p>

One row per motion group; every clip also carries depth, camera-space normals, part segmentation and joints:

<p align="center"><img src="figs/labels_row.png" width="1000"></p>

The stick-figure dataset was the minimal case (lines, an exact oracle). This is the next rung: a real character with volume — a grey, flat-shaded chibi mannequin (~2.8 heads tall, face guide cross, dark outline, the kind of template figure drawing books use) — while every pixel is still a deterministic function of ~30 known parameters. Same motion, same cameras, same clip_id as the stick-figure dataset, so the two can be used together (skeleton → character, stick render ↔ chibi render, pose estimation on chibi proportions).

v0.1 = first public cut. Rendering, labels and splits are final for this version. Captions are templated from the motion labels (see Captions) and may be refined in v0.2 without re-rendering.

Which config?

Quick start

from datasets import load_dataset
ds = load_dataset("sprited/dancing-chibi-figures", "frames", split="validation")
row = ds[0]
row["color"]              # PIL RGBA image, transparent background
row["caption"]            # "Seen from the front-right, a chibi mannequin waves hello with the right hand; it raises the right hand to head level at 0.5 s ..."
row["caption_dynamic"]    # observed motion only (from the labels, never from the prompt)
row["prompt"]             # the text the motion was generated from: "A person waves hello with the right hand."
import json, numpy as np
events = json.loads(row["events"])                                    # per-clip motion events (same on every row of a clip)
xy = np.frombuffer(row["joint_xy"], np.float32).reshape(27, 2)       # normalised [0,1] image coords
seg = np.array(row["seg"])                                            # bone id per pixel (0 = background)

Composite the RGBA colour over any background you like; depth/normal/seg let you relight or recolour it (the render is unlit on purpose — the normals are there so you can "afterlight" it).

What is in a frame

`frames` / `mini` — one row per rendered frame.

All left/right in prompts, captions, events and joint names are the figure's own left/right (egocentric), not the viewer's — a figure facing the camera points "to the left" toward screen-right.

Binary columns are raw little-endian arrays: np.frombuffer(row[col], dtype).reshape(shape). Skeleton (cskel27, index order): Hips, Spine, Spine1, Spine2, Spine3, Neck, Head, RightShoulder, RightArm, RightForeArm, RightHand, RightHandEnd, RightHandThumb1, LeftShoulder, LeftArm, LeftForeArm, LeftHand, LeftHandEnd, LeftHandThumb1, RightUpLeg, RightLeg, RightFoot, RightToeBase, LeftUpLeg, LeftLeg, LeftFoot, LeftToeBase. Head is the head-sphere centre (skull base in the stick dataset).

Captions: prompt ≠ caption

The ARDY prompt is what we asked for; the clip is what the motion model produced, and they disagree more often than you would think ("walks in a circle" turns 62°; "does jumping jacks" jumps twice, then stands). So the captions are generated from the labels, Seedance-style: a static part (appearance, framing, camera), a dynamic part (events in temporal order), and three granularities, with the prompt kept as a separate prompt column and a prompt_mismatch flag when the two clearly contradict. The render is hips-centred, so the captions say "the camera follows the hips" and never claim visible travel; travel lives in events / root_pos.

We also ran an off-the-shelf video VLM (Qwen3-VL-8B) on the clips: it described ~half of them correctly (sitting, waving, kicking) and missed whole-body fast motion (jumps, a flip) and turning on this featureless mannequin — a useful probe, not a caption source; VLM captions are not included in v0.1.

Splits

By prompt, never by seed or camera (identical to the stick dataset): sport is held out entirely; else hash(prompt) → 90/5/5.

How it was made

prompt → NVIDIA ARDY text-to-motion (6 s, 20 fps, seeds 0–9; the same clips as the stick dataset) → chibi retarget (same joint rotations on chibi bone lengths; root travel scaled by the leg ratio so the feet do not slide; ground from foot contacts) → Blender 5.1 procedural rig (Skin-modifier mannequin over the 27-joint graph + sphere head, heat weights, dual-quaternion skinning; flat Emission material, inverted-hull outline, face guide cross) → 3 orthographic cameras per clip (same sampling as the stick dataset) → two headless Eevee passes per frame (anti-aliased colour; 1-sample exact depth / normal / segmentation) at 256², box-downsampled to 128 / 64 → parquet. Deterministic from clip_id; the generator is in the code repo.

Intended use / limitations

Teaching and benchmarking small video/image generative models, conditional generation (skeleton/seg/depth → character), pose estimation on chibi proportions, sprite/character animation research. One template character only (no clothes, hair, faces, props); hips-centred framing (no visible travel); ~0.3 % of frames are partly out of frame (flagged out_of_frame); the rig mesh has small internal defects at the neck and wrists invisible at ≤256 px; prompt_mismatch flags are heuristics; camera pitch sign follows the stick dataset (positive = from slightly below).

Baselines & tutorial

Two tracks, both trained on mini (64px):

🤗 diffusers-standard — load with three lines, everything transfers to any diffusers project:

• —sprited/dancing-chibi-figures-ddpm-64 — unconditional DDPMPipeline (UNet2DModel, 4-channel RGBA).
• —sprited/dancing-chibi-figures-t2i-64 — text-to-image: a miniature Stable Diffusion (UNet2DConditionModel + frozen CLIP + classifier-free guidance, no VAE).

from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained("sprited/dancing-chibi-figures-t2i-64",
                                         custom_pipeline="sprited/dancing-chibi-figures-t2i-64",
                                         trust_remote_code=True)
imgs = pipe(["a person jumps in place"] * 4, num_inference_steps=50, guidance_scale=3.0).images

pure PyTorch — sprited/dancing-chibi-figures-baselines: a 64px image model and a text-conditioned autoregressive video model (chunked diffusion, CLIP prompt embeddings); trainer and scripts/rollout.py in the code repo. A ComfyUI node (comfy/qversion-chibi) wraps the image checkpoints.

📓 Colab tutorial — dancing_chibi_figures_colab.ipynb: look at the data, train the text-to-image model from scratch, turn it into a video model, grade it with a counting robot — free-T4-sized, written for absolute beginners. (Sequel to the stick-figures notebook.)

Versioning

v0.1 (2026-08) first public cut. Planned v0.2: more prompts (a curated set of ~300 from a chibi pose reference book), an anatomy oracle, post-processed outlines, optional colour-coded config, 8-direction sprite views.

License and attribution

Data CC0-1.0. Motion generated with NVIDIA ARDY (NVIDIA Open Model License, which claims no ownership of outputs); character, rig and renderer are procedural code (MIT) — no third-party 3D assets. Paired dataset: sprited/dancing-stick-figures.