pollen-robotics/reachy-mini-text-to-motion

Reachy Mini Text-to-Motion

Natural-language motion generation for Reachy Mini: "type a sentence, get a new expressive move", built on top of the community moves gallery (407 moves / 52 HF datasets tagged reachy_mini_community_moves).

Docs

Prototype (T1: LLM motion-clip generation) - working

Two isolated pieces connected by the standard recorded-move JSON:

generator/   Python: prompt -> LLM -> motion clip (curves) -> 50 Hz recorded-move JSON
player/      Vite + Three.js: plays those JSONs on the reachy-viz rig, no robot needed

Generation is two-stage: a director call first turns the prompt into a short performance brief, then the LLM authors a motion clip: a curve-based source format inspired by Blender F-curves - independent per-channel keyframes with named easing (linear|in|out|inout|overshoot|bounce|hold) plus optional procedural oscillators (trembles, flutter, grooves). The clip is clamped to corpus-derived limits and evaluated into the standard recorded-move JSON, so everything downstream (daemon, gallery, player) works unchanged. Clips also round-trip naturally to/from Blender F-curves for hand-polishing. Clips can carry a sound track too - the robot's voice, grounded in music theory: the director declares a key + mode (the emotional color), the animator composes pitch phrases on that scale, and musictheory.py deterministically snaps every anchor note to it before the audio is rendered with the slide whistle voice (audio/render_curve.py).

# One-time setup
cd generator
python fewshot_author.py                 # few-shot bank + corpus limits

# Promptable UI (recommended): generator server + player
python -m uvicorn serve:app --port 8123  # LLM backend (HF_TOKEN or cached
                                         # huggingface-cli login)
cd ../player
npm install && npm run dev               # type a prompt, get a move (~30-60s)

# Or CLI only
python generate.py "a shy but proud little celebration" --plot --clip-out

generator/movegen.py holds the 9-DOF math (head 4x4 <-> euler+mm, antenna order [right, left]), generator/motionclip.py the clip schema, evaluator and clamps. Output validated against the SDK's RecordedMove.

Public-deployment guardrails (serve.py)

All env-tunable, defaults are fine for local dev:

• —MOVEGEN_ADMIN_SECRET: when set, DELETE /api/moves/{name} requires a matching X-Admin-Secret header (set it as a Space secret; store it browser-side once via localStorage["movegen-admin-secret"]). Unset = delete stays open.
• —MOVEGEN_MAX_CONCURRENT (default 4): simultaneous LLM generations across all clients; 429 beyond.
• —MOVEGEN_RATE_LIMIT / MOVEGEN_RATE_WINDOW_S (default 20 per 3600 s): per-client-IP budget on generate/refine; 0 disables.

The ladder

• —L1 - zero training: LLM in-context generation (keyframes, code over the symbolic-motion skill, retrieval + composition), GenEM/EMOTION style.
• —L2 - light training: per-move variation models (SinMDM), style transfer, a motion-text embedding for retrieval + evaluation.
• —L3 - trained generator: retrieval-augmented masked-token or flow-matching model over the gallery; audio-motion co-generation as the research contribution.

Key ecosystem pieces this builds on

• —Recorded-move format: reachy_mini/src/reachy_mini/motion/recorded_move.py
• —Semantic move search (production): reachy_mini_api/server/{moves.js,search.js}
• —Publishing rails: Marionette (reachy_mini_marionette/lib/hf.js)
• —Keyframe-tweening runtime pattern: reachy_oracle/src/embed.ts
• —3D preview: reachy_mini_mobile_app/src/ui/widgets/reachy-viz/ReachyModel.tsx
• —Evaluation protocol: Rogel, Yadollahi & Laban perception study (arXiv:2605.12786)