ysharma/gr-workflow-microduck-lab

🦆 Microduck Lab

Describe a routine in plain English. A language model compiles it into a move plan, and the real reinforcement-learning policies that ship with [Microduck](https://pollen-robotics.com/microduck/) execute it against MuJoCo physics — then you get the video, the telemetry and a report.

[routine text] ─▶ (fn) choreograph ─┬─▶ 📝 Move plan
                    LLM             │
                                    ▼
[seed] ────────────────▶ (fn) perform ─┬─▶ 🎬 Routine video
              MuJoCo + 9 ONNX policies ├─▶ 📊 Telemetry
                                       └─▶ 📝 Report

This is not a re-implementation

Nothing about the duck is faked or hand-animated. At runtime the app downloads, from the official playground Space `pollen-robotics/microduck-simulator`:

• —robot_allcollisions.xml and robot_allcollisions_rollers.xml — the MJCFs for the walking robot and the roller-skate variant, plus their meshes
• —all nine trained ONNX policies — BEST_alpha_walking, BEST_alpha_stand, BEST_alpha_sitstand, ball_kick_left, ball_kick_right, roulade, alpha_ground_pick, BEST_roller, BEST_roller_crouch

and steps them exactly the way the robot's runtime does:

The 13-slot command is overloaded per mode, and each encoding is the one the policy was trained against:

The one-shot state machines and the fall-recovery machine are ported from that Space's app/src/game/game.js and constants.js, the reference implementation. Sitting hands over gently (0.8 s under the sitstand policy before the sit is commanded, 2.0 s to stand back up) because an abrupt policy switch knocks the duck over — same as the original.

Falls are real. Ask for a barrel roll and the duck genuinely tumbles, the fall detector fires, and the stand policy picks it back up — visible in the uprightness trace.

Moves

On foot, timed — forward, backward, strafe_left, strafe_right, turn_left, turn_right, stand, sit, and the head gestures look_up, look_down, look_left, look_right, tilt_head.

On foot, one-shot — kick_left, kick_right, roll, pick.

On roller skates — skate_forward, skate_backward, skate_turn_left, skate_turn_right, and the one-shot skate_crouch. Switching between the walking body and the skating body rebuilds the model and respawns the duck at the origin, exactly as the official playground does.

Because that respawn is jarring mid-routine, once the skates are on they stay on: a walking move that appears after the first skate move is treated as the planner losing track, and is swapped for its skating equivalent (forward → skate_forward) or dropped if it has none (sit, kick_*). A walking move before any skating is left alone, so "walk around, then put the skates on" still does exactly that. Any such swap is listed in the report.

Running it

pip install -r apps/09_microduck_lab/requirements.txt
python apps/09_microduck_lab/app.py

First launch downloads ~10 MB of meshes and policies into .microduck_cache/ (override with MICRODUCK_CACHE); the roller body is pulled lazily the first time a routine skates.

The choreographer uses Qwen/Qwen3-4B-Instruct-2507 through HF Inference Providers when a token is available — sign in on the Space, or set HF_TOKEN locally. Without a token it still works: it falls back to a keyword planner, so nothing is behind a sign-in wall.

Voice input

Record or upload a spoken routine instead of typing one and openai/whisper-large-v3-turbo transcribes it; a transcript overrides the text box, so speaking replaces the typed routine rather than competing with it.

Speech is the one part that cannot degrade gracefully — there is no offline fallback for it — so transcribe returns two things: the transcript (empty unless it really got one) and a status line. The status is what reaches the Heard output, which is how a signed-out visitor is told that voice needs an account instead of watching the recording silently do nothing. Keeping those on separate ports matters: a status message on the transcript port would be handed to the choreographer as if it were the routine.

samples/spoken_routine.flac is a synthesised clip used by the test suite so the speech path is actually covered.

Files

Notes for anyone extending it

• —The kick policies are blind — there is no ball in the 61-D observation; on the real robot the operator aims the duck first. A scripted routine has no operator, so the ball is placed on the swinging foot's path (0.12 m ahead, 0.06 m to the kicking side, swept empirically). The 0.30–0.35 m the interactive playground spawns at simply misses.
• —Pin `model.stat.extent`. MuJoCo scales the camera near/far planes by it, and it is derived from the bounding box — the ball parked off-screen pushed it to ~52 m, putting the near plane at 0.52 m. With the chase camera 0.62 m out, the near plane sliced through the duck and clipped the neck away, so the head rendered as if detached.
• —Both operators are fn nodes on purpose. The canvas rewrites model node ports to the endpoint schema, so anything wanting a richer control surface has to be an fn.
• —Structured data travels as text, never json or dataframe — the canvas serializes those with String(obj) and the receiver gets "[object Object]".
• —The video and chart outputs are returned as {"path", "url"} dicts. The REST API reads path; the canvas reads url and needs a data: URI.
• —workflow.json is autosaved whenever the app runs — a browser is not required, a gradio_client call is enough. Re-run build_workflow.py after any local launch.