pollen-robotics/reachy-mini-3d-voice

Reachy Mini 3D Voice

Fork of smolagents/hf-realtime-voice where the orb is replaced by a rigged 3D Reachy Mini (Draco GLB, three.js): procedural motion per conversational state, a speech-synced head wobble driven by the assistant's voice, and emotion/dance tools the model calls mid-conversation. Emotions replay the official recorded head, antenna, and body trajectories from Pollen's pinned emotions dataset; dances reproduce the official dance library. Same deployment shape as upstream — a thin FastAPI serving the static front-end, proxying /session, and metering talk time. Its WebSocket transport is driven by the official OpenAI Agents SDK, with browser audio exchanged as 24 kHz PCM16.

Run it locally

Install the SDK bundle and Python dependencies, then run the same FastAPI app used in production:

npm ci
pip install -r requirements.txt
LOAD_BALANCER_URL=<allocator base url> uvicorn server:app --port 7860

Set a speech-to-speech server URL in Settings (the get-session-url.sh helper allocates a session and copies its connect_url). Without a backend the robot idles; localStorage.setItem("s2s.debug", "1") + reload exposes window.__robot.setState(...), window.__robot.playEmotion(...), and window.__robot.playDance(...) in the console.

Fork additions on top of the upstream files:

Microduck companion

Microduck uses approximately its physical size: about 25 cm tall in a normal standing pose, compared with Pollen's advertised 28 cm for Reachy Mini. The CAD asset is in metres and is displayed at 0.95 scale; it is not a miniature mascot. The room camera allows enough space for both robots, including in portrait.

Microduck approaches Reachy, then takes varied walks, pauses, looks around, dips its head and occasionally opens its beak. Routes avoid Reachy's base. It turns with short steps before walking forwards. Its head alternates between watching its path, glancing toward the viewer and looking up at Reachy's face, especially during pauses. Each interest lasts a few seconds with smooth head turns, independently of the direction of travel.

Before a conversation, Reachy turns toward Microduck, then back to the visitor. The body carries about 70% of each turn, with the head sharing the rest and gently tilting down toward the companion while staying clear of the shell. Occasionally it plays a short official nod or welcoming emotion, using the existing cached trajectories. These spontaneous moves yield when a conversation starts. Hidden tabs skip scene updates, and reduced motion disables the extra routine. No model call or audio is used.

Each supporting foot stays planted in world space. The other follows a smooth lift/swing/landing curve, with a short period when both feet touch the table. The body shifts weight toward the supporting foot and settles between steps. A small, bounded three-joint solve for each leg positions its sole, and its ankle keeps pitch level. The knees stay slightly bent to avoid an unstable straight-leg pose. This is browser animation, not a dynamics simulation or robot controller.

Everything runs in the existing Three.js scene, at up to 60 pose updates per second for smooth movement. There is no physics engine, learned walking policy, model inference, audio generation or ongoing server request for the duck. The approximately 302 KB GLB contains 14,917 triangles and one shared material. Its 16 meshes use the existing renderer, lights and shadow pass; two tiny sole patches reinforce foot contact. There is no additional WebGL context or shadow map.

Both robots load together, and the loading screen waits until they are ready and the scene has rendered. A companion asset failure leaves the voice demo usable. Hidden documents skip scene work. The user's reduced-motion preference freezes the duck's routine and closes its beak.

Run npm test for physical-size, forward-walking, planted-foot, joint-solve, randomized-route, reduced-motion and asset-budget checks, and npm run check for type checking. For a silent preview, start the local server and open /tools/microduck-preview.html. Its controls pause and scrub a reproducible routine and report sole-position error and update timing. Timing measures the duck's update callback per rendered frame, not total scene rendering or GPU time.

The model is derived from Pollen Robotics' Microduck simulator at e81974b, using its articulated CAD meshes, joint definitions, physical beak hinge and classic cream/orange colorway. See asset provenance. To regenerate it, download microduck.glb and kinematics.json from that revision's app/public/robot/mjlab/ directory into a local folder, run npm ci, then:

node tools/build-microduck.mjs /path/to/source-folder
npm test

The optimizer is a development dependency only. Neither it nor the original simulator's runtime dependencies are loaded by visitors.

Remote MCP tools

Port of the conversation app's "tool spaces": Gradio Spaces exposing the standard /gradio_api/mcp/ endpoint are declared in MCP_SERVERS (main.js), their tools discovered at page load and offered to the model as alias__tool functions — the browser calls the Space directly (those endpoints answer CORS, no proxy needed). Preinstalled: Pollen's weather and web search tools. To add your own (e.g. a Pollen/Reachy knowledge base): publish a Gradio Space with mcp_server=True and add one { alias, slug } line to MCP_SERVERS.

The page uses the WebSocket route of the Hugging Face speech-to-speech backend. Protocol events and tool execution go through the official OpenAI Agents SDK adapter, while this fork keeps the Reachy 3D, MCP, and motion layers.

How it works

1. 1.App POSTs <lb_url>/session (empty JSON body).
2. 2.The LB picks a ready compute (round-robin) and returns:

   {
     "session_id": "...",
     "websocket_url": "wss://<compute>/v1/realtime",
     "connect_url": "wss://<compute>/v1/realtime?session_token=<JWT>",
     "session_token": "<JWT>",
     "pending_timeout_s": 60
   }

1. 1.App opens a WebSocket directly on connect_url (no rewrite to https://; unlike the WebRTC client which POSTs an SDP offer).
2. 2.The official Agents SDK configures the session using the OpenAI Realtime GA schema.
3. 3.Client streams mic audio as PCM16 24 kHz mono chunks (input_audio_buffer.append, one frame every ~40 ms).
4. 4.Server pushes response.output_audio.delta (PCM16 24 kHz mono base64) and transcript deltas.

The backend exposes one concurrent session per compute (same as WebRTC mode); the LB pins the session via a signed session_token.

Why WebSocket instead of WebRTC

Backend requirement

This app talks to the /v1/realtime WebSocket route in `huggingface/speech-to-speech`. The compute deployment must support the official Agents SDK WebSocket subprotocol/event shapes.

Smoke-test from the shell:

LB="${LOAD_BALANCER_URL:?Set LOAD_BALANCER_URL in your shell}"
curl -X POST "$LB/session" -H "Content-Type: application/json" -d '{}'
# -> { "connect_url": "wss://<compute>/v1/realtime?session_token=..." }
# Feed connect_url into a wscat / websocat and you should get a
# session.created event back immediately.

Tools

The assistant can call tools mid-conversation from the Tools button, top-right:

• —Web search — DuckDuckGo results through Pollen's MCP search-tool Space. It is discovered at page load and requires no API key in this Space.
• —Camera — while enabled, a live self-view shows bottom-left; when the model calls the tool, the current frame is sent to the vision-language model so it can see what you're showing it.
• —Body movement — always on once the 3D robot loads. The model gets the same four movement tool names as the official conversation app: dance, stop_dance, play_emotion, and stop_emotion. The twenty public dances are faithful JavaScript ports of reachy-mini-dances-library 0.2.1, including its default 114 BPM timing, amplitudes, phases, waveforms and paths.

Connecting to a backend

Three modes, picked by env (/api/config tells the client which one is active):

• —`SPEECH_TO_SPEECH_URL` env — highest priority. The browser connects directly to this realtime WebSocket URL; it's shown read-only in Settings. Setting it disables the load-balancer logic entirely (no /api/session proxy, no queue, no metering, no sign-in). Unlike the LB address it is not a secret.
• —`LOAD_BALANCER_URL` env — the original flow: the browser POSTs the same-origin /api/session proxy, the server forwards to the LB, and the browser dials the per-session compute URL the LB hands back. The LB address never reaches the browser; the Settings URL field is hidden. When a visitor signs in with Hugging Face, the proxy forwards their OAuth access token to the allocator through X-Reachy-Mini-Authorization; the token remains server-side. Set REQUIRE_LOGIN=true to reject anonymous allocation and ask visitors to sign in before starting.
• —Neither — Settings → Speech-to-speech server URL: paste a full connect_url (wss://host/v1/realtime?...) or a bare host like localhost:8080 (the app adds /v1/realtime), and the browser connects to it directly.

Settings → Restart reconnects with the current voice, instructions and URL.

Usage limits

Conversation time is metered per UTC day by sign-in tier (see limiter.py / auth.py), but only on the deployed Space — metering turns on only when BOTH LOAD_BALANCER_URL and SPACE_ID (injected automatically by the HF Space runtime) are present. Running locally — even with LOAD_BALANCER_URL exported — leaves the app unmetered. Tunable via env:

PRO members are always unlimited. Members of cerebras, HuggingFaceM4, smolagents, and pollen-robotics are unlimited out of the box (shown as "Team", not "PRO"); set UNLIMITED_ORGS=my-team to add more. Matched case-insensitively against the user's organisations from HF OAuth.

Run locally

The app is a small FastAPI server that serves the front-end, proxies session allocation, and meters usage in the deployed Space.

pip install -r requirements.txt
npm ci
export SPEECH_TO_SPEECH_URL=...    # optional; pin a direct s2s server URL (overrides the LB)
export LOAD_BALANCER_URL=...       # optional; session-proxy flow (set a URL in Settings otherwise)
uvicorn server:app --reload --port 7860
# or, matching production: docker build -t s2s . && docker run -p 7860:7860 -e LOAD_BALANCER_URL=... s2s

Then open <http://localhost:7860/>, click the orb, allow the mic, talk.

Browsers require HTTPS or `localhost` for getUserMedia() (mic + camera). 127.0.0.1 and localhost both work; plain http://192.168.x.y does NOT.

Settings (stored in localStorage)

LocalStorage keys are namespaced s2s.ws.* so this app's settings do NOT collide with the WebRTC variant.

Files

The chat history keeps the exact post-gate audio sent over WebSocket and exposes it through a local replay control. Backend VAD also drives an immediate “Listening…” → “Sending voice…” bubble; when STT is enabled, the same bubble and history row are updated with the transcript.

Audio pipeline notes

• —Input: getUserMedia({ echoCancellation, noiseSuppression, autoGainControl }) feeds the mic-capture worklet at the AudioContext rate. The worklet resamples to 24 kHz (boxcar lowpass + decimation on the 48 -> 24 fast path, linear interpolation fallback for odd rates) and packs Int16 LE.
• —Output: response.output_audio.delta decodes to Int16 -> Float32 and is posted to the audio-playback worklet. The worklet maintains a per-context ring buffer, linearly interpolates 24 -> 48, and applies short 32-frame fades on entry/exit to suppress clicks.
• —Barge-in: when the server VAD detects user speech mid-response (input_audio_buffer.speech_started while ai-speaking), the client posts { kind: "clear" } to the playback worklet to wipe the queue immediately. The server itself cancels the in-flight response.

Credits

• —Backend: huggingface/speech-to-speech
• —UI verbatim from amir-tfrere/minimal-conversation-app-s2s-backend (Pollen Robotics × Hugging Face)