Enricx/reachy_mini_gpt_live

Reachy Mini GPT-Live 🎙️🤖

 ┌──────────────────────────────────────────────────────────────┐
 │  REACHY MINI · GPT-LIVE            Montagehandleiding / Guide │
 │                                                              │
 │   🤖  +  🎙️  +  🔑  →  💬  🕺  📞                           │
 │                                                              │
 │   ⏱ 10 min      🧰 geen gereedschap / no tools               │
 └──────────────────────────────────────────────────────────────┘

📦 In de doos / In the box

🔧 Montage / Assembly

 ① INSTALLEREN                      ② STARTEN
 ┌────────────────────┐             ┌────────────────────┐
 │ Dashboard          │             │ Dashboard          │
 │ http://reachy-mini │             │  ▶ Reachy Mini     │
 │        .local:8000 │   ──────▶   │    GPT-Live        │
 │                    │             │                    │
 │ App store ▸ [⤓]    │             │   ⚙️ (open UI)     │
 └────────────────────┘             └────────────────────┘
   Reachy Mini GPT-Live               poort 8042

 ③ API-KEY                          ④ PRATEN
 ┌────────────────────┐             ┌────────────────────┐
 │ Instellingen ▸     │             │                    │
 │  OpenAI            │             │   🗣️ ─────▶ 🤖     │
 │  ┌──────────────┐  │   ──────▶   │        ◀───── 🔊    │
 │  │ sk-proj-…    │  │             │                    │
 │  └──────────────┘  │             │  "Hoi Reachy!"     │
 │      [Opslaan]     │             │                    │
 └────────────────────┘             └────────────────────┘

 ⑤ PERSOONLIJKHEID (optioneel)      ⑥ BELLEN (optioneel)
 ┌────────────────────┐             ┌────────────────────┐
 │ Profiel ▸          │             │ Instellingen ▸     │
 │  ○ default         │             │  Bellen · Twilio   │
 │  ○ mars_rover      │             │  Bellen · webhook  │
 │  ○ butler          │   ──────▶   │   & tunnel (ngrok) │
 │  ○ storyteller     │             │                    │
 │  [Activeren]       │             │ OpenAI ▸ Webhooks  │
 └────────────────────┘             └────────────────────┘

 ⚠️  Reachy praat alleen met tegoed op je OpenAI-account.     ⚠️  Stop = sessie netjes afsluiten (usage definitief).
 💡  Tik op een antenne: robot wordt wakker en de app start.  💡  Alles werkt ook op je telefoon (zelfde URL).

Talk to your Reachy Mini. The app opens a full-duplex OpenAI GPT-Live voice session over WebRTC and wires it to the robot's own microphone array and speaker. While you talk, Reachy looks at you (face tracking), turns toward your voice (microphone direction of arrival), wobbles its head in sync with its own voice and moves its antennas.

The GPT-Live backend can call robot tools: dance, play an emotion, move or sweep the head, follow your face, look through the camera, remember and forget facts about you, tell the time and the weather, search the web, and go to sleep. Personality profiles (voice, greeting, instructions, enabled tools) are selectable and editable in the web UI, and long-term memory persists on the robot between conversations.

A small web page (port 8042) shows the live transcript (including tool calls), a start/stop button, the profiles, the memory and the settings. The OpenAI API key stays on the machine that runs the app; the browser never sees it.

Snel starten (NL)

1. 1.Installeer de app via het Reachy Mini dashboard (of pip install op een Lite, zie onder).
2. 2.Start de app in het dashboard en open de app-pagina (⚙️, poort 8042).
3. 3.Plak je OpenAI API-key bij Instellingen en klik Opslaan.
4. 4.Klik Start gesprek (of zet automatisch starten aan) en praat tegen Reachy.
5. 5.Stop sluit de sessie netjes af: de app wacht op session.closed zodat het gebruik definitief is.

Requirements

• —A Reachy Mini (Wireless or Lite) with a running daemon (reachy-mini ≥ 1.10).
• —An OpenAI API key with access to gpt-live-1 (project-scoped key).
• —Python ≥ 3.11. Extra Python dependencies: aiortc, av, aiohttp (wheels exist for Linux aarch64, so the Wireless robot installs them fine).

Install

From the dashboard (Wireless or Lite): open http://reachy-mini.local:8000 (Wireless) or http://localhost:8000 (Lite), find Reachy Mini GPT-Live in the app store and click Install, or install this Space by URL:

curl -X POST http://<HOST>:8000/api/apps/install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://huggingface.co/spaces/<user>/reachy_mini_gpt_live"}'

Manually on a Lite (daemon on your laptop, same Python env as reachy-mini):

uv pip install git+https://huggingface.co/spaces/<user>/reachy_mini_gpt_live
# or, from a checkout:
uv pip install -e /path/to/reachy_mini_gpt_live

Manually on a Wireless robot without internet on the robot:

scp -r reachy_mini_gpt_live pollen@reachy-mini.local:/tmp/
ssh pollen@reachy-mini.local "/venvs/apps_venv/bin/pip install /tmp/reachy_mini_gpt_live"

Configure the API key

Pick one (later entries win):

1. 1.Settings page (recommended): start the app, open port 8042, paste the key. It is written to ~/.config/reachy_mini_gpt_live/.env (mode 600) on the machine running the app.
2. 2.A .env file next to the installed package (see .env.example).
3. 3.Environment variables of the daemon process (OPENAI_API_KEY=...).

All options are listed in .env.example. Everything except the key can also be changed from the settings page.

Run

• —Dashboard: start Reachy Mini GPT-Live, then open its page (⚙️) at http://reachy-mini.local:8042 (Wireless) or http://localhost:8042 (Lite).
• —Terminal (daemon must be running): python -m reachy_mini_gpt_live.main
• —Make it the default experience: reachy-mini-daemon --startup-app reachy_mini_gpt_live

With autostart on (default) the session starts as soon as the app starts, so you can simply launch the app from the dashboard and talk.

How it works

 Reachy microphone ──► RobotMicTrack ──► aiortc PeerConnection ──► OpenAI GPT-Live
 (16 kHz, SDK media)    (20 ms frames)        │  ▲  oai-events data channel
                                               ▼  │
 Reachy speaker ◄─── SpeakerSink ◄──── remote audio track
 (push_audio_sample,   (resample 16 kHz)
  head wobbler taps it)
                                        Bridge ──► WebSocket ──► browser UI (transcript, status)
                                          │
                                        RobotBehavior (wake-up, face tracking, DoA look, antennas)

The GPT-Live contract as implemented (reachy_mini_gpt_live/live_client.py):

• —The app (server side, holds the key) creates the session with POST /v1/live/sessions and body {"session": <session_config.json>, "transport": {"type": "webrtc", "sdp": <offer>}} using bearer authentication, then reads transport.sdp (answer) and the opaque session.id.
• —Remote audio arrives on the WebRTC audio track; JSON events on the oai-events data channel.
• —The client waits for session.started and never sends startup configuration again.
• —session.input_transcript.delta / session.output_transcript.delta are shown with their exact delta text and grouped by start_ms / end_ms (see transcript.py), not by turns.
• —Stop sends session.close and keeps audio and transport alive until session.closed (with usage.seconds and reason). A timeout or disconnect before that is reported as incomplete finalization in the UI and the log.

session_config.json is sent verbatim:

{
  "model": "gpt-live-1",
  "audio": { "output": { "voice": "marin" } },
  "delegation": {
    "type": "responses",
    "responses": {
      "parallel_tool_calls": false,
      "model": "gpt-5.6-terra",
      "reasoning": { "effort": "medium" },
      "tools": [ { "type": "web_search" } ]
    }
  }
}

At start the active profile, its tools and the remembered facts are merged into this base (instructions, voice, delegation.responses.instructions and delegation.responses.tools); session_config.json itself is never modified. The greeting is requested right after session.started with session.instructions.append.

Memory works two ways: the voice model delegates "remember this / call me X / I like …" to the backend (tool remember), and after every conversation the app distils stable facts from the transcript into memory (setting auto memory, on by default; "Extract facts" button in the Memory tab). Facts are injected into the next conversation.

Releasing the robot: while the app runs it holds the robot's app lock, so the dashboard and Reachy Mini Control show the power button disabled. Use the header button Sleep & stop app (or say "ga slapen"): the session closes, the robot goes to its sleep pose and the app stops. Because the app is the robot's startup app, it starts again on the next wake-up or antenna touch.

Tools

Tools are declared as function tools of the Responses delegation (delegation.responses.tools, next to web_search). When the backend calls one, the app executes it on the robot and returns the result with response.item.create + response.create; the Live model then tells you what happened.

Which tools a session gets is decided by the active profile's default_tools and by availability (camera enabled, move libraries loaded). Both move libraries are Hugging Face datasets and are downloaded once in the background when the app starts.

Profiles

A profile is one profile.md (TOML metadata between +++ lines, then the instructions as Markdown), the same format as Pollen's conversation app:

+++
schema_version = 1
voice = "cedar"
greeting = "Greet the user in one sentence, in character."
default_tools = ["dance", "play_emotion", "camera", "remember", "forget"]
+++

### IDENTITY
You are Reachy Mini, ...

Built-in profiles: default, mars_rover, butler, storyteller. Your own profiles are saved under ~/.config/reachy_mini_gpt_live/profiles/<name>/profile.md on the machine running the app (create, edit and activate them in the UI). The profile's instructions become the Live instructions; its voice sets audio.output.voice; the backend prompt (delegation.responses.instructions) gets the tool guidance, a summary of the persona and the remembered facts.

Phone calls (Twilio → OpenAI SIP)

Say "bel Pizza Mama in Dronten en reserveer een tafel voor twee om zeven uur" and Reachy's assistant makes the call: it introduces itself as your digital assistant (an AI), pursues the goal, checks with you when a real decision is needed ("een moment, ik leg het even voor", with 2–4 options you answer by voice or with a button in the UI), hangs up, and reports back: agreed / agreed with a change / not possible, plus exceptions.

How it works (outbound calls must be provider-owned, per the GPT-Live SIP guide):

1. 1.phone_call asks Twilio to dial the number. When it is answered, the called party joins a Twilio conference and the app places a second Twilio leg to sip:{project}@sip.api.openai.com;transport=tls;secure=true?X-Reachy-Call={id} that joins the same conference. With Call route set to direct the app uses the older <Dial><Sip> bridge, without keys.
2. 2.OpenAI posts live.transport.incoming to https://<public-url>/api/phone/webhook (Standard-Webhooks signature verified); the app accepts the session with call-specific instructions and a small Responses delegation with two tools: ask_owner and hang_up.
3. 3.The sideband WebSocket (/live/sessions/{id}/attach) streams the transcript; owner questions are relayed to the robot conversation (session.commentary.append) and answered with the answer_phone_question tool or POST /api/phone/answer.
4. 4.After hang-up the transcript is summarised into {status, summary, exceptions} and spoken by Reachy.

Call scenarios: reusable goals with {placeholders} (table reservation, appointment, insurance claim follow-up, order status, complaint, and a few tongue-in-cheek ones such as negotiating an extra pay step with your manager), each in Dutch, English, German, French and Italian. Pick one in the Calls tab, choose the language, fill in the fields, call; or say "call the hairdresser with the appointment scenario" and Reachy asks for the missing details. The assistant then speaks that language on the call (introduction, "one moment, let me check", goodbye). Your own scenarios are saved on the robot (scenarios.json).

Languages: the web UI itself is available in the same five languages (switch in the header; defaults to your browser language).

Keypad and phone menus (DTMF): in real calls GPT-Live said menu choices out loud instead of pressing keys, so Twilio plays the tones. To press keys the app briefly redirects the called party's leg to <Play digits> and straight back into the conference, while the assistant stays connected. Keys can come from three places:

• —Keypad in the Calls tab, as soon as the call is answered. You can also type 0–9, * and # on your keyboard.
• —The call assistant's backend tool press_keys. The assistant is told never to say a menu choice out loud and to press the key that fits the goal.
• —Menu keys on a scenario or a call, pressed right after answering. WWW1 waits three seconds, then presses 1.

Why a call ended: every call records who ended it (you, the assistant, the maximum duration, the other party, OpenAI or the connection to the assistant), OpenAI's close reason and a timeline of the call legs, keys and tool calls. Open Timeline under a call in the Calls tab.

Recording and live listening: with record calls on, Twilio forks the call audio to the app (<Start><Stream> over the tunnel, independent of the SIP leg). The app writes a stereo WAV (left = the person called, right = the assistant) to recordings/ next to the config, shows a player per call in the UI, and streams the mixed audio live to your browser ("🎧 live meeluisteren"). Tell people you record if your local law requires it.

Sharing and cleaning up recordings: every recording has Download, Share and Delete buttons. Share creates a secret link (valid 7 days, revoke it any time with Stop sharing) to a small page with the audio player, the result and the transcript. With the tunnel running the link works for anyone you send it to (copy, WhatsApp or e-mail); without it, only inside your own network. Delete removes the audio, the transcript and any share links. To save space automatically, set Delete recordings after (days): older calls are then removed after each call, at app start and as soon as you save the setting. The UI shows how much space the recordings take. On a free ngrok account, people who open a share link first see ngrok's warning page ("You are about to visit…") and click Visit Site.

What the tunnel exposes: only the OpenAI webhook, Twilio's media stream and share links (/share/…). The web UI, its API and live listening answer 404 through the public URL and stay reachable on your own network only (http://reachy-mini.local:8042).

Setup:

1. 1.Twilio: a (non-trial) account with a voice-capable number. Enter Account SID, Auth Token and the number in the app settings.
2. 2.Public URL for the webhook: the robot is behind your router, so the app opens an ngrok tunnel itself (through pyngrok, no manual install on the robot). Create a free ngrok account, copy your authtoken and claim your free static domain (YOUR-NAME.ngrok-free.app), and enter both in the settings. The tunnel starts with the app and the public URL / webhook URL appear automatically. Make the app the robot's startup app (dashboard, or PUT /api/apps/startup-app) so it, and the tunnel, come back after a reboot or an antenna touch.
3. 3.OpenAI dashboard (same project as the API key): copy the Project ID (Settings → General) and create a webhook (Settings → Webhooks) for the event live.transport.incoming pointing at https://YOUR-NAME.ngrok-free.app/api/phone/webhook; paste the webhook secret (whsec_…) in the settings.
4. 4.Set your name (used in the introduction), the call language and voice.
5. 5.Add contacts (name + number) in the UI, or let the assistant look businesses up with web search. Test with the Bel nu form.

Costs: Twilio per-minute rates for the outbound leg plus the SIP leg, and GPT-Live voice time for the call session. The assistant always discloses that it is an AI; keep it that way.

Settings

Web API (port 8042)

GET /api/status, GET /api/config, POST /api/config, POST /api/start, POST /api/stop, POST /api/mute {"muted": true}, GET /api/transcript, GET /api/profiles, POST /api/profiles, POST /api/profiles/select, DELETE /api/profiles/{name}, GET /api/tools, POST /api/moves/reload, GET /api/memory, DELETE /api/memory[/{id}], GET /api/phone, POST /api/phone/call, POST /api/phone/hangup, POST /api/phone/answer, POST /api/phone/keys {"keys": "1#"}, POST /api/phone/webhook (OpenAI), GET /api/phone/recordings, GET /api/phone/recordings/{id}.wav, DELETE /api/phone/recordings/{id}, POST /api/phone/recordings/{id}/share?days=7, DELETE /api/phone/shares/{token}, the public share page GET /share/{token} (+ .wav), GET/POST/DELETE /api/contacts, and a WebSocket at /ws streaming status, transcript.delta, tool.call, tool.result, delegation, event and finalization messages.

Tests

The test-suite runs without a robot and without OpenAI: a fake GPT-Live server (tests/fake_live_server.py, aiortc + aiohttp) answers the SDP offer, echoes the microphone audio back, emits scripted transcript events and honours session.close. Fake robot media generates a tone and records what would be played on the speaker.

uv pip install -e ".[dev]"
python -m pytest

Troubleshooting

• —"No OPENAI_API_KEY configured" – open the app page (port 8042) and paste the key.
• —HTTP 401/403 from /v1/live/sessions – wrong key, or the project has no GPT-Live access.
• —HTTP 500 "Internal Server Error" from /v1/live/sessions – seen when the OpenAI account has no credits left (a plain Responses call then reports credit_balance_exhausted). Add credits at https://platform.openai.com/settings/organization/billing/ and try again. python -m tests.probe_live tests the endpoint with several config/SDP variants and prints the OpenAI request-id.
• —"No session.started within 20s" – the WebRTC connection could not be established; check that the robot can reach the internet (UDP) and try again.
• —Reachy hears itself – lower the speaker volume in the dashboard; the daemon's echo canceller and GPT-Live's turn handling cope with normal levels.
• —Incomplete finalization – the session was closed but session.closed never arrived (timeout/disconnect). The conversation is over, but the final usage could not be confirmed.
• —Tunnel "error" in the phone card – check the ngrok authtoken/domain; the first start downloads the ngrok binary on the robot, which needs internet access and a few seconds.
• —Port 8042 already in use (e.g. Reachy Mini Control on a laptop) – start the daemon/app with GPT_LIVE_UI_PORT=8047 and open that port instead.
• —Logs: Lite → the terminal running the daemon; Wireless → ssh pollen@reachy-mini.local "sudo journalctl -u reachy-mini-daemon -f".

Credits

Built with the Reachy Mini SDK by Pollen Robotics and the OpenAI GPT-Live API.