pollen-robotics/reachy_mini_conversation_app

Reachy Mini conversation app

Conversational app for the Reachy Mini robot combining realtime voice, vision, personality-aware tools, and choreographed motion.

[image]

Table of contents

• —Overview
• —Architecture
• —Installation
• —Configuration
• —Running the app
• —LLM tools
• —Creating and adding tools
• —Advanced features
• —Contributing
• —License

Overview

• —Low-latency audio conversation through the Hugging Face realtime backend, using the built-in server or a local endpoint.
• —Vision is handled by the realtime backend when the camera tool is used.
• —Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble.
• —Async tools integrate motion, camera capture, and MCP Tool Spaces. The optional web UI (--ui) manages conversations, personalities, tools, and settings.

Architecture

The app connects the user, AI services, and robot hardware:

<p align="center"> <img src="docs/assets/conversationapparch.svg" alt="Architecture Diagram" width="600"/> </p>

Installation

[!IMPORTANT] Install Reachy Mini's SDK before using this app.<br> Windows support is currently experimental and has not been extensively tested. Use with caution.

<details open> <summary>Using uv (recommended)</summary>

Set up with uv:

# macOS (Homebrew)
uv venv --python /opt/homebrew/bin/python3.12 .venv

# Linux / Windows (Python in PATH)
uv venv --python python3.12 .venv

source .venv/bin/activate
uv sync

Include dev dependencies:

uv sync --group dev

</details>

[!NOTE] Run uv sync --frozen to install the exact dependency set from uv.lock without re-resolving versions.

<details> <summary>Using pip</summary>

python -m venv .venv
source .venv/bin/activate
pip install -e .

Install dev dependencies:

pip install -e .[dev]                   # Development tools

</details>

Configuration

The default setup uses the Hugging Face backend and does not require an API key.

Copy .env.example to .env when you want to point Hugging Face at your own local endpoint.

Hugging Face Connection Modes

Use the built-in Hugging Face server through the app-managed Space proxy. This is the default for a new install; set it explicitly only when you want to switch back from a saved local endpoint:

HF_REALTIME_CONNECTION_MODE=deployed

Deployed session allocation falls back to cached hf auth login credentials and reports the daemon-provided hardware ID when available. Cached credentials and the hardware ID are not sent to local endpoints.

Run your own realtime voice backend using speech-to-speech on the same machine as the conversation app:

HF_REALTIME_CONNECTION_MODE=local
HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime

Run your own Hugging Face backend on your laptop and connect to it from Reachy Mini Wireless over the same Wi-Fi network:

HF_REALTIME_CONNECTION_MODE=local
HF_REALTIME_WS_URL=ws://<your-laptop-lan-ip>:8765/v1/realtime

For that LAN setup, make sure the backend listens on an address reachable from the robot, not only on 127.0.0.1.

If the backend stays bound to loopback on your laptop, you can forward it into the robot over SSH instead:

ssh -N -R 8765:127.0.0.1:8765 <robot-user>@<robot-host>

Then set this on the robot:

HF_REALTIME_CONNECTION_MODE=local
HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime

In the web UI's Settings view, the Connection section lets you choose either the built-in server or a local host:port target. The UI writes HF_REALTIME_CONNECTION_MODE for you, and the local path writes HF_REALTIME_WS_URL with a default of localhost:8765.

Running the app

Activate your virtual environment, then launch:

reachy-mini-conversation-app

[!TIP] Make sure the Reachy Mini daemon is running before launching the app. If you see a TimeoutError, it means the daemon isn't started. See Reachy Mini's SDK for setup instructions.

The app runs in console mode. Add --ui to serve the web interface at http://127.0.0.1:7860/.

CLI options

Examples

# Audio-only conversation (no camera)
reachy-mini-conversation-app --no-camera

# Launch with the minimal web UI for personality/mic/settings control
reachy-mini-conversation-app --ui

LLM tools exposed to the assistant

The default profile exposes these tools. Use Tools → Tool access to customize any profile. Every bundled profile enables head_tracking by default; users can still disable it per personality.

[!NOTE] remember/forget facts are stored in memory.v1.json inside the app's instance data directory (~/.local/share/reachy_mini_conversation_app/ by default, or the instance path used by the desktop launcher). forget only removes facts matched by query. To reset all remembered facts, delete this file.

Creating and adding tools

Tools can run locally as Python code or remotely in an MCP-compatible Hugging Face Space. Keep robot, camera, and local-data operations in local tools. A Space is a better fit for shareable, stateless services such as search and external API lookups.

Local tools

Create one Python module per tool, with the file name matching the tool's unique name. See `idle_do_nothing.py` for a minimal implementation.

Each tool subclasses Tool and defines name, a model-facing description, an object-shaped JSON Schema in parameters_schema, and an async __call__ method. Use ToolDependencies for runtime services, and set needs_response = False for actions that should not trigger a spoken follow-up. Catch expected operational failures, log them with the module logger, and return {"error": "..."} so the conversation can continue.

Restart the app after adding the module. Use Tools → Tool access to enable it for a personality, or add its name to that profile's default_tools in profile.md. See External profiles and tools for external directories and autoload behavior.

Hugging Face Space tools

To publish a remote tool, create a Gradio Space, expose its API as MCP with mcp_server=True, and give each function clear type hints and docstrings. Verify that https://<space-subdomain>.hf.space/gradio_api/mcp/schema lists the expected tools before installing the Space.

Use the maintained weather, time, and search Spaces as examples. See Gradio's MCP server guide for additional publishing guidance and Installing Hugging Face Space tools for this app's installation steps.

Advanced features

Built-in motion content is published as open Hugging Face datasets:

• —Emotions: `pollen-robotics/reachy-mini-emotions-library`
• —Dances: `pollen-robotics/reachy-mini-dances-library`

<details> <summary>Custom profiles</summary>

Create custom profiles with dedicated instructions and per-profile tool access.

Select and save a startup profile in the UI. The choice is stored in startup_settings.json. Before one is saved, REACHY_MINI_CUSTOM_PROFILE=<name> can select profiles/<name>/; otherwise the app uses default.

Every profile directory contains one strict schema-version-1 profile.md. TOML metadata is enclosed by +++; the remaining Markdown body is the realtime assistant prompt:

+++
schema_version = 1
voice = "Aiden"
greeting = "Greet me warmly in one sentence, in character, and vary the wording each time."
hidden = false
default_tools = [
  "dance",
  "camera",
  "sweep_look",
]
+++

## Identity

You are a concise, friendly robot guide.

schema_version, default_tools, and a non-empty Markdown body are required. voice, greeting, and hidden are optional. Set hidden = true to omit a profile from the UI. An empty default_tools list is valid and inherits nothing.

default_tools is the authored baseline. Tools → Tool access stores overrides in instance-local profile_toolsets.json without changing bundled profiles. Restoring defaults removes the override. Active-profile changes reconnect the conversation; other changes apply when selected.

Profile directories are data-only. Python tool implementations belong in src/reachy_mini_conversation_app/tools/, or in REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY for external tools. Each enabled tool ID must resolve to a shared tool, an external tool, or a tool from an installed Hugging Face Space.

See Creating and adding tools for the local tool interface and a maintained example.

To manage personalities in the UI:

With --ui, Home lists the available profiles and the built-in default:

• —Tap a card to apply that personality and start talking.
• —Tap "Manage tools" on a saved personality to open its tool access directly.
• —Tap "Custom" to create a personality with a name, instructions, and optional greeting. It inherits the default tools, which can be changed under "Manage tools". Managed instances store it at user_personalities/<name>/profile.md; standalone runs use external_content/user_personalities/<name>/profile.md.

Switching a personality reloads its prompt and effective tools through a quick backend reconnect. Editing profile.md directly requires re-selecting the profile or restarting the app.

</details>

<details> <summary>Locked profile mode</summary>

To create a locked variant of the app that cannot switch profiles, edit src/reachy_mini_conversation_app/config.py and set the LOCKED_PROFILE constant to the desired profile name:

LOCKED_PROFILE: str | None = "mars_rover"  # Lock to this profile

When set, the app ignores saved startup settings, REACHY_MINI_CUSTOM_PROFILE, and UI selection. The UI marks the profile as locked and disables editing.

</details>

<a id="external-profiles-and-tools"></a>

<details> <summary>External profiles and tools</summary>

You can extend the app with profiles/tools stored outside the repository defaults.

• —Core profiles are under profiles/.
• —Core tools are under src/reachy_mini_conversation_app/tools/.

Recommended layout:

external_content/
├── external_profiles/
│   └── my_profile/
│       └── profile.md
├── external_tools/
│   └── my_custom_tool.py
├── user_personalities/
│   └── my_custom_profile/
│       └── profile.md
├── installed_tool_spaces.json
└── profile_toolsets.json

Environment variables:

Set these values in your .env when you want env-driven external profile/tool selection:

# Optional fallback/manual profile selector:
REACHY_MINI_CUSTOM_PROFILE=my_profile
REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=./external_content/external_profiles
REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=./external_content/external_tools
# Optional convenience mode:
# AUTOLOAD_EXTERNAL_TOOLS=1

Loading rules:

• —Profiles: each directory requires a schema-version-1 profile.md with explicit default_tools; there is no cross-profile fallback.
• —Default mode: enabled IDs must resolve to a shared, external, or installed Tool Space tool.
• —Autoload: AUTOLOAD_EXTERNAL_TOOLS=1 adds every valid *.py module from REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY.
• —Web UI: Tools → Tool access enables external modules per profile; it does not upload or edit Python.
• —Separation: profile directories contain data only; external Python belongs in REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY.
• —Tool names: every loaded class needs a unique Tool.name; duplicates fail fast.

</details>

<a id="installing-hugging-face-space-tools"></a>

<details> <summary>Installing Hugging Face Space tools</summary>

You can install MCP-compatible Hugging Face Spaces as remote tool sources for this app. Private Spaces work too, as long as HF_TOKEN is set (or you have run hf auth login) for an account that can access them. To publish a new Space, follow Creating and adding tools.

Tools → Tool Spaces installs or refreshes a global source. Its tools then appear under Tools → Tool access for per-profile selection. Removing a Space removes its tools from every profile. Active-profile changes reconnect the conversation; other changes apply when selected.

The app accepts Hugging Face Spaces exposing the standard /gradio_api/mcp/ endpoint, not arbitrary MCP URLs. Installation discovers the Space's tools and assigns namespaced local IDs, so do not guess or hard-code those IDs beforehand.

# install + enable in active profile
reachy-mini-conversation-app tool-spaces add <owner/space-name>

# enable in a specific profile
reachy-mini-conversation-app tool-spaces add <owner/space-name> --profile NAME

# install without enabling
reachy-mini-conversation-app tool-spaces add <owner/space-name> --install-only

# list installed spaces
reachy-mini-conversation-app tool-spaces list

# remove an installed space
reachy-mini-conversation-app tool-spaces remove owner/space-name

Bundled Pollen Spaces use static specs and are enabled by the default profile. Custom Spaces are validated through the Hugging Face Hub; HF tokens are sent only to private Spaces. Tool metadata is cached in:

• —installed_tool_spaces.json in the managed app instance directory
• —external_content/installed_tool_spaces.json in terminal mode

Startup and profile switching read this cache without discovery or MCP probing. Network access occurs only during install, refresh, or remote tool calls. Per-profile access is stored in profile_toolsets.json beside the manifest, or under external_content/ in terminal mode.

Recommended tags for discoverability on Hugging Face:

• —reachy-mini-tool
• —mcp

Tags are advisory; installation still requires successful MCP validation.

[!NOTE] Preinstalled Pollen Spaces can be removed like any other (tool-spaces remove pollen-robotics/reachy-mini-weather-tool). To restore access, reinstall the Space and restore or update the relevant profile under "Tool access".

</details>

<details> <summary>Multiple robots on the same subnet</summary>

If you run multiple Reachy Mini daemons on the same network, use:

reachy-mini-conversation-app --robot-name <name>

<name> must match the daemon's --robot-name value so the app connects to the correct robot.

</details>

Contributing

See CONTRIBUTING.md for the development workflow and `AGENTS.md` for coding-agent standards.

License

Apache 2.0