pollen-robotics/reachy-mini-chatbox

Reachy Mini ChatBox

A modular voice conversation app for the Reachy Mini robot. Cheap to run, versatile, customizable, and fun.

ChatBox uses a cascade pipeline — ASR → LLM → TTS — where each stage is a swappable provider. Mix cloud APIs and local models, tweak the personality, add live reactions to keywords, and let the robot dance.

[image]

How it works

🎤 Microphone
    │
    ▼
┌─────────┐     ┌─────────┐     ┌─────────┐     🔊 Speaker
│   ASR   │ ──▶ │   LLM   │ ──▶ │   TTS   │ ──▶ 🤖 Robot moves
└─────────┘     └─────────┘     └─────────┘
 Speech to       Reasoning       Text to
 text            + tool calls    speech

Voice Activity Detection (VAD) segments microphone input. The ASR provider transcribes it, the LLM generates a response (with optional tool calls for head movements, dances, emotions, camera…), and the TTS provider speaks it back — all while the robot animates.

Installation

[!IMPORTANT] Before using this app, install Reachy Mini's SDK.

<details open> <summary><b>Using uv (recommended)</b></summary>

# Create venv
uv venv --python python3.12 .venv
source .venv/bin/activate

# Install with cascade pipeline
uv sync --extra cascade

Install additional providers as needed:

uv sync --extra cascade_parakeet_progressive  # Parakeet ASR (Apple Silicon)
uv sync --extra cascade_kokoro                # Kokoro TTS (local)
uv sync --extra cascade_gemini                # Gemini LLM
uv sync --extra cascade_elevenlabs            # ElevenLabs TTS
uv sync --extra cascade_deepgram              # Deepgram ASR
uv sync --extra cascade_parakeet              # Parakeet batch/RNNT ASR (Apple Silicon)
uv sync --extra cascade_voxtral_mlx           # Voxtral ASR (Apple Silicon)
uv sync --extra cascade_nemotron              # NeMo ASR (CUDA)
uv sync --extra cascade_gradium               # Gradium TTS (Python ≥3.12)
uv sync --extra cascade_all                   # All cascade providers

Vision and head-tracking extras:

uv sync --extra yolo_vision          # YOLO head-tracking
uv sync --extra mediapipe_vision     # MediaPipe head-tracking
uv sync --extra local_vision         # Local VLM (SmolVLM2)
uv sync --extra all_vision           # All vision features

Combine extras freely:

uv sync --extra cascade --extra cascade_kokoro --extra cascade_gemini --extra yolo_vision --group dev

Tip: Use uv sync --frozen to install from the lockfile without re-resolving.

</details>

<details> <summary><b>Using pip</b></summary>

python -m venv .venv
source .venv/bin/activate
pip install -e ".[cascade]"

# Add providers
pip install -e ".[cascade_kokoro]"
pip install -e ".[cascade_gemini]"
# etc.

</details>

Optional extras reference

Configuration

Environment variables

Copy .env.example to .env and fill in the API keys for the providers you use:

cascade.yaml

The cascade.yaml file at the project root controls which providers are used and their settings. Structure:

asr:
  provider: parakeet_mlx_progressive   # Which ASR to use
  providers:
    parakeet_mlx_progressive:          # Provider definitions
      module: parakeet_mlx_progressive
      class: ParakeetMLXProgressiveASR
      streaming: true
      location: local
      hardware: apple_silicon
      # Provider-specific settings
      model: mlx-community/parakeet-tdt-0.6b-v3
      precision: float16

llm:
  provider: gemini-2.5-flash-lite      # Which LLM to use
  temperature: 1.0
  providers: { ... }

tts:
  provider: kokoro                     # Which TTS to use
  trim_silence: true
  providers: { ... }

Change provider: under each section to switch. You can also override from the CLI with --asr-provider, --llm-provider, or --tts-provider.

ASR providers

LLM providers

TTS providers

Running the app

[!TIP] Make sure the Reachy Mini daemon is running before launching. See Reachy Mini's SDK for setup.

reachy-mini-conversation-app --gradio

CLI options

Examples

# Gradio UI with YOLO head-tracking
reachy-mini-conversation-app --gradio --head-tracker yolo

# Console mode (no UI, VAD-based)
reachy-mini-conversation-app

# Override providers from CLI
reachy-mini-conversation-app --gradio --asr-provider deepgram --llm-provider gpt-4o-mini

# Audio-only (no camera)
reachy-mini-conversation-app --gradio --no-camera

# Automated testing
reachy-mini-conversation-app --autotest

Profiles

Profiles define the robot's personality: what it says, how it sounds, and which tools it can use.

Each profile is a folder under src/reachy_mini_conversation_app/profiles/<name>/ containing:

Selecting a profile

• —Environment variable: REACHY_MINI_CUSTOM_PROFILE=pirate in .env
• —Gradio UI: Open the "Personality" accordion to switch profiles, edit instructions, or create new ones.

Template placeholders in instructions.txt

Reuse shared prompt snippets by referencing files under src/reachy_mini_conversation_app/prompts/:

[passion_for_lobster_jokes]
[identities/witty_identity]

Locked profile mode

Set LOCKED_PROFILE in src/reachy_mini_conversation_app/config.py to lock the app to a single profile. The Gradio UI shows "(locked)" and disables profile editing. Useful for dedicated app variants.

Live reactions

Reactions let the robot respond to keywords or entities in real time — while the user is still speaking — without waiting for the LLM.

Define them in profiles/<name>/reactions.yaml:

- name: music_excitement
  callback: excited_about_music
  trigger:
    words: [music, guitar, piano, drum, violin]

- name: food_reaction
  callback: react_to_food_entity
  trigger:
    entities: [food]
  repeatable: true

- name: groovy_dance
  callback: do_groovy_dance
  trigger:
    all:
      - words: [danc*]
      - words: [groov*]

- name: reachy_name
  callback: react_to_name
  trigger:
    words: [reachy, richie, reechy]
  params:
    emotion: helpful1

Trigger types

Behavior

• —Reactions fire at most once per conversation turn by default.
• —Set repeatable: true to allow multiple triggers per turn (deduplicated by entity text for entity triggers).
• —params are passed as **kwargs to the callback.

Callback signature

Each callback value maps to a Python module in the profile folder exporting an async function:

async def my_callback(deps: ToolDependencies, match: TriggerMatch, **kwargs) -> None:
    ...

match.words contains matched keywords; match.entities contains matched entities (with .text, .label, .confidence).

LLM tools

Tools the LLM can call during a conversation:

Profiles can also define custom tools (e.g. turn_left, turn_right, center_position in the default profile).

Advanced features

<details> <summary><b>External profiles and tools</b></summary>

Store profiles and tools outside the source tree:

external_content/
├── external_profiles/
│   └── my_profile/
│       ├── instructions.txt
│       ├── tools.txt
│       └── voice.txt
└── external_tools/
    └── my_custom_tool.py

Set in .env:

REACHY_MINI_CUSTOM_PROFILE=my_profile
REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=./external_content/external_profiles
REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=./external_content/external_tools

• —Default mode: tools.txt must list every tool explicitly. Names resolve against built-in tools first, then external tools.
• —Auto-load mode (AUTOLOAD_EXTERNAL_TOOLS=1): all *.py modules in the external tools directory are loaded automatically.

</details>

<details> <summary><b>Multiple robots on the same subnet</b></summary>

reachy-mini-conversation-app --robot-name <name>

<name> must match the daemon's --robot-name value.

</details>

<details> <summary><b>Autotest mode</b></summary>

Run the full pipeline with synthetic text utterances instead of a microphone:

reachy-mini-conversation-app --autotest
reachy-mini-conversation-app --autotest my_test_script.txt

Each line in the test file is treated as a user utterance. Useful for end-to-end testing without audio hardware.

</details>

<details> <summary><b>OpenAI Realtime mode (legacy)</b></summary>

The original audio-to-audio mode using OpenAI's realtime API is still available:

reachy-mini-conversation-app --realtime --gradio

This bypasses the cascade pipeline entirely. Requires OPENAI_API_KEY.

</details>

Contributing

We welcome bug fixes, features, profiles, and documentation improvements. Please review our contribution guide for branch conventions, quality checks, and PR workflow.

Quick start:

• —Fork and clone the repo
• —Follow the installation steps (include the dev dependency group)
• —Run contributor checks listed in CONTRIBUTING.md

License

Apache 2.0