Jacid23/Lyon_chatbox

Lyon Chatbox

A modular voice conversation app for the Reachy Mini robot. Cheap to run, versatile, customizable, and fun.

ChatBox uses a cascade pipeline — ASR → LLM → TTS — where each stage is a swappable provider. Mix cloud APIs and local models, tweak the personality, add live reactions to keywords, and let the robot dance.

[image]

How it works

🎤 Microphone
    │
    ▼
┌─────────┐     ┌─────────┐     ┌─────────┐     🔊 Speaker
│   ASR   │ ──▶ │   LLM   │ ──▶ │   TTS   │ ──▶ 🤖 Robot moves
└─────────┘     └─────────┘     └─────────┘
 Speech to       Reasoning       Text to
 text            + tool calls    speech

Voice Activity Detection (VAD) segments microphone input. The ASR provider transcribes it, the LLM generates a response (with optional tool calls for head movements, dances, emotions, camera…), and the TTS provider speaks it back — all while the robot animates.

Installation

[!IMPORTANT] Before using this app, install Reachy Mini's SDK.

<details open> <summary><b>Using uv (recommended)</b></summary>

# Create venv
uv venv --python python3.12 .venv
source .venv/bin/activate

# Install with cascade pipeline
uv sync --extra cascade

Install additional providers as needed:

uv sync --extra cascade_parakeet_progressive  # Parakeet ASR (Apple Silicon)
uv sync --extra cascade_kokoro                # Kokoro TTS (local)
uv sync --extra cascade_gemini                # Gemini LLM
uv sync --extra cascade_elevenlabs            # ElevenLabs TTS
uv sync --extra cascade_deepgram              # Deepgram ASR
uv sync --extra cascade_parakeet              # Parakeet batch/RNNT ASR (Apple Silicon)
uv sync --extra cascade_voxtral_mlx           # Voxtral ASR (Apple Silicon)
uv sync --extra cascade_nemotron              # NeMo ASR (CUDA)
uv sync --extra cascade_gradium               # Gradium TTS (Python ≥3.12)
uv sync --extra cascade_all                   # All cascade providers

Vision and head-tracking extras:

uv sync --extra yolo_vision          # YOLO head-tracking
uv sync --extra mediapipe_vision     # MediaPipe head-tracking
uv sync --extra local_vision         # Local VLM (SmolVLM2)
uv sync --extra all_vision           # All vision features

Combine extras freely:

uv sync --extra cascade --extra cascade_kokoro --extra cascade_gemini --extra yolo_vision --group dev

Tip: Use uv sync --frozen to install from the lockfile without re-resolving.

</details>

<details> <summary><b>Using pip</b></summary>

python -m venv .venv
source .venv/bin/activate
pip install -e ".[cascade]"

# Add providers
pip install -e ".[cascade_kokoro]"
pip install -e ".[cascade_gemini]"
# etc.

</details>

Optional extras reference

Configuration

Environment variables

Open the Gradio Settings panel to choose providers and save keys or local endpoint settings. Gradio cascade requires explicit ASR, LLM, and TTS choices because every runtime is different. The app writes those values to .env. You can also copy .env.example to .env and fill them manually:

If no .env is found near the working directory, the app falls back to ~/.config/settings/.env. Keys stored there survive app reinstalls and updates.

For preconfigured installs, copy config/examples/lyon_chatbox_cascade_setup.yaml to config/lyon_chatbox.yaml or lyon_chatbox.yaml. You can also point LYON_CHATBOX_CONFIG_FILE to any YAML file. That file chooses providers and can include non-secret endpoint/model settings while cascade.yaml remains the provider-definition catalog. API keys still belong in .env or real environment variables.

cascade.yaml

The cascade.yaml file at the project root controls which providers are used and their settings. Structure:

asr:
  provider: parakeet_mlx_progressive   # Which ASR to use
  cloud_providers:
    whisper_openai:
      module: whisper_openai
      class: WhisperOpenAIASR
      streaming: false
      requires: [OPENAI_API_KEY]
      model: whisper-1
  local_providers:
    parakeet_mlx_progressive:
      module: parakeet_mlx_progressive
      class: ParakeetMLXProgressiveASR
      streaming: true
      hardware: apple_silicon
      # Provider-specific settings
      model: mlx-community/parakeet-tdt-0.6b-v3
      precision: float16

llm:
  provider: gemini-2.5-flash-lite      # Which LLM to use
  temperature: 1.0
  cloud_providers: { ... }
  local_providers: { ... }

tts:
  provider: kokoro                     # Which TTS to use
  trim_silence: true
  cloud_providers: { ... }
  local_providers: { ... }

Change providers from the Gradio Settings panel, cascade.yaml, .env, or the CLI with --asr-provider, --llm-provider, or --tts-provider.

ASR providers

LLM providers

TTS providers

Running the app

[!TIP] Make sure the Reachy Mini daemon is running before launching. See Reachy Mini's SDK for setup.

lyon-chatbox --gradio

CLI options

Examples

# Gradio UI with YOLO head-tracking
lyon-chatbox --gradio --head-tracker yolo

# Console mode (no UI, VAD-based)
lyon-chatbox

# Override providers from CLI
lyon-chatbox --gradio --asr-provider deepgram --llm-provider gpt-4o-mini

# Audio-only (no camera)
lyon-chatbox --gradio --no-camera

# Automated testing
lyon-chatbox --autotest

Profiles

Profiles define the robot's personality: what it says, how it sounds, and which tools it can use.

Each profile is a folder under src/lyon_chatbox/profiles/<name>/ containing:

Selecting a profile

• —Environment variable: LYON_CHATBOX_CUSTOM_PROFILE=pirate in .env
• —Gradio UI: Open the "Personality" accordion to switch profiles, edit instructions, or create new ones.

Template placeholders in instructions.txt

Reuse shared prompt snippets by referencing files under src/lyon_chatbox/prompts/:

[passion_for_lobster_jokes]
[identities/witty_identity]

Locked profile mode

Set LOCKED_PROFILE in src/lyon_chatbox/config.py to lock the app to a single profile. The Gradio UI shows "(locked)" and disables profile editing. Useful for dedicated app variants.

Live reactions

Reactions let the robot respond to keywords or entities in real time — while the user is still speaking — without waiting for the LLM.

Define them in profiles/<name>/reactions.yaml:

- name: music_excitement
  callback: excited_about_music
  trigger:
    words: [music, guitar, piano, drum, violin]

- name: food_reaction
  callback: react_to_food_entity
  trigger:
    entities: [food]
  repeatable: true

- name: groovy_dance
  callback: do_groovy_dance
  trigger:
    all:
      - words: [danc*]
      - words: [groov*]

- name: reachy_name
  callback: react_to_name
  trigger:
    words: [reachy, richie, reechy]
  params:
    emotion: helpful1

Trigger types

Behavior

• —Reactions fire at most once per conversation turn by default.
• —Set repeatable: true to allow multiple triggers per turn (deduplicated by entity text for entity triggers).
• —params are passed as **kwargs to the callback.

Callback signature

Each callback value maps to a Python module in the profile folder exporting an async function:

async def my_callback(deps: ToolDependencies, match: TriggerMatch, **kwargs) -> None:
    ...

match.words contains matched keywords; match.entities contains matched entities (with .text, .label, .confidence).

LLM tools

Tools the LLM can call during a conversation:

Profiles can also define custom tools (e.g. turn_left, turn_right, center_position in the default profile).

Advanced features

<details> <summary><b>External profiles and tools</b></summary>

Store profiles and tools outside the source tree:

external_content/
├── external_profiles/
│   └── my_profile/
│       ├── instructions.txt
│       ├── tools.txt
│       └── voice.txt
└── external_tools/
    └── my_custom_tool.py

Set in .env:

LYON_CHATBOX_CUSTOM_PROFILE=my_profile
LYON_CHATBOX_EXTERNAL_PROFILES_DIRECTORY=./external_content/external_profiles
LYON_CHATBOX_EXTERNAL_TOOLS_DIRECTORY=./external_content/external_tools

• —Default mode: tools.txt must list every tool explicitly. Names resolve against built-in tools first, then external tools.
• —Auto-load mode (AUTOLOAD_EXTERNAL_TOOLS=1): all *.py modules in the external tools directory are loaded automatically.

</details>

<details> <summary><b>Multiple robots on the same subnet</b></summary>

lyon-chatbox --robot-name <name>

<name> must match the daemon's --robot-name value.

</details>

<details> <summary><b>Autotest mode</b></summary>

Run the full pipeline with synthetic text utterances instead of a microphone:

lyon-chatbox --autotest
lyon-chatbox --autotest my_test_script.txt

Each line in the test file is treated as a user utterance. Useful for end-to-end testing without audio hardware.

</details>

<details> <summary><b>OpenAI Realtime mode (legacy)</b></summary>

The original audio-to-audio mode using OpenAI's realtime API is still available:

lyon-chatbox --realtime --gradio

This bypasses the cascade pipeline entirely. Requires OPENAI_API_KEY.

</details>

Contributing

We welcome bug fixes, features, profiles, and documentation improvements. Please review our contribution guide for branch conventions, quality checks, and PR workflow.

Quick start:

• —Fork and clone the repo
• —Follow the installation steps (include the dev dependency group)
• —Run contributor checks listed in CONTRIBUTING.md

License

Apache 2.0