Maksim-1/hf-realtime-voice

Minimal Conversation App (S2S backend, WebSocket transport)

Drop-in alternative to `amir-tfrere/minimal-conversation-app-s2s-backend` that uses the WebSocket route of the Hugging Face speech-to-speech backend instead of the WebRTC SDP proxy. Same load balancer, same /session handshake, same UI, same orb. Just a different wire.

How it works

1. 1.App POSTs <lb_url>/session (empty JSON body).
2. 2.The LB picks a ready compute (round-robin) and returns:

   {
     "session_id": "...",
     "websocket_url": "wss://<compute>/v1/realtime",
     "connect_url": "wss://<compute>/v1/realtime?session_token=<JWT>",
     "session_token": "<JWT>",
     "pending_timeout_s": 60
   }

1. 1.App opens a WebSocket directly on connect_url (no rewrite to https://; unlike the WebRTC client which POSTs an SDP offer).
2. 2.Server pushes session.created on connect. Client replies with session.update (OpenAI Realtime GA schema: session.audio.input, session.audio.output, session.output_modalities).
3. 3.Client streams mic audio as PCM16 16 kHz mono base64 chunks (input_audio_buffer.append, one frame every ~40 ms).
4. 4.Server pushes response.output_audio.delta (PCM16 24 kHz mono base64) and transcript deltas.

The backend exposes one concurrent session per compute (same as WebRTC mode); the LB pins the session via a signed session_token.

Why WebSocket instead of WebRTC

Backend requirement

This app talks to the WebSocket route @app.websocket("/v1/realtime") defined in `websocket_router.py` on the `feat/webrtc-transport` branch. The same compute serves both the WebRTC POST and the WebSocket upgrade on the same path; no backend change required.

Smoke-test from the shell:

LB="https://kaa1l6rplzb1gg3y.us-east-1.aws.endpoints.huggingface.cloud"
curl -X POST "$LB/session" -H "Content-Type: application/json" -d '{}'
# -> { "connect_url": "wss://<compute>/v1/realtime?session_token=..." }
# Feed connect_url into a wscat / websocat and you should get a
# session.created event back immediately.

Tools

The assistant can call two tools mid-conversation (toggle them from the Tools button, top-right):

• —Web search — Google results via Serper.dev, proxied server-side so the key never reaches the browser. Set SERPER_API_KEY as a Space secret. Without it, the tool is disabled unless the user pastes their own key in the Tools panel.
• —Camera — while enabled, a live self-view shows bottom-left; when the model calls the tool, the current frame is sent to the vision-language model so it can see what you're showing it.

Connecting to a backend

Three modes, picked by env (/api/config tells the client which one is active):

• —`SPEECH_TO_SPEECH_URL` env — highest priority. The browser connects directly to this realtime WebSocket URL; it's shown read-only in Settings. Setting it disables the load-balancer logic entirely (no /api/session proxy, no queue, no metering, no sign-in). Unlike the LB address it is not a secret.
• —`LOAD_BALANCER_URL` env — the original flow: the browser POSTs the same-origin /api/session proxy, the server forwards to the LB, and the browser dials the per-session compute URL the LB hands back. The LB address never reaches the browser; the Settings URL field is hidden.
• —Neither — Settings → Speech-to-speech server URL: paste a full connect_url (wss://host/v1/realtime?...) or a bare host like localhost:8080 (the app adds /v1/realtime), and the browser connects to it directly.

Settings → Restart reconnects with the current voice, instructions and URL.

Usage limits

Conversation time is metered per UTC day by sign-in tier (see limiter.py / auth.py), but only on the deployed Space — metering turns on only when BOTH LOAD_BALANCER_URL and SPACE_ID (injected automatically by the HF Space runtime) are present. Running locally — even with LOAD_BALANCER_URL exported — leaves the app unmetered. Tunable via env:

PRO members are always unlimited. Members of cerebras, HuggingFaceM4, smolagents, and pollen-robotics are unlimited out of the box (shown as "Team", not "PRO"); set UNLIMITED_ORGS=my-team to add more. Matched case-insensitively against the user's organisations from HF OAuth.

Run locally

The app is now a small FastAPI server (it serves the front-end and the search proxy from one container).

pip install -r requirements.txt
export SERPER_API_KEY=...          # optional; web search is disabled without it
export SPEECH_TO_SPEECH_URL=...    # optional; pin a direct s2s server URL (overrides the LB)
export LOAD_BALANCER_URL=...       # optional; session-proxy flow (set a URL in Settings otherwise)
uvicorn server:app --reload --port 7860
# or, matching production: docker build -t s2s . && docker run -p 7860:7860 -e SERPER_API_KEY=... -e LOAD_BALANCER_URL=... s2s

Then open <http://localhost:7860/>, click the orb, allow the mic, talk.

Browsers require HTTPS or `localhost` for getUserMedia() (mic + camera). 127.0.0.1 and localhost both work; plain http://192.168.x.y does NOT.

Settings (stored in localStorage)

LocalStorage keys are namespaced s2s.ws.* so this app's settings do NOT collide with the WebRTC variant.

Files

Audio pipeline notes

• —Input: getUserMedia({ echoCancellation, noiseSuppression, autoGainControl }) feeds the mic-capture worklet at the AudioContext rate. The worklet resamples to 16 kHz (boxcar lowpass + decimation on the 48 -> 16 fast path, linear interpolation fallback for odd rates) and packs Int16 LE.
• —Output: response.output_audio.delta decodes to Int16 -> Float32 and is posted to the audio-playback worklet. The worklet maintains a per-context ring buffer, linearly interpolates 24 -> 48, and applies short 32-frame fades on entry/exit to suppress clicks.
• —Barge-in: when the server VAD detects user speech mid-response (input_audio_buffer.speech_started while ai-speaking), the client posts { kind: "clear" } to the playback worklet to wipe the queue immediately. The server itself cancels the in-flight response.

Credits

• —Backend: huggingface/speech-to-speech on feat/webrtc-transport
• —UI verbatim from amir-tfrere/minimal-conversation-app-s2s-backend (Pollen Robotics × Hugging Face)