Solshine/hyperspace-jam-lyria

Hyperspace Jam — Lyria AI Edition

Hand-controlled AI music generation with hyperbolic geometry visuals.

Wave your hands in front of a webcam to DJ real-time AI-generated music. Hand positions, finger spread, wrist angles, and movement speed are interpreted by a local Qwen 3.5 AI model into musical descriptions, which steer Google's Lyria RealTime API to generate a continuous stream of music that responds to your gestures.

Forked from Hyperspace Jam v2 (original by collidingScopes). The original used Tone.js client-side synthesis — this version replaces all audio generation with AI-generated music via Lyria RealTime.

How It Works

Webcam → MediaPipe Hand Tracking → Gesture Data
                                        ↓
                              ┌─────────┴──────────┐
                              │   Python Backend    │
                              │                     │
                              │  Qwen 3.5 (0.8B)   │ ← Interprets gestures
                              │  via Ollama          │   into musical text
                              │         ↓           │   descriptions
                              │  Lyria RealTime API │ ← Generates real-time
                              │  via Google GenAI    │   48kHz stereo audio
                              │         ↓           │
                              │  PCM audio stream   │
                              └─────────┬──────────┘
                                        ↓
                           Browser Web Audio API → Speakers
                           + Three.js hyperbolic geometry visuals

Data Flow Detail

1. 1.MediaPipe HandLandmarker tracks 30 landmarks per hand at ~30fps in the browser
2. 2.game.js computes gesture features: hand height, finger spread, wrist angle, velocity, finger extensions, shape detection
3. 3.MusicManager.js sends gesture data to the Python backend via WebSocket every 100ms
4. 4.Backend does two things in parallel:
5. 5.Direct parameter mapping: Hand height → brightness, finger spread → density, wrist tilt → brightness modulation, drum hand → BPM (sent to Lyria every 500ms, only when hands move)
6. 6.Qwen 3.5 prompt generation: Every 3 seconds (only when gesture changes significantly), the gesture state is described in natural language and sent to Qwen 3.5 0.8B via Ollama, which outputs weighted musical text prompts
7. 7.Lyria RealTime receives the prompts + config and generates 2-second chunks of 48kHz stereo 16-bit PCM audio, streamed back via WebSocket
8. 8.Browser decodes PCM chunks and schedules seamless playback via Web Audio API AudioWorklet, with an AnalyserNode feeding the hyperbolic geometry visualizer

Gesture Controls

Musical Mapping

Groove Lock

When hands stay still, the music stays in its groove — no prompt changes are sent to Lyria. Small movements trigger gradual evolution ("keep the groove, shift subtly"). Large, dramatic movements trigger a direction change ("shift genre entirely").

Transition System

The Qwen model maintains conversation history (last 6 exchanges) to ensure smooth transitions. It builds on previous themes rather than jumping randomly between styles.

Architecture

Frontend (Browser)

Backend (Python)

Key Technologies

Setup

Prerequisites

• —Python 3.10+
• —Ollama running locally with qwen3.5:0.8b model
• —Google Gemini API key with Lyria RealTime access
• —Modern browser with webcam access (Chrome recommended)

Installation

# Clone the repository
git clone https://github.com/SolshineCode/hyperspace-jam-lyria.git
cd hyperspace-jam-lyria

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate        # Linux/Mac
source .venv/Scripts/activate    # Windows/Git Bash

# Install Python dependencies
pip install -r backend/requirements.txt

# Pull the Qwen model (if not already installed)
ollama pull qwen3.5:0.8b

Configuration

Local development: Create a .env file in the project root:

GEMINI_API_KEY=your-gemini-api-key-here

Hugging Face Spaces: Add GEMINI_API_KEY as a Secret in the Space settings (Settings > Repository Secrets).

The .env file is in .gitignore and will not be committed.

Running

# Make sure Ollama is running
ollama serve  # (if not already running)

# Start the server
python -u backend/server.py

# Open in browser
# http://localhost:8088

The server starts on port 8088 by default. It serves both the API (WebSocket at /ws) and the frontend static files.

Quick Start (one command)

bash start.sh

Lyria RealTime API Details

Connection

• —Protocol: WebSocket (persistent, bidirectional, low-latency)
• —Model: models/lyria-realtime-exp
• —SDK: google-genai with api_version='v1alpha'
• —Audio format: Raw 16-bit PCM, 48kHz, stereo
• —Chunk size: 384,000 bytes (2 seconds of audio)
• —First chunk latency: ~3-4 seconds after play()
• —Session limit: 10 minutes (auto-reconnect planned)

Controls Used

Prompt Engineering

The Qwen model receives structured gesture descriptions like:

Hand HIGH (0.8), fingers WIDE, wrist STRAIGHT, movement STILL.

Evolve gradually — keep the groove, shift subtly.

And outputs weighted prompt JSON:

{
  "prompts": [
    {"text": "ethereal ambient floating pads celestial", "weight": 1.5},
    {"text": "shimmering reverb atmosphere", "weight": 1.0}
  ]
}

The conversation history ensures coherent transitions between styles.

Gesture Change Detection

The backend implements a two-tier change detection system:

Significant Change (triggers config update + Qwen prompt)

• —Hand height moved > 0.12
• —Finger spread changed > 0.15
• —Wrist angle changed > 0.25
• —Hand velocity > 0.25
• —Shape (white square) toggled
• —Drum hand toggled

Major Change (triggers Qwen "change direction" mode)

• —Hand height moved > 0.3
• —Finger spread changed > 0.35
• —Hand velocity > 0.5
• —Shape toggled

If neither threshold is met, no updates are sent — the music stays locked in its current groove.

Deployment to Hugging Face Spaces

1. 1.Create a private Space on Hugging Face (SDK: Docker or Static)
2. 2.Add GEMINI_API_KEY as a Secret in Space settings
3. 3.Push the repository to the Space
4. 4.The backend will read the API key from environment variables

Known Limitations

• —Lyria session limit: 10 minutes max per session. Page refresh starts a new session.
• —Audio latency: ~3-4 second initial latency for first chunk, then ~2 second chunks arrive smoothly. Not suitable for rhythm-game-level precision.
• —Qwen inference: ~1-2 seconds per prompt generation on CPU with 0.8B model. Doesn't block audio streaming.
• —No vocals: Lyria RealTime generates instrumental music only.
• —Experimental API: Lyria RealTime is in v1alpha — API may change.

Credits

• —Original Hyperspace Jam: collidingScopes — hand tracking + Tone.js synth + hyperbolic visuals
• —Lyria RealTime: Google DeepMind — real-time AI music generation
• —Qwen 3.5: Alibaba/Qwen — small language model for gesture interpretation
• —MediaPipe: Google — hand landmark detection
• —Three.js: mrdoob — WebGL rendering
• —Ollama: ollama.com — local LLM inference

License

MIT License