Apppublic

WCA0202/Timbre-Audio-to-Brief

sourceHugging Faceotherupdated 4mo agoView on Hugging Face

0likes

App README

Timbre — Audio-to-Brief Engine

![Hugging Face Spaces](https://huggingface.co/spaces/WCA0202/Timbre-Audio-to-Brief) ![Web](https://timbrebrief.com)

Timbre is a two-sided creative marketplace connecting clients who need music made with musicians, producers, and bands who make it. The core product is the Brief Engine — it translates a client's emotional description into a structured acoustic specification that musicians can actually execute.

Repository Structure

.
├── app.py                  # Hugging Face / Gradio entry point + JSON API
├── recommend_v2.py         # Semantic matching engine (imported by app.py)
├── requirements.txt        # Production dependencies
│
├── training/               # Offline data pipeline (run once)
│   ├── download_songs_v2.py   # Bulk YouTube downloader + Kworb Japan scraper
│   ├── build_library.py       # Scan songs/ → data/song_library.csv
│   ├── download_models.py     # Download Essentia .pb models (~500 MB)
│   ├── extract_features.py    # Feature + genre score extraction
│   ├── get_song_emotions.py   # Assign emotion labels
│   └── enrich_metadata.py     # Refresh language + genre columns only
│
├── data/                   # Generated CSVs (gitignored)
│   ├── song_features.csv   # Source of truth — acoustic + genre data (~9,983 songs)
│   └── song_library.csv    # Song index with language/ISRC
│
├── ui/                     # HF Space frontend assets
│   └── emotion_ui.html     # Standalone interactive bubble UI
│
├── docs/                   # Project documentation
│   ├── CONTRIBUTING.md     # Dev setup, scripts, PR checklist
│   ├── RUNBOOK.md          # Deployments, health, rollback
│   ├── ENV.md              # Environment variables & secrets
│   ├── architecture/
│   │   └── backend.md      # D1 + Workers + Stripe design notes
│   └── 創業計劃書.md       # Business plan (Chinese)
│
└── web/                    # timbrebrief.com — Vite + React SPA
    ├── src/
    │   ├── App.jsx             # Root — page router, global state (darkMode, lang)
    │   ├── theme.js            # Light / dark design tokens
    │   ├── data.js             # Sample data (emotions, brief, musicians)
    │   ├── pages/
    │   │   ├── HomePage.jsx         # Hero + Brief preview + trust strip
    │   │   ├── ExplorePage.jsx      # Emotion Explorer (full React port)
    │   │   ├── HowClientsPage.jsx   # 5-step editorial for clients
    │   │   ├── HowMusiciansPage.jsx # 5-step editorial + commission split
    │   │   ├── PricingPage.jsx      # Two-column pricing + FAQ accordion
    │   │   ├── OnboardingFlow.jsx   # 6-step brief builder + Clerk auth
    │   │   └── Dashboards.jsx       # Client dashboard, ProjectPage, MusicianDashboard
    │   └── components/
    │       ├── Nav.jsx              # Global nav — page links, dark mode, lang toggle
    │       ├── BriefDoc.jsx         # Acoustic brief document component
    │       ├── EmotionExplorer.jsx  # Compact bubble selector (legacy; onboarding now uses ExplorePage embedded)
    │       ├── EmotionCanvas.jsx    # Canvas-based emotion visualization
    │       ├── AcousticRadar.jsx    # Radar chart for acoustic features
    │       ├── MusicianCard.jsx     # Musician match card
    │       └── StatusPill.jsx       # Project status badge
    └── public/
        └── songs.json          # ~9,983 songs — normalized acoustic data for offline recs

Product Overview

Pages

Route (state)	Component	Description
`home`	`HomePage`	Hero, brief preview, trust strip, footer
`explore`	`ExplorePage`	Full-screen 4-layer Emotion Explorer
`how-clients`	`HowClientsPage`	Editorial: how the brief flow works for clients
`how-musicians`	`HowMusiciansPage`	Editorial: how matching + commission works
`pricing`	`PricingPage`	Client (free) + musician (commission) pricing
`onboarding`	`OnboardingFlow`	6-step guided brief builder
`signup-client` / `signup-musician`	`SignupPage`	Clerk `<SignUp>`
`login`	`LoginPage`	Clerk `<SignIn>`
`dashboard`	`ClientDashboard`	Sidebar + project list
`project`	`ProjectPage`	3-col: brief / conversation / deliverables
`musician-dashboard`	`MusicianDashboard`	Brief inbox, earnings, profile

Onboarding Flow (6 steps)

1.Start — choose entry point: describe a feeling / upload reference tracks
2.Explore — full ExplorePage (4-layer bubble UI) embedded inline, or upload reference tracks
3.Purpose — placement context, audience, story, commercial vs. personal use
4.Technical — tempo, length, instrumentation, stems, exclusivity, revisions (pre-filled from mood)
5.Review — full BriefDoc preview, all fields editable
6.Match — 3 musician cards matched to the brief; choose one to start the project

Business Model

—Clients — entirely free. Brief engine, Emotion Explorer, matching, project workspace, file delivery, messaging, milestone management.
—Musicians — no subscription, no pay-per-lead. One commission per completed project (the majority of the project fee stays with the musician). Shown as a line item on every transaction.

Trust & Safety Features

—Escrow payments — funds held until client approval
—Brief-as-contract — scope is locked at brief stage
—Dispute resolution — Timbre mediates
—Transparent commission — one line item, always visible

Emotion Explorer

A full-screen interactive 4-layer emotional journey implemented in React (ExplorePage.jsx), ported from the original vanilla JS ui/emotion_ui.html.

Layers:

1.Core emotion — Joy, Calm, Sadness, Anger, Fear, Anticipation (click one or blend two)
2.Nuance — 6 fine-grained descriptors per emotion
3.Somatic sensation — where you feel it in the body
4.Action urge — what the emotion wants to do

After completing the constellation:

—Emotion path is mapped to a weighted acoustic profile
—Top-25 matching songs are surfaced from songs.json (fully offline — no API call)
—Client view: paginated reference track list with YouTube search links
—Musician view: tracks + auto-generated Acoustic Brief (tempo, energy, tone, style tags, production suggestions)

Bilingual: full 中文 / English support. Language toggle is global (in Nav) and switches the entire UI mid-session without losing state.

Recommendation Engine

Web (offline, `ExplorePage.jsx`)

1.Constellation path → emotion key (from LABEL_TO_KEY map)
2.Emotion key → acoustic target profile from EMOTION_PROFILES (valence, arousal, mood scores, danceability)
3.Songs from songs.json scored: score = mood_dot_product − distance × 1.5
4.Top 25 returned, paginated by 5

HF Space (`recommend_v2.py`)

1.User text → cosine similarity against 16 EMOTION_DESCRIPTIONS via paraphrase-multilingual-MiniLM-L12-v2
2.Matched emotion → target feature vector from MOOD_PROFILES
3.Songs ranked by Euclidean distance in normalized feature space
4.Primary emotion label: boost=100; neighbouring emotions: boost=30

Isolated ML Microservices (HF Spaces)

Two single-responsibility Gradio Spaces handle on-demand ML inference. Both are stateless and ephemeral — no CSV reads/writes.

`WCA0202/Timbre-Feature-Extractor`

Extracts acoustic features from an uploaded audio file using Essentia.

—Repo: github.com/willwang0202/Timbre-Feature-Extractor
—Models: Discogs-EffNet (mood heads + danceability), MusiCNN → DEAM (valence/arousal), RhythmExtractor2013 (BPM)
—Input: audio file (MP3, WAV, AAC, FLAC, M4A)
—Output: { "emotion": "melancholic", "features": { "bpm", "valence", "arousal", "mood_*", "danceability" } }
—Web call: @gradio/client with handle_file(file); large WAV uploads use a longer analysis timeout than the Space startup/connect timeout.
—API:

  POST /call/analyze_audio   { "data": [<audio>] }  → { "event_id": "..." }
  GET  /call/analyze_audio/{event_id}               → SSE stream

`WCA0202/Timbre-Text-to-Emotion`

Classifies a free-text mood description into a Timbre emotion + acoustic target profile.

—Repo: github.com/willwang0202/Timbre-Text-to-Emotion
—Model: j-hartmann/emotion-english-distilroberta-base (~82MB, DistilRoBERTa fine-tuned on 6 emotion labels)
—Input: plain text string (e.g. "driving alone at night, feeling weightless")
—Output: { "emotion": "melancholic", "confidence": 0.87, "acoustic_profile": { "valence", "arousal", "bpm", "mood_*", "danceability" } }
—API:

  POST /call/classify_mood   { "data": ["mood text"] }  → { "event_id": "..." }
  GET  /call/classify_mood/{event_id}                   → SSE stream

When to call which:

—Client uploads a track → Timbre-Feature-Extractor
—Client types a mood description → Timbre-Text-to-Emotion
—Client uses the Emotion Explorer bubble UI → offline scoring against songs.json (no API call needed)

Running Locally

HF Space (Python/Gradio)

bash

source .venv/bin/activate
python app.py
# http://0.0.0.0:7860

Requires data/song_features.csv. If missing, run the data pipeline first.

Web Frontend

bash

cd web
npm install
npm run dev      # http://localhost:5173
npm run build    # → web/dist/

Environment variable (create web/.env.local):

VITE_CLERK_PUBLISHABLE_KEY=pk_live_...

Data Pipeline (one-time setup)

bash

# 1. Download songs via YouTube (global top 10,000)
python training/download_songs_v2.py

# 2. Download Japanese songs from Kworb.net
python training/download_songs_v2.py --japan

# 3. Scan songs/ → data/song_library.csv
python training/build_library.py

# 4. Download Essentia models (first time only, ~500 MB)
python training/download_models.py

# 5. Extract acoustic + genre features → data/song_features.csv
python training/extract_features.py

# 6. Assign emotion labels
python training/get_song_emotions.py

To regenerate web/public/songs.json after updating data/song_features.csv:

bash

node web/scripts/export-songs.mjs

Enriching without re-extraction

bash

python training/build_library.py
python training/enrich_metadata.py

Deployment

Web — Vercel

Configured via vercel.json at repo root.

—Build command: cd web && npm install && npm run build
—Build output: web/dist
—Env vars: VITE_CLERK_PUBLISHABLE_KEY, VITE_SUPABASE_URL, VITE_SUPABASE_ANON_KEY — set in Vercel dashboard

Domain: timbrebrief.com — registered on Cloudflare Registrar; DNS points at Vercel (A record 76.76.21.21, grey cloud).

Git Remotes

Remote	URL	Purpose
`origin`	`https://huggingface.co/spaces/WCA0202/Timbre-Audio-to-Brief`	Hugging Face Spaces (Python app)
`github`	`https://github.com/willwang0202/Timbre-Audio-to-Brief`	GitHub → Vercel (web)

Always push to both:

bash

git push origin main   # → Hugging Face
git push github main   # → GitHub / Vercel

Auth — Clerk

—Package: @clerk/react
—Production instance tied to timbrebrief.com
—<ClerkProvider> in main.jsx with global theme tokens
—Protected routes: dashboard, project, musician-dashboard, onboarding — redirect to login when signed out
—unsafeMetadata.role: 'client' or 'musician' saved on signup

Tech Stack

Layer	Tech
Web frontend	Vite + React 18, deployed on Vercel
Auth	Clerk (`@clerk/react`)
Emotion UI	Custom physics bubble engine — ported to React
Offline matching	Scored against `songs.json` (9,983 songs)
HF matching	sentence-transformers `paraphrase-multilingual-MiniLM-L12-v2`
Text-to-emotion	`j-hartmann/emotion-english-distilroberta-base` (HF Space)
Audio feature extraction	Essentia — Discogs-EffNet, MusiCNN, DEAM (HF Space)
Cloud app	Gradio on Hugging Face Spaces
Domain / CDN	Cloudflare Registrar + Vercel

Open TODOs

—Project workspace — project pages read Supabase projects/messages/milestones, store musician deliverables in private Supabase Storage, and move milestones through submit/approve/revision checkpoints; next step is Stripe capture + Connect payout on approved milestones
—Musician onboarding/profile — onboarding now saves matchable profile data to Supabase; next step is runtime audio analysis for uploaded portfolio files
—Clerk social connections — Google OAuth live; Spotify OAuth planned for reference-track import
—HF cold start — free-tier Spaces sleep after 48h; consider $9/mo always-on for Feature Extractor and Text-to-Emotion
—Wire Text-to-Emotion to frontend — StepUpload → Timbre-Feature-Extractor is live; StepExplore free-text input → Timbre-Text-to-Emotion not yet wired (next step: add text input + /classify_mood call in StepExplore)
—ExplorePage dark mode — canvas and bubble UI currently hardcodes light cream palette regardless of darkMode prop

License

All rights reserved. This codebase is proprietary and confidential. Access is granted solely for authorized use. No part of this software may be copied, modified, distributed, or used without explicit written permission from the copyright holder.

Timbre — Audio-to-Brief Engine

Repository Structure

Product Overview

Pages

Onboarding Flow (6 steps)

Business Model

Trust & Safety Features

Emotion Explorer

Recommendation Engine

Web (offline, ExplorePage.jsx)

HF Space (recommend_v2.py)

Isolated ML Microservices (HF Spaces)

WCA0202/Timbre-Feature-Extractor

WCA0202/Timbre-Text-to-Emotion

Running Locally

HF Space (Python/Gradio)

Web Frontend

Data Pipeline (one-time setup)

Enriching without re-extraction

Deployment

Web — Vercel

Git Remotes

Auth — Clerk

Tech Stack

Open TODOs

License

Web (offline, `ExplorePage.jsx`)

HF Space (`recommend_v2.py`)

`WCA0202/Timbre-Feature-Extractor`

`WCA0202/Timbre-Text-to-Emotion`