OnyxMunk/AudioForge

1# AudioForge Architecture
2
3## Overview
4
5AudioForge is a production-ready, open-source music generation platform inspired by Suno. It uses a multi-stage pipeline to generate music from text descriptions.
6
7## System Architecture
8
9```
10┌─────────────┐
11│   Frontend   │ (Next.js + React)
12│  Port 3000   │
13└──────┬───────┘
14       │ HTTP/REST
15       ▼
16┌─────────────┐
17│   Backend    │ (FastAPI)
18│  Port 8000   │
19└──────┬───────┘
20       │
21       ├──► PostgreSQL (Metadata Storage)
22       ├──► Redis (Caching)
23       └──► Storage (Audio Files)
24```
25
26## Generation Pipeline
27
28### Stage 1: Prompt Understanding
29- **Service**: `PromptUnderstandingService`
30- **Purpose**: Analyze user prompt to extract:
31  - Musical style/genre
32  - Tempo/BPM
33  - Mood
34  - Instrumentation hints
35  - Lyrics (if provided)
36  - Duration preferences
37- **Output**: Enriched prompt with metadata
38
39### Stage 2: Music Generation
40- **Service**: `MusicGenerationService`
41- **Model**: Meta MusicGen (via AudioCraft)
42- **Purpose**: Generate instrumental music track
43- **Output**: WAV file with instrumental track
44
45### Stage 3: Vocal Generation (Optional)
46- **Service**: `VocalGenerationService`
47- **Model**: Bark or XTTS
48- **Purpose**: Generate vocals from lyrics
49- **Output**: WAV file with vocals
50
51### Stage 4: Mixing
52- **Service**: `PostProcessingService`
53- **Purpose**: Mix instrumental and vocal tracks
54- **Output**: Mixed audio file
55
56### Stage 5: Post-Processing/Mastering
57- **Service**: `PostProcessingService`
58- **Purpose**: Apply compression, EQ, normalization
59- **Output**: Final mastered audio file
60
61### Stage 6: Metadata Storage
62- **Service**: Database layer
63- **Purpose**: Store generation metadata, paths, status
64- **Output**: Database record
65
66## Technology Stack
67
68### Backend
69- **Framework**: FastAPI (async Python)
70- **Database**: PostgreSQL with SQLAlchemy async
71- **Caching**: Redis
72- **ML Framework**: PyTorch
73- **Music Models**: 
74  - MusicGen (Meta AudioCraft)
75  - Bark (for vocals)
76- **Audio Processing**: librosa, soundfile, scipy
77
78### Frontend
79- **Framework**: Next.js 14+ (App Router)
80- **Language**: TypeScript (strict mode)
81- **Styling**: Tailwind CSS
82- **UI Components**: Radix UI primitives
83- **State Management**: React Query + Zustand
84- **Forms**: React Hook Form + Zod
85
86### Observability
87- **Logging**: structlog (structured JSON logs)
88- **Metrics**: Prometheus
89- **Tracing**: OpenTelemetry (optional)
90
91## Data Flow
92
931. User submits prompt via frontend
942. Frontend sends POST to `/api/v1/generations`
953. Backend creates generation record (status: pending)
964. Background task starts processing
975. Pipeline executes stages 1-6
986. Frontend polls `/api/v1/generations/{id}` for status
997. On completion, audio available at `/api/v1/generations/{id}/audio`
100
101## Database Schema
102
103### Generations Table
104- `id`: UUID (primary key)
105- `prompt`: Text (user input)
106- `lyrics`: Text (optional)
107- `style`: String (extracted style)
108- `duration`: Integer (seconds)
109- `status`: String (pending/processing/completed/failed)
110- `audio_path`: String (final audio file path)
111- `instrumental_path`: String (instrumental track path)
112- `vocal_path`: String (vocal track path, if applicable)
113- `metadata`: JSON (analysis results, etc.)
114- `created_at`, `updated_at`, `completed_at`: Timestamps
115- `error_message`: Text (if failed)
116- `processing_time_seconds`: Float
117
118## API Endpoints
119
120### Generations
121- `POST /api/v1/generations` - Create generation
122- `GET /api/v1/generations/{id}` - Get generation status
123- `GET /api/v1/generations/{id}/audio` - Download audio
124- `GET /api/v1/generations` - List generations (paginated)
125
126## Configuration
127
128All configuration via environment variables (see `.env.example`):
129
130- Database connection
131- Redis connection
132- Model paths and devices (CPU/CUDA)
133- Storage paths
134- Logging levels
135- Feature flags
136
137## Scalability Considerations
138
139- **Horizontal Scaling**: Stateless API, can run multiple instances
140- **Queue System**: Background tasks can be moved to Celery/RQ
141- **Model Serving**: Models can be served separately via TorchServe
142- **Storage**: Audio files can be stored in S3/object storage
143- **Caching**: Redis caches prompt analysis results
144
145## Security
146
147- Input validation via Pydantic schemas
148- SQL injection prevention via SQLAlchemy ORM
149- CORS configuration
150- Rate limiting (to be added)
151- Authentication (to be added)
152
153## Performance Optimizations
154
155- Async/await throughout
156- Model lazy loading
157- Background task processing
158- Connection pooling (database, Redis)
159- Audio file streaming
160
161## Future Enhancements
162
163- User authentication & authorization
164- Rate limiting
165- WebSocket for real-time updates
166- Advanced post-processing (reverb, delay, etc.)
167- Multiple model support (switch between MusicGen variants)
168- Batch generation
169- Playlist creation
170- Social features (sharing, likes)
171