Generate Music with AI

Generate Music with AI is a full-stack web application that turns a text prompt into an original music track using ACE-Step, a diffusion and language-based music generation foundation model, running on serverless GPUs. You describe the style, tempo, and mood — "lo-fi hip hop with rain ambiance" or "aggressive metalcore with a jazz breakdown" — and the system composes and renders an original audio track.

The project demonstrates full-stack AI systems engineering across two linked components:

1. Upstream Open-Source Contribution (ACE-Step 1.5 Modal): Added serverless GPU deployment support (modal_app.py) to the open-source ACE-Step foundation model repository, leveraging CRIU-based GPU memory snapshotting for rapid cold starts and scale-to-zero when idle.
2. Web Application (ai-music-gen): A Next.js 16 frontend and FastAPI stateless proxy handling rate limiting, speculative GPU prewarming, asynchronous job polling, and waveform rendering.

From user input to audio playback, each layer serves a distinct operational role:

1. Speculative GPU prewarm: On the visitor's first form interaction (keystroke or input focus), the frontend dispatches a background prewarm request. This wakes the serverless GPU container on Modal while the user is still crafting their prompt, hiding 30+ seconds of cold start latency. A visibility-gated, capped heartbeat holds the worker warm, bounded by an in-memory monthly budget cap.
2. FastAPI backend (Railway): Acts as a lightweight, stateless proxy. It validates payloads with Pydantic, applies client-IP-keyed rate limiting via slowapi, and proxies the generation task to Modal over a persistent HTTP/2 connection.
3. Modal GPU worker: Launches asynchronous diffusion inference on demand and returns a task ID immediately, avoiding long-lived blocking HTTP connections.
4. Asynchronous job polling: The frontend polls /api/jobs/{task_id} for real-time progress updates until inference completes.
5. Waveform audio playback: Once rendered, the frontend proxies the generated audio stream from Modal and visualizes it using a WaveSurfer.js waveform player.

Browser (Next.js on Vercel)
  ├── Speculative prewarm on first interaction
  └── Fetch / SSE to NEXT_PUBLIC_API_URL
FastAPI (Railway)
  ├── Client-IP rate limiting (slowapi, 5 req/min on /generate)
  ├── In-memory prewarm budget & deduplication state
  ├── Persistent HTTP/2 client (httpx, shared lifespan)
  └── Stateless proxy to ACE-Step REST API
Modal GPU (Serverless ACE-Step 1.5)
  ├── CRIU memory snapshots for sub-second model reloads
  └── Scale-to-zero GPU orchestration (L4 / A10G)

Key Engineering Decisions

• Speculative GPU Prewarm (ADR 0001): Cold starting a multi-gigabyte diffusion model on serverless GPUs can take 30 to 45 seconds. By initiating prewarm on first user interaction rather than form submission, the GPU is warm before the user clicks generate. Heartbeats pause when the browser tab is hidden and terminate after a capped duration to strictly control cloud GPU costs.
• Client-IP Rate Limiting: Rate limits are keyed strictly on client IP rather than session cookies. Because session cookies are client-supplied, rotating cookies previously minted fresh allowances; IP-based limiting provides robust abuse prevention across all endpoints.
• HTTP/2 with httpx: The ACEStepClient is instantiated once during the FastAPI lifespan startup and shared across requests. Driving persistent HTTP/2 with httpx eliminates repeated TCP handshake overhead and multiplexes requests to Modal.
• Production CI/CD & DevSecOps: Comprehensive automated GitHub Actions pipelines enforce code quality (Ruff, ESLint, Prettier, TypeScript verification), unit testing (Pytest and Vitest with uv caching), and automated security scanning including SAST (Semgrep, Bandit), secrets detection (Gitleaks), container vulnerability auditing (Trivy), and automated semantic releases via Release Please.

This project demonstrates practical AI infrastructure and full-stack product engineering:

• Serverless GPU orchestration: Optimizing cold starts with memory snapshotting, lifecycle management, and cost-capped prewarming.
• Open-source foundation model deployment: Packaging complex deep learning models (PyTorch, Hugging Face transformers, DiT) into production cloud APIs.
• Defensive API architecture: Stateless proxy design, IP-keyed rate limiting, strict Pydantic/Zod schemas, and end-to-end CI/CD DevSecOps automation.