This project provides a comprehensive, open-source music analysis suite designed for developers, researchers, and power users. It addresses the need for detailed audio analysis by offering features like chord recognition, beat tracking, real-time piano visualization, interactive guitar diagrams, and AI-assisted lyrics transcription, all accessible through a clean, intuitive interface. The application benefits users by consolidating complex music analysis tasks into a single, powerful tool that can process both uploaded audio files and YouTube videos.

How It Works

The application employs a microservices-like architecture, with a React/Next.js frontend communicating with a Python backend. The backend leverages pre-trained models such as Beat-Transformer for beat detection and Chord-CNN-LSTM for chord recognition. Advanced features like melody transcription are handled by experimental services like Sheet Sage, while lyrics synchronization and contextual analysis are powered by integrations with large language models like Google Gemini and services like Music.ai. This modular approach allows for flexibility and the integration of cutting-edge AI models for deeper musical insights.

Quick Start & Requirements

• Primary Install/Run:
  • Local Development: Clone repository (git clone --recursive), npm install, cd python_backend, pip install -r requirements.txt, python app.py (backend), npm run dev (frontend).
  • Docker Production: Download docker-compose.prod.yml and .env.docker.example, configure .env.docker, then run docker compose -f docker-compose.prod.yml --env-file .env.docker up -d.
• Prerequisites: Node.js 20.9+, npm 10+, Python 3.10.x (3.10.16 recommended), Docker (recommended for melody service), Git LFS, Firebase account, Gemini API key. FluidSynth is required for MIDI synthesis.
• Resource Footprint: Local setup involves running multiple services (frontend, Python backend, optional Docker services), requiring sufficient RAM and CPU. Docker deployment simplifies dependency management but still requires adequate system resources.
• Relevant Links: Setup instructions serve as primary documentation. No explicit demo or homepage link provided beyond the GitHub repository.

Highlighted Details

• Advanced Chord Analysis: Includes Roman Numeral Analysis, Key Modulation Signals, Simplified Chord Notation, and song segmentation (intro, verse, chorus, bridge, outro).
• Interactive Visualizations: Real-time piano roll with falling MIDI notes and interactive guitar chord diagrams synchronized to the beat grid.
• Experimental Melody Transcription: Sheet Sage offers an optional, experimental melodic line transcription with separate playback and MIDI export.
• AI-Powered Lyrics: Synchronized lyrics transcription augmented by an AI chatbot for contextual analysis and translation.

Maintenance & Community

The project acknowledges support from various external APIs and services, including Madmom, Google Gemini, yt-dlp, and Sheetsage. Specific details regarding active maintainers, community channels (like Discord or Slack), or a public roadmap are not provided in the README.

Licensing & Compatibility

The license type is not explicitly stated in the provided README. Native Windows backend installations are noted as unreliable due to dependency conflicts; WSL2/Ubuntu or Docker is strongly recommended for development and production on Windows x86_64 systems. Pinned Docker images are linux/arm64, requiring local builds for amd64 hosts.

Limitations & Caveats

The melody transcription feature is experimental, with potential limitations in inference speed and note accuracy. Native Windows backend installations face dependency issues, making WSL2 or Docker the preferred environments. The Music.ai API is deprecated for individual key access, requiring a business plan. Docker images are built for linux/arm64, necessitating manual builds for amd64 architectures.