This monorepo, formerly expo-audio-stream, provides a comprehensive cross-platform audio processing SDK for React Native developers. It addresses real-time audio streaming, recording, analysis, visualization, and on-device inference, enabling rich audio features in mobile and web applications with native performance.

How It Works

Audiolab is structured as a monorepo containing several specialized packages: @siteed/expo-audio-studio for core audio I/O, @siteed/expo-audio-ui for data visualization, @siteed/react-native-essentia for advanced audio analysis via native C++ bindings, and @siteed/sherpa-onnx.rn (in development) for on-device speech recognition and TTS. This modular approach leverages native implementations for performance-critical tasks, ensuring efficient audio processing across iOS, Android, and web platforms.

Quick Start & Requirements

Example applications demonstrating various use cases are available within the /apps directory. A web-based Audio Playground demo is accessible at https://deeeed.github.io/audiolab/playground/, and a Sherpa Voice demo is at https://deeeed.github.io/audiolab/sherpa-voice/. Installation typically involves standard React Native package management. Specific hardware acceleration requirements for on-device inference are not detailed.

Highlighted Details

• Cross-Platform: Full support for iOS, Android, and Web environments.
• On-Device AI: Integrates Sherpa-ONNX for speech-to-text and text-to-speech, and Essentia for advanced audio feature extraction (e.g., MFCC, Key, Spectrum).
• Real-time Audio: Features include zero-latency recording APIs and real-time streaming capabilities.
• Native Performance: Utilizes C++ for Essentia bindings and optimized native modules for efficient processing.
• Visualization: Dedicated UI components for visualizing audio data streams.

Maintenance & Community

The project is maintained by Arthur Breton, with sponsorship opportunities available. The README includes a roadmap detailing completed and pending features, such as automating changelog generation and integrating Sherpa ONNX models. No explicit community channels like Discord or Slack are listed.

Licensing & Compatibility

This project is licensed under the MIT License, which is permissive for commercial use and integration into closed-source applications.

Limitations & Caveats

The @siteed/sherpa-onnx.rn package is currently under development. The roadmap indicates ongoing work on optimizing advanced audio feature extraction for live scenarios and implementing cross-platform feature validation. Some UI components, like an audio preview waveform, are still pending.