This project provides a secure and reliable gateway for interacting with various Large Language Model (LLM) providers, including OpenAI, Cohere, and Amazon Bedrock. It targets developers and organizations concerned about data privacy and usage policies, offering PII scrubbing and detailed logging to ensure compliance and safe LLM integration.

How It Works

The gateway acts as an intermediary, routing requests to specified LLM providers. It intercepts all data, applies PII scrubbing heuristics, and logs interactions in a PostgreSQL database. This approach mitigates risks associated with third-party LLM providers' data usage policies, particularly for sensitive information, while providing a unified API interface.

Quick Start & Requirements

• Backend: Requires Python 3.11.3, Poetry, and Pyenv for environment management. brew install gitleaks is also recommended.
• Frontend: Uses npm and yarn, with the required Node.js version specified in .node-version.
• Docker: A docker-compose setup is available for simplified development and deployment.
• Setup: Detailed setup instructions for OpenAI, Cohere, and AWS Bedrock are available.

Highlighted Details

• Supports a wide range of LLM providers and models, including OpenAI (GPT-3.5, GPT-4), Anthropic (Claude), Cohere, AI21 Labs, and Meta (Llama-2).
• Implements PII scrubbing and data logging for enhanced privacy and compliance.
• Replicates the ChatGPT frontend experience using the /ChatCompletion endpoint for seamless integration.
• Provides both cURL and Python SDK examples for usage.

Maintenance & Community

The project is maintained by Wealthsimple. Further community or roadmap information is not detailed in the README.

Licensing & Compatibility

The README does not explicitly state the license type. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project is primarily focused on OpenAI and requires specific Python and Node.js versions, potentially limiting broader compatibility without adjustments. The effectiveness and comprehensiveness of the PII scrubbing heuristics are not detailed.