This project provides a Python wrapper to enhance the reliability of LLM applications, specifically targeting developers using OpenAI or Azure OpenAI services. It aims to eliminate dropped requests due to API errors, rate limits, or timeouts by implementing automatic retries with fallback models, cached responses, and key rotation.

How It Works

The core mechanism involves patching the openai.ChatCompletion.create method. When an error occurs, reliableGPT intercepts the failure and applies a configurable fallback strategy, which can include retrying with different OpenAI models (e.g., GPT-4, GPT-3.5-turbo) or using backup API keys. For persistent issues or high load, it offers a hosted caching solution using Supabase and cosine similarity for semantic retrieval, ensuring a response is always returned.

Quick Start & Requirements

• Install: pip install reliableGPT
• Prerequisites: Python 3.x, OpenAI API key, optionally Supabase credentials for caching.
• Setup: Integration involves a single line of code to patch the OpenAI client.

Highlighted Details

• Automatic fallback to alternative models (GPT-4, GPT-3.5-turbo variants) on failure.
• Handles context window errors by retrying with larger context models.
• Provides hosted caching with semantic similarity retrieval for persistent failures or high load.
• Supports key rotation for OpenAI API keys to mitigate invalid key errors.
• Can switch between Azure OpenAI and raw OpenAI endpoints.

Maintenance & Community

The project is actively maintained, with a deprecation warning indicating a migration to LiteLLM. Contact is available via Discord or email (ishaan@berri.ai, krrish@berri.ai).

Licensing & Compatibility

The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project is being deprecated in favor of LiteLLM. The caching mechanism relies on Supabase, which requires separate setup and credentials. The exact performance benefits and resource footprint of the caching layer are not detailed.