This project provides OpenAI-compatible RESTful APIs for Amazon Bedrock, enabling developers to leverage Amazon's foundation models (like Claude 3, Llama 3) using existing OpenAI SDKs and tools without code modifications. It targets developers and researchers seeking to experiment with or integrate Bedrock models into applications originally built for OpenAI.

How It Works

The gateway acts as a proxy, translating OpenAI API requests into Amazon Bedrock API calls. It supports various Bedrock capabilities including chat completions, streaming responses via Server-Sent Events (SSE), tool calls, embeddings, and multimodal inputs. The architecture can be deployed using AWS Lambda or AWS Fargate behind an Application Load Balancer (ALB), with options for Lambda Function URLs for potentially lower latency.

Quick Start & Requirements

• Deployment: Deploy via CloudFormation stack (ALB+Lambda or ALB+Fargate).
• Prerequisites: Access to Amazon Bedrock foundation models, AWS account.
• Setup: Requires creating an API key secret in AWS Secrets Manager. Deployment takes 3-5 minutes.
• Docs: Usage Guide, Troubleshooting Guide

Highlighted Details

• Supports streaming responses via Server-Sent Events (SSE), a feature not supported by AWS API Gateway.
• Offers OpenAI-compatible APIs for Chat Completion, Tool Use, Embeddings, and Multimodal inputs.
• Supports cross-region inference for Bedrock models.
• Includes new reasoning capabilities for models like Claude 3.7 Sonnet and DeepSeek R1.

Maintenance & Community

The project is part of the aws-samples organization, indicating official AWS backing. No specific community links (Discord/Slack) are provided in the README.

Licensing & Compatibility

• License: MIT-0 License.
• Compatibility: Permissive license allows commercial use and integration with closed-source applications.

Limitations & Caveats

Currently does not support SageMaker models or Bedrock custom/fine-tuned models with Provisioned Throughput. Performance may incur additional latency compared to direct AWS SDK calls.