This project provides an OpenAI-compatible API server for Apple's on-device Foundation Models, enabling local AI processing for applications designed to interact with OpenAI endpoints. It targets developers and power users looking to leverage Apple's AI capabilities without relying on cloud services.

How It Works

The server acts as a drop-in replacement for OpenAI's API, specifically implementing the chat completions endpoint with streaming support. It utilizes Apple's Foundation Models for local inference, offering a privacy-focused and potentially faster alternative to cloud-based LLMs. The choice of a GUI application over a command-line tool is a deliberate design decision to circumvent Apple's rate-limiting policies for foreground applications.

Quick Start & Requirements

• Installation: Download the pre-built .zip from the Releases page or build from source using Xcode.
• Requirements: macOS 14 beta, Xcode 15 beta, and Apple Intelligence enabled in system settings.
• Usage: Launch the app, configure server settings (defaulting to 127.0.0.1:11535), and click "Start Server".
• Docs: Apple Foundation Models Documentation, OpenAI API Documentation

Highlighted Details

• OpenAI-compatible API with chat completions endpoint.
• Real-time streaming response support.
• On-device processing using Apple's Foundation Models.
• Function calling capabilities are under development (WIP).

Maintenance & Community

The project was primarily developed using Cursor, Claude Sonnet 4, and ChatGPT. Further community engagement details are not specified in the README.

Licensing & Compatibility

Licensed under the MIT License, permitting commercial use and integration with closed-source applications.

Limitations & Caveats

Users may still encounter rate limits due to current limitations in Apple's Foundation Models, with a recommendation to restart the server if issues arise. The project is also noted as being in beta.