Foundry Local enables on-device execution of generative AI models, targeting developers and power users who need to run AI locally without cloud dependencies. It offers enhanced privacy, reduced latency, and offline capabilities by leveraging ONNX Runtime and hardware acceleration, providing an OpenAI-compatible API for seamless integration.

How It Works

Foundry Local utilizes ONNX Runtime for optimized inference, automatically selecting and downloading model variants best suited for the user's hardware (CPU, GPU, NPU). This approach ensures high performance and efficient resource utilization. The project exposes an OpenAI-compatible API, allowing existing applications and workflows to interact with local models using familiar SDKs and REST calls.

Quick Start & Requirements

• Windows: winget install Microsoft.FoundryLocal
• macOS: brew tap microsoft/foundrylocal && brew install foundrylocal
• Models: foundry model run phi-3.5-mini (automatically downloads optimal variant)
• SDKs: pip install foundry-local-sdk openai (Python), npm install foundry-local-sdk openai (JavaScript)
• Documentation: Foundry Local Download

Highlighted Details

• On-device inference for privacy and reduced latency.
• OpenAI-compatible API for easy integration.
• Optimized performance via ONNX Runtime and hardware acceleration.
• Supports automatic model variant selection based on hardware.

Maintenance & Community

• Actively seeking feedback during preview phase.
• Issues and suggestions can be reported via GitHub Issues.
• Discord community available.

Licensing & Compatibility

• Licensed under the Microsoft Software License Terms.
• Compatible with applications using OpenAI SDKs.

Limitations & Caveats

The project is currently in a preview phase, indicating potential for breaking changes and ongoing development. Specific hardware acceleration support details beyond general mentions of GPU/NPU are not detailed in the README.