This repository provides a community-curated model zoo and knowledge base for deploying AI models, particularly LLMs, on Apple's Core AI framework for iOS and macOS. It targets developers and researchers seeking efficient, verified on-device inference, offering converted models, conversion tools, and performance optimization insights for Apple hardware.

How It Works

The project focuses on converting popular open-source models (e.g., Qwen, Gemma) into Apple's .aimodel format, enabling on-device execution via the Core AI framework. It emphasizes end-to-end verification on target devices like iPhone and Mac, providing detailed conversion scripts, custom Metal kernels for performance bottlenecks, and a Swift runner for integration. This approach facilitates practical deployment of advanced AI capabilities directly on user devices.

Quick Start & Requirements

• Primary install/run command: Utilize the provided demo apps (apps/CoreAIChat, apps/QwenChatFast) or the Swift package (swift/CoreAIRunner). The demo app offers the simplest entry point with in-app model downloads.
• Prerequisites: iOS 27 / macOS 27 beta, Xcode 27 beta, xcodegen. Specific models are Mac-only. On-device inference leverages GPU and ANE.
• Resource Footprint: Model sizes vary; for example, BitCPM-8B requires approximately 2.1 GB on an iPhone GPU.
• Links:
  • Demo App (iOS/macOS): CoreAIChat on TestFlight (upcoming)
  • Mac-only bundles: .dmg (notarized)
  • Swift Runtime: knowledge/swift-runtime.md
  • Conversion Guide: knowledge/conversion-guide.md
  • Custom Kernels: knowledge/custom-metal-kernels.md

Highlighted Details

• Model Breadth: Features a diverse range of models including LLMs, Vision-Language Models (VLMs), OCR, ASR, TTS, generative audio, object detection, super-resolution, depth estimation, 3D Gaussian splats, and text-to-video.
• On-Device Optimization: Verified performance on iPhone 17 Pro (GPU/ANE) and M4 Max, incorporating custom Metal kernels for operations like flash-decode and gather_qmm to overcome stock framework limitations.
• Novel Architectures: Introduces firsts for the zoo, such as ternary LLMs (BitCPM-8B), diffusion LLMs (LLaDA-8B dLLM), Vision-Language-Action models (BitVLA), GUI-grounding VLMs (Holo2-4B), generative audio (Stable Audio Open), 3D Gaussian splats (TripoSplat), and text-to-video (LTX-Video).
• Performance Benchmarks: Provides detailed decode throughput (tok/s) comparisons across different Apple hardware, often contrasting with Hugging Face references.

Maintenance & Community

The repository is described as a "Community model zoo," but specific details regarding active maintainers, sponsorships, or dedicated community channels (like Discord/Slack) are not present in the provided text.

Licensing & Compatibility

The repository code is licensed under BSD-3-Clause. Model weights adhere to their respective original licenses (e.g., Apache-2.0, MIT, Gemma, LFM Open License v1.0, Stability Community). The BSD-3-Clause license is permissive for commercial use.

Limitations & Caveats

The project relies on beta versions of Apple's Core AI framework (iOS 27 / macOS 27), indicating potential instability and breaking changes. A known issue is an "in-graph KV-write crash" within the Core AI beta MPSGraph, for which workarounds are documented. Some advanced models are exclusively Mac-only, and ANE inference is not universally supported across all model types. Certain models require custom Metal kernels to function due to limitations in the stock framework's attention mechanisms.