Floneum provides a Rust-based ecosystem for developing applications with local or remote AI models, targeting developers and researchers. It simplifies interaction with various AI modalities (text, audio, image) and offers a graphical editor for AI workflows, enabling efficient local AI application development.

How It Works

Floneum leverages the candle machine learning library for pure Rust model execution, supporting quantized and accelerated models. Its core, Kalosm, offers a unified interface for diverse models like Llama, Mistral, and Whisper. A key innovation is structured generation, allowing Rust types with #[derive(Parse, Schema)] to constrain model output to specific formats (JSON, regex patterns), enhancing data integrity and control.

Quick Start & Requirements

• Install Rust: cargo new floneum-app && cd floneum-app
• Add dependencies: cargo add kalosm --features language (add metal, cuda, or mkl for acceleration) and cargo add tokio --features full.
• Run: cargo run --release
• Prerequisites: Rust toolchain. GPU with Metal, CUDA, or MKL support is recommended for acceleration.
• Documentation: Kalosm website

Highlighted Details

• Supports Llama, Mistral, Phi, Whisper, RWuerstchen, TrOcr, Segment Anything, and Bert models.
• Achieves performance comparable to llama.cpp on Metal (M2): 39 t/s vs 27 t/s for Mistral 7b.
• Enables structured generation with custom parsers and grammar for complex data structures.
• Includes utilities for context extraction, audio transcription, and web scraping.

Maintenance & Community

• Active development with a Discord community for discussion and support.
• Contributions are welcomed via the issue tracker.

Licensing & Compatibility

• MIT License. Permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

• Fusor, the WGPU runtime, is in early development and not production-ready.
• Segment Anything model support does not currently list GPU acceleration.