Lumigator is an open-source platform by Mozilla.ai designed to help users select the most suitable language models for their specific needs, particularly for summarization and translation tasks. It addresses the challenge of an rapidly evolving AI landscape where choosing the right model can be difficult due to a lack of tailored evaluation tools. Lumigator provides a framework for comparing LLMs using task-specific metrics, enabling more informed and efficient model selection for developers and businesses.

How It Works

Lumigator utilizes a Docker Compose setup with multiple containerized services: Minio for local S3-compatible storage, a FastAPI backend for its REST API, a Ray cluster for job submission, and a web-based UI. It evaluates sequence-to-sequence and causal language models by comparing them against task-specific metrics. This approach allows for transparent and efficient comparison of models tailored to a user's specific domain and use case, moving beyond generic benchmarks.

Quick Start & Requirements

• Installation: Local deployment via Docker Compose.
• Prerequisites: Docker (Desktop 4.3+ on Mac, post-install steps on Linux), Docker Compose (1.28+ on Mac), system Python (version managers deactivated). For GPU support, NVIDIA Container Toolkit is required.
• Setup: Estimated setup time is minimal, but initial model downloads for experiments may cause delays.
• Documentation: https://docs.lumigator.ai/
• Example Notebook: Provided for API walkthrough.

Highlighted Details

• Supports evaluation of summarization and translation tasks using models like BART, T5, GPT, and Mistral.
• Local deployment mimics S3-compatible storage and can be configured for distributed scenarios with cloud storage and remote Ray clusters.
• API keys for Mistral, OpenAI, and Deepseek can be configured via the UI.
• Models are cached locally after the first download for faster subsequent runs.

Maintenance & Community

• Actively developed by Mozilla.ai.
• Bug reports and feature requests should be filed as GitHub issues.
• Contribution guidelines are available in CONTRIBUTING.md.

Licensing & Compatibility

• The specific license is not detailed in the provided README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

Lumigator is in the early stages of development, with missing features and documentation. Users should expect breaking changes in core interfaces and configuration structures. The README does not specify the license, which may impact compatibility.