This project provides a desktop application for evaluating and comparing Large Language Models (LLMs) served via Ollama. It targets users who need to systematically test different models, prompts, and inference parameters to find optimal configurations for their use cases, offering a visual interface for results inspection and A/B testing.

How It Works

The application leverages Ollama's API to interact with various LLMs. It implements a "grid search" concept by iterating through user-defined combinations of models, prompts, and inference parameters (e.g., temperature, top_p). This systematic approach allows for comprehensive testing, with features for A/B testing different prompts or models side-by-side and managing experiment logs for reproducibility.

Quick Start & Requirements

• Install: Download pre-built binaries from the releases page.
• Prerequisites: Ollama must be installed and serving models locally or remotely.
• Setup: Minimal setup required if Ollama is already running.
• Docs: Project Blog Post

Highlighted Details

• Automated fetching of models from local or remote Ollama servers.
• Supports A/B testing of prompts and models simultaneously.
• Includes a prompt archive with "/" autocompletion for prompt selection.
• Outputs inference metadata like time and tokens per second.

Maintenance & Community

• Active development with contributions from multiple individuals.
• Development notes and workflow charts are available for contributors.

Licensing & Compatibility

• License: MIT License.
• Compatibility: Permissive license suitable for commercial use and integration with closed-source applications.

Limitations & Caveats

The project is primarily focused on evaluating models served through Ollama; integration with other LLM serving frameworks is not supported. Future features include result grading and sharing capabilities.