Summary

This project provides a Vitest extension for running and evaluating AI applications and agents. It addresses the need for structured, reproducible testing of AI components by integrating evaluation harnesses, judges, and reporting directly into a familiar JavaScript/TypeScript testing framework. Targeted at developers building AI-powered features and researchers assessing LLM performance, it offers a robust solution for improving the quality and reliability of AI integrations.

How It Works

The project is structured as a monorepo, extending Vitest with specialized constructs for AI evaluations. Core components include describeEval for defining evaluation suites, harnesses for interacting with AI models or agents, and judges for asserting correctness against defined criteria. Evaluations leverage Vitest's execution engine, enhanced with explicit harness calls and judge assertions via expect(...).toSatisfyJudge(...). This approach allows for detailed tracing of tool calls, model outputs, and execution metadata, facilitating deep analysis and automated grading.

Quick Start & Requirements

• Primary install/run command: pnpm install to set up the workspace, followed by pnpm evals to run evaluations.
• Prerequisites: Node.js and pnpm are required. Demo applications may need API keys configured via .env files.
• Documentation: Official guided setup and documentation are available at https://vitest-evals.sentry.dev/docs.

Highlighted Details

• GitHub Reporting: Seamlessly integrates with GitHub Actions, emitting Vitest JSON reports to generate job summaries, annotations for failed evaluations, and optional Check Runs.
• Local Report UI: A local React SPA, launched via pnpm exec vitest-evals serve, provides an interactive interface for inspecting detailed evaluation artifacts, including run summaries, sessions, tool calls, and trace details.
• Tool Replay: Supports automatic recording and replaying of tool calls within evaluations, configurable globally or per-tool, enhancing test determinism and debugging capabilities.
• Pluggable Judges: Features built-in judges like FactualityJudge and allows for custom judge implementations, enabling flexible, model-backed grading and assertion logic across different AI harnesses.

Maintenance & Community

No specific details regarding notable contributors, sponsorships, community channels (e.g., Discord/Slack), or roadmaps were present in the provided text.

Licensing & Compatibility

The license type and any compatibility notes for commercial use or closed-source linking were not explicitly stated in the provided text.

Limitations & Caveats

The project is organized as a monorepo, requiring familiarity with pnpm workspaces for development and contribution. Integration with specific AI providers may necessitate external API key configuration and availability.