The open-source Passmark library enhances Playwright for AI-driven browser regression testing. It addresses the challenge of maintaining stable, end-to-end tests by leveraging AI for natural language step execution, intelligent caching, and multi-model assertion verification, enabling users to catch regressions early with reduced maintenance overhead.

How It Works

Passmark utilizes AI models, specifically from Anthropic (Claude) and Google (Gemini), to interpret natural language descriptions and execute browser actions via Playwright. Intelligent caching, powered by Redis, stores successful step executions for rapid replay, falling back to AI execution with auto-healing when necessary. A multi-model assertion engine employs a consensus mechanism between primary AI models, with an arbiter resolving disagreements, ensuring robust test validation.

Quick Start & Requirements

Installation involves initializing a Playwright project with npm init playwright@latest passmark-project, navigating into the directory (cd passmark-project), and installing Passmark (npm install passmark). Essential prerequisites include Node.js and Playwright. For multi-model features, API keys for Anthropic (ANTHROPIC_API_KEY) and Google (GOOGLE_GENERATIVE_AI_API_KEY) are mandatory, or alternatively, keys for AI gateways like Vercel, OpenRouter, or Cloudflare. Environment variables are managed via a .env file, configured in playwright.config.ts using the dotenv package. A sample test is provided in the README. Run tests with npx playwright test example.spec.ts --project chromium.

Highlighted Details

• runSteps() and runUserFlow() functions enable flexible, natural language test orchestration.
• Redis-based step caching with AI fallback and automatic self-healing capabilities.
• Multi-model assertion engine for consensus-based validation using Claude, Gemini, and an arbiter model.
• Support for AI Gateways (Vercel, OpenRouter, Cloudflare) to route AI requests and manage providers.
• Dynamic placeholders ({{run.*}}, {{global.*}}, {{data.*}}, {{email.*}}) facilitate data-driven and dynamic test scenarios.
• AI-powered data extraction and smart, AI-evaluated wait conditions eliminate rigid selectors.

Maintenance & Community

No specific details regarding maintainers, community channels (e.g., Discord, Slack), sponsorships, or roadmap were provided in the README.

Licensing & Compatibility

The project is licensed under FSL-1.1-Apache-2.0 (Functional Source License, Version 1.1, with Apache 2.0 future license). The FSL component imposes strong copyleft obligations, potentially impacting compatibility with closed-source or commercial applications.

Limitations & Caveats

The README explicitly states that test coverage is not comprehensive, with an invitation for contributions to address edge cases and failure modes.