AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories

AdaRubric addresses the critical challenge of evaluating Large Language Model (LLM) agent performance, where fixed rubrics fail to capture task-specific quality. It provides a dynamic, task-adaptive evaluation framework for multi-step agent tasks like code debugging and web navigation. This enables more reliable LLM agent evaluation and reward learning, leading to improved task success rates and more accurate reward signals for training.

How It Works

AdaRubric employs a three-stage pipeline. First, the Rubric Generator uses an LLM to dynamically create N orthogonal evaluation dimensions and corresponding 5-point scoring criteria based on a given task description, caching rubrics for cost efficiency. Second, the Trajectory Evaluator scores each step (Thought → Action → Observation) per-dimension, assigning a confidence-weighted score. It supports pluggable aggregation strategies like Weighted Mean (default), Geometric Mean, and Min Score. Third, the Data Filter curates high-quality DPO preference pairs using composable filters, with the novel DimensionAwareFilter provably preventing high-scoring dimensions from masking critical dimension-level failures.

Quick Start & Requirements

• Installation: Clone the repository and install dependencies using pip install -e ".[dev]".
• Prerequisites: Requires an OPENAI_API_KEY environment variable. pyyaml is needed for YAML configuration support.
• Links: See examples/quickstart.py for a runnable example.

Highlighted Details

• Achieves 0.79 Pearson correlation with human judgment, a +0.15 improvement over static baselines.
• Demonstrates deployment-grade inter-run reliability with Krippendorff's α of 0.83.
• DPO task success rates improve by +6.8–+8.5% over Prometheus across WebArena, ToolBench, and AgentBench benchmarks.
• Shows a +4.9% resolve rate improvement on SWE-bench code repair tasks with zero rubric engineering.
• Accelerates PPO convergence, achieving +6.6% SR at 5K steps.

Maintenance & Community

No specific details regarding maintainers, community channels (e.g., Discord/Slack), sponsorships, or roadmaps were found in the provided README.

Licensing & Compatibility

The project is licensed under the Apache 2.0 license, which is permissive and generally compatible with commercial use and closed-source linking.

Limitations & Caveats

The framework relies on external LLM APIs (specifically OpenAI, as indicated by the API key requirement), which may incur costs and introduce external dependencies. While demonstrating strong performance metrics, the project appears research-oriented, and its direct applicability to production systems without further adaptation is not explicitly detailed.