UQLM is a Python library designed for detecting hallucinations in Large Language Models (LLMs) by employing uncertainty quantification (UQ) techniques. It offers a flexible framework for developers and researchers to assess the reliability of LLM outputs, providing confidence scores to identify potential errors or fabricated information.

How It Works

UQLM categorizes UQ scorers into four types: Black-Box (consistency-based), White-Box (token-probability-based), LLM-as-a-Judge, and Ensemble. Black-box methods measure response consistency across multiple generations, offering universal compatibility but higher latency and cost. White-box methods leverage internal token probabilities for efficiency but require model access. LLM-as-a-Judge uses other LLMs for evaluation, allowing customization. Ensemble scorers combine multiple methods for robust estimation, with options for off-the-shelf use or fine-tuning.

Quick Start & Requirements

• Primary install: pip install uqlm
• Prerequisites: LangChain compatible LLM.
• Resources: Requires multiple LLM calls for Black-Box and LLM-as-a-Judge methods, potentially increasing cost and latency.
• Demos: Black-Box UQ Demo, White-Box UQ Demo, LLM-as-a-Judge Demo, Ensemble Demos

Highlighted Details

• Supports a wide range of UQ scorers including semantic negentropy, exact match, BERT-score, minimum token probability, and various LLM-as-a-Judge approaches.
• Offers an ensemble scorer that can be tuned on custom datasets for improved performance.
• Integrates with LangChain, allowing seamless use with various LLM providers.
• Provides detailed documentation and example notebooks for each scorer category.

Maintenance & Community

• Associated research paper: Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers.
• The project appears to be primarily driven by Dylan Bouchard and Mohit Singh Chauhan.

Licensing & Compatibility

• License: Not explicitly stated in the README. Compatibility for commercial use or closed-source linking is undetermined.

Limitations & Caveats

• The README does not specify the license, which is crucial for determining commercial usability.
• Black-box methods incur significant computational cost and latency due to multiple LLM calls.
• White-box methods are limited to LLMs that provide access to token probabilities.