Kani is a lightweight, highly customizable Python framework for building chat-based language model applications, specifically designed for researchers and developers who need fine-grained control over the interaction flow. It simplifies integrating various language models, managing chat history, and implementing function calling, offering a more flexible alternative to opinionated, heavyweight frameworks.

How It Works

Kani employs a model-agnostic engine architecture, allowing seamless switching between different language models (OpenAI, Anthropic, Hugging Face via transformers or llama.cpp) with minimal code changes. It handles automatic chat memory management to prevent token limits from breaking conversations and provides a straightforward mechanism for function calling with built-in feedback and retry logic. The framework is built with an asynchronous design from the ground up for parallel processing.

Quick Start & Requirements

• Install with pip install "kani[<engine>]" (e.g., kani[openai], kani[huggingface], or kani[all]).
• Requires Python 3.10+.
• OpenAI API key is needed for kani[openai].
• See Model Zoo for engine-specific dependencies.

Highlighted Details

• Supports streaming responses token-by-token, even with function calls.
• Function calling is exposed via a simple @ai_function decorator.
• Offers deep customizability over prompts and underlying language models.
• Asynchronous design allows for easy scaling of multiple chat sessions.

Maintenance & Community

• Developed by PhD students from the University of Pennsylvania's NLP lab.
• Actively used in research, aiming to stay current with NLP practices.
• Documentation available on ReadTheDocs.
• Paper available on arXiv.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.

Limitations & Caveats

The README does not specify a license, which may pose a barrier for commercial or closed-source use.