This project implements a voice-enabled ReAct agent leveraging OpenAI's Realtime API for natural language interaction and tool usage. It's designed for developers and researchers looking to build conversational AI applications that can interact with external tools through voice commands. The primary benefit is a functional, voice-first agent prototype that can be extended with custom tools and instructions.

How It Works

The agent operates on a ReAct (Reasoning and Acting) framework, enabling it to reason about tasks and then act by calling tools. It utilizes OpenAI's Realtime API for speech-to-text and text-to-speech, facilitating a natural voice interaction. The agent is designed to accept a list of LangChain tools, allowing for flexible integration with various functionalities.

Quick Start & Requirements

• Python Server: cd server && uv run src/server/app.py
• TypeScript Client: cd js_server && yarn dev
• Prerequisites: Python 3.10+, Node.js/Yarn, OPENAI_API_KEY, TAVILY_API_KEY (optional, for search tool).
• Access: Requires browser microphone access.
• Docs: http://localhost:3000

Highlighted Details

• Implements a voice-first ReAct agent.
• Integrates with OpenAI's Realtime API.
• Supports custom tools via LangChain.
• Allows customization of agent instructions.

Maintenance & Community

The project is maintained by langchain-ai. Further community engagement details are not provided in the README.

Licensing & Compatibility

The licensing information is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project is a prototype with planned next steps including enabling AI interruption and dynamic instruction/tool changes. WebSocket connection errors may indicate OpenAI Realtime API access or funding issues.