Lumos is an open-source framework for training language agents, designed for researchers and developers working with complex interactive tasks. It offers a modular architecture and a unified data format to achieve competitive performance against proprietary models like GPT-4 and larger open-source agents across diverse benchmarks.

How It Works

Lumos employs a modular design, separating agent functionality into planning, grounding, and execution modules. These modules are built upon Llama-2 (7B/13B) and can integrate off-the-shelf APIs. A key innovation is its unified data format, which consolidates various task types, enabling the framework to support a wide range of interactive tasks efficiently. This approach facilitates the development of robust, generalizable agents.

Quick Start & Requirements

• Install via ./setup.sh. Ensure cudatoolkit version matches your local CUDA installation.
• Download training data from a provided Google Drive link.
• Training commands: ./train.sh [MODULE] [FORMULATION], where [MODULE] is plan or ground, and [FORMULATION] is iterative or onetime.
• Evaluation: ./scripts/eval/hotpotqa.sh for HotpotQA.
• Official demo and code for annotation generation, training, and evaluation are available.

Highlighted Details

• Achieves competitive or superior performance compared to GPT-series agents and other open-source agents on tasks like Mind2Web, HotpotQA, WebShop, and InterCode_SQL.
• Trained on ~56K diverse, high-quality subgoal/action annotations generated using GPT-4.
• Offers fine-tuned checkpoints for planning and grounding modules on Hugging Face.
• Includes code for generating training annotations from existing benchmarks.

Maintenance & Community

The project released its latest version in March 2024, including new multimodal tasks and 13B-scale model experiments, along with a demo. Initial release of training/evaluation code and checkpoints was in November 2023.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README does not specify the exact license, which may impact commercial adoption. The setup requires aligning cudatoolkit with the local CUDA version, potentially adding complexity for users with different setups.