This project trains open-source Large Language Models (LLMs) for optimization modeling tasks. It addresses the need for specialized LLMs by introducing OR-Instruct for synthetic data generation and the IndustryOR benchmark for real-world problem evaluation, significantly enhancing LLM capabilities in optimization.

How It Works

ORLM fine-tunes open-source LLMs using OR-Instruct, a semi-automated process for creating tailored synthetic data crucial for optimization modeling. It also introduces the IndustryOR benchmark, the first industrial-scale dataset for evaluating LLMs on real-world optimization problems, enabling robust performance assessment.

Quick Start & Requirements

• Primary install / run command: Clone the repository (https://github.com/Cardinal-Operations/ORLM.git), navigate into the directory (cd ORLM), and install dependencies (pip install -r requirements.txt).
• Non-default prerequisites and dependencies: GPU acceleration is implied for training and inference. The COPT solver is required for executing generated optimization models. Access to NVIDIA-A100 GPUs is mentioned for demos.
• Links: The README mentions links to the paper, IndustryOR Benchmark, an interactive demo, and a sample of OR-Instruct data, but these URLs are not directly provided in the text. A Hugging Face (HF) link is provided for the ORLM-LLaMA-3-8B model checkpoint.

Highlighted Details

• ORLM-LLaMA-3-8B achieves State-of-the-Art (SOTA) performance on the NL4OPT, MAMO, and IndustryOR benchmarks.
• The IndustryOR benchmark comprises 100 real-world optimization problems, serving as the first industrial-grade evaluation suite.
• OR-Instruct provides a semi-automated method for generating high-quality synthetic data specifically for training optimization-focused LLMs.
• An interactive demo of the ORLM-LLaMA-3-8B model is available.

Maintenance & Community

The project is actively maintained, with continuous updates to the IndustryOR Benchmark. Feedback is welcomed. The ORLM project has been accepted by the Operations Research journal.

Licensing & Compatibility

• License type: The ORLM-LLaMA-3-8B model checkpoint follows the Llama 3 license. The repository's code license is not explicitly stated.
• Compatibility notes: Commercial use may be subject to Llama 3 license terms. The reliance on the COPT solver for execution may introduce additional licensing considerations or costs for extensive deployment.

Limitations & Caveats

Program execution is currently limited to the COPT solver. For complex problems, particularly those in the IndustryOR benchmark, obtaining COPT web licenses might be necessary, potentially incurring costs or access restrictions. Execution results can vary based on computing resources.