Theseus is a PyTorch library for building custom, end-to-end differentiable nonlinear optimization layers, targeting robotics and computer vision researchers and engineers. It enables integrating domain-specific priors into neural networks by allowing optimization problems to be solved within the forward pass and gradients to be computed through the optimizer, facilitating end-to-end training.

How It Works

Theseus provides an application-agnostic interface for defining optimization problems as Objectives, composed of CostFunctions. It supports various second-order (Gauss-Newton, Levenberg-Marquardt) and other nonlinear optimizers, coupled with efficient linear solvers (dense and sparse, including GPU-accelerated options like CHOLMOD and BaSpaCho). The library leverages Lie groups (via torchlie) for efficient representation of rotations and poses, crucial for many robotics and vision tasks. Gradients can be computed using implicit, truncated, or direct loss minimization (DLM) backward modes for efficiency.

Quick Start & Requirements

• Install via pip: pip install theseus-ai
• Requires PyTorch.
• GPU support necessitates matching CUDA versions for PyTorch and Theseus custom CUDA operations (wheels provided for CUDA 11.6/Python 3.10).
• Requires suitesparse (install via apt or conda).
• For BaSpaCho sparse solver, compile from source and set BASPACHO_ROOT_DIR.
• Tutorials and examples are available.

Highlighted Details

• Supports multiple nonlinear optimization algorithms (Gauss-Newton, Levenberg-Marquardt, CEM).
• Offers efficient linear solvers including sparse options with GPU acceleration.
• Integrates Lie group operations for pose and rotation representation.
• Provides various backward modes (Implicit, Truncated, DLM, LEO) for gradient computation.

Maintenance & Community

Developed by Meta AI (facebookresearch). Community discussions and issue tracking are managed via GitHub.

Licensing & Compatibility

MIT licensed, permitting commercial use and integration into closed-source projects.

Limitations & Caveats

The PyPI installation does not include experimental "Theseus Labs" features; these require installation from source. Compilation from source may be necessary for specific CUDA or dependency versions not covered by pre-built wheels.