Aqueduct is an open-source MLOps framework designed to simplify the deployment and management of machine learning and LLM workloads across diverse cloud infrastructures. It targets ML engineers and data scientists seeking a unified Python-native interface to abstract away infrastructure complexities, enabling seamless execution and monitoring of pipelines on platforms like Kubernetes, Spark, and AWS Lambda.

How It Works

Aqueduct utilizes a Python-native API, allowing users to define ML tasks and workflows using standard Python code without requiring specialized DSLs or YAML configurations. It abstracts underlying cloud infrastructure, enabling code to run seamlessly across different environments (e.g., Kubernetes, Spark, Lambda) and facilitating data movement between them. Workflows consist of Artifacts (data) transformed by Operators (compute), with execution managed on user-specified cloud infrastructure for enhanced security and data privacy.

Quick Start & Requirements

• Install: pip3 install aqueduct-ml
• Run: aqueduct start
• Prerequisites: Python 3.x, Unix environment.
• Documentation: Quickstart Guide, Examples

Highlighted Details

• Python-native API for defining ML workflows.
• Seamless integration with existing cloud infrastructure (Kubernetes, Spark, Lambda, etc.).
• Centralized visibility into code, data, and metadata for workflow runs.
• Runs securely within the user's cloud environment.

Maintenance & Community

• Project is no longer maintained.
• Community: Slack
• Roadmap: Roadmap

Licensing & Compatibility

• License: Apache 2.0
• Compatibility: Suitable for commercial use and integration with closed-source systems.

Limitations & Caveats

The project is explicitly stated as no longer being maintained, which may lead to a lack of future updates, bug fixes, or community support.