This SDK integrates Large Language Models (LLMs) and AI Agents into Apache Airflow pipelines, enabling users to orchestrate complex AI workflows using familiar Airflow decorators. It targets Airflow users and data engineers looking to leverage LLMs for tasks like text summarization, structured data extraction, and agent-based research directly within their scheduled data pipelines.

How It Works

The SDK extends Airflow's @task decorator with specialized versions: @task.llm for direct LLM calls, @task.agent for orchestrating AI agents with tools, and @task.llm_branch for LLM-driven DAG control flow. It leverages Pydantic AI for model support and automatic output parsing via type hints, including Pydantic models, simplifying the integration of LLM outputs into downstream tasks.

Quick Start & Requirements

• Install with pip install airflow-ai-sdk[<provider>] (e.g., airflow-ai-sdk[openai,duckduckgo]).
• Requires an Airflow environment.
• For local testing, clone the ai-sdk-examples repository and run astro dev start.
• See examples: https://github.com/astronomer/ai-sdk-examples

Highlighted Details

• Supports multiple LLM providers (OpenAI, Anthropic, Gemini, Ollama, Groq, Mistral, Cohere, Bedrock).
• Enables structured LLM outputs using Pydantic models.
• Facilitates agentic workflows with custom tools and web search capabilities.
• Allows LLM-driven conditional logic for DAG branching.

Maintenance & Community

• Developed by Astronomer.
• Active development and examples provided.

Licensing & Compatibility

• Apache License 2.0.
• Compatible with commercial and closed-source applications.

Limitations & Caveats

The SDK relies on external LLM providers, which may incur costs and have rate limits. The slim version requires manual installation of LLM provider dependencies.