Fenic is a DataFrame framework designed to streamline the development of AI and agentic applications, particularly those involving large language models (LLMs). It offers a PySpark-inspired API with specialized "semantic operators" for natural language processing tasks, enabling users to process and derive insights from both structured and unstructured data. The framework aims to bring the reliability and scalability of traditional data pipelines to AI workloads, making it suitable for engineers and data practitioners alike.

How It Works

Fenic's core innovation lies in its DataFrame engine, purpose-built for LLM inference. Unlike traditional data tools retrofitted for AI, Fenic's engine is designed from the ground up to handle inference efficiently. It features automatic batch optimization for API calls, built-in retry logic, rate limiting, and cost tracking. Key to its approach are "semantic operators" that integrate LLM capabilities directly into DataFrame operations, such as sentiment analysis, text extraction, classification, and semantic joins, allowing for natural language transformations and filtering.

Quick Start & Requirements

• Installation: pip install fenic
• Prerequisites: Requires an API key from at least one LLM provider (OpenAI, Anthropic, Google, Cohere). Set the corresponding environment variable (e.g., OPENAI_API_KEY).
• Examples: The repository includes several examples demonstrating various functionalities, accessible via Colab links.

Highlighted Details

• Purpose-built query engine for LLM inference with automatic batch optimization.
• First-class support for semantic operators (e.g., semantic.analyze_sentiment, semantic.extract, semantic.join).
• Native support for unstructured data types like Markdown, transcripts (SRT, WebVTT), and JSON with specialized processing.
• Multi-provider support (OpenAI, Anthropic, Gemini) and familiar DataFrame API (PySpark/SQL compatible).

Maintenance & Community

• Community support is available via Discord.
• Contributions are welcomed, with guidelines provided for developers.

Licensing & Compatibility

• The README does not explicitly state the license type or compatibility for commercial use.

Limitations & Caveats

• The project requires API keys for LLM providers, which may incur costs.
• The README does not specify the license, which could be a blocker for commercial adoption.