This project addresses the need for a high-performance data engine capable of handling diverse AI and multimodal workloads at scale. It targets AI engineers and researchers seeking to process images, audio, video, and structured data efficiently within a unified framework, offering built-in AI operations and seamless scalability.

How It Works

Daft is a Python-native data engine, leveraging Rust for high-performance execution. Its core design emphasizes native multimodal data processing, allowing images, audio, video, and embeddings to be handled alongside structured data. It integrates built-in AI operations, such as LLM prompting and embedding generation, and supports distributed execution across Ray, Kubernetes, or Daft Cloud.

Quick Start & Requirements

• Primary install: pip install daft
• Prerequisites: Python 3.10 or higher.
• Links:
  • Installation Guide: https://docs.daft.ai/en/stable/install/
  • Quickstart: https://docs.daft.ai/en/latest/quickstart/

Highlighted Details

• Native multimodal processing for images, audio, video, embeddings, and structured data in a single framework.
• Built-in AI operations (LLM prompts, embeddings, classification) supporting OpenAI, Transformers, or custom models.
• Python-native interface with a Rust backend for optimized performance.
• Seamless scaling capabilities for distributed clusters (Ray, Kubernetes, Daft Cloud).
• Universal data connectivity, supporting sources like S3, GCS, Iceberg, Delta Lake, Hugging Face, and Unity Catalog.

Maintenance & Community

• Community support is available via Slack https://join.slack.com/t/dist-data/shared_invite/zt-2e77olvxw-uyZcPPV1SRchhi8ah6ZCtg and GitHub Discussions https://github.com/Eventual-Inc/Daft/discussions.
• Contribution guidelines are detailed in CONTRIBUTING.md https://github.com/Eventual-Inc/Daft/blob/main/CONTRIBUTING.md.
• Telemetry collection is enabled by default via Scarf but can be disabled by setting the DO_NOT_TRACK=true environment variable.

Licensing & Compatibility

• License: Apache 2.0.
• Compatibility: The Apache 2.0 license permits commercial use and integration into closed-source projects.

Limitations & Caveats

The project actively collects telemetry data for improvement, which can be opted out of. The quickstart documentation link currently points to /latest/, suggesting potential instability or ongoing development in that specific area.