PiFlow is a research-oriented prototype for big data processing and AI agent orchestration. It targets engineers and researchers needing a flexible platform for complex data pipelines, offering features like streaming agent chat, conversation management, and integrated skill execution. The system aims to streamline AI-driven data workflows by combining a robust backend with an interactive frontend.

How It Works

The architecture leverages DeepAgents for intelligent agent orchestration, FastAPI for a high-performance backend API, and a React frontend for user interaction. It supports Spark for distributed big data processing. Core functionalities include real-time streaming agent chat, persistent conversation history management, and a comprehensive workspace for file uploads and downloads. The system's novelty lies in its integration of AI agents with a data flow engine, enabling dynamic skill execution and complex pipeline construction.

Quick Start & Requirements

• Prerequisites: Python 3.10+ (recommended), Node.js 18+ (frontend), PostgreSQL, LLM API keys (e.g., DASHSCOPE_API_KEY, OPENAI_API_KEY), and optionally MinerU API key and Pandoc.
• Installation: Install the bundled PiFlow Python engine (pip install ./third_party/piflow/piflow_engine-0.1.1-py3-none-any.whl), backend dependencies (pip install -r requirements.txt), and frontend dependencies (cd vue-web && npm install).
• Docker Deployment: Requires Docker and Docker Compose. Set API keys and run docker compose -f docker/docker-compose.yml up -d --build.
• Local Run: Backend: python -m uvicorn server:app --host 0.0.0.0 --port 8080 --reload. Frontend: cd vue-web && npm run dev.
• Links: docs/workspace_file_api.md (note: contains historical content).

Highlighted Details

• Streaming agent chat via /chat/stream.
• Full conversation history management (create, switch, delete, load threads).
• Workspace file handling: upload inputs, download outputs, message-level attachments.
• Pre-configured example pipelines for one-click task initiation.
• Skills center for discovering and managing agent capabilities.

Maintenance & Community

No specific details regarding maintainers, community channels (e.g., Discord, Slack), or roadmap are provided in the README.

Licensing & Compatibility

The repository README does not specify a software license. This lack of clarity may impact commercial use or integration with other projects.

Limitations & Caveats

The project is explicitly described as a "research data processing prototype." Documentation, particularly docs/workspace_file_api.md, may contain outdated information, with the codebase serving as the definitive source of truth. The absence of a stated license is a significant caveat.