Bistro is a distributed task scheduler and computation framework designed for engineers to build and manage complex distributed systems. It offers high performance, flexibility, and ease of configuration, supporting various paradigms like data-parallel jobs and enforcing resource constraints.

How It Works

Bistro operates as a scheduler that tracks jobs and data shards, initiating tasks only when resources are available. It supports remote worker pools and can aggregate multiple schedulers for a share-nothing architecture. Its extensibility allows for custom plugins to tailor functionality to specific needs.

Quick Start & Requirements

• Install: Build from source using provided CMake scripts.
• Prerequisites: 64-bit Linux, GCC 4.9+, Folly, FBThrift, Proxygen, Boost, libsqlite3. Requires 2-3GB RAM for building. Docker-based build scripts are available for Ubuntu 14.04/16.04 and Debian 8.6.
• Documentation: http://facebook.github.io/bistro

Highlighted Details

• Replaced Hadoop for an HBase compression task, improving time-to-completion and reliability by over 10x.
• Safely runs ETL tasks against live production databases (MySQL, HBase, Postgres).
• Supports resource-aware job queues for CPU/GPU compute jobs.
• Offers command-line and web UIs for management and monitoring.

Maintenance & Community

• The project originated at Facebook and has been in production for over 3 years.
• Community support is available via a Facebook group: https://www.facebook.com/groups/bistro.scheduler.

Licensing & Compatibility

• The license is available in the LICENSE file. (Specific license type not detailed in README).
• Compatibility for commercial use or closed-source linking is not explicitly stated.

Limitations & Caveats

The current public release is partial, including only server components. Building and dependency management may require careful attention to the CMake setup script for a full list of dependencies.