LemonGrenade is a data-driven automation platform designed for complex, dynamic dataflow orchestration. It targets users who need to link disparate systems, data sources, or capabilities without manual configuration of dependencies, enabling iterative correlation and recursive processing. The platform's core benefit is its ability to build processing graphs on-demand, driven by data and user intent, rather than relying on pre-defined workflows.

How It Works

LemonGrenade utilizes a unique approach where data, rather than a static workflow definition, drives the execution graph. It connects adapters (which perform specific tasks) through controllers. Each job generates a dynamic handling graph, tracking processing stages and new data insertions. The underlying graph is managed by LemonGraph, a custom library built on Symas LMDB for transactional key-value storage. Job state is persisted in MongoDB and accessed via a REST API. This on-demand graph construction avoids the overhead of precomputation and allows for greater flexibility in handling evolving data relationships.

Quick Start & Requirements

• Installation: Linux (CentOS 7, Ubuntu 17.10 tested). Requires manual installation of dependencies including RabbitMQ, ZooKeeper, Storm, and MongoDB.
• Prerequisites: Java 8, Maven, Python 2.7 with msgpack and cffi, git, make, gcc/g++, zlib, libffi, supervisor.
• Setup: Extensive manual setup involving downloading and configuring multiple distributed systems (ZooKeeper, Storm, MongoDB, RabbitMQ) and then launching them via supervisor. Configuration of Storm topology ports and memory is critical.
• Links: GitHub Repo

Highlighted Details

• Leverages Symas LMDB for its custom graph library (LemonGraph).
• Integrates with Apache Storm for distributed processing.
• Job state managed via MongoDB and exposed through a REST API.
• Provides a UI for monitoring topologies and adapter metrics.

Maintenance & Community

• Developed by the National Security Agency.
• The project appears to be a technical preview with no explicit mention of active community engagement or ongoing maintenance beyond the initial release.

Licensing & Compatibility

• Apache 2.0 License.
• Permissive license suitable for commercial use and integration with closed-source systems.

Limitations & Caveats

This project is explicitly labeled as a "Technical Preview" and should not be considered stable in terms of functionality or APIs. The installation and setup process is highly complex, manual, and requires significant expertise in distributed systems management.