AutoMQ offers a stateless, diskless Kafka alternative built on S3-compatible storage, targeting users seeking significant cost reductions and improved scalability over traditional Kafka deployments. It aims to provide a 10x more cost-effective, auto-scaling, low-latency, and highly available streaming data platform.

How It Works

AutoMQ forks Apache Kafka and replaces its shared-nothing architecture with a shared-storage model leveraging object storage (like S3). Brokers are stateless, with data persisted in S3 via a custom S3 Storage Adapter and S3Stream library. This design eliminates cross-AZ traffic costs, enables rapid scaling by adding/removing stateless brokers, and utilizes an Auto Balancer for automatic partition and traffic management, removing the need for manual rebalancing.

Quick Start & Requirements

• Install/Run: docker compose -f docker/docker-compose.yaml up -d for a single-node setup.
• Prerequisites: Docker.
• Resources: Minimal for single-node evaluation. Cluster and production deployments require more.
• Links: Documentation, AWS Marketplace Trial, Alibaba Cloud Marketplace Trial.

Highlighted Details

• 10x cost-effective compared to Kafka, eliminating cross-AZ traffic costs.
• Stateless brokers scale in seconds, offering serverless auto-scaling.
• 100% Kafka compatibility.
• New "Table Topic" feature integrates stream and table functionalities with Apache Iceberg and S3 tables.

Maintenance & Community

• Active community via Slack and GitHub Issues.
• Contributions from major companies like Grab, JD.com, Palmpay, Geely Auto, and XPENG Motors.
• Slack

Licensing & Compatibility

• Apache 2.0 License.
• Compatible with commercial and closed-source applications.

Limitations & Caveats

• The provided Docker Quick Start is for evaluation only, not production.
• Kubernetes deployment is currently Enterprise-only, with open-source availability planned.
• Single-digit millisecond latency is available in the enterprise version; open-source latency is in the hundreds of milliseconds.