pipes  by joboccara

C++ header-only library for expressive collection processing via pipelines

created 8 years ago
832 stars

Top 43.7% on sourcepulse

GitHubView on GitHub
Project Summary

This C++ library provides a header-only, C++14 implementation for creating expressive data processing pipelines. It targets developers working with collections, offering a "push-based" alternative to C++20 ranges, enabling complex data transformations and routing with a fluent, pipe-like syntax.

How It Works

The library uses a "push-based" model where data flows sequentially through a chain of components called "pipes." Each pipe receives data, performs an operation (e.g., filtering, transforming), and passes the result to the next pipe in the pipeline. This design allows for flexible data routing, including branching (fork), merging (mux), and conditional processing (switch), which are distinct from the "pull-based" nature of C++20 ranges.

Quick Start & Requirements

  • Install: Header-only, no installation required. Include <pipes/pipes.hpp>.
  • Requirements: C++14 compliant compiler.
  • Demo: Fluent C++ article and examples in the repository.

Highlighted Details

  • Advanced Routing: Supports complex data flow patterns like fork (broadcast to multiple pipes), mux (process multiple collections in parallel), cartesian_product, and unzip.
  • STL Integration: Pipes can be used as output iterators for standard algorithms (e.g., std::copy, std::set_difference).
  • Stream Processing: Includes pipes for reading from and writing to streams (read_in_stream, to_out_stream).
  • Custom Aggregation: map_aggregator and set_aggregator allow custom merging logic for elements with duplicate keys or values.

Maintenance & Community

  • Developed by joboccara.
  • Contributions are welcome; issues can be logged for enhancements or bugs.

Licensing & Compatibility

  • License: MIT License.
  • Compatibility: Permissive license allows for commercial use and integration with closed-source projects.

Limitations & Caveats

The library is under active development and subject to change. While it offers advanced routing, it lacks features like infinite ranges found in C++20 ranges.

Health Check
Last commit

9 months ago

Responsiveness

1+ week

Pull Requests (30d)
0
Issues (30d)
0
Star History
9 stars in the last 90 days

Explore Similar Projects

Starred by Jeff Hammerbacher Jeff Hammerbacher(Cofounder of Cloudera) and Chip Huyen Chip Huyen(Author of AI Engineering, Designing Machine Learning Systems).

bytewax by bytewax

0.2%
2k
Python framework for stateful stream processing
created 3 years ago
updated 4 months ago
Starred by Chip Huyen Chip Huyen(Author of AI Engineering, Designing Machine Learning Systems) and Jeff Hammerbacher Jeff Hammerbacher(Cofounder of Cloudera).

towhee by towhee-io

0.2%
3k
Framework for neural data processing pipelines
created 4 years ago
updated 9 months ago
Feedback? Help us improve.