clewdr  by Xerxes-2

High-performance LLM reverse proxy

Created 1 year ago
1,157 stars

Top 33.0% on SourcePulse

GitHubView on GitHub
1 Expert Loves This Project
Project Summary

ClewdR is a high-performance LLM reverse proxy designed for Claude and Gemini, offering a full-featured React frontend and an efficient, event-driven architecture. It targets users needing to manage and route requests to these large language models, providing significant performance gains and resource efficiency over script-based solutions.

How It Works

ClewdR leverages Rust's Tokio and Axum frameworks for multi-threaded asynchronous processing, enabling it to handle thousands of requests per second with minimal memory footprint (single-digit MB). Its design emphasizes efficiency through an event-driven approach, decoupled logic, and hot reloading capabilities. It features fingerprint-level Chrome simulation for requests, intelligent cookie management with automatic account status classification, and a fine-grained polling mechanism for optimal resource utilization. Response caching is supported via Moka technology.

Quick Start & Requirements

  • Download the pre-compiled binary for your platform.
  • Access the frontend at http://127.0.0.1:8484 and use the console-displayed Web Admin Password for login.
  • Configure proxy addresses, cookies, and keys via the frontend.
  • For third-party applications (e.g., SillyTavern), use the console-provided API access addresses and API Password.
  • Official Wiki: https://github.com/Xerxes-2/clewdr/wiki

Highlighted Details

  • Claims ten times the performance and one-tenth the resource usage of script language implementations.
  • Supports OpenAI compatible modes for both Claude and Gemini.
  • Includes built-in proxy server support, eliminating the need for TUN.
  • Implements stop sequences and image attachment uploads on the proxy side.

Maintenance & Community

  • Primary development by Xerxes-2.
  • Community resources available via GitHub Wiki.

Licensing & Compatibility

  • License details are not explicitly stated in the README.
  • Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README does not specify the license, which may impact commercial adoption. Compatibility details for closed-source linking are also absent.

Health Check
Last Commit

2 days ago

Responsiveness

1 day

Pull Requests (30d)
3
Issues (30d)
2
Star History
54 stars in the last 30 days

Explore Similar Projects

Starred by Yaowei Zheng Yaowei Zheng(Author of LLaMA-Factory), Chip Huyen Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"), and
11 more.

GPTCache by zilliztech

0.2%
8k
Semantic cache for LLM queries, integrated with LangChain and LlamaIndex
Created 3 years ago
Updated 10 months ago
Starred by Jason Knight Jason Knight(Director AI Compilers at NVIDIA; Cofounder of OctoML), Omar Sanseviero Omar Sanseviero(DevRel at Google DeepMind), and
12 more.

mistral.rs by EricLBuehler

0.3%
7k
LLM inference engine for blazing fast performance
Created 2 years ago
Updated 3 weeks ago
Feedback? Help us improve.