tgmeng-api  by tgmeng-com

Real-time web hot list aggregator with AI integration

Created 1 year ago
329 stars

Top 84.1% on SourcePulse

GitHubView on GitHub
Project Summary

Summary

Tgmeng API provides a backend service for aggregating real-time trending topics ("hot lists") from numerous sources across the web. It targets users and developers needing up-to-date information streams, offering features like minute-level updates, RSS subscriptions, historical data tracking, and multi-platform push notifications. The project aims to simplify access to trending information and facilitate its integration into other applications or workflows, with a focus on ease of deployment.

How It Works

The backend service fetches data from over a hundred official sources, updating most minute-by-minute, with specific sources like GitHub and NetEase Cloud Music having slightly longer intervals. It employs a robust caching mechanism using Caffeine for in-memory storage, with configurable expiration times and automatic refresh timers. The system also supports proxy pools for accessing third-party services and includes comprehensive exception handling, request header simulation, and global logging. Historical data is stored locally in Parquet format, queried via DuckDB, requiring careful disk space management.

Quick Start & Requirements

  • Installation: One-click deployment via GitHub Actions (requires server configuration), Docker (pre-built images available on DockerHub/GHCR), or Docker Compose.
  • Prerequisites: For AI features, an AI_PLATFORM_CONFIG environment variable with specific JSON format is required. Docker deployment requires volume mounts for logs (/app/logs), data (/app/data), and heap dumps (/app/heapdumps). Historical data storage can consume significant disk space (~600MB per day).
  • Links: Project homepage: https://trend.tgmeng.com.

Highlighted Details

  • Supports 100+ data sources with minute-level updates for most.
  • Real-time push notifications to platforms like Telegram, DingTalk, Feishu, and more.
  • Historical data tracking with local storage (Parquet/DuckDB) for up to 10 days.
  • Flexible proxy support and configurable in-memory caching strategies.
  • Automated deployment pipelines via GitHub Actions to servers or container registries.

Maintenance & Community

The project is actively developed by tgmeng-com. Specific details regarding community channels (like Discord/Slack), notable contributors, or sponsorships are not detailed in the provided README.

Licensing & Compatibility

  • License: GNU GENERAL PUBLIC LICENSE Version 3 (GPL v3).
  • Compatibility: GPL v3 is a strong copyleft license. Derived works must also be licensed under GPL v3. Commercial use or linking with closed-source software may be restricted due to the copyleft nature of the license.

Limitations & Caveats

Historical data storage requires substantial disk space and monitoring. The AI platform configuration is optional and requires specific formatting. The GPL v3 license may impose restrictions on integration into proprietary software.

Health Check
Last Commit

9 months ago

Responsiveness

Inactive

Pull Requests (30d)
0
Issues (30d)
0
Star History
0 stars in the last 30 days

Explore Similar Projects

Feedback? Help us improve.