This repository houses the core recommendation system powering X's "For You" feed, designed to personalize content discovery. It targets engineers and researchers by providing a sophisticated, ML-driven approach that combines content from followed accounts with ML-discovered posts. The system's primary benefit lies in its elimination of manual feature engineering, relying on a powerful Grok-based transformer model for nuanced content ranking based on predicted user engagement.

How It Works

The system orchestrates content retrieval and ranking through a multi-stage pipeline. It sources posts from two primary candidates: "Thunder" for in-network content (from followed accounts) and "Phoenix Retrieval" for out-of-network content discovered via ML similarity search across a global corpus. These candidates are enriched, filtered, and then scored by a Grok-based transformer model that predicts probabilities for various engagement actions (likes, replies, etc.). A weighted combination of these predictions, alongside diversity considerations, determines the final ranking. A key design decision is the complete removal of hand-engineered features, with the transformer learning relevance directly from user engagement history.

Quick Start & Requirements

The provided README focuses on the system's architecture and components rather than direct installation or quick-start instructions. Specific setup commands, non-default prerequisites (like GPU, CUDA, or Python versions), estimated setup times, or resource footprints are not detailed. Links to official quick-start guides, documentation, or demos are also absent.

Highlighted Details

• Grok-Based Transformer: Leverages a transformer model adapted from Grok-1 to predict multiple user engagement probabilities, forming the core of the ranking system.
• No Hand-Engineered Features: Relies entirely on the ML model to learn content relevance from user engagement sequences, simplifying data pipelines and infrastructure.
• Candidate Isolation: During ranking inference, candidate posts cannot attend to each other, ensuring consistent and cacheable scores independent of the batch composition.
• Composable Pipeline: Employs a reusable candidate-pipeline framework for flexible recommendation system construction, supporting parallel execution and easy extension of sources, hydrators, filters, and scorers.

Maintenance & Community

The provided README does not contain information regarding notable contributors, sponsorships, partnerships, community channels (like Discord/Slack), or roadmaps.

Licensing & Compatibility

• License: Apache License 2.0.
• Compatibility: The Apache 2.0 license is generally permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

The README details the system's architecture and design principles but does not explicitly list limitations, known bugs, or unsupported platforms. The absence of quick-start instructions or setup details suggests that deploying or experimenting with this system may require significant engineering effort and domain expertise.