This repository provides HLLM, a framework for enhancing sequential recommendation systems by leveraging Hierarchical Large Language Models for item and user modeling. It targets researchers and practitioners in recommendation systems seeking to improve performance by incorporating rich textual item descriptions and user interaction histories. The primary benefit is improved recommendation accuracy through sophisticated language understanding.

How It Works

HLLM employs a hierarchical approach, processing item text (title, description) and user interaction sequences using separate LLMs. It then fuses these representations to model user preferences and predict the next item. This method allows for capturing nuanced item semantics and long-range user behavior patterns, outperforming traditional ID-based methods by effectively utilizing rich textual metadata.

Quick Start & Requirements

• Install: pip3 install -r requirements.txt
• Prerequisites: PyTorch 2.1.0, DeepSpeed 0.14.2, Transformers 4.41.1, Lightning 2.4.0, Flash-attn 2.5.9post1. Optional: fbgemm-gpu for HSTU, sentencepiece for Baichuan2.
• Data: Requires PixelRec and Amazon Book Reviews datasets, with specific folder structures for interactions and item information.
• Models: Needs pre-trained LLMs like TinyLlama or Baichuan2.
• Setup: Data preparation and LLM download are required. Training commands are provided for both HLLM and ID-based models.
• Docs: arXiv Paper, Hugging Face Weights

Highlighted Details

• Achieves state-of-the-art results on Pixel8M and Amazon Book Reviews datasets, outperforming HSTU and SASRec.
• Supports both 1B and 7B parameter LLM variants (e.g., HLLM-1B, HLLM-7B).
• Offers DeepSpeed integration for efficient distributed training and memory saving (e.g., gradient_checkpointing, stage 3).
• Provides pre-trained weights for evaluation on specific datasets.

Maintenance & Community

• Released by ByteDance.
• Codebase includes modifications from RecBole, VisRec, PixelRec, and HSTU.
• Citation details provided for academic use.

Licensing & Compatibility

• HLLM is released under the Apache License 2.0.
• Modified code from PixelRec is under MIT License.
• Users must comply with the licenses of underlying LLMs (TinyLlama, Baichuan2).

Limitations & Caveats

The framework requires significant computational resources for training and inference due to the use of large language models. Specific dataset formatting and pre-trained model acquisition are necessary prerequisites.