NBCE  by bojone

Context extension technique for LLMs (research paper)

created 2 years ago
325 stars

Top 85.0% on sourcepulse

GitHubView on GitHub
Project Summary

NBCE (Naive Bayes-based Context Extension) is a plug-and-play library that enables any Large Language Model (LLM) to process arbitrarily long contexts without fine-tuning. It is designed for researchers and developers working with LLMs who need to overcome context length limitations for tasks involving extensive documents or conversations.

How It Works

NBCE leverages a formula inspired by Naive Bayes to extend context handling. This approach allows LLMs to effectively process and reason over much longer inputs than their native training limits, achieving linear efficiency with respect to context length. The method is model-agnostic, meaning it can be applied to various LLMs without requiring architectural changes or retraining.

Quick Start & Requirements

  • Installation: Typically via pip.
  • Prerequisites: Requires a compatible LLM (e.g., OpenBuddy 7B) and sufficient compute resources (e.g., 160GB VRAM for 50k context with a 7B model on 8 A800 GPUs).
  • Demo: Available for testing with provided context and questions.

Highlighted Details

  • Enables LLMs to handle contexts exceeding 50k tokens (e.g., 7B model with 160GB VRAM).
  • "Plug-and-play" functionality requires no model fine-tuning.
  • Achieves linear efficiency with respect to context length.
  • Demonstrated success in tasks like resume matching and essay scoring.

Maintenance & Community

  • The project is associated with authors from NAACL 2024.
  • Community support is available via QQ group: 808623966.

Licensing & Compatibility

  • The README does not explicitly state a license. Further clarification is needed for commercial use or integration into closed-source projects.

Limitations & Caveats

The project is described as having "decent results" and is suitable for experimentation, suggesting it may not yet offer state-of-the-art performance across all tasks. The exact computational requirements can be substantial, especially for very long contexts.

Health Check
Last commit

7 months ago

Responsiveness

1 day

Pull Requests (30d)
0
Issues (30d)
0
Star History
0 stars in the last 90 days

Explore Similar Projects

Starred by Chip Huyen Chip Huyen(Author of AI Engineering, Designing Machine Learning Systems), Jeff Hammerbacher Jeff Hammerbacher(Cofounder of Cloudera), and
1 more.

yarn by jquesnelle

1.0%
2k
Context window extension method for LLMs (research paper, models)
created 2 years ago
updated 1 year ago
Feedback? Help us improve.