This library provides a streaming syntax highlighter for Shiki, designed to efficiently highlight text streams, particularly useful for real-time outputs from Large Language Models (LLMs). It enables developers to integrate syntax highlighting into dynamic content without blocking the main thread.

How It Works

The core of the library is the CodeToTokenTransformStream, a Web Stream API transform stream. It takes raw text input and pipes it through Shiki's highlighter, converting code into an array of ThemedToken objects. This approach allows for incremental processing of text data as it arrives, making it ideal for asynchronous operations. The allowRecalls option enables more granular token output by handling context-dependent highlighting changes via special "recall" tokens.

Quick Start & Requirements

• Install via npm: npm install shiki-stream
• Requires Node.js or a modern browser environment supporting Web Streams.
• Initialize Shiki highlighter with desired languages and themes.
• Pipe your text stream through CodeToTokenTransformStream.

Highlighted Details

• Streaming tokenization for efficient LLM output highlighting.
• Supports allowRecalls for fine-grained, context-aware highlighting.
• Provides renderers for Vue and React for easy integration.

Maintenance & Community

• Developed by Anthony Fu.
• MIT License.

Licensing & Compatibility

• MIT License.
• Compatible with commercial and closed-source applications.

Limitations & Caveats

The allowRecalls feature requires custom handling by the stream consumer to correctly manage context-dependent highlighting updates.