This library provides highly optimized tokenizers for natural language processing research and production, addressing the need for fast and versatile text processing. It's designed for developers and researchers working with large text datasets who require efficient pre-processing pipelines.

How It Works

The core of the library is implemented in Rust, ensuring exceptional performance for both training new vocabularies and tokenizing text. It supports popular algorithms like Byte-Pair Encoding (BPE), WordPiece, and Unigram. A key advantage is its ability to track alignments between original text and tokens after normalization, enabling precise mapping back to the source.

Quick Start & Requirements

• Install via pip: pip install tokenizers or pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
• Python 3.x
• Official documentation: https://huggingface.co/docs/tokenizers/index
• Quick tour: https://huggingface.co/docs/tokenizers/quicktour

Highlighted Details

• Implemented in Rust for high performance, capable of tokenizing 1GB of text in under 20 seconds on a server CPU.
• Supports BPE, WordPiece, and Unigram models.
• Offers comprehensive pre-processing: truncation, padding, and special token insertion.
• Provides normalization with alignment tracking for mapping tokens back to original text.
• Bindings available for Rust, Python, and Node.js, with a community-contributed Ruby binding.

Maintenance & Community

• Developed and maintained by Hugging Face.
• Active community support and development.

Licensing & Compatibility

• Apache 2.0 License.
• Permissive license suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

The provided performance benchmarks are specific to a particular AWS instance and may vary across different hardware configurations.