This library provides robust, efficient, and adaptable text segmentation into sentences or other semantic units, targeting NLP researchers and developers. It offers state-of-the-art performance across 85 languages with its SaT models, significantly outperforming previous methods and existing tools.

How It Works

The core is the SaT (Segment Any Text) model, a universal approach to sentence segmentation. It leverages a transformer architecture, offering improved performance and reduced computational cost compared to the previous WtP (Where's the Point?) model. SaT models can be further adapted to specific domains or languages using LoRA, enabling highly customized segmentation.

Quick Start & Requirements

• Install: pip install wtpsplit or pip install wtpsplit[onnx-gpu] / pip install wtpsplit[onnx-cpu] for ONNX support.
• Requirements: Python, PyTorch. GPU with CUDA is recommended for performance.
• Docs: https://github.com/segment-any-text/wtpsplit

Highlighted Details

• State-of-the-art performance across 85 languages, outperforming SpaCy, PySBD, and others.
• ONNX support for up to ~50% faster inference on GPUs.
• Domain and style adaptation via LoRA for 81 languages, including legal documents, code-switching, and tweets.
• Supports paragraph segmentation by leveraging newline prediction capabilities.

Maintenance & Community

• Active development with a focus on the SaT model.
• Contact: markus.frohmann@gmail.com for questions.

Licensing & Compatibility

• Apache 2.0 License.
• Compatible with commercial use and closed-source linking.

Limitations & Caveats

The library includes previous WtP models for reproducibility, but SaT is the recommended and actively developed model. Some advanced features like LoRA export for ONNX are experimental.