NeMo Curator is a Python toolkit for scalable, GPU-accelerated data preprocessing and curation for generative AI models, targeting researchers and engineers building foundation language models, text-to-image models, and performing domain-adaptive pretraining or fine-tuning. It significantly speeds up data preparation by leveraging GPUs via Dask and RAPIDS, enabling the creation of high-quality datasets for improved model convergence.

How It Works

The framework employs a modular, pipeline-based approach, allowing users to chain various text and image processing modules. It utilizes Dask for distributed computing and RAPIDS for GPU acceleration, enabling efficient handling of large datasets. Key features include GPU-accelerated deduplication (exact, fuzzy, semantic), language identification, text cleaning, and various classification filters (quality, safety, domain) powered by fastText or Hugging Face models. Modules can be reordered and scaled across multiple nodes for increased throughput.

Quick Start & Requirements

• Install: pip install --extra-index-url https://pypi.nvidia.com nemo-curator[all]
• Prerequisites: Python 3.10+, packaging >= 22.0, Ubuntu 22.04/20.04, NVIDIA GPU (Volta/compute capability 7.0+), CUDA 12+.
• Resources: Installation via PyPI, source, or NeMo Framework Container. Nightly builds for RAPIDS are available.
• Docs: https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/data_curation/intro.html
• Examples: https://github.com/NVIDIA/NeMo-Curator/tree/main/examples

Highlighted Details

• GPU-accelerated fuzzy deduplication on 1.96T tokens completed in 0.5 hours with 32x H100 GPUs.
• Offers multilingual text curation capabilities.
• Includes modules for PII redaction and downstream-task decontamination.
• Provides pre-trained classifiers for quality, safety, and domain classification.

Maintenance & Community

• Actively developed by NVIDIA.
• Community contributions are welcomed via CONTRIBUTING.md.
• Blog posts and tutorials are available for guidance.

Licensing & Compatibility

• Apache 2.0 License.
• Permissive license suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

• While optional, GPU acceleration is highly recommended for performance, with specific CUDA and GPU compute capability requirements. CPU-only installation is available but significantly slower for large-scale tasks.