Summary

Addresses efficient PII/NER entity span detection using a minimal C++/GGML runtime. Targets developers needing high-performance, low-memory footprint solutions for sensitive data identification. Offers significant speedups and memory savings over standard Transformer models.

How It Works

Implements OpenAI's privacy filter NER architecture within a custom C++ GGML runtime. Employs a banded/near-linear attention mechanism, contrasting with the quadratic complexity of standard Transformers. This design enables sustained throughput on long documents and drastically reduces memory requirements. Supports multiple model variants (openai-privacy-filter, privacy-filter-multilingual, privacy-filter-nemotron) and offers pre-converted GGUF files.

Quick Start & Requirements

• Primary install/run command: Clone recursively (git clone --recursive <repo>), then build using CMake presets: cmake --preset release && cmake --build --preset release -j. Runtime execution via ./build/release/pf-cli.
• Non-default prerequisites: CMake, C++ compiler. Optional GPU acceleration requires Vulkan (headers/loader, glslc) or CUDA (toolkit). Model conversion scripts require Python 3.x and pip install -r scripts/requirements.txt (torch, safetensors, gguf).
• Pre-converted GGUF models are available for direct use.

Highlighted Details

• Achieves 7.7x faster CPU and superior GPU performance (handling 131k tokens vs. HF's ~16k OOM) for PII/NER tasks.
• Demonstrates real-time, on-device PII classification on a Raspberry Pi 5 (360 tok/s).
• Banded attention ensures near-linear scaling and flat throughput with increasing sequence length, unlike O(n²) Transformer attention.
• Minimal VRAM usage (~2.8 GiB on a 16 GiB GPU).
• Optimized CPU runtime dispatches optimal ISA (e.g., AVX-512).

Maintenance & Community

Continuous integration (CI) is present for model conversion parity checks. No explicit community channels (Discord/Slack) or core contributor information is detailed in the provided text.

Licensing & Compatibility

The specific open-source license is not mentioned in the provided README excerpt. Compatibility for commercial use or closed-source linking is therefore undetermined.

Limitations & Caveats

The project focuses exclusively on PII/NER classification, not generative tasks. GPU backend setup requires specific system configurations (Vulkan or CUDA). The license status requires clarification for adoption decisions.