This library provides efficient computation of various similarity measures between vectors, targeting data scientists, machine learning engineers, and researchers. It offers optimized Rust implementations with Python bindings for seamless integration, enabling faster and more robust analysis of vector relationships, particularly with high-dimensional data from LLMs.

How It Works

The library leverages Rust's performance capabilities, utilizing the rayon crate for parallel processing and ndarray for vectorized operations. It implements several similarity measures including Spearman's Rho, Kendall's Tau (optimized with merge sort for inversion counting), Approximate Distance Correlation, Jensen-Shannon Dependency Measure, Hoeffding's D, and Normalized Mutual Information. A key feature is its bootstrapping functionality for robust estimation and confidence intervals.

Quick Start & Requirements

• Python: pip install fast_vector_similarity
• Rust: Add to Cargo.toml.
• Dependencies: Python 3.x, Rust. No specific hardware requirements mentioned beyond standard CPU.
• Docs: https://github.com/Dicklesworthstone/fast_vector_similarity

Highlighted Details

• Implements 6 distinct similarity measures: Spearman's Rho, Kendall's Tau, Approximate Distance Correlation, Jensen-Shannon Dependency Measure, Hoeffding's D, and Normalized Mutual Information.
• Features parallel processing via rayon and vectorized operations via ndarray for performance.
• Includes bootstrapping for robust estimation and confidence intervals.
• Compatible with high-dimensional vectors (e.g., 4096-dim) from LLMs like Llama2.

Maintenance & Community

• The project is maintained by Dicklesworthstone. No other contributors, sponsorships, or community links (Discord/Slack) are mentioned in the README.

Licensing & Compatibility

• The README does not explicitly state a license. This is a critical omission for adoption.

Limitations & Caveats

The README does not specify a license, which is a significant blocker for determining commercial use or integration compatibility. Some measures like Jensen-Shannon Dependency Measure have been "revised for improved utility," implying potential breaking changes or shifts in interpretation.