MLServer is an open-source inference server designed for deploying machine learning models via REST and gRPC interfaces. It targets ML engineers and data scientists seeking a standardized, scalable, and framework-agnostic solution for model deployment, offering compatibility with the KFServing V2 Dataplane spec.

How It Works

MLServer utilizes an "inference runtime" system, acting as a bridge between the server and various ML frameworks. This modular design allows for easy integration of new frameworks and custom model serving logic. It supports multi-model serving within a single process, parallel inference across worker pools, and adaptive batching for request optimization, all while adhering to the V2 Inference Protocol for broad compatibility.

Quick Start & Requirements

• Install: pip install mlserver
• Optional Runtimes: pip install mlserver-<framework> (e.g., mlserver-sklearn)
• Supported Python: 3.9, 3.10, 3.11, 3.12
• Examples: https://github.com/SeldonIO/MLServer/tree/master/examples

Highlighted Details

• Supports Scikit-Learn, XGBoost, Spark MLlib, LightGBM, CatBoost, Tempo, MLflow, Alibi-Detect, Alibi-Explain, and HuggingFace.
• Compliant with KFServing V2 Dataplane spec for REST and gRPC.
• Features multi-model serving, adaptive batching, and parallel inference workers.
• Integrates with Kubernetes-native frameworks like Seldon Core and KServe.

Maintenance & Community

• Developed by SeldonIO.
• Versioning managed via ./hack/update-version.sh.
• Testing commands: make test, tox -e py3 -- <test_file>.

Licensing & Compatibility

• Licensed under Apache License 2.0.
• Note: Associated libraries (e.g., Alibi Detect/Explain) may have different licenses (e.g., Business Source License 1.1).

Limitations & Caveats

Python versions 3.7, 3.13, and earlier are unsupported. The README notes that associated software may have different licensing terms, requiring careful review for commercial use.