MetaSpore is an end-to-end machine learning platform addressing the full ML lifecycle from data preprocessing to online A/B testing. It targets ML engineers and data scientists seeking a unified development experience, offering efficient offline training and high-performance online prediction.

How It Works

MetaSpore provides a unified framework for data preprocessing, distributed deep learning training (PyTorch compatible), and high-performance online serving. Key advantages include seamless PySpark integration for data access, unified feature extraction across offline/online environments, and an online application framework supporting dynamic parameter hot-loading and traffic splitting.

Quick Start & Requirements

• Installation: Offline training via pip install metaspore. Requires Python >= 3.8. PySpark and PyTorch must be installed separately (e.g., pip install pyspark, pip install torch==1.11.0+cpu). Prebuilt Docker images are available for serving (CPU/GPU).
• Prerequisites: Python 3.8+, PyTorch, PySpark.
• Docs/Demos: Links provided for tutorials, online serving, feature extraction, experiment pipelines, and a MovieLens recommender demo.

Highlighted Details

• Supports distributed large-scale sparse feature learning within a PyTorch-compatible training framework.
• Features a high-performance online prediction service with heterogeneous hardware acceleration.
• Enables unified feature extraction logic for both offline training and online inference.
• Online framework includes model prediction, experiment bucketing, traffic splitting, and dynamic parameter hot-loading.

Maintenance & Community

Developed by the DMetaSoul team. Community support is available via GitHub Discussions, GitHub Issues, and a Slack channel. Contributions are welcomed.

Licensing & Compatibility

Released under the Apache License 2.0, which is permissive for commercial use and integration with closed-source projects.

Limitations & Caveats

The specified PyTorch dependency (1.11.0) is notably outdated, potentially limiting compatibility with newer models and features. The online application framework mentions a Java implementation, suggesting a mixed-language environment for deployment.