This tool addresses the need to convert large language models (LLMs) into ONNX and MNN formats for efficient deployment. It targets developers and researchers working with LLMs who require cross-platform compatibility and optimized inference, offering a streamlined process for model conversion and testing.

How It Works

The project leverages Hugging Face model repositories and provides a command-line interface for conversion. It supports dynamic shape optimization, constant folding, and ONNX model optimization via OnnxSlim, claiming up to 5% performance improvement. The tool also facilitates the export of LoRA weights and includes integrated testing capabilities for conversational and multimodal models.

Quick Start & Requirements

• Install: pip install llmexport or pip install git+https://github.com/wangzhaode/llm-export@master
• Prerequisites: Python, Git. Specific LLM model weights are required for conversion.
• Usage: Download models via git clone from Hugging Face or ModelScope. Convert using llmexport --path <model_path> --export onnx or llmexport --path <model_path> --export mnn. Test with llmexport --path <model_path> --test "<query>".
• Docs: https://github.com/wangzhaode/llm-export

Highlighted Details

• Supports ONNX and MNN export formats.
• Includes ONNX model optimization with OnnxSlim.
• Facilitates LoRA weight merging and splitting.
• Supports AWQ and GPTQ quantized model weights.

Maintenance & Community

The repository is actively maintained by wangzhaode. Community engagement details such as Discord/Slack channels are not explicitly mentioned in the README.

Licensing & Compatibility

The README does not specify a license. This lack of explicit licensing information may pose compatibility issues for commercial or closed-source use.

Limitations & Caveats

The project does not explicitly state its license, which could impact commercial adoption. While it lists many supported models, compatibility with all LLM architectures or specific quantization methods is not guaranteed.