This repository serves as a personal knowledge base for Computer Vision (CV) algorithm engineers, documenting their growth path and sharing notes on CV and model compression/deployment technologies. It also promotes a paid course on building a custom large model inference framework.

How It Works

The project's core is a collection of notes and resources for CV engineers, covering topics from foundational programming and machine learning to advanced areas like model compression, high-performance computing, and deployment. A significant portion highlights a custom inference framework built with Triton and PyTorch, designed for ease of use and GPU acceleration via Triton kernels, aiming to simplify CUDA programming.

Quick Start & Requirements

The repository itself is primarily a collection of notes and documentation. The associated course project, however, requires Python and PyTorch. Specific hardware requirements for running the inference framework are not detailed in the README, but performance claims suggest GPU acceleration is essential.

Highlighted Details

• A custom large model inference framework is developed using Triton and PyTorch, offering a PyTorch-like syntax for Triton kernels.
• The framework claims up to 4x speedup on Llama3 1B and 3B models compared to the transformers library.
• It supports advanced features like FlashAttention (V1-V3), GQA, and PageAttention, with custom fused kernels.
• The repository includes a comprehensive learning roadmap and a list of companies hiring CV algorithm engineers.

Maintenance & Community

The project is actively maintained by the author, harleyszhang, with content being continuously updated. The author also promotes a paid course and a WeChat public account ("嵌入式视觉") for further engagement.

Licensing & Compatibility

The repository's licensing is not explicitly stated in the README. The content is presented as personal notes and educational material.

Limitations & Caveats

The README explicitly states that "This project is gradually being deprecated, and most of the content will no longer be updated." Users are directed to other repositories (dl_note, lite_llama) for updated information on deep learning and inference frameworks.