This repository serves as a comprehensive knowledge base and interview preparation guide for AI engineers specializing in Large Language Models (LLMs). It covers fundamental concepts, architectural details, training methodologies, inference techniques, and practical applications, aiming to equip individuals for LLM-focused roles.

How It Works

The project is structured as a curated collection of notes and explanations, drawing from various online resources and personal insights. It delves into core LLM components like Transformer architecture, attention mechanisms (MHA, MQA, GQA), and decoding strategies. Practical implementation details are provided through associated projects like tiny-llm-zh for building small LLMs, tiny-rag for RAG systems, tiny-mcp for agent development, and llama3-from-scratch-zh for local debugging of Llama 3.

Quick Start & Requirements

• Experience Projects: Links to deployed demos and code repositories for tiny-llm-zh, tiny-rag, tiny-mcp, and llama3-from-scratch-zh are provided within the README.
• Prerequisites: General understanding of deep learning, machine learning, and Python is assumed. Specific projects may have varying hardware requirements (e.g., 16GB RAM for llama3-from-scratch-zh).
• Resources: Extensive documentation and code examples are available directly within the repository.

Highlighted Details

• Detailed breakdown of Transformer architecture, including attention variants (MHA, MQA, GQA).
• Coverage of distributed training strategies (Data, Pipeline, Tensor, Sequence, Hybrid Parallelism) and frameworks like DeepSpeed and Megatron.
• Practical guides on fine-tuning techniques (LoRA, Adapter-tuning) and inference optimization (vLLM, TGI, TRT-LLM).
• Exploration of advanced topics like RLHF, DPO, RAG, and LLM Agents.

Maintenance & Community

The repository is maintained by the author, who welcomes contributions and corrections. Links to a WeChat public account for updates and interview experiences are provided.

Licensing & Compatibility

The repository content is primarily for educational and personal use. Specific code projects within the repository may have their own licenses.

Limitations & Caveats

The answers and explanations are self-authored and may contain inaccuracies; users are encouraged to provide feedback for correction. The focus is on interview preparation, and while practical projects are included, it's not a production-ready framework itself.