This repository provides a unified PyTorch implementation of attention-based models for both Natural Language Processing (NLP) and Computer Vision (CV) tasks. It aims to establish a strong baseline for future research by offering a streamlined codebase for core Transformer components and common downstream applications, enabling efficient training and ONNX-based deployment.

How It Works

The project focuses on implementing the core encoder and decoder stages of the Transformer architecture, drawing inspiration from Hugging Face's transformers library but with a simplified codebase. This approach allows for direct comparison and validation against established models like BERT, ensuring consistency in results. The goal is to create a foundational backbone that can achieve state-of-the-art performance across diverse NLP and CV tasks.

Quick Start & Requirements

• Installation: PyTorch.
• Prerequisites: Python, Hugging Face transformers library for model downloading.
• Resources: Access to domestic download mirrors for Hugging Face models is provided.
• Documentation: transformers国内下载镜像

Highlighted Details

• PyTorch implementation of Transformer encoder and BERT.
• Achieves consistent results with Hugging Face's encoder.
• Provides domestic download mirrors for popular Chinese NLP models (bert-base-chinese, chinese-roberta-wwm-ext, etc.).
• Planned implementation of Transformer decoder for seq2seq tasks.

Maintenance & Community

The project is actively maintained by mmmwhy. Further community engagement details are not specified in the README.

Licensing & Compatibility

The repository's license is not explicitly stated in the provided README.

Limitations & Caveats

The project is currently in its first phase, with planned implementations for NLP downstream tasks (sequence labeling, classification), ViT, UNILM, MAE, GPT series, and ONNX export/inference. The current focus is on validating the core encoder and decoder implementations.