This repository is a curated list of research papers and resources focused on the intersection of Transformer architectures and Neural Architecture Search (NAS). It serves as a valuable reference for researchers and engineers exploring efficient and novel Transformer designs across NLP, computer vision, and speech processing. The list aims to track recent advancements in automating Transformer development.

How It Works

The project categorizes papers into key areas: General Transformer Search, Domain-Specific Applications (NLP, Vision, ASR), Transformer Knowledge (parameters, attention), Surveys, Foundation Models, and Miscellaneous Resources. Each entry typically includes the paper title, venue, and contributing research group, providing a structured overview of the field.

Quick Start & Requirements

This is a curated list, not a software package. No installation or execution is required. The primary resource is the list of papers and their associated venues and research groups.

Highlighted Details

• Comprehensive categorization of Transformer NAS research.
• Includes papers from top-tier conferences (NeurIPS, CVPR, ICLR, ACL, ICML, ICCV).
• Covers a wide range of applications, from NLP and Vision to ASR.
• Features foundational work and recent advancements in efficient Transformer design.

Maintenance & Community

The list is maintained by Yash Mehta. Contributions are welcomed via pull requests or issues. A Google Doc is linked for a comprehensive list of foundation model papers from ICML 2023.

Licensing & Compatibility

The repository itself is not software and does not have a license. The linked papers are subject to their respective publication licenses and copyright.

Limitations & Caveats

As a curated list, it reflects the state of research at the time of its last update and may not include the very latest publications. It is a reference, not an implementation.