The Transformer Family Version 2.0

Many new Transformer architecture improvements have been proposed since my last post on “The Transformer Family” about three years ago. Here I did a big refactoring and enrichment of that 2020 post — restructure the hierarchy of sections and improve many sections with more recent papers. Version 2.0 is a superset of the old version, about twice the length.
Notations Symbol Meaning $d$ The model size / hidden state dimension / positional encoding size.

Related Keywords

Mostafa Dehghani ,Olah Carter ,Emilio Parisotto ,Sainbayar Sukhbaatar ,Alex Graves ,Longformer Beltagy ,Niki Parmar ,Ashish Vaswani ,Nikita Kitaev ,Zihang Dai ,Linformer Wang ,Rahimi Recht ,Aidann Gomez ,Adaptive Computation Time For Recurrent Neural Networks ,A Survey ,Recurrent Neural Networks ,Rotary Position Embedding ,Memorizing Transformer ,Aware Transformer ,Linear Biases ,Universal Transformer ,Adaptive Attention ,Adaptive Computation Time ,Depth Adaptive Transformer ,Confident Adaptive Language Model ,Efficient Transformers ,Image Transformer ,Local Attention ,Sparse Transformer ,Factorized Self Attention ,Blockwise Attention ,Extended Transformer Construction ,Big Bird ,Locality Sensitive Hashing ,Routing Transformer ,Feature Attention ,Reinforcement Learning ,Gated Recurrent Unit ,Augmented Recurrent Neural ,Attention Span ,Long Sequences ,Computation Time ,Recurrent Neural ,Attentive Language Models Beyond ,Fixed Length Context ,Reversible Residual Network ,Backpropagation Without Storing ,Long Range Sequence Modelling ,Test Long ,Distance Aware ,Adaptive Transformer ,Adaptive Language ,Content Based Sparse Attention ,Routing Transformers ,Encoding Long ,Structured Inputs ,Longer Sequences ,Linear Complexity ,Sinkhorn Attention ,Architecture ,Attention ,Transformer ,Foundation ,Long Read ,