Efficient Transformer Models

A summary of architectures that make improvements around computational and memory efficiency of the original Transformer architecture. This is taken from the paper Efficient Transformers: A Survey, published in ACM Computing Surveys. In time-complexity, N refers to the sequence length, B is the local window or block size, $$N_g$$ denotes global model memory length, and $$N_c$$ refers to convolutionally-compressed sequence lengths respectively. The taxonomy of algorithms that improve the general efficiency of the Transformer architecture are presented in https://orkg.org/paper/R211075/R211081.