Large Transformer Model Inference Optimization

Large transformer models are mainstream nowadays, creating SoTA results for a variety of tasks. They are powerful but very expensive to train and use. The extremely high inference cost, in both time and memory, is a big bottleneck for adopting a powerful transformer for solving real-world tasks at scale.
Why is it hard to run inference for large transformer models? Besides the increasing size of SoTA models, there are two main factors contributing to the inference challenge (Pope et al.

Related Keywords

Noam Shazeer ,Zhou Ma ,Zhu Gupta ,Elsen Hooker ,Zeroquant Yao ,Xiao Lin ,Xiao Lin Smoothquant ,Frantar Alistarh ,Smoothquant Xiao Lin ,Frankle Carbin ,Neural Network Compression ,Nvidia ,Trainable Neural Networks ,Sinkhorn Sorting Network ,A Survey ,Neural Networks ,Training Quantization ,Aware Training ,Optimal Brain Quantization ,Layer By Knowledge Distillation ,Lottery Ticket Hypothesis ,Gradual Magnitude Pruning ,Ticket Hypothesis ,Straight Through Estimator ,Scaling Transformer ,Vision Moe ,Vision Transformer ,Batch Priority Routing ,Task Level Mixture Of Experts ,Task Moe ,Efficient Transformers ,Axial Transformer ,Sorting Network ,Sparse Transformer ,Large Transformer Model Inference ,Matrix Multiplication ,Accurate Quantization ,Generative Pre Trained ,Topk Always Sparse Training ,Deep Neural Networks ,Model Compression ,Scaling Transformers ,Sparse Expert Models ,Deep Learning ,Efficient Inference ,Transformer Decoding ,One Write Head ,All You Need ,Scaling Transformer Inference ,Finding Sparse ,Trainable Neural ,Adaptive Language ,Language Model ,Ong Read 34 ,Architecture ,Attention ,Transformer ,Foundation ,