Quantization of large scale language models

Quantization is a powerful technique for reducing model size and speeding up inference, and it has demonstrated effectiveness across various convolutional neural networks. The increasing size of NLP models leads to a memory wall problem during the generation phase. To address this challenge, recent efforts have concentrated on quantizing models.