Quantization of large scale language models

Quantization is a powerful technique for reducing model size and speeding up inference, and it has demonstrated effectiveness across various convolutional neural networks. The increasing size of NLP models leads to a memory wall problem during the generation phase. To address this challenge, recent efforts have concentrated on quantizing models. Quantization techniques can reduce the computational and resource requirements of training Large Language Models (LLMs), thereby helping to lower the overall training costs.