From Autoregressive to Non-Autoregressive: Studies on Text Generation in Neural Sequence Models

There is a balancing act between quality and time as we move from autoregressive to non-autoregressive models. Autoregressive models are slower but produce better quality output compared to non-autoregressive models which are faster. Semi-autoregressive models provide the best of both worlds by decoding faster but at the expense of output quality. This thesis, therefore, explores the problem of text generation to improve generation across the three model families. It identifies and addresses various literature gaps such as ineffective decoding methods for autoregressive models, lack of length flexibility in autoregressive and non-autoregressive models, and inferior output quality by non-autoregressive models compared to autoregressive models.