Inference for the Number of Topics in the Latent Dirichlet Allocation Model via Bayesian Mixture Modelling

In Latent Dirichlet Allocation, the number of topics, <i>T</i>, is a hyperparameter of the model that must be specified before one can fit the model. The need to specify <i>T</i> in advance is restrictive. One way of dealing with this problem is to put a prior on <i>T</i>, but unfortunately the distribution on the latent variables of the model is then a mixture of distributions on spaces of different dimensions, and estimating this mixture distribution by Markov chain Monte Carlo is very difficult. We present a variant of the Metropolis-Hastings algorithm that can be used to estimate this mixture distribution, and in particular the posterior distribution of the number of topics. We evaluate our methodology on synthetic data, and compare it with procedures that are currently used in the machine learning literature. We also give an illustration on two collections of articles from Wikipedia. Supplemental materials for the paper are available online.