Penalized Versus Constrained Generalized Eigenvalue Problems

We investigate the difference between using an ℓ1 penalty versus an ℓ1 constraint in generalized eigenvalue problems arising in multivariate analysis. Our main finding is that the ℓ1 penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an ℓ1 constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of the ℓ1 constraint in the context of discriminant analysis and principal component analysis. Supplementary materials for this article are available online.