Penalized Versus Constrained Generalized Eigenvalue Problems

We investigate the difference between using an ℓ₁ penalty versus an ℓ₁ constraint in generalized eigenvalue problems arising in multivariate analysis. Our main finding is that the ℓ₁ penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an ℓ₁ constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of the ℓ₁ constraint in the context of discriminant analysis and principal component analysis. Supplementary materials for this article are available online.