Discrete Representation Learning in the New Era: Tokens, Robustness and Transferability

This thesis proposes a unified framework for deep discrete representation learning, addressing the fragmented landscape of ad-hoc, problem-specific solutions. Grounded in a task-driven compression principle, the framework extracts task-relevant information through discrete bottlenecks. We show this formulation admits an equivalent generative view where token distributions serve as discrete latent priors, unifying existing methods as instantiations of the same framework. This unification enables theoretical analysis revealing a fundamental trade-off between expressivity and sample complexity, providing principled guidance for architecture design. We further demonstrate how discrete representations improve robustness under distribution shift and enhance cross-modal alignment in modern pre-trained models.