Representative Learning for Distributed Learning with Heterogeneity and Asynchrony

Representative Learning (RepL) is a distributed learning framework in which nodes transmit pseudo data, called representatives, instead of model parameters or gradients. These representatives retain the original data format while encoding key statistical features, enabling them to support asynchronous communication and heterogeneous tasks. This paper introduces two new representative constructions: the Transformed Mean Representative (TMR), which generalizes the mean representative by incorporating model-specific link functions; and the Anchored Score-Matching Representative (Anchored-SMR), which modifies score-matching equations to ensure uniqueness and stability. Anchored-SMR is further extended to accommodate general smooth loss functions with optional non-smooth penalties. We analyze RepL in decentralized, asynchronous systems where gradients and models from other nodes may be delayed or misaligned. Theoretical results and extensive simulations demonstrate that the proposed representatives maintain accuracy and convergence under heterogeneity and asynchrony, offering a scalable and interpretable alternative to gradient-based distributed optimization.