Relaxed Contrastive Learning for Federated Learning
A relaxed contrastive learning loss is introduced that imposes a divergence penalty on excessively similar sample pairs within each class, which prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements.
Abstract
We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity infederated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a naïve integration of SCL into federated learning incurs representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework out-performs all existing federated learning approaches by significant margins on the standard benchmarks, as demonstrated by extensive experimental results. The source code is available at our project page11https://github.com/skynbe/FedRCL: