Similarity Based Technique for Enhancing Personalized Federated Learning via Adaptive Knowledge Injection

Modern applications of federated learning often require solving personalized problems. Local training remains impractical due to the small sizes of datasets and the limited computational power of devices, while global training suffers from poor performance due to the heterogeneity of data across clients. These issues form a significant area of research. We propose a novel strategy that leverages a powerful server as an assistant. The server employs its large dataset to selectively sample objects for each device based on cosine similarity, allowing for more effective personalization. Our insights are supported by the theory and validated through a series of experiments, including next word prediction and image classification. Our approach outperforms most state-of-the-art techniques both theoretically and empirically.

1. J. Hestness et al., “Deep learning scaling is predictable, empirically”, arXiv preprint arXiv:1712.00409, 2017.

2. N. Kiryati and Y. Landau, “Dataset growth in medical image analysis research”, Journal of imaging, vol. 7, no. 8, p. 155, 2021.

3. L. Mangasarian, “Parallel gradient distribution in unconstrained optimization”, SIAM Journal on Control and Optimization, vol. 33, no. 6, pp. 1916–1925, 1995.

4. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs”, in Interspeech, Singapore, 2014, pp. 1058–1062.

5. D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic, “QSGD: Communication-efficient SGD via gradient quantization and encoding”, Advances in neural information processing systems, vol. 30, 2017.

6. J. Konečný, H. B. McMahan, D. Ramage, and P. Richtárik, “Federated optimization: Distributed machine learning for on-device intelligence”, arXiv preprint arXiv:1610.02527, 2016.

7. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data”, in Artificial intelligence and statistics, PMLR, 2017, pp. 1273 1282.

8. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions”, IEEE signal processing magazine, vol. 37, no. 3, pp. 50–60, 2020.

9. P. Kairouz et al., “Advances and open problems in federated learning”, Foundations and trends® in machine learning, vol. 14, no. 1–2, pp. 1–210, 2021.

10. C. Zhang, Y. Xie, H. Bai, B. Yu, W. Li, and Y. Gao, “A survey on federated learning”, Knowledge-Based Systems, vol. 216, p. 106775, 2021.

11. S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan, “Learnability, stability and uniform convergence”, The Journal of Machine Learning Research, vol. 11, pp. 2635–2670, 2010.

12. A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning”, IEEE transactions on neural networks and learning systems, vol. 34, no. 12, pp. 9587–9603, 2022.

13. Y. Deng, M. M. Kamani, and M. Mahdavi, “Adaptive personalized federated learning”, arXiv preprint arXiv:2003.13461, 2020.

14. A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning: A meta-learning approach”, arXiv preprint arXiv:2002.07948, 2020.

15. J. Zhang, S. Guo, X. Ma, H. Wang, W. Xu, and F. Wu, “Parameterized knowledge transfer for personalized federated learning”, Advances in Neural Information Processing Systems, vol. 34, pp. 10092–10104, 2021.

16. D. C. Nguyen et al., “Federated learning for smart healthcare: A survey”, ACM Computing Surveys (Csur), vol. 55, no. 3, pp. 1–37, 2022.

17. Q. Wu, K. He, and X. Chen, “Personalized federated learning for intelligent IoT applications: A cloud-edge based framework”, IEEE Open Journal of the Computer Society, vol. 1, pp. 35–44, 2020.

18. H. Li et al., “Fedtp: Federated learning by transformer personalization”, IEEE transactions on neural networks and learning systems, 2023.

19. X. Ying, “An overview of overfitting and its solutions”, Journal of Physics: Conference Series, vol. 1168, p. 022022, Feb. 2019.

20. Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data”, arXiv preprint arXiv:1806.00582, 2018.

21. F. Hanzely and P. Richtárik, “Federated learning of a mixture of global and local models”, arXiv preprint arXiv:2002.05516, 2020.

22. F. Hanzely, S. Hanzely, S. Horváth, and P. Richtárik, “Lower bounds and optimal algorithms for personalized federated learning”, Advances in Neural Information Processing Systems, vol. 33, pp. 2304–2315, 2020.

23. C. T Dinh, N. Tran, and J. Nguyen, “Personalized federated learning with moreau envelopes”, Advances in neural information processing systems, vol. 33, pp. 21394–21405, 2020.

24. K. Pillutla, K. Malik, A.-R. Mohamed, M. Rabbat, M. Sanjabi, and L. Xiao, “Federated learning with partial model personalization”, in International conference on machine learning, PMLR, 2022, pp.17716 17758.

25. K. Mishchenko, R. Islamov, E. Gorbunov, and S. Horváth, “Partially personalized federated learning: Breaking the curse of data heterogeneity”, arXiv preprint arXiv:2305.18285, 2023.

26. X. Tang, S. Guo, and J. Guo “Personalized federated learning with contextualized generalization”, in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 2022 pp. 2241–2247.

27. J. Zhang and Y. Shi, “A personalized federated learning method based on clustering and knowledge distillation”, Electronics, vol. 13, p. 857, Feb. 2024.

28. Z. Ni and M. Hashemi, “Efficient cluster selection for personalized federated learning: A multi-armed bandit approach”, in 2023 IEEE virtual conference on communications (VCC), IEEE, 2023, pp. 115–120.

29. J. Wang, Y. Chen, Y. Wu, M. Das, H. Yang, and F. Ma, “Rethinking personalized federated learning with clustering-based dynamic graph propagation”, in Pacific-Asia conference on knowledge discovery and data mining, Springer, 2024, pp. 155–167.

30. M. Khodak, M.-F. F. Balcan, and A. S. Talwalkar, “Adaptive gradient-based meta-learning methods”, Advances in Neural Information Processing Systems, vol. 32, 2019.

31. D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation”, arXiv preprint arXiv:1910.03581, 2019.

32. Z. Tang, S. Xu, H. Jin, S. Liu, R. Zhai, and K. Lu, “Personalized federated learning via decoupling self-knowledge distillation and global adaptive aggregation”, Multimedia Systems, vol. 31, Feb. 2025.

33. A. Hard et al., “Federated learning for mobile keyboard prediction”, arXiv preprint arXiv:1811.03604, 2018.

34. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning”, in International conference on machine learning, PMLR, 2020, pp. 5132–5143.

35. P. Vanhaesebrouck, A. Bellet, and M. Tommasi, “Decentralized collaborative learning of personalized models over networks”, in Artificial intelligence and statistics, PMLR, 2017, pp. 509–517.

36. V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning”, Advances in neural information processing systems, vol. 30, 2017.

37. D. Peterson, P. Kanani, and V. J. Marathe, “Private federated learning with domain adaptation”, arXiv preprint arXiv:1912.06733, 2019.

38. J. Wang, Y. Jin, and L. Wang, “Personalizing federated medical image segmentation via local calibration”, in European conference on computer vision, Springer, 2022, pp. 456–472.

39. Y. Zhao, Q. Liu, X. Liu, and K. He, “Medical federated model with mixture of personalized and sharing components”, arXiv preprint arXiv:2306.14483, 2023.

40. T. H. Kim et al., “PPFL: A personalized progressive federated learning method for leveraging different healthcare institution-specific features”, iScience, vol. 27, no. 10, 2024.

41. A. Ghosh, J. Hong, D. Yin, and K. Ramchandran, “Robust federated learning in a heterogeneous environment”, in ICML 2019 workshop on Privacy and Security, 2019.

42. A. Ghosh, J. Chung, D. Yin, and K. Ramchandran, “An efficient framework for clustered federated learning”, Advances in Neural Information Processing Systems, vol. 33, pp. 19586–19597, 2020.

43. C. Briggs, Z. Fan, and P. Andras, “Federated learning with hierarchical clustering of local updates to improve training on non-IID data”, in 2020 international joint conference on neural networks (IJCNN), IEEE, 2020, pp. 1–9.

44. Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh, “Three approaches for personalization with applications to federated learning”, arXiv preprint arXiv:2002.10619, 2020.

45. G. Long, M. Xie, T. Shen, T. Zhou, X. Wang, and J. Jiang, “Multi-center federated learning: Clients clustering for better personalization”, World Wide Web, vol. 26, no. 1, pp. 481–500, 2023.

46. O. Marfoq, G. Neglia, A. Bellet, L. Kameni, and R. Vidal, “Federated multi-task learning under a mixture of distributions”, Advances in Neural Information Processing Systems, vol. 34, pp. 15434–15447, 2021.

47. I. Lin, O. Yagan, C. Joe-Wong, “FedSPD: A soft-clustering approach for personalized decentralized federated learning”, in Forty-first Conference on Uncertainty in Artificial Intelligence, PMLR 2025, pp. 2618–2641.

48. T. Liang, C. Yuan, C. Lu, Y. Li, J. Yuan, and Y. Yin, “Efficient one-off clustering for personalized federated learning”, Knowledge-Based Systems, vol. 277, p. 110813, 2023.

49. Y. J. Cho, J. Wang, T. Chirvolu, and G. Joshi, “Communication-efficient and model-heterogeneous personalized federated learning via clustered knowledge transfer”, IEEE Journal of Selected Topics in Signal Processing, vol. 17, no. 1, pp. 234–247, 2023.

50. L. Yang, J. Huang, W. Lin, and J. Cao, “Personalized federated learning on non-IID data via group-based meta-learning”, ACM Transactions on Knowledge Discovery from Data, vol. 17, no. 4, pp. 1–20, 2023.

51. Z. Yang, Y. Liu, S. Zhang, and K. Zhou, “Personalized federated learning with model interpolation among client clusters and its application in smart home”, World Wide Web, vol. 26, no. 4, pp. 2175–2200, 2023.

52. M. Werner, L. He, M. Jordan, M. Jaggi, and S. P. Karimireddy, “Provably personalized and robust federated learning”, Transactions on Machine Learning Research, 2023.

53. C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks”, in International conference on machine learning, PMLR, 2017, pp. 1126–1135.

54. A. Fallah, A. Mokhtari, and A. Ozdaglar, “On the convergence theory of gradient-based model-agnostic meta-learning algorithms”, in International conference on artificial intelligence and statistics, PMLR, 2020, pp. 1082–1092.

55. G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network”, arXiv preprint arXiv:1503.02531, 2015.

56. Y. Jiang, X. Zhao, H. Li, and Y. Xue, “A personalized federated learning method based on knowledge distillation and differential privacy”, Electronics, vol. 13, p. 3538, Sept. 2024, DOI: 10.3390/electronics13173538.

57. F. Lv, P. Qian, Y. Lu, and H. Wang, “Personalized federated learning on long-tailed data via knowledge distillation and generated features”, Pattern Recognition Letters, vol. 186, pp. 178–183, 2024.

58. H. Hu, A. N. Kothari, and A. Banerjee, “A novel algorithm for personalized federated learning: Knowledge distillation with weighted combination loss”, Algorithms, vol. 18, no. 5, p. 274, 2025.

59. F. Gauthier, V. C. Gogineni, and S. Werner, “Networked personalized federated learning using reinforcement learning”, in ICC 2023-IEEE international conference on communications, IEEE, 2023, pp. 4397–4402.

60. W. Xiong, Q. Liu, F. Li, B. Wang, and F. Zhu, “Personalized federated reinforcement learning: Balancing personalization and experience sharing via distance constraint”, Expert Systems with Applications, vol. 238, p. 122290, 2024.

61. T. Wu, X. Li, P. Gao, W. Yu, L. Xin, and M. Guo, “Resource-aware personalized federated learning based on reinforcement learning”, IEEE Communications Letters, vol. 29, no. 1, pp. 175–179, 2025, DOI: 10.1109/LCOMM.2024.3506015.

62. H. Yang, J. Li, M. Hao, W. Zhang, H. He, and A. K. Sangaiah, “An efficient personalized federated learning approach in heterogeneous environments: A reinforcement learning perspective”, Scientific Reports, vol. 14, no. 1, p. 28877, 2024.

63. X. Lu, Z. Liu, L. Xiao, and H. Dai, “Reinforcement learning-based personalized differentially private federated learning”, IEEE Transactions on Information Forensics and Security, 2024.

64. S. Vahidian, M. Morafah, and B. Lin, “Personalized federated learning by structured and unstructured pruning under data heterogeneity”, in 2021 IEEE 41st international conference on distributed computing systems workshops (ICDCSW), IEEE, 2021, pp. 27–34.

65. W. Jeong and S. J. Hwang, “Factorized-fl: Personalized federated learning with parameter factorization & similarity matching”, Advances in Neural Information Processing Systems, vol. 35, pp. 35684–35695, 2022.

66. X. Zhang, Y. Li, W. Li, K. Guo, and Y. Shao, “Personalized federated learning via variational bayesian inference”, in International conference on machine learning, PMLR, 2022, pp. 26293–26310.

67. J. Zhang et al., “Fedala: Adaptive local aggregation for personalized federated learning”, in Proceedings of the AAAI conference on artificial intelligence, 2023, pp. 11237–11244.

68. J. Zhang et al., “Gpfl: Simultaneously learning global and personalized feature information for personalized federated learning”, in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 5041–5051.

69. J. Wu, W. Bao, E. Ainsworth, and J. He, “Personalized federated learning with parameter propagation”, in Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, 2023, pp. 2594–2605.

70. R. Zhang, Y. Chen, C. Wu, and F. Wang, “Multi-level personalized federated learning on heterogeneous and long-tailed data”, IEEE Transactions on Mobile Computing, 2024.

71. F. Sattler, S. Wiedemann, K.-R. Müller, and W. Samek, “Robust and communication-efficient federated learning from non-iid data”, IEEE transactions on neural networks and learning systems, vol. 31, no. 9, pp. 3400–3413, 2019.

72. K. A. Sankararaman, S. De, Z. Xu, W. R. Huang, and T. Goldstein, “The impact of neural network overparameterization on gradient confusion and stochastic gradient descent”, in International conference on machine learning, PMLR, 2020, pp. 8469–8479.

73. A. Khaled and P. Richtárik, “Better theory for SGD in the nonconvex world”, arXiv preprint arXiv:2002.03329, 2020.

74. F. Sattler, K.-R. Müller, and W. Samek, “Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints”, IEEE transactions on neural networks and learning systems, vol. 32, no. 8, pp. 3710–3722, 2020.

75. H.-Y. Hsu, K. H. Keoy, J.-R. Chen, H.-C. Chao, and C.-F. Lai, “Personalized federated learning algorithm with adaptive clustering for non-IID IoT data incorporating multi-task learning and neural network model characteristics”, Sensors, vol. 23, no. 22, p. 9016, 2023.

76. P. Ren, K. Qi, J. Li, T. Yan, and Q. Dai, “CosPer: An adaptive personalized approach for enhancing fairness and robustness of federated learning”, Information Sciences, vol. 675, p. 120760, 2024.

77. R. Ye, Z. Ni, F. Wu, S. Chen, and Y. Wang, “Personalized federated learning with inferred collaboration graphs”, in International conference on machine learning, PMLR, 2023, pp. 39801–39817.

78. Z. Ma, Y. Lu, W. Li, J. Yi, and S. Cui, “PFedAtt: Attention-based personalized federated learning on heterogeneous clients”, in Asian conference on machine learning, PMLR, 2021, pp. 1253–1268.

79. F. Rosenblatt, “The perceptron: A probabilistic model for information storage and organization in the brain”. Psychological review, vol. 65, no. 6, p. 386, 1958.

80. G. Cybenko, “Approximation by superpositions of a sigmoidal function”, Mathematics of control, signals and systems, vol. 2, no. 4, pp. 303–314, 1989.

81. Q. Nguyen and M. Hein, “Optimization landscape and expressivity of deep CNNs”, in International conference on machine learning, PMLR, 2018, pp. 3730–3739.

82. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data”, in International Conference on Learning Representations, 2020.

83. J. A. Nelder and R. Mead, “A simplex method for function minimization”, The Computer journal, vol. 7, no. 4, pp. 308–313, 1965.

84. A. Beck and M. Teboulle, “Mirror descent and nonlinear projected subgradient methods for convex optimization”, Operations Research Letters, vol. 31, no. 3, pp. 167–175, 2003.

85. M. B. Cohen, Y. T. Lee, and Z. Song, “Solving linear programs in the current matrix multiplication time”, Journal of the ACM (JACM), vol. 68, no. 1, pp. 1–39, 2021.

86. C. Dwork, “Differential privacy”, in International colloquium on automata, languages, and programming, Springer, 2006, pp. 1–12.

87. S. Khirirat, E. Gorbunov, S. Horváth, R. Islamov, F. Karray, and P. Richtárik, “Clip21: Error feedback for gradient clipping”, arXiv preprint arXiv:2305.18929, 2023.

88. R. Islamov, S. Horvath, A. Lucchi, P. Richtarik, and E. Gorbunov, “Double momentum and error feedback for clipping with fast rates and differential privacy”, arXiv preprint arXiv:2502.11682, 2025.

89. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition”, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

90. A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images”, 2009.

91. S. Hochreiter and J. Schmidhuber, “Long short-term memory”, Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.

92. A. Paszke et al., “Automatic differentiation in PyTorch”, 2017.

93. M. Priya, “English-Italian Sentence Translation”, [Dataset], Kaggle, 2020. [Online]. Available: https://www.kaggle.com/datasets/ncsaayali/english-italian-sentence-translation/data.

94. D. Pollard, Convergence of stochastic processes. Springer Science & Business Media, 2012.