• Outstanding Paper Award: Alex Iacob, Lorenzo Sani, Bill Marino, Preslav Aleksandrov, William F. Shen & Nicholas Donald Lane. Worldwide Federated Training of Language Models
• Outstanding Paper Award: Wang Lu, Hao Yu, Jindong Wang, Damien Teney, Haohan Wang, Yao Zhu, Yiqiang Chen, Qiang Yang, Xing Xie & Xiangyang Ji. ZOOPFL: Exploring Black-box Foundation Models for Personalized Federated Learning
• Outstanding Student Paper Award: Lei Shen, Zhenheng Tang, Lijun Wu, Yonggang Zhang, Xiaowen Chu, Tao Qin & Bo Han. Hot Pluggable Federated Learning
• Outstanding Student Paper Award: Xianjie Guo, Liping Yi, Xiaohu Wu, Kui Yu & Gang Wang. Enhancing Causal Discovery in Federated Settings with Limited Local Samples

1. Alex Iacob, Lorenzo Sani, Bill Marino, Preslav Aleksandrov, William F. Shen & Nicholas Donald Lane. Worldwide Federated Training of Language Models
2. Alexander Bienstock, Antigoni Polychroniadou & Ujjwal Kumar. Distributed Matrix Mechanism for Federated Learning
3. Christian Internò, Elena Raponi, Niki van Stein, Thomas Bäck, Markus Olhofer, Yaochu Jin & Barbara Hammer. Adaptive Model Hybrid Pruning in Federated Learning through Loss Exploration
4. Filip Granqvist, Congzheng Song, Áine Cahill, Rogier van Dalen, Martin Pelikan, Yi Sheng Chan, Xiaojun Feng, Natarajan Krishnaswami, Vojta J & Mona Chitnis. pfl-research: Simulation Framework for Accelerating Research in Private Federated Learning
5. Harish Karthikeyan & Antigoni Polychroniadou. OPA: One-shot Private Aggregation with Single Client Interaction and its Applications to Federated Learning
6. Kai Yi, Timur Kharisov, Igor Sokolov & Peter Richtárik. Cohort Squeeze: Beyond a Single Communication Round per Cohort in Cross-Device Federated Learning
7. Lei Shen, Zhenheng Tang, Lijun Wu, Yonggang Zhang, Xiaowen Chu, Tao Qin & Bo Han. Hot Pluggable Federated Learning
8. Lorenzo Sani, Alex Iacob, Zeyu Cao, Bill Marino, Yan Gao, Tomas Paulik, Wanru Zhao, William F. Shen, Preslav Aleksandrov, Xinchi Qiu & Nicholas Donald Lane. The Future of Large Language Model Pre-training is Federated
9. Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen & Yoshua Bengio. MAP: Model Merging with Amortized Pareto Front Using Limited Computation
10. Mariel Werner, Sai Praneeth Karimireddy & Michael Jordan. Defection-Free Collaboration between Competitors in a Learning System
11. Muxing Wang, Pengkun Yang & Lili Su. On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments
12. Rui Ye, Jingyi Chai, Xiangrui Liu, Yaodong Yang, Yanfeng Wang & Siheng Chen. Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models
13. Rui Ye, Rui Ge, Fengting Yuchi, Jingyi Chai, Yanfeng Wang & Siheng Chen. Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models
14. Rui Ye, Xinyu Zhu, Jingyi Chai, Lingjuan Lyu, Chen Xie, Yanfeng Wang & Siheng Chen. Federated Learning with Generative Content
15. Sergio Zaera Mata & Roberto Gómez-Espinosa Martín. The SynapticCity Phenomenon: When All Foundation Models Marry Federated Learning and Blockchain
16. Steffen Schotthöfer & M. Paul Laiu. Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees
17. Sunny Gupta & Amit Sethi. FedStein: Enhancing Multi-Domain Federated Learning Through James-Stein Estimator
18. Tao Yu, Congzheng Song, Jianyu Wang & Mona Chitnis. Momentum Approximation in Asynchronous Private Federated Learning
19. Vasileios Tsouvalas, Samaneh Mohammadi, Ali Balador, Tanir Özçelebi, Francesco Flammini & Nirvana Meratnia. EncCluster: Bringing Functional Encryption in Federated Foundational Models
20. Wang Lu, Hao Yu, Jindong Wang, Damien Teney, Haohan Wang, Yao Zhu, Yiqiang Chen, Qiang Yang, Xing Xie & Xiangyang Ji. ZOOPFL: Exploring Black-box Foundation Models for Personalized Federated Learning
21. Xianjie Guo, Liping Yi, Xiaohu Wu, Kui Yu & Gang Wang. Enhancing Causal Discovery in Federated Settings with Limited Local Samples
22. Xiaochun Niu, Lili Su, Jiaming Xu & Pengkun Yang. Collaborative Learning with Shared Linear Representation: Statistical Rates and Optimal Algorithms
23. Yao Shu, Wenyang Hu, See-Kiong Ng, Bryan Kian Hsiang Low & Fei Richard Yu. Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
24. Zexi Li, Jie Lin, Zhiqi Li, Didi Zhu, Rui Ye, Tao Shen, Tao Lin & Chao Wu. Improving Group Connectivity for Generalization of Federated Deep Learning
25. Zhe Li, Bicheng Ying, Zidong Liu, Chaosheng Dong & Haibo Yang. DeComFL: Federated Learning with Dimension-Free Communication
26. Zhilong Li, Xiaohu Wu, Xiaoli Tang, Tiantian He, Yew-Soon Ong, Mengmeng Chen, Qiqi Liu, Qicheng Lao & Han Yu. Benchmarking Data Heterogeneity Evaluation Approaches for Personalized Federated Learning

Foundation models (FMs) are typically associated with large language models (LLMs), like ChatGPT, and are characterized by their scale and broad applicability. While these models provide transformative capabilities, they also introduce significant challenges, particularly concerning dis-tributed model management and related data privacy, efficiency, and scalability. The training of foundation models is data and resource intensive and the conventional methods are typically centralized; this creates significant challenges including regulatory and privacy concerns in real-world use cases. These include distributed training data, computational resources to manage distributed data repositories, and development of and alignment with regulatory guidelines (e.g., GDPR) that restrict sharing sensitive data.

Federated learning (FL) is an emerging paradigm that can mitigate these challenges by training a global but distributed model using distributed data. The extensive application of machine learning to analyze and draw insight from real-world, distributed, and sensitive data necessitates familiarity with and adoption of this relevant and timely topic within the general scientific community. As FL allows self-interested data owners to collaboratively train models, end-users can become co-creators of AI solutions. By adopting federated learning approaches, we can leverage distributed data and computing power available across different sources while respecting user privacy.

The rise of FMs amplifies the importance and relevance of FL as a crucial research direction. With FMs becoming the norm in machine learning development, the focus shifts from model architecture design to tackling the issues surrounding privacy-preserving and distributed learning. Advancements in FL methods have the potential to unlock the use of FMs, enabling efficient and scalable training while safeguarding sensitive data.

FMs such as GPT-4 encoded with vast knowledge and powerful emergent abilities have achieved remarkable success in various natural language processing and computer vision tasks. Grounding FMs by adapting them to domain-specific tasks or augmenting them with domain-specific knowledge enables us to exploit the full potential of FMs. However, grounding FMs faces several challenges, stemming primarily from constrained computing resources, data privacy, model heterogeneity, and model ownership. Federated Transfer Learning (FTL), the combination of FL and transfer learning, provides promising solutions to address these challenges. In recent years, the need for grounding FMs leveraging FTL, coined FTL-FM, has arisen strongly in both academia and industry.

With this in mind, we invite original research contributions, position papers, and work-in-progress reports on various aspects of federated learning in the era of foundation models. Since the emergence of foundation models has been a relatively recent phenomenon, their full impact on federated learning has not yet been well explored or understood. We hope to provide a platform to facilitate interaction among students, scholars, and industry professionals from around the world to discuss the latest advancements, share insights, and identify future directions in this exciting field.

This workshop aims to bring together academic researchers and industry practitioners to address open issues in this interdisciplinary research area. For industry participants, we intend to create a forum to communicate problems are practically relevant. For academic participants, we hope to make it easier to become productive in this area. The workshop will focus on the theme of combining FL with FM to open up opportunities to address new challenges. The workshop topics include but are not limited to:

Theory and algorithmic foundations: Federated in-context learning Federated neuro-symbolic learning Impact of heterogeneity in FL of large models Multi-stage model training (e.g., base model + fine tuning) Optimization advances in FL (e.g., beyond first-order and local methods) Privacy-preserving machine learning Prompt tuning and design in federated settings Self-supervised learning in federated settings Leveraging foundation models to improve federated learning: Adaptive aggregation strategies for FL in heterogeneous environments Foundation model enhanced FL knowledge distillation Overcoming data interoperability challenges using foundation models Personalization of FL with foundation models

Federated learning for training and tuning foundation models: Fairness, bias, and interpretability challenges in FL with foundation models Federated transfer learning with foundation models FL-empowered multi-agent foundation model systems FL techniques for training large-scale foundation models Hardware for FL with foundation models Optimization algorithms for federated training of foundation models Privacy-preserving mechanisms in FL with foundation models Resource-efficient FL with foundation models Security and robustness considerations in FL with foundation models Systems and infrastructure for FL with foundation models Vertical federated learning with foundation models Vulnerabilities of FL with foundation models

More information on previous workshops can be found here.

The main text of a submitted paper can be between 4 to 9 content pages, including all figures and tables, following NeurIPS'24 template. Additional pages containing references don't count as content pages. An optional appendix of any length is allowed and should be put at the end of the paper (after references). Submissions are double-blind (author identity shall not be revealed to the reviewers), so the submitted PDF file should not include any identifiable information of authors.

Late breaking papers refer to those papers that have been reviewed by NeurIPS'24 but were not accepted. Authors who wish to submit such papers can follow the same submission link and do so by 27 September 2024. In your submission, please include the NeurIPS'24 review comments in the appendix of your paper. These papers will not need to go through another round of peer review. Instead, the organizing committee will determine whether they are accepted into the FL@FM-NeurIPS'24 workshop.

Submissions are collected on OpenReview at the following link: https://openreview.net/group?id=NeurIPS.cc/2024/Workshop/Federated_Learning.
Accepted papers and their review comments will be posted on OpenReview in public. Due to the short timeline, we will not have a rebuttal period, but the authors are encouraged to interact and discuss with reviewers on OpenReview after the acceptance notifications are sent out. Rejected papers and their reviews will remain private and not posted in public.