Guangxu Zhu
Shenzhen Research Institute of Big Data, China
Guangxu Zhu (Member, IEEE) received the Ph.D. degree in electrical and electronic engineering from The University of Hong Kong in 2019. Currently he is a senior research scientist and deputy director of network and machine intelligence center at the Shenzhen research institute of big data, and an adjunct associate professor with the Chinese University of Hong Kong, Shenzhen. His recent research interests include edge intelligence, large foundation model, and integrated sensing and communication. He is a recipient of the 2023 IEEE ComSoc Asia-Pacific Best Young Researcher Award and Outstanding Paper Award, the World's Top 2% Scientists by Stanford University, the "AI 2000 Most Influential Scholar Award Honorable Mention", the Young Scientist Award from UCOM 2023. He serves as associate editors at top-tier journals in IEEE, including IEEE TMC, TWC and WCL. He is the vice co-chair of the IEEE ComSoc Asia-Pacific Board Young Professionals Committee.
Federated fine-tuning (FFT) attempts to fine-tune a pre-trained model with private data from distributed clients by exchanging models rather than data under the orchestration of a parameter server (PS). To overcome the bottleneck forged by the growing communication and memory overhead for clients in such systems due to the ever-increasing model sizes, we propose FeedSign, an FFT algorithm in which the upload and download payload is exactly 1 bit per aggregation step. Meanwhile, the memory overhead is squeezed to the amount needed for inference. The reduction is realized by utilizing zeroth-order (ZO) optimizers on large models and shared pseudo-random number generators (PRNG) across devices to represent the gradient estimates as seed-sign pairs. We conduct theoretical analysis on FeedSign and show that it converges at an exponential rate under the widely used Polyak-Łojaciewicz assumption. Moreover, FeedSign is robust against data heterogeneity and Byzantine attacks. We extensively tested FeedSign on models across different structures and sizes (11M to 13B). We found that the proposed method performs better or closely, depending on the scenarios, than its ZO and FO counterparts, despite orders-of-magnitude lower communication overhead. We also discuss some interesting advantages as byproducts guaranteed by the design of FeedSign.