v1v2v3 (latest)

Private Fine-tuning of Large Language Models with Zeroth-order Optimization

9 January 2024

Papers citing "Private Fine-tuning of Large Language Models with Zeroth-order Optimization"

50 / 82 papers shown

Title
KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning Zhendong Mi Qitao Tan Xiaodong Yu Zining Zhu Geng Yuan Shaoyi Huang 200 0 0 24 May 2025
Forward Learning with Differential Privacy Mingqian Feng Zeliang Zhang Jinyang Jiang Yijie Peng Chenliang Xu 83 0 0 01 Apr 2025
Privacy Auditing of Large Language Models Ashwinee Panda Xinyu Tang Milad Nasr Christopher A. Choquette-Choo Prateek Mittal PILM 97 9 0 09 Mar 2025
Towards hyperparameter-free optimization with differential privacy Zhiqi Bu Ruixuan Liu 77 2 0 02 Mar 2025
Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning Raghav Singhal Kaustubh Ponkshe Rohit Vartak Lav R. Varshney Praneeth Vepakomma FedML 119 4 0 24 Feb 2025
Private Text Generation by Seeding Large Language Model Prompts Supriya Nagesh Justin Y. Chen Nina Mishra Tal Wagner SyDa SILM 94 2 0 20 Feb 2025
ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimization Keisuke Sugiura Hiroki Matsutani MQ 71 2 0 08 Jan 2025
Balls-and-Bins Sampling for DP-SGD Lynn Chua Badih Ghazi Charlie Harrison Ethan Leeman Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chiyuan Zhang 142 5 0 21 Dec 2024
Scalable DP-SGD: Shuffling vs. Poisson Subsampling Lynn Chua Badih Ghazi Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chiyuan Zhang 88 8 0 06 Nov 2024
Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning Fengyu Gao Ruida Zhou T. Wang Cong Shen Jing Yang 66 3 0 15 Oct 2024
Private Zeroth-Order Nonsmooth Nonconvex Optimization Qinzi Zhang Hoang Tran Ashok Cutkosky 100 4 0 27 Jun 2024
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs Charlie Hou Akshat Shrivastava Hongyuan Zhan Rylan Conway Trang Le Adithya Sagar Giulia Fanti Daniel Lazar 95 13 0 05 Jun 2024
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity Wentao Guo Jikai Long Yimeng Zeng Zirui Liu Xinyu Yang ... Osbert Bastani Christopher De Sa Xiaodong Yu Beidi Chen Zhaozhuo Xu 60 21 0 05 Jun 2024
AI Risk Management Should Incorporate Both Safety and Security Xiangyu Qi Yangsibo Huang Yi Zeng Edoardo Debenedetti Jonas Geiping ... Chaowei Xiao Yue Liu Dawn Song Peter Henderson Prateek Mittal AAML 80 12 0 29 May 2024
Delving into Differentially Private Transformer Youlong Ding Xueyang Wu Yining Meng Yonggang Luo Hao Wang Weike Pan 102 5 0 28 May 2024
Avoiding Pitfalls for Privacy Accounting of Subsampled Mechanisms under Composition C. Lebeda Matthew Regehr Gautam Kamath Thomas Steinke 112 10 0 27 May 2024
Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimization Zhe Li Bicheng Ying Zidong Liu Chaosheng Dong Haibo Yang FedML 118 3 0 24 May 2024
How Private are DP-SGD Implementations? Lynn Chua Badih Ghazi Pritish Kamath Ravi Kumar Pasin Manurangsi Amer Sinha Chiyuan Zhang 73 13 0 26 Mar 2024
Privacy-preserving Fine-tuning of Large Language Models through Flatness Tiejin Chen Longchao Da Huixue Zhou Pingzhi Li Kaixiong Zhou Tianlong Chen Hua Wei 68 5 0 07 Mar 2024
Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer Yanjun Zhao Sizhe Dang Haishan Ye Guang Dai Yi Qian Ivor W.Tsang 118 13 0 23 Feb 2024
Privacy-Preserving Instructions for Aligning Large Language Models Da Yu Peter Kairouz Sewoong Oh Zheng Xu 91 24 0 21 Feb 2024
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark Yihua Zhang Pingzhi Li Junyuan Hong Jiaxiang Li Yimeng Zhang ... Wotao Yin Mingyi Hong Zhangyang Wang Sijia Liu Tianlong Chen 88 58 0 18 Feb 2024
Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning Zhicheng Liu Jian Lou Wenxuan Bao Yihan Hu Baochun Li Zhan Qin K. Ren 74 10 0 12 Feb 2024
Zero redundancy distributed learning with differential privacy Zhiqi Bu Justin Chiu Ruixuan Liu Sheng Zha George Karypis 76 8 0 20 Nov 2023
DPZero: Private Fine-Tuning of Language Models without Backpropagation Liang Zhang Bingcong Li K. K. Thekumparampil Sewoong Oh Niao He 65 15 0 14 Oct 2023
Mistral 7B Albert Q. Jiang Alexandre Sablayrolles A. Mensch Chris Bamford Devendra Singh Chaplot ... Teven Le Scao Thibaut Lavril Thomas Wang Timothée Lacroix William El Sayed MoE LRM 81 2,229 0 10 Oct 2023
Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation Xinyu Tang Richard Shin Huseyin A. Inan Andre Manoel Fatemehsadat Mireshghallah Zinan Lin Sivakanth Gopi Janardhan Kulkarni Robert Sim 93 59 0 21 Sep 2023
DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass Minxin Du Xiang Yue Sherman S. M. Chow Tianhao Wang Chenyu Huang Huan Sun SILM 78 64 0 13 Sep 2023
Differentially Private Image Classification by Learning Priors from Random Processes Xinyu Tang Ashwinee Panda Vikash Sehwag Prateek Mittal 68 21 0 08 Jun 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 115 196 0 27 May 2023
Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models Haonan Duan Adam Dziedzic Nicolas Papernot Franziska Boenisch AAML 63 66 0 24 May 2023
Privacy-Preserving In-Context Learning for Large Language Models Tong Wu Ashwinee Panda Jiachen T. Wang Prateek Mittal 89 31 0 02 May 2023
A Randomized Approach for Tight Privacy Accounting Jiachen T. Wang Saeed Mahloujifar Tong Wu R. Jia Prateek Mittal 77 10 0 17 Apr 2023
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy Natalia Ponomareva Hussein Hazimeh Alexey Kurakin Zheng Xu Carson E. Denison H. B. McMahan Sergei Vassilvitskii Steve Chien Abhradeep Thakurta 150 180 0 01 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.5K 13,420 0 27 Feb 2023
Why Is Public Pretraining Necessary for Private Model Training? Arun Ganesh Mahdi Haghifam Milad Nasr Sewoong Oh Thomas Steinke Om Thakkar Abhradeep Thakurta Lun Wang 59 39 0 19 Feb 2023
Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining Florian Tramèr Gautam Kamath Nicholas Carlini SILM 81 72 0 13 Dec 2022
A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization Ashwinee Panda Xinyu Tang Saeed Mahloujifar Vikash Sehwag Prateek Mittal 87 12 0 08 Dec 2022
Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping Jiyan He Xuechen Li Da Yu Huishuai Zhang Janardhan Kulkarni Y. Lee A. Backurs Nenghai Yu Jiang Bian 101 48 0 03 Dec 2022
Differentially Private Optimization on Large Model at Small Cost Zhiqi Bu Yu Wang Sheng Zha George Karypis 93 54 0 30 Sep 2022
Differentially Private Bias-Term Fine-tuning of Foundation Models Zhiqi Bu Yu Wang Sheng Zha George Karypis 99 48 0 30 Sep 2022
Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributions Vadym Doroshenko Badih Ghazi Pritish Kamath Ravi Kumar Pasin Manurangsi 65 43 0 10 Jul 2022
When Does Differentially Private Learning Not Suffer in High Dimensions? Xuechen Li Daogao Liu Tatsunori Hashimoto Huseyin A. Inan Janardhan Kulkarni Y. Lee Abhradeep Thakurta 63 57 0 01 Jul 2022
Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger Zhiqi Bu Yu Wang Sheng Zha George Karypis 119 71 0 14 Jun 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 338 3,685 0 02 May 2022
Unlocking High-Accuracy Differentially Private Image Classification through Scale Soham De Leonard Berrada Jamie Hayes Samuel L. Smith Borja Balle 88 232 0 28 Apr 2022
Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks Fatemehsadat Mireshghallah Kartik Goyal Archit Uniyal Taylor Berg-Kirkpatrick Reza Shokri MIALM 59 164 0 08 Mar 2022
Differentially Private Fine-tuning of Language Models Da Yu Saurabh Naik A. Backurs Sivakanth Gopi Huseyin A. Inan ... Y. Lee Andre Manoel Lukas Wutschitz Sergey Yekhanin Huishuai Zhang 236 369 0 13 Oct 2021
Opacus: User-Friendly Differential Privacy Library in PyTorch Ashkan Yousefpour I. Shilov Alexandre Sablayrolles Davide Testuggine Karthik Prasad ... Sayan Gosh Akash Bharadwaj Jessica Zhao Graham Cormode Ilya Mironov VLM 284 368 0 25 Sep 2021
Large Scale Private Learning via Low-rank Reparametrization Da Yu Huishuai Zhang Wei Chen Jian Yin Tie-Yan Liu 58 104 0 17 Jun 2021