Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data

23 May 2024

Hanlin Gu

Papers citing "Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data"

50 / 60 papers shown

Title
Differentially Private Knowledge Distillation via Synthetic Text Generation James Flemings Murali Annavaram SyDa 62 14 0 01 Mar 2024
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning Rui Ye Wenhao Wang Jingyi Chai Dihan Li Zexi Li Yinda Xu Yaxin Du Yanfeng Wang Siheng Chen ALM FedML AIFin 56 89 0 10 Feb 2024
Mutual Enhancement of Large and Small Language Models with Cross-Silo Knowledge Transfer Yongheng Deng Ziqing Qiao Ju Ren Yang Liu Yaoxue Zhang 69 11 0 10 Dec 2023
Grounding Foundation Models through Federated Transfer Learning: A General Framework Yan Kang Tao Fan Hanlin Gu Xiaojin Zhang Lixin Fan Qiang Yang AI4CE 96 19 0 29 Nov 2023
PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Models Haoran Li Dadi Guo Donghao Li Wei Fan Qi Hu Xin Liu Chunkit Chan Duanyi Yao Yuan Yao Yangqiu Song PILM 54 25 0 07 Nov 2023
FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models Tao Fan Yan Kang Guoqiang Ma Weijing Chen Wenbin Wei Lixin Fan Qiang Yang 62 65 0 16 Oct 2023
Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuning Liam Collins Shanshan Wu Sewoong Oh K. Sim FedML 79 10 0 06 Oct 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 213 1,781 0 28 Sep 2023
SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models Sara Babakniya A. Elkordy Yahya H. Ezzeldin Qingfeng Liu Kee-Bong Song Mostafa El-Khamy Salman Avestimehr 47 65 0 12 Aug 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 312 4,253 0 09 Jun 2023
Harnessing large-language models to generate private synthetic text Alexey Kurakin Natalia Ponomareva Umar Syed Liam MacDermed Andreas Terzis SILM SyDa 52 41 0 02 Jun 2023
Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models Haonan Duan Adam Dziedzic Nicolas Papernot Franziska Boenisch AAML 37 64 0 24 May 2023
Privacy Implications of Retrieval-Based Language Models Yangsibo Huang Samyak Gupta Zexuan Zhong Keqin Li Danqi Chen RALM 42 30 0 24 May 2023
Selective Pre-training for Private Fine-tuning Da Yu Sivakanth Gopi Janardhan Kulkarni Zinan Lin Saurabh Naik Tomasz Religa Jian Yin Huishuai Zhang 64 19 0 23 May 2023
Can Public Large Language Models Help Private Cross-device Federated Learning? Wei Ping Yibo Jacky Zhang Yuan Cao Yue Liu H. B. McMahan Sewoong Oh Zheng Xu Manzil Zaheer FedML 67 38 0 20 May 2023
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning Mustafa Safa Ozdayi Charith Peris Jack G. M. FitzGerald Christophe Dupuy Jimit Majmudar Haidar Khan Rahil Parikh Rahul Gupta 54 34 0 19 May 2023
Privacy-Preserving Prompt Tuning for Large Language Model Services Yansong Li Zhixing Tan Yang Liu SILM VLM 79 67 0 10 May 2023
Distilling Script Knowledge from Large Language Models for Constrained Language Planning Siyu Yuan Jiangjie Chen Ziquan Fu Xuyang Ge Soham Shah C. R. Jankowski Yanghua Xiao Deqing Yang 63 54 0 09 May 2023
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision Zhiqing Sun Songlin Yang Qinhong Zhou Hongxin Zhang Zhenfang Chen David D. Cox Yiming Yang Chuang Gan SyDa ALM 74 330 0 04 May 2023
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond Jingfeng Yang Hongye Jin Ruixiang Tang Xiaotian Han Qizhang Feng Haoming Jiang Bing Yin Xia Hu LM&MA 165 656 0 26 Apr 2023
Multi-step Jailbreaking Privacy Attacks on ChatGPT Haoran Li Dadi Guo Wei Fan Mingshi Xu Jie Huang Fanpu Meng Yangqiu Song SILM 91 338 0 11 Apr 2023
Generative Agents: Interactive Simulacra of Human Behavior J. Park Joseph C. O'Brien Carrie J. Cai Meredith Ringel Morris Percy Liang Michael S. Bernstein LM&Ro AI4CE 374 1,907 0 07 Apr 2023
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face Yongliang Shen Kaitao Song Xu Tan Dongsheng Li Weiming Lu Yueting Zhuang MLLM 91 884 0 30 Mar 2023
TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs Yaobo Liang Chenfei Wu Ting Song Wenshan Wu Yan Xia ... Shaoguang Mao Yuntao Wang Linjun Shou Ming Gong Nan Duan LLMAG CLL 56 198 0 29 Mar 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.0K 14,179 0 15 Mar 2023
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection Kai Greshake Sahar Abdelnabi Shailesh Mishra C. Endres Thorsten Holz Mario Fritz SILM 115 483 0 23 Feb 2023
A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT Ce Zhou Qian Li Chen Li Jun Yu Yixin Liu ... P. Xie Caiming Xiong Jian Pei Philip S. Yu U. Chicago AI4CE 52 516 0 18 Feb 2023
DP-BART for Privatized Text Rewriting under Local Differential Privacy Timour Igamberdiev Ivan Habernal 69 18 0 15 Feb 2023
GPTScore: Evaluate as You Desire Jinlan Fu See-Kiong Ng Zhengbao Jiang Pengfei Liu LM&MA ALM ELM 117 281 0 08 Feb 2023
Self-Instruct: Aligning Language Models with Self-Generated Instructions Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi Hannaneh Hajishirzi ALM SyDa LRM 94 2,204 0 20 Dec 2022
Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe Xiang Yue Huseyin A. Inan Xuechen Li Girish Kumar Julia McAnallen Hoda Shajari Huan Sun David Levitan Robert Sim 77 84 0 25 Oct 2022
Differentially Private Language Models for Secure Data Sharing Justus Mattern Zhijing Jin Benjamin Weggenmann Bernhard Schoelkopf Mrinmaya Sachan SyDa 54 49 0 25 Oct 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 167 3,110 0 20 Oct 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 488 4,077 0 24 May 2022
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models Denny Zhou Nathanael Scharli Le Hou Jason W. Wei Nathan Scales ... Dale Schuurmans Claire Cui Olivier Bousquet Quoc Le Ed H. Chi RALM LRM AI4CE 70 1,101 0 21 May 2022
Just Fine-tune Twice: Selective Differential Privacy for Large Language Models Weiyan Shi Ryan Shea Si-An Chen Chiyuan Zhang R. Jia Zhou Yu AAML 59 40 0 15 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 754 12,835 0 04 Mar 2022
ZeroGen: Efficient Zero-shot Learning via Dataset Generation Jiacheng Ye Jiahui Gao Qintong Li Hang Xu Jiangtao Feng Zhiyong Wu Tao Yu Lingpeng Kong SyDa 88 220 0 16 Feb 2022
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents Wenlong Huang Pieter Abbeel Deepak Pathak Igor Mordatch LM&Ro 76 1,095 0 18 Jan 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 335 1,696 0 15 Oct 2021
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models Peter West Chandrasekhar Bhagavatula Jack Hessel Jena D. Hwang Liwei Jiang Ronan Le Bras Ximing Lu Sean Welleck Yejin Choi SyDa 89 333 0 14 Oct 2021
Differentially Private Fine-tuning of Language Models Da Yu Saurabh Naik A. Backurs Sivakanth Gopi Huseyin A. Inan ... Y. Lee Andre Manoel Lukas Wutschitz Sergey Yekhanin Huishuai Zhang 207 364 0 13 Oct 2021
Towards a Unified View of Parameter-Efficient Transfer Learning Junxian He Chunting Zhou Xuezhe Ma Taylor Berg-Kirkpatrick Graham Neubig AAML 105 930 0 08 Oct 2021
Opacus: User-Friendly Differential Privacy Library in PyTorch Ashkan Yousefpour I. Shilov Alexandre Sablayrolles Davide Testuggine Karthik Prasad ... Sayan Gosh Akash Bharadwaj Jessica Zhao Graham Cormode Ilya Mironov VLM 264 363 0 25 Sep 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 201 5,454 0 07 Jul 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 371 10,226 0 17 Jun 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 509 4,021 0 18 Apr 2021
Natural Language Understanding with Privacy-Preserving BERT Chen Qu Weize Kong Liu Yang Mingyang Zhang Michael Bendersky Marc Najork 67 74 0 15 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 213 4,238 0 01 Jan 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 602 41,736 0 28 May 2020