Exploring the Benefits of Training Expert Language Models over Instruction Tuning

7 February 2023

Papers citing "Exploring the Benefits of Training Expert Language Models over Instruction Tuning"

50 / 72 papers shown

Title
LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews Sukannya Purkayastha Zhuang Li Anne Lauscher Lizhen Qu Iryna Gurevych 27 0 0 15 Apr 2025
MASS: MoErging through Adaptive Subspace Selection Donato Crisostomi Alessandro Zirilli Antonio Andrea Gargiulo Maria Sofia Bucarelli Simone Scardapane Fabrizio Silvestri Iacopo Masi Emanuele Rodolà MoMe 40 0 0 06 Apr 2025
RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs Zhongzhan Huang Guoming Ling Vincent S. Liang Yupei Lin Yandong Chen Shanshan Zhong Hefeng Wu Liang Lin LRM 54 2 0 08 Mar 2025
RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis Jianwei Wang Junyao Yang Haoran Li Huiping Zhuang Cen Chen Ziqian Zeng SyDa 44 0 0 23 Feb 2025
A Unified Approach to Routing and Cascading for LLMs Jasper Dekoninck Maximilian Baader Martin Vechev 60 2 0 17 Feb 2025
Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models Yulei Qin Yuncheng Yang Pengcheng Guo Gang Li Hang Shao Yuchen Shi Zihan Xu Yun Gu Ke Li Xing Sun ALM 90 12 0 31 Dec 2024
Smoothie: Label Free Language Model Routing Neel Guha Mayee F. Chen Trevor Chow Ishan S. Khare Christopher Ré 71 4 0 06 Dec 2024
Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs Sumanth Doddapaneni Mohammed Safi Ur Rahman Khan Dilip Venkatesh Raj Dabre Anoop Kunchukuttan Mitesh M. Khapra ELM 35 1 0 17 Oct 2024
Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence Shangbin Feng Zifeng Wang Yike Wang Sayna Ebrahimi Hamid Palangi ... Nathalie Rauschmayr Yejin Choi Yulia Tsvetkov Chen-Yu Lee Tomas Pfister MoMe 35 3 0 15 Oct 2024
Glider: Global and Local Instruction-Driven Expert Router Pingzhi Li Prateek Yadav Jaehong Yoon Jie Peng Yi-Lin Sung Mohit Bansal Tianlong Chen MoMe MoE 27 1 0 09 Oct 2024
KnowledgeSG: Privacy-Preserving Synthetic Text Generation with Knowledge Distillation from Server Wenhao Wang Xiaoyu Liang Rui Ye Jingyi Chai Siheng Chen Yanfeng Wang SyDa 26 3 0 08 Oct 2024
What Matters for Model Merging at Scale? Prateek Yadav Tu Vu Jonathan Lai Alexandra Chronopoulou Manaal Faruqui Mohit Bansal Tsendsuren Munkhdalai MoMe 46 13 0 04 Oct 2024
Exploring the Benefit of Activation Sparsity in Pre-training Zhengyan Zhang Chaojun Xiao Qiujieli Qin Yankai Lin Zhiyuan Zeng Xu Han Zhiyuan Liu Ruobing Xie Maosong Sun Jie Zhou MoE 64 3 0 04 Oct 2024
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models Yuncheng Yang Yulei Qin Tong Wu Zihan Xu Gang Li ... Yuchen Shi Ke Li Xing Sun Jie Yang Yun Gu ALM OffRL MoE 50 0 0 28 Aug 2024
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning Prateek Yadav Colin Raffel Mohammed Muqeeth Lucas Page-Caccia Haokun Liu Tianlong Chen Mohit Bansal Leshem Choshen Alessandro Sordoni MoMe 43 21 0 13 Aug 2024
'Finance Wizard' at the FinLLM Challenge Task: Financial Text Summarization Meisin Lee Soon Lay-Ki 34 2 0 07 Aug 2024
A Survey on Employing Large Language Models for Text-to-SQL Tasks Liang Shi Zhengju Tang Nan Zhang Xiaotong Zhang Zhi Yang 34 19 0 21 Jul 2024
Efficacy of Various Large Language Models in Generating Smart Contracts Siddhartha Chatterjee B. Ramamurthy 26 3 0 28 Jun 2024
Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts Zhaoxuan Tan Zheyuan Liu Meng-Long Jiang 34 20 0 15 Jun 2024
Deep Exploration of Cross-Lingual Zero-Shot Generalization in Instruction Tuning Janghoon Han Changho Lee Joongbo Shin Stanley Jungkyu Choi Honglak Lee Kynghoon Bae ALM 24 0 0 13 Jun 2024
Towards Lifelong Learning of Large Language Models: A Survey Junhao Zheng Shengjie Qiu Chengming Shi Qianli Ma KELM CLL 28 14 0 10 Jun 2024
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models Seungone Kim Juyoung Suk Ji Yong Cho Shayne Longpre Chaeeun Kim ... Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo ELM ALM LM&MA 97 31 0 09 Jun 2024
Towards Modular LLMs by Building and Reusing a Library of LoRAs O. Ostapenko Zhan Su E. Ponti Laurent Charlin Nicolas Le Roux Matheus Pereira Lucas Page-Caccia Alessandro Sordoni MoMe 39 31 0 18 May 2024
Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models Seungone Kim Juyoung Suk Shayne Longpre Bill Yuchen Lin Jamin Shin Sean Welleck Graham Neubig Moontae Lee Kyungjae Lee Minjoon Seo MoMe ALM ELM 49 167 0 02 May 2024
From Matching to Generation: A Survey on Generative Information Retrieval Xiaoxi Li Jiajie Jin Yujia Zhou Yuyao Zhang Peitian Zhang Yutao Zhu Zhicheng Dou 3DV 75 46 0 23 Apr 2024
InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions Yifan Wang Yafei Liu Chufan Shi Haoling Li Chen Chen H. Lu Yujiu Yang CLL 39 25 0 18 Mar 2024
DAM: Dynamic Adapter Merging for Continual Video QA Learning Feng Cheng Ziyang Wang Yi-Lin Sung Yan-Bo Lin Mohit Bansal Gedas Bertasius CLL MoMe 31 10 0 13 Mar 2024
SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models Yu Yang Siddhartha Mishra Jeffrey N Chiang Baharan Mirzasoleiman 34 17 0 12 Mar 2024
Learning or Self-aligning? Rethinking Instruction Fine-tuning Mengjie Ren Boxi Cao Hongyu Lin Liu Cao Xianpei Han Ke Zeng Guanglu Wan Xunliang Cai Le Sun 30 24 0 28 Feb 2024
Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence Weixiang Zhao Zhuojun Li Shilong Wang Yang Wang Yulin Hu Yanyan Zhao Chen Wei Bing Qin 22 4 0 15 Feb 2024
OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning Rui Ye Wenhao Wang Jingyi Chai Dihan Li Zexi Li Yinda Xu Yaxin Du Yanfeng Wang Siheng Chen ALM FedML AIFin 11 76 0 10 Feb 2024
Learning to Route Among Specialized Experts for Zero-Shot Generalization Mohammed Muqeeth Haokun Liu Yufan Liu Colin Raffel MoMe 37 34 0 08 Feb 2024
Continual Learning for Large Language Models: A Survey Tongtong Wu Linhao Luo Yuan-Fang Li Shirui Pan Thuy-Trang Vu Gholamreza Haffari CLL LRM KELM 21 102 0 02 Feb 2024
SelectLLM: Can LLMs Select Important Instructions to Annotate? Long Lei Jaehyung Kim Yueming Jin Dongyeop Kang SyDa 37 10 0 29 Jan 2024
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models Terra Blevins Tomasz Limisiewicz Suchin Gururangan Margaret Li Hila Gonen Noah A. Smith Luke Zettlemoyer 44 22 0 19 Jan 2024
Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Models Alan Chan Ben Bucknall Herbie Bradley David M. Krueger 14 6 0 22 Dec 2023
Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning Yunhao Gou Zhili Liu Kai Chen Lanqing Hong Hang Xu Aoxue Li Dit-Yan Yeung James T. Kwok Yu Zhang MoE MLLM VLM 34 62 0 19 Dec 2023
OrchestraLLM: Efficient Orchestration of Language Models for Dialogue State Tracking Chia-Hsuan Lee Hao Cheng Mari Ostendorf 47 4 0 16 Nov 2023
Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks Po-Nien Kung Fan Yin Di Wu Kai-Wei Chang Nanyun Peng 71 40 0 01 Nov 2023
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging Joel Jang Seungone Kim Bill Yuchen Lin Yizhong Wang Jack Hessel Luke Zettlemoyer Hannaneh Hajishirzi Yejin Choi Prithviraj Ammanabrolu MoMe 43 132 0 17 Oct 2023
Fusing Models with Complementary Expertise Hongyi Wang Felipe Maia Polo Yuekai Sun Souvik Kundu Eric P. Xing Mikhail Yurochkin FedML MoMe 23 26 0 02 Oct 2023
Deep Model Fusion: A Survey Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen FedML MoMe 28 51 0 27 Sep 2023
SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding Tianyu Yu Chengyue Jiang Chao Lou Shen Huang Xiaobin Wang ... Haitao Zheng Ningyu Zhang Pengjun Xie Fei Huang Yong-jia Jiang LRM 57 17 0 21 Aug 2023
A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment Ying Zhao Yu Bowen Binyuan Hui Haiyang Yu Fei Huang Yongbin Li N. Zhang 42 22 0 10 Aug 2023
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore Sewon Min Suchin Gururangan Eric Wallace Hannaneh Hajishirzi Noah A. Smith Luke Zettlemoyer AILaw 22 63 0 08 Aug 2023
An Empirical Study of AI-based Smart Contract Creation Rabimba Karanjai Edward Li Lei Xu W. Shi 16 9 0 05 Aug 2023
UnIVAL: Unified Model for Image, Video, Audio and Language Tasks Mustafa Shukor Corentin Dancette Alexandre Ramé Matthieu Cord MoMe MLLM 61 42 0 30 Jul 2023
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition Chengsong Huang Qian Liu Bill Yuchen Lin Tianyu Pang Chao Du Min-Bin Lin MoMe 36 183 0 25 Jul 2023
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets Seonghyeon Ye Doyoung Kim Sungdong Kim Hyeonbin Hwang Seungone Kim Yongrae Jo James Thorne Juho Kim Minjoon Seo ALM 35 97 0 20 Jul 2023
Derivative Free Weight-space Ensembling Dean Ninalga MoMe 24 0 0 07 Jul 2023