On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion

17 June 2024

Wei Wei

Xiaoye Qu

Papers citing "On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion"

50 / 50 papers shown

Title
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment Chenghao Fan Zhenyi Lu Sichen Liu Xiaoye Qu Xiaoye Qu Wei Wei Yu Cheng MoE 444 1 0 24 Feb 2025
ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM Zhaochen Su Jun Zhang Xiaoye Qu Tong Zhu Yanshu Li Jiashuo Sun Juntao Li Min Zhang Yu Cheng 16 30 0 22 Aug 2024
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training Tong Zhu Xiaoye Qu Daize Dong Jiacheng Ruan Jingqi Tong Conghui He Yu Cheng MoE ALM 71 82 0 24 Jun 2024
Timo: Towards Better Temporal Reasoning for Language Models Zhaochen Su Jun Zhang Tong Zhu Xiaoye Qu Juntao Li Min Zhang Yu Cheng LRM 66 22 0 20 Jun 2024
Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts Tong Zhu Daize Dong Xiaoye Qu Jiacheng Ruan Wenliang Chen Yu Cheng MoE 69 9 0 17 Jun 2024
Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging Zhenyi Lu Chenghao Fan Wei Wei Xiaoye Qu Dangyang Chen Yu Cheng MoMe 87 60 0 17 Jun 2024
Living in the Moment: Can Large Language Models Grasp Co-Temporal Reasoning? Zhaochen Su Juntao Li Jun Zhang Tong Zhu Xiaoye Qu Pan Zhou Yan Bowen Yu Cheng Min zhang LRM 81 23 0 13 Jun 2024
Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models Zhenyi Lu Jie Tian Wei Wei Xiaoye Qu Yu Cheng Wenfeng Xie Dangyang Chen 52 6 0 11 Jun 2024
Learning to Decode Collaboratively with Multiple Language Models Zejiang Shen Hunter Lang Bailin Wang Yoon Kim David Sontag 67 33 0 06 Mar 2024
LoRA+: Efficient Low Rank Adaptation of Large Models Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 81 173 0 19 Feb 2024
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong ... Yu-Huan Wu Yiming Li Fuli Luo Yingfei Xiong W. Liang ELM 109 753 0 25 Jan 2024
Tuning Language Models by Proxy Alisa Liu Xiaochuang Han Yizhong Wang Yulia Tsvetkov Yejin Choi Noah A. Smith ALM 59 48 0 16 Jan 2024
TinyLlama: An Open-Source Small Language Model Peiyuan Zhang Guangtao Zeng Tianduo Wang Wei Lu ALM LRM 126 389 0 04 Jan 2024
Enhancing Low-Resource Relation Representations through Multi-View Decoupling Chenghao Fan Wei Wei Xiaoye Qu Zhenyi Lu Wenfeng Xie Yu Cheng Dangyang Chen 54 4 0 26 Dec 2023
An Emulator for Fine-Tuning Large Language Models using Small Language Models Eric Mitchell Rafael Rafailov Archit Sharma Chelsea Finn Christopher D. Manning ALM 75 55 0 19 Oct 2023
VeRA: Vector-based Random Matrix Adaptation D. J. Kopiczko Tijmen Blankevoort Yuki Markus Asano VLM 74 155 0 17 Oct 2023
AdaMerging: Adaptive Model Merging for Multi-Task Learning Enneng Yang Zhenyi Wang Li Shen Shiwei Liu Guibing Guo Xingwei Wang Dacheng Tao MoMe 59 115 0 04 Oct 2023
Fusing Models with Complementary Expertise Hongyi Wang Felipe Maia Polo Yuekai Sun Souvik Kundu Eric Xing Mikhail Yurochkin FedML MoMe 69 29 0 02 Oct 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 267 11,791 0 18 Jul 2023
TIES-Merging: Resolving Interference When Merging Models Prateek Yadav Derek Tam Leshem Choshen Colin Raffel Joey Tianyi Zhou MoMe 111 293 0 02 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 313 3,895 0 29 May 2023
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Zeyu Wang ... Chunan Shi Zhuoming Chen Daiyaan Arfeen Reyna Abhyankar Zhihao Jia LRM 93 143 0 16 May 2023
Toolformer: Language Models Can Teach Themselves to Use Tools Timo Schick Jane Dwivedi-Yu Roberto Dessì Roberta Raileanu Maria Lomeli Luke Zettlemoyer Nicola Cancedda Thomas Scialom SyDa RALM 129 1,719 0 09 Feb 2023
Dataless Knowledge Fusion by Merging Weights of Language Models Xisen Jin Xiang Ren Daniel Preoţiuc-Pietro Pengxiang Cheng FedML MoMe 57 236 0 19 Dec 2022
Editing Models with Task Arithmetic Gabriel Ilharco Marco Tulio Ribeiro Mitchell Wortsman Suchin Gururangan Ludwig Schmidt Hannaneh Hajishirzi Ali Farhadi KELM MoMe MU 173 486 0 08 Dec 2022
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 100 701 0 30 Nov 2022
Contrastive Decoding: Open-ended Text Generation as Optimization Xiang Lisa Li Ari Holtzman Daniel Fried Percy Liang Jason Eisner Tatsunori Hashimoto Luke Zettlemoyer M. Lewis 89 358 0 27 Oct 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 167 3,110 0 20 Oct 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 282 328 0 11 Sep 2022
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time Mitchell Wortsman Gabriel Ilharco S. Gadre Rebecca Roelofs Raphael Gontijo-Lopes ... Hongseok Namkoong Ali Farhadi Y. Carmon Simon Kornblith Ludwig Schmidt MoMe 116 976 1 10 Mar 2022
Merging Models with Fisher-Weighted Averaging Michael Matena Colin Raffel FedML MoMe 85 389 0 18 Nov 2021
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 227 4,354 0 27 Oct 2021
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 125 1,881 0 08 Sep 2021
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Elad Ben-Zaken Shauli Ravfogel Yoav Goldberg 158 1,216 0 18 Jun 2021
Does Knowledge Distillation Really Work? Samuel Stanton Pavel Izmailov Polina Kirichenko Alexander A. Alemi A. Wilson FedML 57 220 0 10 Jun 2021
Knowledge distillation: A good teacher is patient and consistent Lucas Beyer Xiaohua Zhai Amelie Royer L. Markeeva Rohan Anil Alexander Kolesnikov VLM 103 295 0 09 Jun 2021
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks Rabeeh Karimi Mahabadi Sebastian Ruder Mostafa Dehghani James Henderson MoE 66 307 0 08 Jun 2021
Carbon Emissions and Large Neural Network Training David A. Patterson Joseph E. Gonzalez Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 331 667 0 21 Apr 2021
Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy Lucas Liebenwein Cenk Baykal Brandon Carter David K Gifford Daniela Rus AAML 51 73 0 04 Mar 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 215 4,244 0 01 Jan 2021
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika D. Song Jacob Steinhardt ELM RALM 157 4,377 0 07 Sep 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 702 41,736 0 28 May 2020
Versatile Black-Box Optimization Jialin Liu A. Moreau Mike Preuss Baptiste Roziere Jérémy Rapin F. Teytaud O. Teytaud 28 38 0 29 Apr 2020
Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS Petro B. Liashchynskyi Pavlo Liashchynskyi 43 541 0 12 Dec 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 381 20,053 0 23 Oct 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 208 4,439 0 02 Feb 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.6K 94,511 0 11 Oct 2018
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 195 2,643 0 09 May 2017
Get To The Point: Summarization with Pointer-Generator Networks A. See Peter J. Liu Christopher D. Manning 3DPC 265 4,014 0 14 Apr 2017
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 322 19,609 0 09 Mar 2015