QLoRA: Efficient Finetuning of Quantized LLMs

23 May 2023

Tim Dettmers

Luke Zettlemoyer

Papers citing "QLoRA: Efficient Finetuning of Quantized LLMs"

50 / 407 papers shown

Title
Query-Guided Self-Supervised Summarization of Nursing Notes Ya Gao H. Moen S. Koivusalo M. Koskinen Pekka Marttinen 44 1 0 04 Jul 2024
Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge Young-Jun Lee Dokyong Lee Junyoung Youn Kyeongjin Oh ByungSoo Ko Jonghwan Hyeon Ho-Jin Choi 36 2 0 04 Jul 2024
How Does Quantization Affect Multilingual LLMs? Kelly Marchisio Saurabh Dash Hongyu Chen Dennis Aumiller Ahmet Üstün Sara Hooker Sebastian Ruder MQ 52 8 0 03 Jul 2024
e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Models Jinghui Liu Aaron Nicolson Jason Dowling Bevan Koopman Anthony N. Nguyen 43 5 0 03 Jul 2024
Badllama 3: removing safety finetuning from Llama 3 in minutes Dmitrii Volkov 26 4 0 01 Jul 2024
FAFE: Immune Complex Modeling with Geodesic Distance Loss on Noisy Group Frames R. Wu Ruihan Guo Rui Wang Shitong Luo Yue Xu Jiahan Li Jianzhu Ma Qiang Liu Yunan Luo Jian Peng 39 1 0 01 Jul 2024
Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs Tamzeed Mahfuz Satak Kumar Dey Ruwad Naswan Hasnaen Adil Khondker Salman Sayeed Haz Sameen Shahgir 39 0 0 29 Jun 2024
Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft Chalamalasetti Kranti Sherzod Hakimov David Schlangen 34 2 0 25 Jun 2024
LABOR-LLM: Language-Based Occupational Representations with Large Language Models Tianyu Du Ayush Kanodia Herman Brunborg Keyon Vafa Susan Athey 37 3 0 25 Jun 2024
Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024 Sai Koneru Thai-Binh Nguyen Ngoc-Quan Pham Danni Liu Zhaolin Li Alexander Waibel Jan Niehues OffRL 41 3 0 24 Jun 2024
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation Yuchen Yang Yingdong Shi Cheems Wang Xiantong Zhen Yuxuan Shi Jun Xu 40 1 0 24 Jun 2024
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging Deyuan Liu Zhanyue Qin Han Wang Zhao Yang Zecheng Wang ... Zhao Lv Zhiying Tu Dianhui Chu Bo Li Dianbo Sui 29 2 0 24 Jun 2024
70B-parameter large language models in Japanese medical question-answering Issey Sukeda Risa Kishikawa Satoshi Kodera LM&MA 50 3 0 21 Jun 2024
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning Geewook Kim Minjoon Seo VLM 44 2 0 17 Jun 2024
Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead Rickard Brüel-Gabrielsson Jiacheng Zhu Onkar Bhardwaj Leshem Choshen Kristjan Greenewald Mikhail Yurochkin Justin Solomon 48 5 0 17 Jun 2024
Promoting Data and Model Privacy in Federated Learning through Quantized LoRA Jianhao Zhu Changze Lv Xiaohua Wang Muling Wu Wenhao Liu Tianlong Li Zixuan Ling Cenyuan Zhang Xiaoqing Zheng Xuanjing Huang 44 3 0 16 Jun 2024
ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation Yurun Song Junchen Zhao Ian G. Harris Sangeetha Abdu Jyothi 32 3 0 16 Jun 2024
DIEKAE: Difference Injection for Efficient Knowledge Augmentation and Editing of Large Language Models Alessio Galatolo Meriem Beloucif Katie Winkle 41 0 0 15 Jun 2024
A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations Jinqiang Wang Huansheng Ning Yi Peng Qikai Wei Daniel Tesfai Wenwei Mao Tao Zhu Runhe Huang LM&MA AI4MH ELM 44 5 0 14 Jun 2024
Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning Chung-Ming Chien Andros Tjandra Apoorv Vyas Matt Le Bowen Shi Wei-Ning Hsu 32 0 0 10 Jun 2024
The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMs Mert Yazan Suzan Verberne F. Situmeang MQ 36 3 0 10 Jun 2024
CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning Yibo Yang Xiaojie Li Zhongzhu Zhou Shuaiwen Leon Song Jianlong Wu Liqiang Nie Guohao Li 45 6 0 07 Jun 2024
LLM-based speaker diarization correction: A generalizable approach Georgios Efstathiadis Vijay Yadav Anzar Abbas 45 3 0 07 Jun 2024
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning Naibin Gu Peng Fu Xiyu Liu Bowen Shen Zheng-Shen Lin Weiping Wang 38 6 0 06 Jun 2024
From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation Ali Malik Stephen Mayhew Chris Piech K. Bicknell 34 3 0 05 Jun 2024
Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities Wenyue Hua Kaijie Zhu Lingyao Li Lizhou Fan Shuhang Lin Mingyu Jin Haochen Xue Zelong Li Jindong Wang Yongfeng Zhang LRM 67 9 0 04 Jun 2024
LCQ: Low-Rank Codebook based Quantization for Large Language Models Wen-Pu Cai Wu-Jun Li Wu-Jun Li MQ 46 0 0 31 May 2024
One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models Yutao Zhu Zhaoheng Huang Zhicheng Dou Ji-Rong Wen RALM 56 5 0 30 May 2024
Unlearning Climate Misinformation in Large Language Models Michael Fore Simranjit Singh Chaehong Lee Amritanshu Pandey Antonios Anastasopoulos Dimitrios Stamoulis MU 62 1 0 29 May 2024
4-bit Shampoo for Memory-Efficient Network Training Sike Wang Jia Li Pan Zhou Hua Huang MQ 44 5 0 28 May 2024
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections Roy Miles Pradyumna Reddy Ismail Elezi Jiankang Deng VLM 43 3 0 28 May 2024
I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models Xing Hu Yuan Cheng Dawei Yang Zhihang Yuan Jiangyong Yu Chen Xu Sifan Zhou MQ 36 7 0 28 May 2024
More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs Chengyuan Liu Shihang Wang Yangyang Kang Lizhi Qing Fubang Zhao Changlong Sun Kun Kuang Fei Wu ELM AILaw CLL 27 6 0 28 May 2024
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design Rui Kong Qiyang Li Xinyu Fang Qingtian Feng Qingfeng He Yazhu Dong Weijun Wang Yuanchun Li Linghe Kong Yunxin Liu MoE 40 4 0 28 May 2024
Exploiting the Layered Intrinsic Dimensionality of Deep Models for Practical Adversarial Training Enes Altinisik Safa Messaoud Husrev Taha Sencar Hassan Sajjad Sanjay Chawla AAML 48 0 0 27 May 2024
On Fairness of Low-Rank Adaptation of Large Models Zhoujie Ding Ken Ziyu Liu Pura Peetathawatchai Berivan Isik Sanmi Koyejo 48 4 0 27 May 2024
The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspective Nils Philipp Walter Linara Adilova Jilles Vreeken Michael Kamp AAML 48 2 0 27 May 2024
5W1H Extraction With Large Language Models Yang Cao Yangsong Lan Feiyan Zhai Piji Li 65 1 0 25 May 2024
Towards Completeness-Oriented Tool Retrieval for Large Language Models Changle Qu Sunhao Dai Xiaochi Wei Hengyi Cai Shuaiqiang Wang Dawei Yin Jun Xu Jirong Wen KELM 31 7 0 25 May 2024
Bridging The Gap between Low-rank and Orthogonal Adaptation via Householder Reflection Adaptation Shen Yuan Haotian Liu Hongteng Xu 44 2 0 24 May 2024
Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models Byung-Kwan Lee Chae Won Kim Beomchan Park Yonghyun Ro MLLM LRM 41 18 0 24 May 2024
BiSup: Bidirectional Quantization Error Suppression for Large Language Models Minghui Zou Ronghui Guo Sai Zhang Xiaowang Zhang Zhiyong Feng MQ 36 1 0 24 May 2024
Sparse Matrix in Large Language Model Fine-tuning Haoze He Juncheng Billy Li Xuan Jiang Heather Miller MoE 27 3 0 24 May 2024
Large Language Models Meet NLP: A Survey Libo Qin Qiguang Chen Xiachong Feng Yang Wu Yongheng Zhang Hai-Tao Zheng Min Li Wanxiang Che Philip S. Yu ALM LM&MA ELM LRM 52 48 0 21 May 2024
Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging Xiaobo Liang Haoke Zhang Helan hu Juntao Li Jun Xu Min Zhang ALM 49 2 0 20 May 2024
TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models Junlong Jia Ying Hu Xi Weng Yiming Shi Miao Li ... Baichuan Zhou Ziyu Liu Jie Luo Lei Huang Ji Wu 34 9 0 20 May 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 119 22 0 20 May 2024
Tailoring Vaccine Messaging with Common-Ground Opinions Rickard Stureborg Sanxing Chen Ruoyu Xie Aayushi Patel Christopher Li Chloe Qinyu Zhu Tingnan Hu Jun Yang Bhuwan Dhingra 42 0 0 17 May 2024
Contextual Emotion Recognition using Large Vision Language Models Yasaman Etesam Özge Nilay Yalçin Chuxuan Zhang Angelica Lim VLM 73 3 0 14 May 2024
HoneyBee: A Scalable Modular Framework for Creating Multimodal Oncology Datasets with Foundational Embedding Models Aakash Tripathi Asim Waqas Yasin Yilmaz Ghulam Rasool 36 5 0 13 May 2024