LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

21 September 2023

Zhijian Liu

Song Han

Jiaya Jia

ArXiv PDF HTML

Papers citing "LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models"

50 / 124 papers shown

Title
What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices Zhi Chen Qiguang Chen Libo Qin Qipeng Guo Haijun Lv Yicheng Zou Wanxiang Che Hang Yan K. Chen Dahua Lin SyDa 53 4 0 03 Sep 2024
VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation Shiwei Wu Joya Chen Kevin Qinghong Lin Qimeng Wang Yan Gao Qianli Xu Tong Xu Yao Hu Enhong Chen Mike Zheng Shou VLM 52 12 0 29 Aug 2024
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs Yushi Bai Jiajie Zhang Xin Lv Linzhi Zheng Siqi Zhu Lei Hou Yuxiao Dong Jie Tang Juanzi Li VGen LLMAG ALM 42 39 0 13 Aug 2024
MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training Cheng Luo Jiawei Zhao Zhuoming Chen Beidi Chen A. Anandkumar 31 3 0 22 Jul 2024
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference Qichen Fu Minsik Cho Thomas Merth Sachin Mehta Mohammad Rastegari Mahyar Najibi 50 26 0 19 Jul 2024
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities Peng-Tao Xu Ming-Yu Liu Xianchao Wu Zihan Liu M. Shoeybi Mohammad Shoeybi Bryan Catanzaro RALM 52 14 0 19 Jul 2024
DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion Huiguo He Huan Yang Zixi Tuo Yuan Zhou Qiuyue Wang Yuhang Zhang Zeyu Liu Wenhao Huang Hongyang Chao Jian Yin DiffM VGen 62 12 0 17 Jul 2024
Human-like Episodic Memory for Infinite Context LLMs Z. Fountas Martin A Benfeghoul Adnan Oomerjee Fenia Christopoulou Gerasimos Lampouras Haitham Bou-Ammar Jun Wang 31 18 0 12 Jul 2024
SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers Shraman Pramanick Rama Chellappa Subhashini Venugopalan 50 13 0 12 Jul 2024
SBoRA: Low-Rank Adaptation with Regional Weight Updates L. Po Yuyang Liu Haoxuan Wu Tianqi Zhang W. Yu Zeyu Jiang Kun Li 38 1 0 07 Jul 2024
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations Yubo Ma Yuhang Zang Liangyu Chen Meiqi Chen Yizhu Jiao ... Liangming Pan Yu-Gang Jiang Jiaqi Wang Yixin Cao Aixin Sun ELM RALM VLM 36 24 0 01 Jul 2024
Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning Haobo Song Hao Zhao Soumajit Majumder Tao Lin 30 3 0 01 Jul 2024
Revisiting Random Walks for Learning on Graphs Jinwoo Kim Olga Zaghen Ayhan Suleymanzade Youngmin Ryou Seunghoon Hong 62 0 0 01 Jul 2024
UIO-LLMs: Unbiased Incremental Optimization for Long-Context LLMs Wenhao Li Mingbao Lin Mingliang Xu Shuicheng Yan Rongrong Ji 40 0 0 26 Jun 2024
Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Making O. Amujo S. Yang 35 0 0 25 Jun 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 35 18 0 24 Jun 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 77 13 0 20 Jun 2024
Efficient Prompting for LLM-based Generative Internet of Things Bin Xiao B. Kantarci Jiawen Kang Dusit Niyato Mohsen Guizani 35 16 0 14 Jun 2024
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding Xindian Ma Wenyuan Liu Peng Zhang Nan Xu 42 3 0 14 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 74 56 0 11 Jun 2024
SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models Hengyu Zhang RALM 41 2 0 09 Jun 2024
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling Zefan Cai Yichi Zhang Bofei Gao Yuliang Liu Yongqian Li ... Wayne Xiong Yue Dong Baobao Chang Junjie Hu Wen Xiao 62 84 0 04 Jun 2024
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model Chaochen Gao Xing Wu Qingfang Fu Songlin Hu SyDa 34 5 0 30 May 2024
TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations Zhengwu Zhang Yuntong Hu Bo Pan Chen Ling Liang Zhao 41 2 0 27 May 2024
Compressing Lengthy Context With UltraGist Peitian Zhang Zheng Liu Shitao Xiao Ninglu Shao Qiwei Ye Zhicheng Dou 27 4 0 26 May 2024
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models Bernal Jiménez Gutiérrez Yiheng Shu Yu Gu Michihiro Yasunaga Yu-Chuan Su RALM CLL 68 33 0 23 May 2024
From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT Jace Grandinetti R. Mcbeth AI4CE LRM LM&MA 41 0 0 17 May 2024
Long Context Alignment with Short Instructions and Synthesized Positions Wenhao Wu Yizhong Wang Yao Fu Xiang Yue Dawei Zhu Sujian Li SyDa 41 18 0 07 May 2024
Gaussian Stochastic Weight Averaging for Bayesian Low-Rank Adaptation of Large Language Models Emre Onal Klemens Flöge Emma Caldwell A. Sheverdin Vincent Fortuin UQCV BDL 45 9 0 06 May 2024
From Persona to Personalization: A Survey on Role-Playing Language Agents Jiangjie Chen Xintao Wang Rui Xu Siyu Yuan Yikai Zhang ... Caiyu Hu Siye Wu Scott Ren Ziquan Fu Yanghua Xiao 62 77 0 28 Apr 2024
LLoCO: Learning Long Contexts Offline Sijun Tan Xiuyu Li Shishir G. Patil Ziyang Wu Tianjun Zhang Kurt Keutzer Joseph E. Gonzalez Raluca A. Popa RALM OffRL LLMAG 40 6 0 11 Apr 2024
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Tsendsuren Munkhdalai Manaal Faruqui Siddharth Gopal LRM LLMAG CLL 91 102 0 10 Apr 2024
Long-context LLMs Struggle with Long In-context Learning Tianle Li Ge Zhang Quy Duc Do Xiang Yue Wenhu Chen 47 156 0 02 Apr 2024
MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learning Ahmed A. Agiza Marina Neseem Sherief Reda MoE 48 12 0 29 Mar 2024
MLDT: Multi-Level Decomposition for Complex Long-Horizon Robotic Task Planning with Open-Source Large Language Model Yike Wu Jiatao Zhang Nan Hu LanLing Tang Guilin Qi Jun Shao Jie Ren Wei Song 62 10 0 27 Mar 2024
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning Hao Shao Shengju Qian Han Xiao Guanglu Song Zhuofan Zong Letian Wang Yu Liu Hongsheng Li VGen LRM MLLM 66 37 0 25 Mar 2024
CodeS: Natural Language to Code Repository via Multi-Layer Sketch Daoguang Zan Ailun Yu Wei Liu Dong Chen Bo Shen ... Bei Guan Zhiguang Yang Yongji Wang Qianxiang Wang Li-zhen Cui 33 14 0 25 Mar 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 310 0 21 Mar 2024
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses Jia-Nan Li Quan Tu Cunli Mao Zhengtao Yu Ji-Rong Wen Rui Yan OffRL 24 3 0 13 Mar 2024
Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning Yao Liang Yuwei Wang Yang Li Yi Zeng 44 0 0 12 Mar 2024
ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models Jiaxiang Cheng Pan Xie Xin Xia Jiashi Li Jie Wu Yuxi Ren Huixia Li Xuefeng Xiao Min Zheng Lean Fu 38 12 0 04 Mar 2024
DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation Sunghyeon Woo Baeseong Park Byeongwook Kim Minjung Jo S. Kwon Dongsuk Jeon Dongsoo Lee 65 2 0 27 Feb 2024
Me LLaMA: Foundation Large Language Models for Medical Applications Qianqian Xie Qingyu Chen Aokun Chen C.A.I. Peng Yan Hu ... Huan He Lucila Ohno-Machido Yonghui Wu Hua Xu Jiang Bian LM&MA AI4MH 70 4 0 20 Feb 2024
BitDelta: Your Fine-Tune May Only Be Worth One Bit James Liu Guangxuan Xiao Kai Li Jason D. Lee Song Han Tri Dao Tianle Cai 31 21 0 15 Feb 2024
A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts Kuang-Huei Lee Xinyun Chen Hiroki Furuta John F. Canny Ian S. Fischer RALM 55 29 0 15 Feb 2024
The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models Ayo Adedeji Sarita Joshi Brendan Doohan LM&MA 24 14 0 12 Feb 2024
UniMem: Towards a Unified View of Long-Context Large Language Models Junjie Fang Likai Tang Hongzhe Bi Yujia Qin Si Sun ... Xiaodong Shi Sen Song Yankai Lin Zhiyuan Liu Maosong Sun 24 3 0 05 Feb 2024
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization Coleman Hooper Sehoon Kim Hiva Mohammadzadeh Michael W. Mahoney Y. Shao Kurt Keutzer A. Gholami MQ 25 177 0 31 Jan 2024
In-context Learning with Retrieved Demonstrations for Language Models: A Survey an Luo Xin Xu Yue Liu Panupong Pasupat Mehran Kazemi RALM 31 55 0 21 Jan 2024
With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation Y. Wang D. Ma D. Cai RALM 46 18 0 21 Jan 2024