Title
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 173 0 0 31 Mar 2025
CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement Gaifan Zhang Yi Zhou Danushka Bollegala 171 0 0 21 Mar 2025
KV-Distill: Nearly Lossless Learnable Context Compression for LLMs Vivek Chari Guanghui Qin Benjamin Van Durme VLM 76 0 0 13 Mar 2025
Self-Adjust Softmax Chuanyang Zheng Yihang Gao Guoxuan Chen Han Shi Jing Xiong Xiaozhe Ren Chao Huang Xin Jiang Zhiyu Li Yu Li 50 0 0 25 Feb 2025
M2R2: Mixture of Multi-Rate Residuals for Efficient Transformer Inference Nikhil Bhendawade Mahyar Najibi Devang Naik Irina Belousova MoE 85 0 0 04 Feb 2025
MDCure: A Scalable Pipeline for Multi-Document Instruction-Following Gabrielle Kaili-May Liu Bowen Shi Avi Caciularu Idan Szpektor Arman Cohan 72 4 0 30 Oct 2024
Skipping Computations in Multimodal LLMs Mustafa Shukor Matthieu Cord 26 2 0 12 Oct 2024
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Chuanyang Zheng Yihang Gao Han Shi Jing Xiong Jiankai Sun ... Xiaozhe Ren Michael Ng Xin Jiang Zhenguo Li Yu Li 36 2 0 07 Oct 2024
Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression Haowen Hou Fei Ma Binwen Bai Xinxin Zhu Fei Yu 33 0 0 28 Aug 2024
HDT: Hierarchical Document Transformer Haoyu He Markus Flicke Jan Buchmann Iryna Gurevych Andreas Geiger 43 0 0 11 Jul 2024
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules Zhuocheng Gong Ang Lv Jian Guan Junxi Yan Wei Wu Huishuai Zhang Minlie Huang Dongyan Zhao Rui Yan MoE 52 6 0 09 Jul 2024
DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs Zhen Tan Daize Dong Xinyu Zhao Jie Peng Yu Cheng Tianlong Chen MoE 42 4 0 03 Jul 2024
Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformers Chao Lou Zixia Jia Zilong Zheng Kewei Tu ODL 35 19 0 24 Jun 2024
QRMeM: Unleash the Length Limitation through Question then Reflection Memory Mechanism Bo Wang Heyan Huang Yixin Cao Jiahao Ying Wei Tang Chong Feng KELM 45 0 0 19 Jun 2024
Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens Weiyao Luo Suncong Zheng Heming Xia Weikang Wang Yan Lei Tianyu Liu Shuang Chen Zhifang Sui 37 1 0 16 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 49 0 0 03 Jun 2024
Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysis Yao Fu 32 19 0 14 May 2024
Conditional computation in neural networks: principles and research trends Simone Scardapane Alessandro Baiocchi Alessio Devoto V. Marsocci Pasquale Minervini Jary Pomponi 34 1 0 12 Mar 2024
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding Zhuoming Chen Avner May Ruslan Svirschevski Yuhsun Huang Max Ryabinin Zhihao Jia Beidi Chen 40 39 0 19 Feb 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 39 1 0 01 Feb 2024
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval Parth Sarthi Salman Abdullah Aditi Tuli Shubh Khanna Anna Goldie Christopher D. Manning RALM 19 122 0 31 Jan 2024
Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation Zhenyu He Guhao Feng Shengjie Luo Kai-Bo Yang Liwei Wang Jingjing Xu Zhi Zhang Hongxia Yang Di He 32 14 0 29 Jan 2024
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey Saurav Pawar S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Aman Chadha Amitava Das 37 28 0 15 Jan 2024
GRAM: Global Reasoning for Multi-Page VQA Tsachi Blau Sharon Fogel Roi Ronen Alona Golts Roy Ganz Elad Ben Avraham Aviad Aberdam Shahar Tsiper Ron Litman 16 12 0 07 Jan 2024
Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference Bartosz Wójcik Alessio Devoto Karol Pustelnik Pasquale Minervini Simone Scardapane 23 5 0 15 Dec 2023
Learning to Skip for Language Modeling Dewen Zeng Nan Du Tao Wang Yuanzhong Xu Tao Lei Zhifeng Chen Claire Cui 25 11 0 26 Nov 2023
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering Inderjeet Nair Shwetha Somasundaram Apoorv Saxena Koustava Goswami RALM 29 8 0 22 Nov 2023
Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models Simon Stepputtis Joseph Campbell Yaqi Xie Zhengyang Qi W. Zhang Ruiyi Wang Sanketh Rangreji Michael Lewis Katia P. Sycara LLMAG 24 8 0 09 Nov 2023
Optimizing Retrieval-augmented Reader Models via Token Elimination Moshe Berchansky Peter Izsak Avi Caciularu Ido Dagan Moshe Wasserblat RALM 50 12 0 20 Oct 2023
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention Huiyin Xue Nikolaos Aletras 30 0 0 11 Oct 2023
Large Language Models Are Zero-Shot Time Series Forecasters Nate Gruver Marc Finzi Shikai Qiu Andrew Gordon Wilson AI4TS 33 319 0 11 Oct 2023
Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading Howard Chen Ramakanth Pasunuru Jason Weston Asli Celikyilmaz RALM 68 72 0 08 Oct 2023
Dodo: Dynamic Contextual Compression for Decoder-only LMs Guanghui Qin Corby Rosset Ethan C. Chau Nikhil Rao Benjamin Van Durme 19 8 0 03 Oct 2023
LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding Yushi Bai Xin Lv Jiajie Zhang Hong Lyu Jiankai Tang ... Aohan Zeng Lei Hou Yuxiao Dong Jie Tang Juanzi Li LLMAG RALM 31 499 0 28 Aug 2023
Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers Jiawen Xie Pengyu Cheng Xiao Liang Yong Dai Nan Du 40 7 0 25 Aug 2023
Sparks of Large Audio Models: A Survey and Outlook S. Latif Moazzam Shoukat Fahad Shamshad Muhammad Usama Yi Ren ... Wenwu Wang Xulong Zhang Roberto Togneri Min Zhang Björn W. Schuller LM&MA AuLLM 33 38 0 24 Aug 2023
ConcatPlexer: Additional Dim1 Batching for Faster ViTs D. Han Seunghyeon Seo D. Jeon Jiho Jang Chaerin Kong Nojun Kwak ViT MoE 28 0 0 22 Aug 2023
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis Izzeddin Gur Hiroki Furuta Austin Huang Mustafa Safdari Yutaka Matsuo Douglas Eck Aleksandra Faust LM&Ro LLMAG 39 198 0 24 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 22 41 0 12 Jul 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Mian OffRL 70 529 0 12 Jul 2023
Focused Transformer: Contrastive Training for Context Scaling Szymon Tworkowski Konrad Staniszewski Mikolaj Pacek Yuhuai Wu Henryk Michalewski Piotr Milo's 29 136 0 06 Jul 2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens Jiayu Ding Shuming Ma Li Dong Xingxing Zhang Shaohan Huang Wenhui Wang Nanning Zheng Furu Wei CLL 41 151 0 05 Jul 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 14 494 0 27 Jun 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu (Allen) Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 52 254 0 24 Jun 2023
GLIMMER: generalized late-interaction memory reranker Michiel de Jong Yury Zemlyanskiy Nicholas FitzGerald Sumit Sanghai William W. Cohen Joshua Ainslie RALM 49 4 0 17 Jun 2023
RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems Tianyang Liu Canwen Xu Julian McAuley ALM 25 148 0 05 Jun 2023
ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games Ruoyao Wang Graham Todd Xingdi Yuan Ziang Xiao Marc-Alexandre Côté Peter Alexander Jansen LRM 16 12 0 24 May 2023
Adapting Language Models to Compress Contexts Alexis Chevalier Alexander Wettig Anirudh Ajith Danqi Chen LLMAG 16 174 0 24 May 2023
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding Uri Shaham Maor Ivgi Avia Efrat Jonathan Berant Omer Levy VLM 41 126 0 23 May 2023
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models Orevaoghene Ahia Sachin Kumar Hila Gonen Jungo Kasai David R. Mortensen Noah A. Smith Yulia Tsvetkov 51 81 0 23 May 2023