Efficient Long-Text Understanding with Short-Text Models

1 August 2022

Papers citing "Efficient Long-Text Understanding with Short-Text Models"

50 / 52 papers shown

Title
MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learning Murtadha Ahmed Wenbo Liu yunfeng 41 0 0 02 May 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 160 1 0 03 Apr 2025
Resona: Improving Context Copying in Linear Recurrence Models with Retrieval Xinyu Wang Linrui Ma Jerry Huang Peng Lu Prasanna Parthasarathi Xiao-Wen Chang Boxing Chen Yufei Cui KELM 45 1 0 28 Mar 2025
An Attempt to Unraveling Token Prediction Refinement and Identifying Essential Layers of Large Language Models Jaturong Kongmanee 39 1 0 28 Jan 2025
Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks Hamed Firooz Maziar Sanjabi Wenlong Jiang Xiaoling Zhai 68 3 0 03 Jan 2025
Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator Frederic Kirstein Terry Ruas Bela Gipp 85 2 0 27 Nov 2024
LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems Nan Xu Xuezhe Ma LRM 59 3 0 18 Oct 2024
ChuLo: Chunk-Level Key Information Representation for Long Document Processing Yan Li Soyeon Caren Han Yue Dai Feiqi Cao 28 0 0 14 Oct 2024
LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English T. Y. S. S. Santosh Cornelius Weiss Matthias Grabmair AILaw ELM 49 2 0 12 Oct 2024
On The Adaptation of Unlimiformer for Decoder-Only Transformers Kian Ahrabian Alon Benhaim Barun Patra Jay Pujara Saksham Singhal Xia Song 38 0 0 02 Oct 2024
Writing in the Margins: Better Inference Pattern for Long Context Retrieval M. Russak Umar Jamil Christopher Bryant Kiran Kamble Axel Magnuson Mateusz Russak Waseem Alshikh 27 2 0 27 Aug 2024
Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models Amey Hengle Prasoon Bajpai Soham Dan Tanmoy Chakraborty LRM 29 2 0 19 Aug 2024
Arctic-TILT. Business Document Understanding at Sub-Billion Scale Łukasz Borchmann Michał Pietruszka Wojciech Ja'skowski Dawid Jurkiewicz Piotr Halama ... Gabriela Nowakowska Artur Zawłocki Łukasz Duhr Paweł Dyda Michał Turski VLM 34 1 0 08 Aug 2024
BLAZE: Cross-Language and Cross-Project Bug Localization via Dynamic Chunking and Hard Example Learning Partha Chakraborty Mahmoud Alfadel Mei Nagappan 25 2 0 24 Jul 2024
When Can Transformers Count to n? Gilad Yehudai Haim Kaplan Asma Ghandeharioun Mor Geva Amir Globerson 39 11 0 21 Jul 2024
Human-like Episodic Memory for Infinite Context LLMs Z. Fountas Martin A Benfeghoul Adnan Oomerjee Fenia Christopoulou Gerasimos Lampouras Haitham Bou-Ammar Jun Wang 31 18 0 12 Jul 2024
Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP Omer Goldman Alon Jacovi Aviv Slobodkin Aviya Maimon Ido Dagan Reut Tsarfaty 64 11 0 29 Jun 2024
Position Debiasing Fine-Tuning for Causal Perception in Long-Term Dialogue Shixuan Fan Wei Wei Wendi Li Xian-Ling Mao Wenfeng Xie Dangyang Chen 101 1 0 04 Jun 2024
Equipping Transformer with Random-Access Reading for Long-Context Understanding Chenghao Yang Zi Yang Nan Hua 32 1 0 21 May 2024
Improving Long Text Understanding with Knowledge Distilled from Summarization Model Yan Liu Yazheng Yang Xiaokang Chen VLM RALM 35 1 0 08 May 2024
In-Context Learning with Long-Context Models: An In-Depth Exploration Amanda Bertsch Maor Ivgi Uri Alon Jonathan Berant Matthew R. Gormley Matthew R. Gormley Graham Neubig ReLM AIMat 93 64 0 30 Apr 2024
Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs Woomin Song Seunghyuk Oh Sangwoo Mo Jaehyung Kim Sukmin Yun Jung-Woo Ha Jinwoo Shin 30 14 0 16 Apr 2024
Select and Summarize: Scene Saliency for Movie Script Summarization Rohit Saxena Frank Keller 27 2 0 04 Apr 2024
LexAbSumm: Aspect-based Summarization of Legal Decisions Santosh T.Y.S.S Mahmoud Aly Matthias Grabmair AILaw ELM 30 6 0 31 Mar 2024
Naive Bayes-based Context Extension for Large Language Models Jianlin Su Murtadha Ahmed Wenbo Luo Abhishek Rao Denny Zhou Hyeontaek Lim 29 5 0 26 Mar 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu (Allen) Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 28 26 0 05 Mar 2024
Improving Legal Judgement Prediction in Romanian with Long Text Encoders Mihai Masala Traian Rebedea Horia Velicu AILaw 43 2 0 29 Feb 2024
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents Tamara Czinczoll Christoph Hones Maximilian Schall Gerard de Melo 35 2 0 27 Feb 2024
Long-Context Language Modeling with Parallel Context Encoding Howard Yen Tianyu Gao Danqi Chen 33 43 0 26 Feb 2024
BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models Kun Luo Zheng Liu Shitao Xiao Kang Liu 39 11 0 18 Feb 2024
Document Structure in Long Document Transformers Jan Buchmann Max Eichler Jan-Micha Bodensohn Ilia Kuznetsov Iryna Gurevych 13 2 0 31 Jan 2024
Large Language Models for Social Networks: Applications, Challenges, and Solutions Jingying Zeng Richard Huang Waleed Malik Langxuan Yin Bojan Babic Danny Shacham Xiao Yan Jaewon Yang Qi He 22 7 0 04 Jan 2024
On the Long Range Abilities of Transformers Itamar Zimerman Lior Wolf 27 7 0 28 Nov 2023
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen ... Xiaoxing Ma Lijuan Yang Zhou Xin Shupeng Li Penghao Zhao LLMAG KELM 36 54 0 21 Nov 2023
Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention Ziwei He Jian Yuan Le Zhou Jingwen Leng Bo Jiang 32 1 0 13 Nov 2023
Guess & Sketch: Language Model Guided Transpilation Celine Lee Abdulrahman Mahmoud Michal Kurek Simone Campanoni David Brooks Stephen Chong Gu-Yeon Wei Alexander M. Rush 39 5 0 25 Sep 2023
Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformers Jiawen Xie Pengyu Cheng Xiao Liang Yong Dai Nan Du 40 7 0 25 Aug 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 40 1,404 0 06 Jul 2023
Long-range Language Modeling with Self-retrieval Ohad Rubin Jonathan Berant RALM KELM 19 18 0 23 Jun 2023
Slide, Constrain, Parse, Repeat: Synchronous SlidingWindows for Document AMR Parsing Sadhana Kumaravel Tahira Naseem Ramón Fernández Astudillo Radu Florian Salim Roukos 49 0 0 26 May 2023
Focus Your Attention (with Adaptive IIR Filters) Shahar Lutati Itamar Zimerman Lior Wolf 32 9 0 24 May 2023
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding Uri Shaham Maor Ivgi Avia Efrat Jonathan Berant Omer Levy VLM 35 126 0 23 May 2023
Unlimiformer: Long-Range Transformers with Unlimited Length Input Amanda Bertsch Uri Alon Graham Neubig Matthew R. Gormley RALM 96 122 0 02 May 2023
A Survey on Long Text Modeling with Transformers Zican Dong Tianyi Tang Lunyi Li Wayne Xin Zhao VLM 21 54 0 28 Feb 2023
LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain Joel Niklaus Veton Matoshi Pooja Rani Andrea Galassi Matthias Sturmer Ilias Chalkidis ELM AILaw 19 55 0 30 Jan 2023
Parallel Context Windows for Large Language Models Nir Ratner Yoav Levine Yonatan Belinkov Ori Ram Inbal Magar Omri Abend Ehud D. Karpas Amnon Shashua Kevin Leyton-Brown Y. Shoham RALM 17 69 0 21 Dec 2022
CAB: Comprehensive Attention Benchmarking on Long Sequence Modeling Jinchao Zhang Shuyang Jiang Jiangtao Feng Lin Zheng Lingpeng Kong 3DV 43 9 0 14 Oct 2022
Investigating Efficiently Extending Transformers for Long Input Summarization Jason Phang Yao-Min Zhao Peter J. Liu RALM LLMAG 29 63 0 08 Aug 2022
Modeling Multi-hop Question Answering as Single Sequence Prediction Semih Yavuz Kazuma Hashimoto Yingbo Zhou N. Keskar Caiming Xiong 43 27 0 18 May 2022
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts Yuta Koreeda Christopher D. Manning AILaw 94 96 0 05 Oct 2021