v1v2 (latest)

In-Context Learning with Long-Context Models: An In-Depth Exploration

30 April 2024

Graham Neubig

Papers citing "In-Context Learning with Long-Context Models: An In-Depth Exploration"

50 / 102 papers shown

Title
Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP Omer Goldman Alon Jacovi Aviv Slobodkin Aviya Maimon Ido Dagan Reut Tsarfaty 133 11 0 29 Jun 2024
Can we teach language models to gloss endangered languages? Michael Ginn Mans Hulden Alexis Palmer 112 7 0 27 Jun 2024
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning Brandon Huang Chancharik Mitra Assaf Arbelle Leonid Karlinsky Trevor Darrell Roei Herzig 101 21 0 21 Jun 2024
When Parts are Greater Than Sums: Individual LLM Components Can Outperform Full Models Ting-Yun Chang Jesse Thomason Robin Jia 115 5 0 19 Jun 2024
Probing the Decision Boundaries of In-context Learning in Large Language Models Siyan Zhao Tung Nguyen Aditya Grover 132 7 0 17 Jun 2024
Is In-Context Learning Sufficient for Instruction Following in LLMs? Hao Zhao Maksym Andriushchenko Francesco Croce Nicolas Flammarion 135 14 0 30 May 2024
Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words? G. Yona Roee Aharoni Mor Geva HILM 101 32 0 27 May 2024
Many-Shot In-Context Learning in Multimodal Foundation Models Yixing Jiang Jeremy Irvin Ji Hun Wang Muhammad Ahmed Chaudhry Jonathan H. Chen Andrew Y. Ng VLM 122 34 0 16 May 2024
Many-Shot In-Context Learning Rishabh Agarwal Avi Singh Lei M. Zhang Bernd Bohnet Luis Rosias ... John D. Co-Reyes Eric Chu Feryal M. P. Behbahani Aleksandra Faust Hugo Larochelle ReLM OffRL BDL 153 121 0 17 Apr 2024
Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs Woomin Song Seunghyuk Oh Sangwoo Mo Jaehyung Kim Sukmin Yun Jung-Woo Ha Jinwoo Shin 79 21 0 16 Apr 2024
Long-Context Language Modeling with Parallel Context Encoding Howard Yen Tianyu Gao Danqi Chen 95 50 0 26 Feb 2024
Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models Mosh Levy Alon Jacoby Yoav Goldberg 136 89 0 19 Feb 2024
Data Engineering for Scaling Language Models to 128K Context Yao Fu Yikang Shen Xinyao Niu Xiang Yue Hanna Hajishirzi Yoon Kim Hao-Chun Peng MoE 119 145 0 15 Feb 2024
In-context Learning with Retrieved Demonstrations for Language Models: A Survey an Luo Xin Xu Yue Liu Panupong Pasupat Mehran Kazemi RALM 164 70 0 21 Jan 2024
A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA Damjan Kalajdzievski ALM 91 103 0 28 Nov 2023
LooGLE: Can Long-Context Language Models Understand Long Contexts? Jiaqi Li Mengmeng Wang Zilong Zheng Muhan Zhang ELM RALM 100 134 0 08 Nov 2023
In-Context Learning Creates Task Vectors Roee Hendel Mor Geva Amir Globerson 121 168 0 24 Oct 2023
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting Melanie Sclar Yejin Choi Yulia Tsvetkov Alane Suhr 119 362 0 17 Oct 2023
Ring Attention with Blockwise Transformers for Near-Infinite Context Hao Liu Matei A. Zaharia Pieter Abbeel 133 258 0 03 Oct 2023
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 180 791 0 29 Sep 2023
In-Context Learning for Text Classification with Many Labels Aristides Milios Siva Reddy Dzmitry Bahdanau 81 37 0 19 Sep 2023
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 181 89 0 19 Sep 2023
LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models Chi Han Qifan Wang Hao Peng Wenhan Xiong Yu Chen Heng Ji Sinong Wang 161 61 0 30 Aug 2023
Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps Fuxiao Liu Paiheng Xu Zongxi Li Yue Feng Hyemi Song 118 35 0 11 Jul 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 142 1,664 0 06 Jul 2023
Focused Transformer: Contrastive Training for Context Scaling Szymon Tworkowski Konrad Staniszewski Mikolaj Pacek Yuhuai Wu Henryk Michalewski Piotr Milo's 83 141 0 06 Jul 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 204 544 0 27 Jun 2023
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation Marius Mosbach Tiago Pimentel Shauli Ravfogel Dietrich Klakow Yanai Elazar 112 135 0 26 May 2023
Coverage-based Example Selection for In-Context Learning Shivanshu Gupta Matt Gardner Sameer Singh 115 49 0 24 May 2023
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer Akari Asai Sneha Kudugunta Xinyan Velocity Yu Terra Blevins Hila Gonen Machel Reid Yulia Tsvetkov Sebastian Ruder Hannaneh Hajishirzi 122 63 0 24 May 2023
What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning Jane Pan Tianyu Gao Howard Chen Danqi Chen 84 128 0 16 May 2023
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers L. Yu Daniel Simig Colin Flaherty Armen Aghajanyan Luke Zettlemoyer M. Lewis 127 93 0 12 May 2023
Unlimiformer: Long-Range Transformers with Unlimited Length Input Amanda Bertsch Uri Alon Graham Neubig Matthew R. Gormley RALM 211 130 0 02 May 2023
Emergent and Predictable Memorization in Large Language Models Stella Biderman USVSN Sai Prashanth Lintang Sutawika Hailey Schoelkopf Quentin G. Anthony Shivanshu Purohit Edward Raf 94 125 0 21 Apr 2023
A Survey on In-context Learning Qingxiu Dong Lei Li Damai Dai Ce Zheng Jingyuan Ma ... Zhiyong Wu Baobao Chang Xu Sun Lei Li Zhifang Sui ReLM AIMat 161 547 0 31 Dec 2022
Parallel Context Windows for Large Language Models Nir Ratner Yoav Levine Yonatan Belinkov Ori Ram Inbal Magar Omri Abend Ehud D. Karpas Amnon Shashua Kevin Leyton-Brown Y. Shoham RALM 126 75 0 21 Dec 2022
Transformers learn in-context by gradient descent J. Oswald Eyvind Niklasson E. Randazzo João Sacramento A. Mordvintsev A. Zhmoginov Max Vladymyrov MLT 160 497 0 15 Dec 2022
Diverse Demonstrations Improve In-context Compositional Generalization Itay Levy Ben Bogin Jonathan Berant 113 146 0 13 Dec 2022
Structured Prompting: Scaling In-Context Learning to 1,000 Examples Y. Hao Yutao Sun Li Dong Zhixiong Han Yuxian Gu Furu Wei LRM 76 75 0 13 Dec 2022
Efficient Long-Text Understanding with Short-Text Models Maor Ivgi Uri Shaham Jonathan Berant VLM 132 84 0 01 Aug 2022
Prototypical Calibration for Few-shot Learning of Language Models Zhixiong Han Y. Hao Li Dong Yutao Sun Furu Wei 268 56 0 20 May 2022
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning Haokun Liu Derek Tam Mohammed Muqeeth Jay Mohta Tenghao Huang Joey Tianyi Zhou Colin Raffel 126 944 0 11 May 2022
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? Sewon Min Xinxi Lyu Ari Holtzman Mikel Artetxe M. Lewis Hannaneh Hajishirzi Luke Zettlemoyer LLMAG LRM 200 1,507 0 25 Feb 2022
MetaICL: Learning to Learn In Context Sewon Min M. Lewis Luke Zettlemoyer Hannaneh Hajishirzi LRM 258 493 0 29 Oct 2021
True Few-Shot Learning with Language Models Ethan Perez Douwe Kiela Kyunghyun Cho 163 440 0 24 May 2021
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity Yao Lu Max Bartolo Alastair Moore Sebastian Riedel Pontus Stenetorp AILaw LRM 466 1,200 0 18 Apr 2021
Efficient Intent Detection with Dual Sentence Encoders I. Casanueva Tadas Temvcinas D. Gerz Matthew Henderson Ivan Vulić VLM 379 481 0 10 Mar 2020
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization Bogdan Gliwa Iwona Mochol M. Biesek A. Wawer 172 640 0 27 Nov 2019
An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction Stefan Larson Anish Mahendran Joseph Peper Christopher Clarke Andrew Lee ... Jonathan K. Kummerfeld Kevin Leach M. Laurenzano Lingjia Tang Jason Mars 147 534 0 04 Sep 2019
BERTScore: Evaluating Text Generation with BERT Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger Yoav Artzi 705 5,897 0 21 Apr 2019