v1v2 (latest)

In-Context Language Learning: Architectures and Algorithms

23 January 2024

Bailin Wang

Papers citing "In-Context Language Learning: Architectures and Algorithms"

32 / 32 papers shown

Title
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning Jingcheng Niu Subhabrata Dutta Ahmed Elshabrawy Harish Tayyar Madabushi Iryna Gurevych 118 1 0 16 May 2025
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models Tyler A. Chang Benjamin Bergen 119 0 0 21 Apr 2025
Yes, Q-learning Helps Offline In-Context RL Denis Tarasov Alexander Nikulin Ilya Zisman Albina Klepach Andrei Polubarov Nikita Lyubaykin Alexander Derevyagin Igor Kiselev Vladislav Kurenkov OffRL OnRL 472 3 0 24 Feb 2025
From Markov to Laplace: How Mamba In-Context Learns Markov Chains Marco Bondaschi Nived Rajaraman Xiuying Wei Kannan Ramchandran Razvan Pascanu Çağlar Gülçehre Michael C. Gastpar Ashok Vardhan Makkuva 140 0 0 17 Feb 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 145 10 0 31 Dec 2024
Training Neural Networks as Recognizers of Formal Languages Alexandra Butoi Ghazal Khalighinejad Anej Svete Josef Valvoda Ryan Cotterell Brian DuSell NAI 96 7 0 11 Nov 2024
N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs Ilya Zisman Alexander Nikulin Andrei Polubarov Nikita Lyubaykin Vladislav Kurenkov Andrei Polubarov Igor Kiselev Vladislav Kurenkov OffRL 114 2 0 04 Nov 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 126 1 0 15 Jul 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 154 69 0 11 Jun 2024
Memory Mosaics Jianyu Zhang Niklas Nolte Ranajoy Sadhukhan Beidi Chen Léon Bottou VLM 105 4 0 10 May 2024
Gated Linear Attention Transformers with Hardware-Efficient Training Aaron Courville Bailin Wang Songlin Yang Yikang Shen Yoon Kim 106 180 0 11 Dec 2023
Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions S. Bhattamishra Arkil Patel Phil Blunsom Varun Kanade 86 55 0 04 Oct 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 233 607 0 22 May 2023
A Theory of Emergent In-Context Learning as Implicit Structure Induction Michael Hahn Navin Goyal LRM 60 86 0 14 Mar 2023
Hyena Hierarchy: Towards Larger Convolutional Language Models Michael Poli Stefano Massaroli Eric Q. Nguyen Daniel Y. Fu Tri Dao S. Baccus Yoshua Bengio Stefano Ermon Christopher Ré VLM 119 307 0 21 Feb 2023
Transformers learn in-context by gradient descent J. Oswald Eyvind Niklasson E. Randazzo João Sacramento A. Mordvintsev A. Zhmoginov Max Vladymyrov MLT 116 494 0 15 Dec 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 319 525 0 24 Sep 2022
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes Shivam Garg Dimitris Tsipras Percy Liang Gregory Valiant 141 513 0 01 Aug 2022
The Parallelism Tradeoff: Limitations of Log-Precision Transformers William Merrill Ashish Sabharwal 78 114 0 02 Jul 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 107 242 0 27 Jun 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 89 324 0 23 Jun 2022
Data Distributional Properties Drive Emergent In-Context Learning in Transformers Stephanie C. Y. Chan Adam Santoro Andrew Kyle Lampinen Jane X. Wang Aaditya K. Singh Pierre Harvey Richemond J. Mcclelland Felix Hill 157 265 0 22 Apr 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 126 231 0 21 Feb 2022
Few-Shot Semantic Parsing with Language Models Trained On Code Richard Shin Benjamin Van Durme 63 66 0 16 Dec 2021
An Explanation of In-context Learning as Implicit Bayesian Inference Sang Michael Xie Aditi Raghunathan Percy Liang Tengyu Ma ReLM BDL VPVLM LRM 213 763 0 03 Nov 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 284 2,521 0 20 Apr 2021
An Interpretability Illusion for BERT Tolga Bolukbasi Adam Pearce Ann Yuan Andy Coenen Emily Reif Fernanda Viégas Martin Wattenberg MILM FAtt 77 80 0 14 Apr 2021
Probing Classifiers: Promises, Shortcomings, and Advances Yonatan Belinkov 288 453 0 24 Feb 2021
RNNs can generate bounded hierarchical languages with optimal memory John Hewitt Michael Hahn Surya Ganguli Percy Liang Christopher D. Manning LRM 51 54 0 15 Oct 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 201 1,786 0 29 Jun 2020
On the Linguistic Capacity of Real-Time Counter Automata William Merrill 41 23 0 15 Apr 2020
Memory-Augmented Recurrent Neural Networks Can Learn Generalized Dyck Languages Mirac Suzgun Sebastian Gehrmann Yonatan Belinkov Stuart M. Shieber 56 50 0 08 Nov 2019