Investigating Recurrent Transformers with Dynamic Halt

1 February 2024

Jishnu Ray Chowdhury

Cornelia Caragea

Papers citing "Investigating Recurrent Transformers with Dynamic Halt"

50 / 86 papers shown

Title
A Transformer with Stack Attention Jiaoda Li Jennifer C. White Mrinmaya Sachan Ryan Cotterell 37 2 0 07 May 2024
TransformerFAM: Feedback attention is working memory Dongseong Hwang Weiran Wang Zhuoyuan Huo K. Sim P. M. Mengibar 70 12 0 14 Apr 2024
The Illusion of State in State-Space Models William Merrill Jackson Petty Ashish Sabharwal 67 54 0 12 Apr 2024
HGRN2: Gated Linear RNNs with State Expansion Zhen Qin Aaron Courville Weixuan Sun Xuyang Shen Dong Li Weigao Sun Yiran Zhong LRM 65 51 0 11 Apr 2024
Jamba: A Hybrid Transformer-Mamba Language Model Opher Lieber Barak Lenz Hofit Bata Gal Cohen Jhonathan Osin ... Nir Ratner N. Rozen Erez Shwartz Mor Zusman Y. Shoham 74 218 0 28 Mar 2024
Gated Linear Attention Transformers with Hardware-Efficient Training Aaron Courville Bailin Wang Songlin Yang Yikang Shen Yoon Kim 69 167 0 11 Dec 2023
Mamba: Linear-Time Sequence Modeling with Selective State Spaces Albert Gu Tri Dao Mamba 124 2,636 0 01 Dec 2023
On the Long Range Abilities of Transformers Itamar Zimerman Lior Wolf 51 8 0 28 Nov 2023
Hierarchically Gated Recurrent Neural Network for Sequence Modeling Zhen Qin Aaron Courville Yiran Zhong 41 77 0 08 Nov 2023
Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalability Jishnu Ray Chowdhury Cornelia Caragea 42 5 0 08 Nov 2023
What Algorithms can Transformers Learn? A Study in Length Generalization Hattie Zhou Arwen Bradley Etai Littwin Noam Razin Omid Saremi Josh Susskind Samy Bengio Preetum Nakkiran 63 121 0 24 Oct 2023
The Expressive Power of Transformers with Chain of Thought William Merrill Ashish Sabharwal LRM AI4CE ReLM 55 41 0 11 Oct 2023
Sparse Universal Transformer Shawn Tan Songlin Yang Zhenfang Chen Aaron Courville Chuang Gan MoE 56 13 0 11 Oct 2023
Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns Brian DuSell David Chiang 61 12 0 03 Oct 2023
Efficient Beam Tree Recursion Jishnu Ray Chowdhury Cornelia Caragea 47 3 0 20 Jul 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 105 1,269 0 17 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 101 328 0 17 Jul 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 78 14 0 19 Jun 2023
Block-State Transformers Mahan Fathi Jonathan Pilault Orhan Firat C. Pal Pierre-Luc Bacon Ross Goroshin 62 17 0 15 Jun 2023
Exposing Attention Glitches with Flip-Flop Language Modeling Bingbin Liu Jordan T. Ash Surbhi Goel A. Krishnamurthy Cyril Zhang LRM 57 51 0 01 Jun 2023
Beam Tree Recursive Cells Jishnu Ray Chowdhury Cornelia Caragea 52 6 0 31 May 2023
Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective Guhao Feng Bohang Zhang Yuntian Gu Haotian Ye Di He Liwei Wang LRM 82 245 0 24 May 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 189 590 0 22 May 2023
Transformer Working Memory Enables Regular Language Reasoning and Natural Language Length Extrapolation Ta-Chung Chi Ting-Han Fan Alexander I. Rudnicky Peter J. Ramadge LRM 39 13 0 05 May 2023
CoLT5: Faster Long-Range Transformers with Conditional Computation Joshua Ainslie Tao Lei Michiel de Jong Santiago Ontañón Siddhartha Brahma ... Mandy Guo James Lee-Thorp Yi Tay Yun-hsuan Sung Sumit Sanghai LLMAG 61 68 0 17 Mar 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 264 287 0 11 Mar 2023
Modular Deep Learning Jonas Pfeiffer Sebastian Ruder Ivan Vulić Edoardo Ponti MoMe OOD 119 77 0 22 Feb 2023
Adaptive Computation with Elastic Input Sequence Fuzhao Xue Valerii Likhosherstov Anurag Arnab N. Houlsby Mostafa Dehghani Yang You 54 20 0 30 Jan 2023
A Length-Extrapolatable Transformer Yutao Sun Li Dong Barun Patra Shuming Ma Shaohan Huang Alon Benhaim Vishrav Chaudhary Xia Song Furu Wei 73 120 0 20 Dec 2022
Towards Reasoning in Large Language Models: A Survey Jie Huang Kevin Chen-Chuan Chang LM&MA ELM LRM 119 621 0 20 Dec 2022
Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions S. Bhattamishra Arkil Patel Varun Kanade Phil Blunsom 68 48 0 22 Nov 2022
Transformers Learn Shortcuts to Automata Bingbin Liu Jordan T. Ash Surbhi Goel A. Krishnamurthy Cyril Zhang OffRL LRM 115 174 0 19 Oct 2022
Neural Attentive Circuits Nasim Rahaman M. Weiß Francesco Locatello C. Pal Yoshua Bengio Bernhard Schölkopf Erran L. Li Nicolas Ballas 95 7 0 14 Oct 2022
Mega: Moving Average Equipped Gated Attention Xuezhe Ma Chunting Zhou Xiang Kong Junxian He Liangke Gui Graham Neubig Jonathan May Luke Zettlemoyer 66 183 0 21 Sep 2022
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay Mostafa Dehghani Samira Abnar Hyung Won Chung W. Fedus J. Rao Sharan Narang Vinh Q. Tran Dani Yogatama Donald Metzler AI4CE 89 102 0 21 Jul 2022
Confident Adaptive Language Modeling Tal Schuster Adam Fisch Jai Gupta Mostafa Dehghani Dara Bahri Vinh Q. Tran Yi Tay Donald Metzler 107 168 0 14 Jul 2022
Recurrent Memory Transformer Aydar Bulatov Yuri Kuratov Andrey Kravchenko CLL 34 108 0 14 Jul 2022
Neural Networks and the Chomsky Hierarchy Grégoire Delétang Anian Ruoss Jordi Grau-Moya Tim Genewein L. Wenliang ... Chris Cundy Marcus Hutter Shane Legg Joel Veness Pedro A. Ortega UQCV 129 148 0 05 Jul 2022
The Parallelism Tradeoff: Limitations of Log-Precision Transformers William Merrill Ashish Sabharwal 47 110 0 02 Jul 2022
Long Range Language Modeling via Gated State Spaces Harsh Mehta Ankit Gupta Ashok Cutkosky Behnam Neyshabur Mamba 74 238 0 27 Jun 2022
On the Parameterization and Initialization of Diagonal State Space Models Albert Gu Ankit Gupta Karan Goel Christopher Ré 71 314 0 23 Jun 2022
Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning Aniket Didolkar Kshitij Gupta Anirudh Goyal Nitesh B. Gundavarapu Alex Lamb Nan Rosemary Ke Yoshua Bengio AI4CE 156 18 0 30 May 2022
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity Sophie Hao Dana Angluin Robert Frank 43 77 0 13 Apr 2022
Block-Recurrent Transformers DeLesley S. Hutchins Imanol Schlag Yuhuai Wu Ethan Dyer Behnam Neyshabur 68 96 0 11 Mar 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 103 228 0 21 Feb 2022
Flowformer: Linearizing Transformers with Conservation Flows Haixu Wu Jialong Wu Jiehui Xu Jianmin Wang Mingsheng Long 53 92 0 13 Feb 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 738 9,267 0 28 Jan 2022
Show Your Work: Scratchpads for Intermediate Computation with Language Models Maxwell Nye Anders Andreassen Guy Gur-Ari Henryk Michalewski Jacob Austin ... Aitor Lewkowycz Maarten Bosma D. Luan Charles Sutton Augustus Odena ReLM LRM 159 737 0 30 Nov 2021
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 184 1,761 0 31 Oct 2021
Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method Yifan Chen Qi Zeng Heng Ji Yun Yang 33 49 0 29 Oct 2021