Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity

13 April 2022

Papers citing "Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity"

50 / 57 papers shown

Title
NoPE: The Counting Power of Transformers with No Positional Encodings Chris Köcher Alexander Kozachinskiy Anthony Widjaja Lin Marco Sälzer Georg Zetzsche 12 0 0 16 May 2025
Generative Linguistics, Large Language Models, and the Social Nature of Scientific Success Sophie Hao ELM AI4CE 54 0 0 25 Mar 2025
Unique Hard Attention: A Tale of Two Sides Selim Jerad Anej Svete Jiaoda Li Ryan Cotterell 61 0 0 18 Mar 2025
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More Arvid Frydenlund LRM 48 0 0 13 Mar 2025
Compositional Reasoning with Transformers, RNNs, and Chain of Thought Gilad Yehudai Noah Amsel Joan Bruna LRM 60 1 0 03 Mar 2025
Depth-Width tradeoffs in Algorithmic Reasoning of Graph Tasks with Transformers Gilad Yehudai Clayton Sanford Maya Bechler-Speicher Orr Fischer Ran Gilad-Bachrach Amir Globerson 55 0 0 03 Mar 2025
Ask, and it shall be given: On the Turing completeness of prompting Ruizhong Qiu Zhe Xu W. Bao Hanghang Tong ReLM LRM AI4CE 70 0 0 24 Feb 2025
Can Language Models Learn Typologically Implausible Languages? Tianyang Xu Tatsuki Kuribayashi Yohei Oseki Ryan Cotterell Alex Warstadt 75 1 0 17 Feb 2025
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers Alireza Amiri Xinting Huang Mark Rofin Michael Hahn LRM 180 0 0 04 Feb 2025
Ehrenfeucht-Haussler Rank and Chain of Thought Pablo Barceló Alexander Kozachinskiy Tomasz Steifer LRM 79 1 0 22 Jan 2025
Circuit Complexity Bounds for Visual Autoregressive Model Yekun Ke Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 45 5 0 08 Jan 2025
Lower bounds on transformers with infinite precision Alexander Kozachinskiy 34 2 0 31 Dec 2024
Mastering NIM and Impartial Games with Weak Neural Networks: An AlphaZero-inspired Multi-Frame Approach Søren Riis 18 0 0 10 Nov 2024
How Numerical Precision Affects Mathematical Reasoning Capabilities of LLMs Guhao Feng Kai-Bo Yang Yuntian Gu Xinyue Ai Shengjie Luo Jiacheng Sun Di He ZeLin Li Liwei Wang LRM 37 6 0 17 Oct 2024
Learning Linear Attention in Polynomial Time Morris Yau Ekin Akyürek Jiayuan Mao Joshua B. Tenenbaum Stefanie Jegelka Jacob Andreas 19 2 0 14 Oct 2024
Fundamental Limitations on Subquadratic Alternatives to Transformers Josh Alman Hantao Yu 23 1 0 05 Oct 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 66 1 0 15 Jul 2024
$$\text{Memory}^3$: Language Modeling with Explicit Memory$ $\text{Memory}^3$ : Language Modeling with Explicit Memory Hongkang Yang Zehao Lin Wenjin Wang Hao Wu Zhiyu Li ... Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E 50 11 0 01 Jul 2024
Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference Anton Xue Avishree Khare Rajeev Alur Surbhi Goel Eric Wong 58 2 0 21 Jun 2024
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning Franz Nowak Anej Svete Alexandra Butoi Ryan Cotterell ReLM LRM 54 12 0 20 Jun 2024
Separations in the Representational Capabilities of Transformers and Recurrent Architectures S. Bhattamishra Michael Hahn Phil Blunsom Varun Kanade GNN 44 9 0 13 Jun 2024
Automata Extraction from Transformers Yihao Zhang Zeming Wei Meng Sun AI4CE 42 1 0 08 Jun 2024
Language Models Need Inductive Biases to Count Inductively Yingshan Chang Yonatan Bisk LRM 32 5 0 30 May 2024
Understanding Transformer Reasoning Capabilities via Graph Algorithms Clayton Sanford Bahare Fatemi Ethan Hall Anton Tsitsulin Seyed Mehran Kazemi Jonathan J. Halcrow Bryan Perozzi Vahab Mirrokni 44 30 0 28 May 2024
A Transformer with Stack Attention Jiaoda Li Jennifer C. White Mrinmaya Sachan Ryan Cotterell 30 2 0 07 May 2024
Transformers Can Represent $n$ -gram Language Models Anej Svete Ryan Cotterell 37 17 0 23 Apr 2024
The Illusion of State in State-Space Models William Merrill Jackson Petty Ashish Sabharwal 54 43 0 12 Apr 2024
Transformers as Transducers Lena Strobl Dana Angluin David Chiang Jonathan Rawski Ashish Sabharwal 29 5 0 02 Apr 2024
Simulating Weighted Automata over Sequences and Trees with Transformers Michael Rizvi M. Lizaire Clara Lacroce Guillaume Rabusseau AI4CE 53 0 0 12 Mar 2024
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval Kaiyue Wen Xingyu Dang Kaifeng Lyu 52 24 0 28 Feb 2024
Transformers are Expressive, But Are They Expressive Enough for Regression? Swaroop Nath H. Khadilkar Pushpak Bhattacharyya 34 3 0 23 Feb 2024
Do Efficient Transformers Really Save Computation? Kai-Bo Yang Jan Ackermann Zhenyu He Guhao Feng Bohang Zhang Yunzhen Feng Qiwei Ye Di He Liwei Wang 42 8 0 21 Feb 2024
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems Zhiyuan Li Hong Liu Denny Zhou Tengyu Ma LRM AI4CE 28 96 0 20 Feb 2024
Why are Sensitive Functions Hard for Transformers? Michael Hahn Mark Rofin 41 25 0 15 Feb 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 39 1 0 01 Feb 2024
On The Expressivity of Recurrent Neural Cascades Nadezda A. Knorozova Alessandro Ronca 18 1 0 14 Dec 2023
What Formal Languages Can Transformers Express? A Survey Lena Strobl William Merrill Gail Weiss David Chiang Dana Angluin AI4CE 20 48 0 01 Nov 2023
Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions Kazuki Irie Róbert Csordás Jürgen Schmidhuber 36 11 0 24 Oct 2023
Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages Andy Yang David Chiang Dana Angluin 30 15 0 21 Oct 2023
The Expressive Power of Transformers with Chain of Thought William Merrill Ashish Sabharwal LRM AI4CE ReLM 27 41 0 11 Oct 2023
Sparse Universal Transformer Shawn Tan Yikang Shen Zhenfang Chen Aaron Courville Chuang Gan MoE 32 13 0 11 Oct 2023
Logical Languages Accepted by Transformer Encoders with Hard Attention Pablo Barceló Alexander Kozachinskiy Anthony Widjaja Lin Vladimir Podolskii 25 16 0 05 Oct 2023
Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages Shunjie Wang Shane Steinert-Threlkeld 25 4 0 02 Sep 2023
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? T. Kajitsuka Issei Sato 31 16 0 26 Jul 2023
Representational Strengths and Limitations of Transformers Clayton Sanford Daniel J. Hsu Matus Telgarsky 22 81 0 05 Jun 2023
Learning Transformer Programs Dan Friedman Alexander Wettig Danqi Chen 28 32 0 01 Jun 2023
Randomized Positional Encodings Boost Length Generalization of Transformers Anian Ruoss Grégoire Delétang Tim Genewein Jordi Grau-Moya Róbert Csordás Mehdi Abbana Bennani Shane Legg J. Veness LLMAG 36 99 0 26 May 2023
Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective Guhao Feng Bohang Zhang Yuntian Gu Haotian Ye Di He Liwei Wang LRM 30 218 0 24 May 2023
Tighter Bounds on the Expressivity of Transformer Encoders David Chiang Peter A. Cholak A. Pillay 27 53 0 25 Jan 2023
Implementation of general formal translators Iosif Iulian Petrila AI4CE 43 2 0 16 Dec 2022