The Parallelism Tradeoff: Limitations of Log-Precision Transformers

2 July 2022

Papers citing "The Parallelism Tradeoff: Limitations of Log-Precision Transformers"

26 / 26 papers shown

Title
Exact Expressive Power of Transformers with Padding William Merrill Ashish Sabharwal 39 0 0 25 May 2025
GeoNav: Empowering MLLMs with Explicit Geospatial Reasoning Abilities for Language-Goal Aerial Navigation Haotian Xu Yue Hu Chen Gao Zhengqiu Zhu Yong Zhao Yongqian Li Quanjun Yin 70 2 0 13 Apr 2025
A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers William Merrill Ashish Sabharwal 59 8 0 05 Mar 2025
Looped ReLU MLPs May Be All You Need as Practical Programmable Computers Yingyu Liang Zhizhou Sha Zhenmei Shi Zhao Song Yufa Zhou 114 18 0 21 Feb 2025
Provably Overwhelming Transformer Models with Designed Inputs Lev Stambler Seyed Sajjad Nezhadi Matthew Coudron 100 1 0 09 Feb 2025
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers Alireza Amiri Xinting Huang Mark Rofin Michael Hahn LRM 446 1 0 04 Feb 2025
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues Riccardo Grazzi Julien N. Siems Jörg Franke Arber Zela Frank Hutter Massimiliano Pontil 121 16 0 19 Nov 2024
Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning Md Rifat Arefin G. Subbaraj Nicolas Angelard-Gontier Yann LeCun Irina Rish Ravid Shwartz-Ziv C. Pal LRM 386 1 0 04 Nov 2024
Can Transformers Reason Logically? A Study in SAT Solving Leyan Pan Vijay Ganesh Jacob Abernethy Chris Esposo Wenke Lee ReLM LRM 45 2 0 09 Oct 2024
On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding Kevin Xu Issei Sato 74 4 0 02 Oct 2024
Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference Anton Xue Avishree Khare Rajeev Alur Surbhi Goel Eric Wong 88 2 0 21 Jun 2024
What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages Nadav Borenstein Anej Svete R. Chan Josef Valvoda Franz Nowak Isabelle Augenstein Eleanor Chodroff Ryan Cotterell 54 13 0 06 Jun 2024
Investigating Recurrent Transformers with Dynamic Halt Jishnu Ray Chowdhury Cornelia Caragea 70 1 0 01 Feb 2024
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 64 648 0 15 Aug 2022
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment Matthew Finlayson Kyle Richardson Ashish Sabharwal Peter Clark 46 12 0 19 Apr 2022
Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity Sophie Hao Dana Angluin Robert Frank 33 77 0 13 Apr 2022
Saturated Transformers are Constant-Depth Threshold Circuits William Merrill Ashish Sabharwal Noah A. Smith 49 102 0 30 Jun 2021
Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent William Merrill Vivek Ramanujan Yoav Goldberg Roy Schwartz Noah A. Smith AI4CE 29 36 0 19 Oct 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 500 41,106 0 28 May 2020
On the Linguistic Capacity of Real-Time Counter Automata William Merrill 26 23 0 15 Apr 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 270 19,824 0 23 Oct 2019
Theoretical Limitations of Self-Attention in Neural Sequence Models Michael Hahn 44 267 0 16 Jun 2019
On the Turing Completeness of Modern Neural Network Architectures Jorge A. Pérez Javier Marinkovic Pablo Barceló BDL 48 144 0 10 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 969 93,936 0 11 Oct 2018
Universal Transformers Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser 64 752 0 10 Jul 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 453 129,831 0 12 Jun 2017