Transformers Can Represent $n$-gram Language Models

Transformers Can Represent $n$ -gram Language Models

23 April 2024

Papers citing "Transformers Can Represent $n$-gram Language Models"

12 / 12 papers shown

Title
Learning curves theory for hierarchically compositional data with power-law distributed features Francesco Cagnetta Hyunmo Kang M. Wyart 36 0 0 11 May 2025
Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures Francesco Cagnetta Alessandro Favero Antonio Sclocchi M. Wyart 26 0 0 11 May 2025
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models Tyler A. Chang Benjamin Bergen 50 0 0 21 Apr 2025
Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More Arvid Frydenlund LRM 52 0 0 13 Mar 2025
Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers Alireza Amiri Xinting Huang Mark Rofin Michael Hahn LRM 195 0 0 04 Feb 2025
Can Transformers Learn $n$ -gram Language Models? Anej Svete Nadav Borenstein M. Zhou Isabelle Augenstein Ryan Cotterell 47 6 0 03 Oct 2024
Representing Rule-based Chatbots with Transformers Dan Friedman Abhishek Panigrahi Danqi Chen 66 1 0 15 Jul 2024
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning Franz Nowak Anej Svete Alexandra Butoi Ryan Cotterell ReLM LRM 54 13 0 20 Jun 2024
What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages Nadav Borenstein Anej Svete R. Chan Josef Valvoda Franz Nowak Isabelle Augenstein Eleanor Chodroff Ryan Cotterell 42 12 0 06 Jun 2024
Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory Xueyan Niu Bo Bai Lei Deng Wei Han 39 6 0 14 May 2024
A Logic for Expressing Log-Precision Transformers William Merrill Ashish Sabharwal ReLM NAI LRM 48 47 0 06 Oct 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 463 0 24 Sep 2022