v1v2 (latest)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency

7 October 2024

Jingzhao Zhang

Papers citing "From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency"

50 / 58 papers shown

Title
Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers Yixiao Huang Hanlin Zhu Tianyu Guo Jiantao Jiao Somayeh Sojoudi Michael I. Jordan Stuart Russell Song Mei LRM 144 0 0 12 Jun 2025
The Coverage Principle: A Framework for Understanding Compositional Generalization Hoyeon Chang Jinho Park Hanseul Cho Sohee Yang Miyoung Ko Hyeonbin Hwang Seungpil Won Dohaeng Lee Youbin Ahn Minjoon Seo 63 0 0 26 May 2025
Minimalist Softmax Attention Provably Learns Constrained Boolean Functions Jerry Yao-Chieh Hu Xiwen Zhang Maojiang Su Zhao Song Han Liu MLT 245 1 0 26 May 2025
How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias Ruiquan Huang Yingbin Liang Jing Yang 131 0 0 02 May 2025
Qwen2 Technical Report An Yang Baosong Yang Binyuan Hui Jian Xu Bowen Yu ... Yuqiong Liu Zeyu Cui Zhenru Zhang Zhifang Guo Zhi-Wei Fan OSLM VLM MU 236 989 0 15 Jul 2024
Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis Hongkang Li Meng Wang Shuai Zhang Sijia Liu Pin-Yu Chen 119 7 0 24 Jun 2024
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning Franz Nowak Anej Svete Alexandra Butoi Ryan Cotterell ReLM LRM 115 17 0 20 Jun 2024
Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot Zixuan Wang Stanley Wei Daniel Hsu Jason D. Lee 67 16 0 11 Jun 2024
Matching the Statistical Query Lower Bound for k-sparse Parity Problems with Stochastic Gradient Descent Yiwen Kou Zixiang Chen Quanquan Gu Sham Kakade 94 0 0 18 Apr 2024
Implicit Regularization of Gradient Flow on One-Layer Softmax Attention Heejune Sheen Siyu Chen Tianhao Wang Harrison H. Zhou MLT 87 13 0 13 Mar 2024
Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality Siyu Chen Heejune Sheen Tianhao Wang Zhuoran Yang MLT 83 46 0 29 Feb 2024
RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval Kaiyue Wen Xingyu Dang Kaifeng Lyu 117 33 0 28 Feb 2024
How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning Subhabrata Dutta Joykirat Singh Soumen Chakrabarti Tanmoy Chakraborty LRM 97 26 0 28 Feb 2024
Chain of Thought Empowers Transformers to Solve Inherently Serial Problems Zhiyuan Li Hong Liu Denny Zhou Tengyu Ma LRM AI4CE 103 133 0 20 Feb 2024
Chain-of-Thought Reasoning Without Prompting Xuezhi Wang Denny Zhou ReLM LRM 269 125 0 15 Feb 2024
Why are Sensitive Functions Hard for Transformers? Michael Hahn Mark Rofin 103 29 0 15 Feb 2024
Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape Juno Kim Taiji Suzuki 133 24 0 02 Feb 2024
Tight Time-Space Lower Bounds for Constant-Pass Learning Xin Lyu Avishay Tal Hongxun Wu Junzhao Yang 85 1 0 12 Oct 2023
The Expressive Power of Transformers with Chain of Thought William Merrill Ashish Sabharwal LRM AI4CE ReLM 105 0 0 11 Oct 2023
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models L. Yu Weisen Jiang Han Shi Jincheng Yu Zhengying Liu Yu Zhang James T. Kwok Zheng Li Adrian Weller Weiyang Liu OSLM LRM 122 395 0 21 Sep 2023
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning Xiang Yue Xingwei Qu Ge Zhang Yao Fu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen AIMat LRM 204 404 0 11 Sep 2023
Graph of Thoughts: Solving Elaborate Problems with Large Language Models Maciej Besta Nils Blach Aleš Kubíček Robert Gerstenberger Michal Podstawski ... Joanna Gajda Tomasz Lehmann H. Niewiadomski Piotr Nyczyk Torsten Hoefler LRM AI4CE LM&Ro 189 718 0 18 Aug 2023
Let's Verify Step by Step Hunter Lightman V. Kosaraju Yura Burda Harrison Edwards Bowen Baker Teddy Lee Jan Leike John Schulman Ilya Sutskever K. Cobbe ALM OffRL LRM 242 1,241 0 31 May 2023
Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer Yuandong Tian Yiping Wang Beidi Chen S. Du MLT 109 79 0 25 May 2023
Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective Guhao Feng Bohang Zhang Yuntian Gu Haotian Ye Di He Liwei Wang LRM 149 262 0 24 May 2023
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning Seungone Kim Se June Joo Doyoung Kim Joel Jang Seonghyeon Ye Jamin Shin Minjoon Seo ALM RALM LRM 129 107 0 23 May 2023
Tree of Thoughts: Deliberate Problem Solving with Large Language Models Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas Griffiths Yuan Cao Karthik Narasimhan LM&Ro LRM AI4CE 241 2,055 0 17 May 2023
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models Lei Wang Wanyu Xu Yihuai Lan Zhiqiang Hu Yunshi Lan Roy Ka-wei Lee Ee-Peng Lim ReLM LRM 134 358 0 06 May 2023
Why think step by step? Reasoning emerges from the locality of experience Ben Prystawski Michael Y. Li Noah D. Goodman LRM ReLM 89 107 0 07 Apr 2023
How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding Yuchen Li Yuan-Fang Li Andrej Risteski 173 65 0 07 Mar 2023
SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics Emmanuel Abbe Enric Boix-Adserà Theodor Misiakiewicz FedML MLT 167 86 0 21 Feb 2023
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 314 3,178 0 20 Oct 2022
Automatic Chain of Thought Prompting in Large Language Models Zhuosheng Zhang Aston Zhang Mu Li Alexander J. Smola ReLM LRM 180 641 0 07 Oct 2022
Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought Abulhair Saparov He He ELM LRM ReLM 304 315 0 03 Oct 2022
Complexity-Based Prompting for Multi-Step Reasoning Yao Fu Hao-Chun Peng Ashish Sabharwal Peter Clark Tushar Khot ReLM LRM 259 446 0 03 Oct 2022
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit Boaz Barak Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 114 133 0 18 Jul 2022
Rationale-Augmented Ensembles in Language Models Xuezhi Wang Jason W. Wei Dale Schuurmans Quoc Le Ed H. Chi Denny Zhou ReLM LRM 121 126 0 02 Jul 2022
Solving Quantitative Reasoning Problems with Language Models Aitor Lewkowycz Anders Andreassen David Dohan Ethan Dyer Henryk Michalewski ... Theo Gutman-Solo Yuhuai Wu Behnam Neyshabur Guy Gur-Ari Vedant Misra ReLM ELM LRM 242 865 0 29 Jun 2022
Large Language Models are Zero-Shot Reasoners Takeshi Kojima S. Gu Machel Reid Yutaka Matsuo Yusuke Iwasawa ReLM LRM 612 4,546 0 24 May 2022
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models Denny Zhou Nathanael Scharli Le Hou Jason W. Wei Nathan Scales ... Dale Schuurmans Claire Cui Olivier Bousquet Quoc Le Ed H. Chi RALM LRM AI4CE 109 1,138 0 21 May 2022
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning Antonia Creswell Murray Shanahan I. Higgins ReLM LRM 121 366 0 19 May 2022
Data Distributional Properties Drive Emergent In-Context Learning in Transformers Stephanie C. Y. Chan Adam Santoro Andrew Kyle Lampinen Jane X. Wang Aaditya K. Singh Pierre Harvey Richemond J. Mcclelland Felix Hill 183 266 0 22 Apr 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 596 6,325 0 05 Apr 2022
STaR: Bootstrapping Reasoning With Reasoning E. Zelikman Yuhuai Wu Jesse Mu Noah D. Goodman ReLM LRM 157 512 0 28 Mar 2022
Self-Consistency Improves Chain of Thought Reasoning in Language Models Xuezhi Wang Jason W. Wei Dale Schuurmans Quoc Le Ed H. Chi Sharan Narang Aakanksha Chowdhery Denny Zhou ReLM BDL LRM AI4CE 726 3,760 0 21 Mar 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 1.1K 9,823 0 28 Jan 2022
Show Your Work: Scratchpads for Intermediate Computation with Language Models Maxwell Nye Anders Andreassen Guy Gur-Ari Henryk Michalewski Jacob Austin ... Aitor Lewkowycz Maarten Bosma D. Luan Charles Sutton Augustus Odena ReLM LRM 223 757 0 30 Nov 2021
Training Verifiers to Solve Math Word Problems K. Cobbe V. Kosaraju Mohammad Bavarian Mark Chen Heewoo Jun ... Jerry Tworek Jacob Hilton Reiichiro Nakano Christopher Hesse John Schulman ReLM OffRL LRM 434 4,609 0 27 Oct 2021
On the Provable Generalization of Recurrent Neural Networks Lifu Wang Bo Shen Bo Hu Xing Cao 144 8 0 29 Sep 2021
When Hardness of Approximation Meets Hardness of Learning Eran Malach Shai Shalev-Shwartz 55 9 0 18 Aug 2020