The Expressibility of Polynomial based Attention Scheme

30 October 2023

Papers citing "The Expressibility of Polynomial based Attention Scheme"

32 / 32 papers shown

Title
Fast Gradient Computation for RoPE Attention in Almost Linear Time Yifang Chen Jiayan Huo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao Song 131 14 0 03 Jan 2025
Do pretrained Transformers Learn In-Context by Gradient Descent? Lingfeng Shen Aayush Mishra Daniel Khashabi 107 9 0 12 Oct 2023
A Unified Scheme of ResNet and Softmax Zhao Song Weixin Wang Junze Yin 54 9 0 23 Sep 2023
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models L. Yu Weisen Jiang Han Shi Jincheng Yu Zhengying Liu Yu Zhang James T. Kwok Zheng Li Adrian Weller Weiyang Liu OSLM LRM 104 394 0 21 Sep 2023
A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time Yeqi Gao Zhao Song Weixin Wang Junze Yin 74 29 0 14 Sep 2023
In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trick Yeqi Gao Zhao Song Shenghao Xie 60 28 0 05 Jul 2023
Trainable Transformer in Transformer A. Panigrahi Sadhika Malladi Mengzhou Xia Sanjeev Arora VLM 82 13 0 03 Jul 2023
A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Models Ritwik Sinha Zhao Song Dinesh Manocha 60 25 0 04 Jun 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 117 205 0 27 May 2023
Fast Submodular Function Maximization Lianke Qin Zhao Song Yitan Wang 71 10 0 15 May 2023
Fast Attention Requires Bounded Entries Josh Alman Zhao Song 76 86 0 26 Feb 2023
$A Nearly-Optimal Bound for Fast Regression with $\ell_\infty$ Guarantee$ A Nearly-Optimal Bound for Fast Regression with $\ell_\infty$ Guarantee Zhao Song Mingquan Ye Junze Yin Licheng Zhang 90 14 0 01 Feb 2023
Mathematical Capabilities of ChatGPT Simon Frieder Luca Pinchetti Alexis Chevalier Ryan-Rhys Griffiths Tommaso Salvatori Thomas Lukasiewicz P. Petersen Julius Berner ELM AI4MH 134 430 0 31 Jan 2023
Transformers learn in-context by gradient descent J. Oswald Eyvind Niklasson E. Randazzo João Sacramento A. Mordvintsev A. Zhmoginov Max Vladymyrov MLT 116 494 0 15 Dec 2022
Discovering Latent Knowledge in Language Models Without Supervision Collin Burns Haotian Ye Dan Klein Jacob Steinhardt 149 386 0 07 Dec 2022
Finding Skill Neurons in Pre-trained Transformer-based Language Models Xiaozhi Wang Kaiyue Wen Zhengyan Zhang Lei Hou Zhiyuan Liu Juanzi Li MILM MoE 65 52 0 14 Nov 2022
What Can Transformers Learn In-Context? A Case Study of Simple Function Classes Shivam Garg Dimitris Tsipras Percy Liang Gregory Valiant 141 513 0 01 Aug 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 529 6,293 0 05 Apr 2022
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 251 1,381 0 10 Feb 2022
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 166 1,130 0 08 Jun 2021
Knowledge Neurons in Pretrained Transformers Damai Dai Li Dong Y. Hao Zhifang Sui Baobao Chang Furu Wei KELM MU 97 464 0 18 Apr 2021
Approximating How Single Head Attention Learns Charles Burton Snell Ruiqi Zhong Dan Klein Jacob Steinhardt MLT 48 31 0 13 Mar 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 252 4,299 0 01 Jan 2021
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 186 1,602 0 30 Sep 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 203 1,786 0 29 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 216 1,716 0 08 Jun 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 485 20,342 0 23 Oct 2019
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee SSL VLM 243 3,695 0 06 Aug 2019
Adaptive Attention Span in Transformers Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin 76 286 0 19 May 2019
Self-Attention with Relative Position Representations Peter Shaw Jakob Uszkoreit Ashish Vaswani 182 2,299 0 06 Mar 2018
Graph Attention Networks Petar Velickovic Guillem Cucurull Arantxa Casanova Adriana Romero Pietro Lio Yoshua Bengio GNN 481 20,233 0 30 Oct 2017
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Ke Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhutdinov R. Zemel Yoshua Bengio DiffM 350 10,083 0 10 Feb 2015