A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time

14 September 2023

Papers citing "A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time"

41 / 41 papers shown

Title
Fast Gradient Computation for RoPE Attention in Almost Linear Time Yifang Chen Jiayan Huo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao Song 89 13 0 03 Jan 2025
HSR-Enhanced Sparse Attention Acceleration Bo Chen Yingyu Liang Zhizhou Sha Zhenmei Shi Zhao Song 142 20 0 14 Oct 2024
In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trick Yeqi Gao Zhao Song Shenghao Xie 23 26 0 05 Jul 2023
CMATH: Can Your Language Model Pass Chinese Elementary School Math Test? Tianwen Wei Jian Luan Wen Liu Shuang Dong Bin Wang ELM 44 33 0 29 Jun 2023
Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence John J. Nay David Karamardian Sarah Lawsky Wenting Tao Meghana Moorthy Bhat Raghav Jain Aaron Travis Lee Jonathan H. Choi Jungo Kasai ELM AILaw 67 57 0 12 Jun 2023
How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization? Aniket Deroy Kripabandhu Ghosh Saptarshi Ghosh ELM AILaw 36 60 0 02 Jun 2023
Fine-Tuning Language Models with Just Forward Passes Sadhika Malladi Tianyu Gao Eshaan Nichani Alexandru Damian Jason D. Lee Danqi Chen Sanjeev Arora 100 186 0 27 May 2023
On Evaluating Adversarial Robustness of Large Vision-Language Models Yunqing Zhao Tianyu Pang Chao Du Xiao Yang Chongxuan Li Ngai-Man Cheung Min Lin VLM AAML MLLM 61 174 0 26 May 2023
Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models Daman Arora H. Singh Mausam ELM LRM 66 53 0 24 May 2023
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark Minje Choi Jiaxin Pei Sagar Kumar Chang Shu David Jurgens ALM LLMAG 66 71 0 24 May 2023
Fast Submodular Function Maximization Lianke Qin Zhao Song Yitan Wang 53 10 0 15 May 2023
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models Emilio Ferrara SILM 82 253 0 07 Apr 2023
Document-Level Machine Translation with Large Language Models Longyue Wang Chenyang Lyu Tianbo Ji Zhirui Zhang Dian Yu Shuming Shi Zhaopeng Tu ELM 50 122 0 05 Apr 2023
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective Jindong Wang Xixu Hu Wenxin Hou Hao Chen Runkai Zheng ... Weirong Ye Xiubo Geng Binxing Jiao Yue Zhang Xingxu Xie AI4MH 83 227 0 22 Feb 2023
Discovering Latent Knowledge in Language Models Without Supervision Collin Burns Haotian Ye Dan Klein Jacob Steinhardt 102 350 0 07 Dec 2022
Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation Preprocessing Josh Alman Jiehao Liang Zhao Song Ruizhe Zhang Danyang Zhuo 110 31 0 25 Nov 2022
Finding Skill Neurons in Pre-trained Transformer-based Language Models Xiaozhi Wang Kaiyue Wen Zhengyan Zhang Lei Hou Zhiyuan Liu Juanzi Li MILM MoE 42 51 0 14 Nov 2022
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 156 1,308 0 10 Feb 2022
Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Time Zhao Song Licheng Zhang Ruizhe Zhang 60 65 0 14 Dec 2021
Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structures Anshumali Shrivastava Zhao Song Zhaozhuo Xu 57 28 0 30 Nov 2021
Does Preprocessing Help Training Over-parameterized Neural Networks? Zhao Song Shuo Yang Ruizhe Zhang 56 49 0 09 Oct 2021
Fast Margin Maximization via Dual Acceleration Ziwei Ji Nathan Srebro Matus Telgarsky 28 38 0 01 Jul 2021
Knowledge Neurons in Pretrained Transformers Damai Dai Li Dong Y. Hao Zhifang Sui Baobao Chang Furu Wei KELM MU 60 440 0 18 Apr 2021
Approximating How Single Head Attention Learns Charles Burton Snell Ruiqi Zhong Dan Klein Jacob Steinhardt MLT 24 30 0 13 Mar 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 174 4,209 0 01 Jan 2021
Improved Guarantees for k-means++ and k-means++ Parallel K. Makarychev Aravind Reddy Liren Shan DRL 52 24 0 27 Oct 2020
Generalized Leverage Score Sampling for Neural Networks Jason D. Lee Ruoqi Shen Zhao Song Mengdi Wang Zheng Yu 38 43 0 21 Sep 2020
Shape Matters: Understanding the Implicit Bias of the Noise Covariance Jeff Z. HaoChen Colin Wei Jason D. Lee Tengyu Ma 97 94 0 15 Jun 2020
An Improved Cutting Plane Method for Convex Optimization, Convex-Concave Games and its Applications Haotian Jiang Y. Lee Zhao Song Sam Chiu-wai Wong 48 106 0 08 Apr 2020
Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality Yi Zhang Orestis Plevrakis S. Du Xingguo Li Zhao Song Sanjeev Arora 96 51 0 16 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 451 4,662 0 23 Jan 2020
Implicit Regularization for Optimal Sparse Recovery Tomas Vaskevicius Varun Kanade Patrick Rebeschini 32 100 0 11 Sep 2019
Kernel and Rich Regimes in Overparametrized Models Blake E. Woodworth Suriya Gunasekar Pedro H. P. Savarese E. Moroshko Itay Golan Jason D. Lee Daniel Soudry Nathan Srebro 63 358 0 13 Jun 2019
Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems Tianle Cai Ruiqi Gao Jikai Hou Siyu Chen Dong Wang Di He Zhihua Zhang Liwei Wang ODL 44 57 0 28 May 2019
Solving Empirical Risk Minimization in the Current Matrix Multiplication Time Y. Lee Zhao Song Qiuyi Zhang 62 116 0 11 May 2019
Gradient Descent Provably Optimizes Over-parameterized Neural Networks S. Du Xiyu Zhai Barnabás Póczós Aarti Singh MLT ODL 151 1,261 0 04 Oct 2018
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data Yuanzhi Li Yingyu Liang MLT 138 652 0 03 Aug 2018
Convergence of Gradient Descent on Separable Data Mor Shpigel Nacson Jason D. Lee Suriya Gunasekar Pedro H. P. Savarese Nathan Srebro Daniel Soudry 60 167 0 05 Mar 2018
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 316 7,951 0 17 Aug 2015
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Ke Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhutdinov R. Zemel Yoshua Bengio DiffM 286 10,034 0 10 Feb 2015
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 390 27,205 0 01 Sep 2014