Attention Scheme Inspired Softmax Regression

20 April 2023

Papers citing "Attention Scheme Inspired Softmax Regression"

41 / 41 papers shown

Title
Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approaches Yifang Chen Xuyang Guo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 67 3 0 03 Mar 2025
When Can We Solve the Weighted Low Rank Approximation Problem in Truly Subquadratic Time? Chenyang Li Yingyu Liang Zhenmei Shi Zhao-quan Song 36 3 0 24 Feb 2025
Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation Yang Cao Zhao-quan Song Chiwun Yang VGen 46 2 0 01 Feb 2025
Fast Gradient Computation for RoPE Attention in Almost Linear Time Yifang Chen Jiayan Huo Xiaoyu Li Yingyu Liang Zhenmei Shi Zhao-quan Song 61 11 0 03 Jan 2025
Binary Hypothesis Testing for Softmax Models and Leverage Score Models Yeqi Gao Yuzhou Gu Zhao-quan Song 33 0 0 09 May 2024
Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence Yichuan Deng Zhao-quan Song Chiwun Yang 29 1 0 02 Feb 2024
Superiority of Multi-Head Attention in In-Context Linear Regression Yingqian Cui Jie Ren Pengfei He Jiliang Tang Yue Xing 37 12 0 30 Jan 2024
One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space Raghav Addanki Chenyang Li Zhao-quan Song Chiwun Yang 47 3 0 24 Nov 2023
Fast Heavy Inner Product Identification Between Weights and Inputs in Neural Network Training Lianke Qin Saayan Mitra Zhao-quan Song Yuanyuan Yang Dinesh Manocha 27 0 0 19 Nov 2023
The Expressibility of Polynomial based Attention Scheme Zhao-quan Song Guangyi Xu Junze Yin 32 5 0 30 Oct 2023
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time Zichang Liu Jue Wang Tri Dao Dinesh Manocha Binhang Yuan ... Anshumali Shrivastava Ce Zhang Yuandong Tian Christopher Ré Beidi Chen BDL 19 191 0 26 Oct 2023
An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent Zhao-quan Song Chiwun Yang 29 9 0 17 Oct 2023
Fine-tune Language Models to Approximate Unbiased In-context Learning Timothy Chu Zhao-quan Song Chiwun Yang 27 15 0 05 Oct 2023
Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network? Lianke Qin Zhao-quan Song Baocheng Sun 23 7 0 14 Sep 2023
A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Time Yeqi Gao Zhao-quan Song Weixin Wang Junze Yin 20 25 0 14 Sep 2023
Online Adaptive Mahalanobis Distance Estimation Lianke Qin Aravind Reddy Zhao-quan Song 41 1 0 02 Sep 2023
Solving Attention Kernel Regression Problem via Pre-conditioner Zhao-quan Song Junze Yin Licheng Zhang 28 9 0 28 Aug 2023
How to Protect Copyright Data in Optimization of Large Language Models? T. Chu Zhao-quan Song Chiwun Yang 37 29 0 23 Aug 2023
GradientCoin: A Peer-to-Peer Decentralized Large Language Models Yeqi Gao Zhao-quan Song Junze Yin 29 18 0 21 Aug 2023
Convergence of Two-Layer Regression with Nonlinear Units Yichuan Deng Zhao-quan Song Shenghao Xie 26 7 0 16 Aug 2023
Zero-th Order Algorithm for Softmax Attention Optimization Yichuan Deng Zhihang Li Sridhar Mahadevan Zhao-quan Song 35 13 0 17 Jul 2023
Fast Quantum Algorithm for Attention Computation Yeqi Gao Zhao-quan Song Xin Yang Ruizhe Zhang LRM 31 19 0 16 Jul 2023
Efficient SGD Neural Network Training via Sublinear Activated Neuron Identification Lianke Qin Zhao-quan Song Yuanyuan Yang 25 9 0 13 Jul 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu (Allen) Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 49 254 0 24 Jun 2023
InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understanding Junda Wu Tong Yu Rui Wang Zhao-quan Song Ruiyi Zhang Handong Zhao Chaochao Lu Shuai Li Ricardo Henao VLM 33 23 0 08 Jun 2023
Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation Zhao-quan Song Mingquan Ye Junze Yin Licheng Zhang 24 7 0 07 Jun 2023
Query Complexity of Active Learning for Function Family With Nearly Orthogonal Basis Xiangyi Chen Zhao-quan Song Baochen Sun Junze Yin Danyang Zhuo 36 3 0 06 Jun 2023
A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Models Ritwik Sinha Zhao-quan Song Dinesh Manocha 22 23 0 04 Jun 2023
Federated Empirical Risk Minimization via Second-Order Method S. Bian Zhao-quan Song Junze Yin FedML 33 8 0 27 May 2023
Fast Submodular Function Maximization Lianke Qin Zhao-quan Song Yitan Wang 21 10 0 15 May 2023
Fast and Efficient Matching Algorithm with Deadline Instances Zhao-quan Song Weixin Wang Chenbo Yin Junze Yin 14 7 0 15 May 2023
Efficient Asynchronize Stochastic Gradient Algorithm with Structured Data Zhao-quan Song Mingquan Ye 22 4 0 13 May 2023
Differentially Private Attention Computation Yeqi Gao Zhao-quan Song Xin Yang 47 19 0 08 May 2023
An Iterative Algorithm for Rescaled Hyperbolic Functions Regression Yeqi Gao Zhao-quan Song Junze Yin 28 33 0 01 May 2023
The Closeness of In-Context Learning and Weight Shifting for Softmax Regression Shuai Li Zhao-quan Song Yu Xia Tong Yu Dinesh Manocha 33 36 0 26 Apr 2023
Solving Tensor Low Cycle Rank Approximation Yichuan Deng Yeqi Gao Zhao-quan Song 34 6 0 13 Apr 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 298 3,007 0 22 Mar 2023
How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding Yuchen Li Yuan-Fang Li Andrej Risteski 120 61 0 07 Mar 2023
Fast Attention Requires Bounded Entries Josh Alman Zhao-quan Song 25 78 0 26 Feb 2023
Low Rank Matrix Completion via Robust Alternating Minimization in Nearly Linear Time Yuzhou Gu Zhao-quan Song Junze Yin Licheng Zhang 16 26 0 21 Feb 2023
Federated Adversarial Learning: A Framework with Convergence Analysis Xiaoxiao Li Zhao-quan Song Jiaming Yang FedML 27 19 0 07 Aug 2022