Quantizable Transformers: Removing Outliers by Helping Attention Heads
Do Nothing

v1v2 (latest)

Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing

22 June 2023

Yelysei Bondarenko

Tijmen Blankevoort

ArXiv (abs)PDF HTML

Papers citing "Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing"

19 / 19 papers shown

Title
SLICK: Selective Localization and Instance Calibration for Knowledge-Enhanced Car Damage Segmentation in Automotive Insurance Teerapong Panboonyuen 150 0 0 12 Jun 2025
Why Do Some Inputs Break Low-Bit LLM Quantization? Ting-Yun Chang Muru Zhang Jesse Thomason Robin Jia MQ 17 0 0 24 May 2025
Resource-Efficient Language Models: Quantization for Fast and Accessible Inference Tollef Emil Jørgensen MQ 95 0 0 13 May 2025
Fast and Low-Cost Genomic Foundation Models via Outlier Removal Haozheng Luo Chenghao Qiu Maojiang Su Zhihan Zhou Zoe Mehta Guo Ye Jerry Yao-Chieh Hu Han Liu AAML 107 1 0 01 May 2025
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax Zayd Muhammad Kawakibi Zuhri Erland Hilman Fuadi Alham Fikri Aji 54 0 0 29 Apr 2025
Outlier dimensions favor frequent tokens in language models Iuri Macocco Nora Graichen Gemma Boleda Marco Baroni 132 1 0 27 Mar 2025
ClusComp: A Simple Paradigm for Model Compression and Efficient Finetuning Baohao Liao Christian Herold Seyyed Hadi Hashemi Stefan Vasilev Shahram Khadivi Christof Monz MQ 130 0 0 17 Mar 2025
See What You Are Told: Visual Attention Sink in Large Multimodal Models Seil Kang Jinyeong Kim Junhyeok Kim Seong Jae Hwang VLM 166 10 0 05 Mar 2025
SpinQuant: LLM quantization with learned rotations Zechun Liu Changsheng Zhao Igor Fedorov Bilge Soran Dhruv Choudhary Raghuraman Krishnamoorthi Vikas Chandra Yuandong Tian Tijmen Blankevoort MQ 261 126 0 21 Feb 2025
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval Changhun Lee Jun-gyu Jin Jun-gyu Jin Younghyun Cho Eunhyeok Park RALM LRM 114 0 0 25 Jan 2025
DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization Y. Park Jake Hyun Hojoon Kim Jae W. Lee MQ 122 0 0 28 Dec 2024
ControlMM: Controllable Masked Motion Generation Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 92 7 0 14 Oct 2024
Differential Transformer Tianzhu Ye Li Dong Yuqing Xia Yutao Sun Yi Zhu Gao Huang Furu Wei 501 0 0 07 Oct 2024
Beat this! Accurate beat tracking without DBN postprocessing Francesco Foscarin Jan Schluter Gerhard Widmer 74 7 0 31 Jul 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 123 10 0 24 Jul 2024
QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuning Jiun-Man Chen Yu-Hsuan Chao Yu-Jie Wang Ming-Der Shieh Chih-Chung Hsu Wei-Fen Lin MQ 84 1 0 11 Mar 2024
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact Ruikang Liu Haoli Bai Haokun Lin Yuening Li Han Gao Zheng-Jun Xu Lu Hou Jun Yao Chun Yuan MQ 84 32 0 02 Mar 2024
Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models Jung Hwan Heo Jeonghoon Kim Beomseok Kwon Byeongwook Kim Se Jung Kwon Dongsoo Lee MQ 129 10 0 27 Sep 2023
Training-Free Acceleration of ViTs with Delayed Spatial Merging J. Heo Seyedarmin Azizi A. Fayyazi Massoud Pedram 123 3 0 04 Mar 2023

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.