Adaptively Sparse Transformers

30 August 2019

André F. T. Martins

Papers citing "Adaptively Sparse Transformers"

50 / 57 papers shown

Title
SFi-Former: Sparse Flow Induced Attention for Graph Transformer Zechao Li J. Q. Shi Xinming Zhang Miao Zhang B. Li 44 0 0 29 Apr 2025
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity Ruifeng Ren Yong Liu 210 0 0 26 Apr 2025
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation Tiansheng Wen Yifei Wang Zequn Zeng Zhong Peng Yudi Su Xinyang Liu Bo Chen Hongwei Liu Stefanie Jegelka Chenyu You CLL 74 3 0 03 Mar 2025
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models J. P. Muñoz Jinjie Yuan Nilesh Jain Mamba 74 1 0 28 Jan 2025
From Tokens to Words: On the Inner Lexicon of LLMs Guy Kaplan Matanel Oren Yuval Reif Roy Schwartz 58 14 0 08 Oct 2024
Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective Jinhao Li Jiaming Xu Shan Huang Yonghua Chen Wen Li ... Jiayi Pan Li Ding Hao Zhou Yu Wang Guohao Dai 62 17 0 06 Oct 2024
Attention layers provably solve single-location regression Pierre Marion Raphael Berthier Gérard Biau Claire Boyer 215 3 0 02 Oct 2024
Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models Dennis Wu Jerry Yao-Chieh Hu Teng-Yun Hsiao Han Liu 45 28 0 04 Apr 2024
On the Theoretical Expressive Power and the Design Space of Higher-Order Graph Transformers Cai Zhou Rose Yu Yusu Wang 45 7 0 04 Apr 2024
Segmentation Guided Sparse Transformer for Under-Display Camera Image Restoration Jingyun Xue Tao Wang Jun Wang Kaihao Zhang ViT 51 2 0 09 Mar 2024
When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges Wang Chao Jiaxuan Zhao Licheng Jiao Lingling Li Fang Liu Shuyuan Yang 75 13 0 19 Jan 2024
Sparse Autoencoders Find Highly Interpretable Features in Language Models Hoagy Cunningham Aidan Ewart Logan Riggs R. Huben Lee Sharkey MILM 33 347 0 15 Sep 2023
From Latent Graph to Latent Topology Inference: Differentiable Cell Complex Module Claudio Battiloro Indro Spinelli Lev Telyatnikov Michael M. Bronstein Simone Scardapane P. Lorenzo BDL 31 14 0 25 May 2023
Align and Attend: Multimodal Summarization with Dual Contrastive Losses Bo He Jun Wang Jielin Qiu Trung Bui Abhinav Shrivastava Zhaowen Wang 22 66 0 13 Mar 2023
Weakly Supervised Learning Significantly Reduces the Number of Labels Required for Intracranial Hemorrhage Detection on Head CT Jacopo Teneggi Paul H. Yi Jeremias Sulam 32 3 0 29 Nov 2022
Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention Wenhao Li Xiaoyuan Yi Jinyi Hu Maosong Sun Xing Xie 44 0 0 14 Nov 2022
ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention Jyotikrishna Dass Shang Wu Huihong Shi Chaojian Li Zhifan Ye Zhongfeng Wang Yingyan Lin 20 53 0 09 Nov 2022
Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost Sungjun Cho Seonwoo Min Jinwoo Kim Moontae Lee Honglak Lee Seunghoon Hong 42 3 0 27 Oct 2022
SIMPLE: A Gradient Estimator for $k$ -Subset Sampling Kareem Ahmed Zhe Zeng Mathias Niepert Mathias Niepert BDL 48 25 0 04 Oct 2022
Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognition Kartik Audhkhasi Yinghui Huang Bhuvana Ramabhadran Pedro J. Moreno 27 3 0 13 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 35 109 0 31 Aug 2022
Unveiling Transformers with LEGO: a synthetic reasoning task Yi Zhang A. Backurs Sébastien Bubeck Ronen Eldan Suriya Gunasekar Tal Wagner LRM 36 85 0 09 Jun 2022
Attention Mechanism in Neural Networks: Where it Comes and Where it Goes Derya Soydaner 3DV 46 150 0 27 Apr 2022
Malceiver: Perceiver with Hierarchical and Multi-modal Features for Android Malware Detection Niall McLaughlin 30 2 0 12 Apr 2022
Accelerating Attention through Gradient-Based Learned Runtime Pruning Zheng Li Soroush Ghodrati Amir Yazdanbakhsh H. Esmaeilzadeh Mingu Kang 27 17 0 07 Apr 2022
Do Long-Range Language Models Actually Use Long-Range Context? Simeng Sun Kalpesh Krishna Andrew Mattarella-Micke Mohit Iyyer RALM 25 82 0 19 Sep 2021
Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval Zhiqi Huang Hamed Bonab Sheikh Muhammad Sarwar Razieh Rahimi James Allan 40 10 0 07 Sep 2021
FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks Sheng-Chun Kao Suvinay Subramanian Gaurav Agrawal Amir Yazdanbakhsh T. Krishna 48 58 0 13 Jul 2021
KVT: k-NN Attention for Boosting Vision Transformers Pichao Wang Xue Wang F. Wang Ming Lin Shuning Chang Hao Li Rong Jin ViT 51 105 0 28 May 2021
BASE Layers: Simplifying Training of Large, Sparse Models M. Lewis Shruti Bhosale Tim Dettmers Naman Goyal Luke Zettlemoyer MoE 38 274 0 30 Mar 2021
SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation Boxiang Yun Yan Wang Jieneng Chen Huiyu Wang Wei Shen Qingli Li ViT MedIm 23 49 0 05 Mar 2021
Perceiver: General Perception with Iterative Attention Andrew Jaegle Felix Gimeno Andrew Brock Andrew Zisserman Oriol Vinyals João Carreira VLM ViT MDE 91 977 0 04 Mar 2021
Mind the Gap: Assessing Temporal Generalization in Neural Language Models Angeliki Lazaridou A. Kuncoro E. Gribovskaya Devang Agrawal Adam Liska ... Sebastian Ruder Dani Yogatama Kris Cao Susannah Young Phil Blunsom VLM 41 207 0 03 Feb 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 11 2,087 0 11 Jan 2021
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections Mingda Chen Sam Wiseman Kevin Gimpel 27 30 0 29 Dec 2020
Rethinking Transformer-based Set Prediction for Object Detection Zhiqing Sun Shengcao Cao Yiming Yang Kris Kitani ViT 27 319 0 21 Nov 2020
Long Document Ranking with Query-Directed Sparse Transformer Jyun-Yu Jiang Chenyan Xiong Chia-Jung Lee Wei Wang 33 25 0 23 Oct 2020
Rethinking Document-level Neural Machine Translation Zewei Sun Mingxuan Wang Hao Zhou Chengqi Zhao Shujian Huang Jiajun Chen Lei Li VLM 83 47 0 18 Oct 2020
SMYRF: Efficient Attention using Asymmetric Clustering Giannis Daras Nikita Kitaev Augustus Odena A. Dimakis 31 44 0 11 Oct 2020
Transformer-GCRF: Recovering Chinese Dropped Pronouns with General Conditional Random Fields Jingxuan Yang Kerui Xu Jun Xu Si Li Sheng Gao Jun Guo Ji-Rong Wen Nianwen Xue 24 5 0 07 Oct 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 114 1,104 0 14 Sep 2020
AutoTrans: Automating Transformer Design via Reinforced Architecture Search Wei-wei Zhu Xiaoling Wang Xipeng Qiu Yuan Ni Guotong Xie 32 18 0 04 Sep 2020
Sparse Graph to Sequence Learning for Vision Conditioned Long Textual Sequence Generation Aditya Mogadala Marius Mosbach Dietrich Klakow VLM 166 0 0 12 Jul 2020
Data Movement Is All You Need: A Case Study on Optimizing Transformers A. Ivanov Nikoli Dryden Tal Ben-Nun Shigang Li Torsten Hoefler 36 131 0 30 Jun 2020
BERTology Meets Biology: Interpreting Attention in Protein Language Models Jesse Vig Ali Madani Lav Varshney Caiming Xiong R. Socher Nazneen Rajani 29 289 0 26 Jun 2020
Hard-Coded Gaussian Attention for Neural Machine Translation Weiqiu You Simeng Sun Mohit Iyyer 33 67 0 02 May 2020
Vector Quantized Contrastive Predictive Coding for Template-based Music Generation Gaëtan Hadjeres Léopold Crestel 34 18 0 21 Apr 2020
SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection Xiaoya Li Yuxian Meng Mingxin Zhou Qinghong Han Fei Wu Jiwei Li 27 20 0 22 Mar 2020
Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation Alessandro Raganato Yves Scherrer Jörg Tiedemann 32 92 0 24 Feb 2020
Low-Rank Bottleneck in Multi-head Attention Models Srinadh Bhojanapalli Chulhee Yun A. S. Rawat Sashank J. Reddi Sanjiv Kumar 24 94 0 17 Feb 2020