ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

19 March 2021

Papers citing "ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases"

49 / 399 papers shown

Title
Study of positional encoding approaches for Audio Spectrogram Transformers L. Pepino Pablo Riera Luciana Ferrer ViT 28 6 0 13 Oct 2021
Global Vision Transformer Pruning with Hessian-Aware Saliency Huanrui Yang Hongxu Yin Maying Shen Pavlo Molchanov Hai Helen Li Jan Kautz ViT 30 39 0 10 Oct 2021
PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions Eleonora Grassucci Aston Zhang Danilo Comminiello 25 38 0 08 Oct 2021
UniNet: Unified Architecture Search with Convolution, Transformer, and MLP Jihao Liu Hongsheng Li Guanglu Song Xin Huang Yu Liu ViT 37 35 0 08 Oct 2021
Token Pooling in Vision Transformers D. Marin Jen-Hao Rick Chang Anurag Ranjan Anish K. Prabhu Mohammad Rastegari Oncel Tuzel ViT 76 66 0 08 Oct 2021
Ripple Attention for Visual Perception with Sub-quadratic Complexity Lin Zheng Huijie Pan Lingpeng Kong 26 3 0 06 Oct 2021
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer Sachin Mehta Mohammad Rastegari ViT 218 1,213 0 05 Oct 2021
Infrared Small-Dim Target Detection with Transformer under Complex Backgrounds Fangcen Liu Chenqiang Gao Fangge Chen Deyu Meng W. Zuo Xinbo Gao ViT 39 37 0 29 Sep 2021
OH-Former: Omni-Relational High-Order Transformer for Person Re-Identification Xianing Chen Chunlin Xu Qiong Cao Jialang Xu Yujie Zhong Jiale Xu Zhengxin Li Jingya Wang Shenghua Gao ViT 71 18 0 23 Sep 2021
DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers Changlin Li Guangrun Wang Bing Wang Xiaodan Liang Zhihui Li Xiaojun Chang 30 9 0 21 Sep 2021
Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning Shikha Dubey Farrukh Olimov M. Rafique Joonmo Kim M. Jeon ViT 23 37 0 16 Sep 2021
Towards Transferable Adversarial Attacks on Vision Transformers Zhipeng Wei Jingjing Chen Micah Goldblum Zuxuan Wu Tom Goldstein Yu-Gang Jiang ViT AAML 24 111 0 09 Sep 2021
Learning the Physics of Particle Transport via Transformers O. Pastor-Serrano Zoltán Perkó MedIm 21 13 0 08 Sep 2021
Scaled ReLU Matters for Training Vision Transformers Pichao Wang Xue Wang Haowen Luo Jingkai Zhou Zhipeng Zhou Fan Wang Hao Li R. L. Jin 19 41 0 08 Sep 2021
Ultra-high Resolution Image Segmentation via Locality-aware Context Fusion and Alternating Local Enhancement Wenxi Liu Qi Li Xin Lin Weixiang Yang Shengfeng He Yuanlong Yu 29 7 0 06 Sep 2021
Searching for Efficient Multi-Stage Vision Transformers Yi-Lun Liao S. Karaman Vivienne Sze ViT 16 19 0 01 Sep 2021
A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP Yucheng Zhao Guangting Wang Chuanxin Tang Chong Luo Wenjun Zeng Zhengjun Zha 35 69 0 30 Aug 2021
Do Vision Transformers See Like Convolutional Neural Networks? M. Raghu Thomas Unterthiner Simon Kornblith Chiyuan Zhang Alexey Dosovitskiy ViT 52 924 0 19 Aug 2021
Mobile-Former: Bridging MobileNet and Transformer Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Xiaoyi Dong Lu Yuan Zicheng Liu ViT 177 476 0 12 Aug 2021
ICAF: Iterative Contrastive Alignment Framework for Multimodal Abstractive Summarization Zijian Zhang Chang Shu Youxin Chen Jing Xiao Qian Zhang Lu Zheng 18 5 0 11 Aug 2021
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer Yifan Xu Zhijie Zhang Mengdan Zhang Kekai Sheng Ke Li Weiming Dong Liqing Zhang Changsheng Xu Xing Sun ViT 32 201 0 03 Aug 2021
DPT: Deformable Patch-based Transformer for Visual Recognition Zhiyang Chen Yousong Zhu Chaoyang Zhao Guosheng Hu Wei Zeng Jinqiao Wang Ming Tang ViT 16 98 0 30 Jul 2021
Contextual Transformer Networks for Visual Recognition Yehao Li Ting Yao Yingwei Pan Tao Mei ViT 22 468 0 26 Jul 2021
Visual Parser: Representing Part-whole Hierarchies with Transformers Shuyang Sun Xiaoyu Yue S. Bai Philip H. S. Torr 50 27 0 13 Jul 2021
Physics-Guided Deep Learning for Dynamical Systems: A Survey Rui Wang Rose Yu AI4CE PINN 39 64 0 02 Jul 2021
Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognition Vittorio Mazzia Simone Angarano Francesco Salvetti Federico Angelini Marcello Chiaberge ViT 25 136 0 01 Jul 2021
Early Convolutions Help Transformers See Better Tete Xiao Mannat Singh Eric Mintun Trevor Darrell Piotr Dollár Ross B. Girshick 20 752 0 28 Jun 2021
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers Andreas Steiner Alexander Kolesnikov Xiaohua Zhai Ross Wightman Jakob Uszkoreit Lucas Beyer ViT 39 614 0 18 Jun 2021
XCiT: Cross-Covariance Image Transformers Alaaeldin El-Nouby Hugo Touvron Mathilde Caron Piotr Bojanowski Matthijs Douze ... Ivan Laptev Natalia Neverova Gabriel Synnaeve Jakob Verbeek Hervé Jégou ViT 39 497 0 17 Jun 2021
Transformed CNNs: recasting pre-trained convolutional layers with self-attention Stéphane dÁscoli Levent Sagun Giulio Biroli Ari S. Morcos ViT 13 6 0 10 Jun 2021
On the Connection between Local Attention and Dynamic Depth-wise Convolution Qi Han Zejia Fan Qi Dai Lei-huan Sun Ming-Ming Cheng Jiaying Liu Jingdong Wang ViT 21 105 0 08 Jun 2021
Refiner: Refining Self-attention for Vision Transformers Daquan Zhou Yujun Shi Bingyi Kang Weihao Yu Zihang Jiang Yuan Li Xiaojie Jin Qibin Hou Jiashi Feng ViT 29 59 0 07 Jun 2021
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias Yufei Xu Qiming Zhang Jing Zhang Dacheng Tao ViT 56 329 0 07 Jun 2021
RegionViT: Regional-to-Local Attention for Vision Transformers Chun-Fu Chen Rameswar Panda Quanfu Fan ViT 16 194 0 04 Jun 2021
Container: Context Aggregation Network Peng Gao Jiasen Lu Hongsheng Li Roozbeh Mottaghi Aniruddha Kembhavi ViT 17 69 0 02 Jun 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 29 22 0 31 May 2021
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition Yulin Wang Rui Huang S. Song Zeyi Huang Gao Huang ViT 27 189 0 31 May 2021
Less is More: Pay Less Attention in Vision Transformers Zizheng Pan Bohan Zhuang Haoyu He Jing Liu Jianfei Cai ViT 21 82 0 29 May 2021
KVT: k-NN Attention for Boosting Vision Transformers Pichao Wang Xue Wang F. Wang Ming Lin Shuning Chang Hao Li R. L. Jin ViT 51 105 0 28 May 2021
Towards Robust Vision Transformer Xiaofeng Mao Gege Qi YueFeng Chen Xiaodan Li Ranjie Duan Shaokai Ye Yuan He Hui Xue ViT 17 186 0 17 May 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 271 2,603 0 04 May 2021
Vehicle Re-identification Method Based on Vehicle Attribute and Mutual Exclusion Between Cameras Junru Chen S. Geng Yongluan Yan Danyang Huang Hao Liu Yadong Li 31 3 0 30 Apr 2021
All Tokens Matter: Token Labeling for Training Better Vision Transformers Zihang Jiang Qibin Hou Li-xin Yuan Daquan Zhou Yujun Shi Xiaojie Jin Anran Wang Jiashi Feng ViT 16 203 0 22 Apr 2021
Escaping the Big Data Paradigm with Compact Transformers Ali Hassani Steven Walton Nikhil Shah Abulikemu Abuduweili Jiachen Li Humphrey Shi 54 462 0 12 Apr 2021
DA-DETR: Domain Adaptive Detection Transformer with Information Fusion Jingyi Zhang Jiaxing Huang Zhipeng Luo Gongjie Zhang Xiaoqin Zhang Shijian Lu ViT 4 35 0 31 Mar 2021
Conditional Positional Encodings for Vision Transformers Xiangxiang Chu Zhi Tian Bo-Wen Zhang Xinlong Wang Chunhua Shen ViT 22 602 0 22 Feb 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 290 979 0 27 Jan 2021
Low-latency Perception in Off-Road Dynamical Low Visibility Environments Nelson Alves Ferreira Neto Marco Ruiz M. Reis Tiago Cajahyba David F. N. Oliveira Ana Barreto Eduardo F. Simas Filho Wagner Luiz Alves de Oliveira L. Schnitman Roberto L. S. Monteiro 16 9 0 23 Dec 2020
Towards Learning Convolutions from Scratch Behnam Neyshabur SSL 220 71 0 27 Jul 2020