Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers

24 November 2021

Papers citing "Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers"

49 / 49 papers shown

Title
FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis Ziqi Ni Ao Fu Yi Zhou 147 0 0 06 Mar 2025
DGenNO: A Novel Physics-aware Neural Operator for Solving Forward and Inverse PDE Problems based on Deep, Generative Probabilistic Modeling Yaohua Zang P. Koutsourelakis AI4CE 84 1 0 10 Feb 2025
Efficient Fourier Filtering Network with Contrastive Learning for UAV-based Unaligned Bi-modal Salient Object Detection Pengfei Lyu Pak-Hei Yeung Xiufei Cheng Xiaosheng Yu Chengdong Wu Jagath C. Rajapakse 64 0 0 06 Nov 2024
Can AI weather models predict out-of-distribution gray swan tropical cyclones? Y. Qiang Sun Pedram Hassanzadeh Mohsen Zand Ashesh Chattopadhyay Jonathan Weare D. Abbot 80 7 0 19 Oct 2024
Mamba Neural Operator: Who Wins? Transformers vs. State-Space Models for PDEs Chun-Wun Cheng Jiahao Huang Yi Zhang Guang Yang Carola-Bibiane Schonlieb Angelica I Aviles-Rivero Mamba AI4CE 108 4 0 03 Oct 2024
Reduced-Order Neural Operators: Learning Lagrangian Dynamics on Highly Sparse Graphs Hrishikesh Viswanath Yue Chang Julius Berner Julius Berner Peter Yichen Chen Aniket Bera AI4CE 97 2 0 04 Jul 2024
Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling Wanghan Xu Fenghua Ling Wenlong Zhang Tao Han Hao Chen Wanli Ouyang Lei Bai AI4CE 140 5 0 22 May 2024
Data Assimilation with Machine Learning Surrogate Models: A Case Study with FourCastNet Melissa Adrian Daniel Sanz-Alonso Rebecca Willett 88 5 0 21 May 2024
Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention Ethan N. Evans Matthew G. Cook Zachary P. Bradshaw Margarite L. LaBorde 133 5 0 21 Mar 2024
PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction Hao Wu Wei Xion Fan Xu Xian-Sheng Hua C. L. Philip Chen Xiansheng Hua AI4TS 150 28 0 19 May 2023
A Survey on Deep Learning based Time Series Analysis with Frequency Transformation Kun Yi Qi Zhang LongBing Cao Shoujin Wang Guodong Long Liang Hu Hui He ZhenDong Niu Wei Fan Hui Xiong AI4TS 114 23 0 04 Feb 2023
Multiwavelet-based Operator Learning for Differential Equations Gaurav Gupta Xiongye Xiao P. Bogdan 154 213 0 28 Sep 2021
U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow Gege Wen Zong-Yi Li Kamyar Azizzadenesheli Anima Anandkumar S. Benson AI4CE 63 383 0 03 Sep 2021
Neural Operator: Learning Maps Between Function Spaces Nikola B. Kovachki Zong-Yi Li Burigede Liu Kamyar Azizzadenesheli K. Bhattacharya Andrew M. Stuart Anima Anandkumar AI4CE 97 446 0 19 Aug 2021
AS-MLP: An Axial Shifted MLP Architecture for Vision Dongze Lian Zehao Yu Xing Sun Shenghua Gao 112 189 0 18 Jul 2021
Long-Short Transformer: Efficient Transformers for Language and Vision Chen Zhu Ming-Yu Liu Chaowei Xiao Mohammad Shoeybi Tom Goldstein Anima Anandkumar Bryan Catanzaro ViT VLM 66 133 0 05 Jul 2021
Global Filter Networks for Image Classification Yongming Rao Wenliang Zhao Zheng Zhu Jiwen Lu Jie Zhou ViT 60 463 0 01 Jul 2021
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers Enze Xie Wenhai Wang Zhiding Yu Anima Anandkumar J. Álvarez Ping Luo ViT 231 4,990 0 31 May 2021
Choose a Transformer: Fourier or Galerkin Shuhao Cao 60 240 0 31 May 2021
Pay Attention to MLPs Hanxiao Liu Zihang Dai David R. So Quoc V. Le AI4CE 108 659 0 17 May 2021
FNet: Mixing Tokens with Fourier Transforms James Lee-Thorp Joshua Ainslie Ilya Eckstein Santiago Ontanon 90 526 0 09 May 2021
ResMLP: Feedforward networks for image classification with data-efficient training Hugo Touvron Piotr Bojanowski Mathilde Caron Matthieu Cord Alaaeldin El-Nouby ... Gautier Izacard Armand Joulin Gabriel Synnaeve Jakob Verbeek Hervé Jégou VLM 75 660 0 07 May 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 399 2,658 0 04 May 2021
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort Yuxuan Zhang Huan Ling Jun Gao K. Yin Jean-Francois Lafleche Adela Barriuso Antonio Torralba Sanja Fidler 3DH GAN VLM 74 334 0 13 Apr 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 398 21,281 0 25 Mar 2021
Random Feature Attention Hao Peng Nikolaos Pappas Dani Yogatama Roy Schwartz Noah A. Smith Lingpeng Kong 80 356 0 03 Mar 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 483 3,699 0 24 Feb 2021
Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention Yunyang Xiong Zhanpeng Zeng Rudrasis Chakraborty Mingxing Tan G. Fung Yin Li Vikas Singh 84 517 0 07 Feb 2021
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 526 40,739 0 22 Oct 2020
Fourier Neural Operator for Parametric Partial Differential Equations Zong-Yi Li Nikola B. Kovachki Kamyar Azizzadenesheli Burigede Liu K. Bhattacharya Andrew M. Stuart Anima Anandkumar AI4CE 453 2,384 0 18 Oct 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 165 1,570 0 30 Sep 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 144 1,115 0 14 Sep 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 166 1,755 0 29 Jun 2020
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 185 1,694 0 08 Jun 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 128 4,048 0 10 Apr 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 294 596 0 12 Mar 2020
Neural Operator: Graph Kernel Network for Partial Differential Equations Zong-Yi Li Nikola B. Kovachki Kamyar Azizzadenesheli Burigede Liu K. Bhattacharya Andrew M. Stuart Anima Anandkumar 157 727 0 07 Mar 2020
Sparse Sinkhorn Attention Yi Tay Dara Bahri Liu Yang Donald Metzler Da-Cheng Juan 76 339 0 26 Feb 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 181 2,307 0 13 Jan 2020
Axial Attention in Multidimensional Transformers Jonathan Ho Nal Kalchbrenner Dirk Weissenborn Tim Salimans 98 528 0 20 Dec 2019
Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel Yao-Hung Hubert Tsai Shaojie Bai M. Yamada Louis-Philippe Morency Ruslan Salakhutdinov 98 254 0 30 Aug 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 93 1,894 0 23 Apr 2019
Image Transformer Niki Parmar Ashish Vaswani Jakob Uszkoreit Lukasz Kaiser Noam M. Shazeer Alexander Ku Dustin Tran ViT 104 1,678 0 15 Feb 2018
MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark Sandler Andrew G. Howard Menglong Zhu A. Zhmoginov Liang-Chieh Chen 169 19,204 0 13 Jan 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 624 130,942 0 12 Jun 2017
The Cityscapes Dataset for Semantic Urban Scene Understanding Marius Cordts Mohamed Omran Sebastian Ramos Timo Rehfeld Markus Enzweiler Rodrigo Benenson Uwe Franke Stefan Roth Bernt Schiele 904 11,587 0 06 Apr 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.9K 193,426 0 10 Dec 2015
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop Feng Yu Ari Seff Yinda Zhang Shuran Song Thomas Funkhouser Jianxiong Xiao 67 2,331 0 10 Jun 2015
Deep Learning Face Attributes in the Wild Ziwei Liu Ping Luo Xiaogang Wang Xiaoou Tang CVBM 224 8,391 0 28 Nov 2014