FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

27 May 2022

Papers citing "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness"

50 / 1,436 papers shown

Title
RoFormer for Position Aware Multiple Instance Learning in Whole Slide Image Classification Etienne Pochet Rami Maroun Roger Trullo MedIm 23 2 0 03 Oct 2023
Ring Attention with Blockwise Transformers for Near-Infinite Context Hao Liu Matei A. Zaharia Pieter Abbeel 38 218 0 03 Oct 2023
SEA: Sparse Linear Attention with Estimated Attention Mask Heejun Lee Jina Kim Jeffrey Willette Sung Ju Hwang 38 6 0 03 Oct 2023
PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels Praneeth Kacham Vahab Mirrokni Peilin Zhong 44 7 0 02 Oct 2023
CAT-LM: Training Language Models on Aligned Code And Tests Nikitha Rao Kush Jain Uri Alon Claire Le Goues Vincent J. Hellendoorn ALM 42 42 0 02 Oct 2023
GRID: A Platform for General Robot Intelligence Development Sai H. Vemprala Shuhang Chen Abhinav Shukla Dinesh Narayanan Ashish Kapoor 25 10 0 02 Oct 2023
Learning Type Inference for Enhanced Dataflow Analysis Lukas Seidel Sedick Baker Effendi Xavier Pinho Konrad Rieck Brink van der Merwe Fabian Yamaguchi 25 2 0 01 Oct 2023
GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Chia-Yuan Chang Xia Hu 33 11 0 01 Oct 2023
Efficient Streaming Language Models with Attention Sinks Michel Lang Yuandong Tian Beidi Chen Song Han Mike Lewis AI4TS RALM 44 649 0 29 Sep 2023
GAIA-1: A Generative World Model for Autonomous Driving Masane Fuchi Lloyd Russell Hudson Yeo Zak Murez Hiroto Minami Alex Kendall Tomohiro Takagi Gianluca Corrado VGen 30 217 0 29 Sep 2023
Training a Large Video Model on a Single Machine in a Day Yue Zhao Philipp Krahenbuhl VLM 34 15 0 28 Sep 2023
Qwen Technical Report Jinze Bai Shuai Bai Yunfei Chu Zeyu Cui Kai Dang ... Zhenru Zhang Chang Zhou Jingren Zhou Xiaohuan Zhou Tianhang Zhu OSLM 64 1,607 0 28 Sep 2023
AtomSurf : Surface Representation for Learning on Protein Structures Vincent Mallet Souhaib Attaiki M. Ovsjanikov 42 3 0 28 Sep 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 34 15 0 28 Sep 2023
Predicting performance difficulty from piano sheet music images Yingwei Ma J. J. Valero-Mas Yu Jiang Changjian Wang 23 2 0 28 Sep 2023
Attention Sorting Combats Recency Bias In Long Context Language Models A. Peysakhovich Adam Lerer LRM RALM 42 42 0 28 Sep 2023
Masked Autoencoders are Scalable Learners of Cellular Morphology Oren Z. Kraus Kian Kenyon-Dean Saber Saberian Maryam Fallah Peter McLean ... Chi Vicky Cheng Kristen Morse Maureen Makes Ben Mabey Berton Earnshaw 27 14 0 27 Sep 2023
Effective Long-Context Scaling of Foundation Models Wenhan Xiong Jingyu Liu Igor Molybog Hejia Zhang Prajjwal Bhargava ... Dániel Baráth Sergey Edunov Mike Lewis Sinong Wang Hao Ma 37 207 0 27 Sep 2023
Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning William Chen Jiatong Shi Brian Yan Dan Berrebbi Wangyou Zhang Yifan Peng Xuankai Chang Soumi Maiti Shinji Watanabe 32 8 0 26 Sep 2023
DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models S. A. Jacobs Masahiro Tanaka Chengming Zhang Minjia Zhang L. Song Samyam Rajbhandari Yuxiong He 25 103 0 25 Sep 2023
LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression Ayush Kaushal Tejas Vaidhya Irina Rish 64 15 0 25 Sep 2023
MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models Kailai Yang Tianlin Zhang Zi-Zhou Kuang Qianqian Xie Jimin Huang Sophia Ananiadou AI4MH 38 47 0 24 Sep 2023
BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models Zican Dong Tianyi Tang Junyi Li Wayne Xin Zhao Ji-Rong Wen RALM ALM 33 33 0 23 Sep 2023
AntiBARTy Diffusion for Property Guided Antibody Design Jordan Venderley DiffM 16 1 0 22 Sep 2023
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Yukang Chen Shengju Qian Haotian Tang Xin Lai Zhijian Liu Song Han Jiaya Jia 53 152 0 21 Sep 2023
DreamLLM: Synergistic Multimodal Comprehension and Creation Runpei Dong Chunrui Han Yuang Peng Zekun Qi Zheng Ge ... Hao-Ran Wei Xiangwen Kong Xiangyu Zhang Kaisheng Ma Li Yi MLLM 39 174 0 20 Sep 2023
The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute Aleksandar Stanić Dylan R. Ashley Oleg Serikov Louis Kirsch Francesco Faccio Jürgen Schmidhuber Thomas Hofmann Imanol Schlag MoE 43 9 0 20 Sep 2023
SlimPajama-DC: Understanding Data Combinations for LLM Training Zhiqiang Shen Tianhua Tao Liqun Ma Willie Neiswanger Zhengzhong Liu ... Bowen Tan Joel Hestness Natalia Vassilieva Daria Soboleva Eric Xing 27 45 0 19 Sep 2023
FoleyGen: Visually-Guided Audio Generation Xinhao Mei Varun K. Nagaraja Gaël Le Lan Zhaoheng Ni Ernie Chang Yangyang Shi Vikas Chandra VGen 26 21 0 19 Sep 2023
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 76 78 0 19 Sep 2023
Baichuan 2: Open Large-scale Language Models Ai Ming Yang Bin Xiao Bingning Wang Borong Zhang Ce Bian ... Youxin Jiang Yuchen Gao Yupeng Zhang Zenan Zhou Zhiying Wu ELM LRM 77 703 0 19 Sep 2023
Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF Simeng Sun Dhawal Gupta Mohit Iyyer 27 17 0 16 Sep 2023
Enhance audio generation controllability through representation similarity regularization Yangyang Shi Gaël Le Lan Varun K. Nagaraja Zhaoheng Ni Xinhao Mei Ernie Chang Forrest N. Iandola Yang Liu Vikas Chandra 42 1 0 15 Sep 2023
Replacing softmax with ReLU in Vision Transformers Mitchell Wortsman Jaehoon Lee Justin Gilmer Simon Kornblith ViT 33 33 0 15 Sep 2023
CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending Shiyi Zhu Jingting Ye Wei Jiang Siqiao Xue Qi Zhang Yifan Wu Jianguo Li 27 4 0 15 Sep 2023
Less is More for Long Document Summary Evaluation by LLMs Yunshu Wu Hayate Iso Pouya Pezeshkpour Nikita Bhutani Estevam R. Hruschka 24 34 0 14 Sep 2023
Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors J. Pata Eric Wulff Farouk Mokhtar D. Southwick Mengke Zhang M. Girone Javier Duarte 30 1 0 13 Sep 2023
Efficient Memory Management for Large Language Model Serving with PagedAttention Woosuk Kwon Zhuohan Li Siyuan Zhuang Ying Sheng Lianmin Zheng Cody Hao Yu Joseph E. Gonzalez Haotong Zhang Ion Stoica VLM 29 1,803 0 12 Sep 2023
CaloClouds II: Ultra-Fast Geometry-Independent Highly-Granular Calorimeter Simulation E. Buhmann F. Gaede Gregor Kasieczka A. Korol W. Korcari K. Krüger Peter McKeown DiffM 38 24 0 11 Sep 2023
Textbooks Are All You Need II: phi-1.5 technical report Yuan-Fang Li Sébastien Bubeck Ronen Eldan Allison Del Giorno Suriya Gunasekar Yin Tat Lee ALM LRM 36 445 0 11 Sep 2023
Evaluating the Deductive Competence of Large Language Models S. M. Seals V. Shalin ELM ReLM LRM 19 8 0 11 Sep 2023
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models Liang Li Qingyuan Li Bo-Wen Zhang Xiangxiang Chu MQ 47 29 0 06 Sep 2023
Music Source Separation with Band-Split RoPE Transformer Wei-Tsung Lu Ju-Chiang Wang Qiuqiang Kong Yun-Ning Hung 27 19 0 05 Sep 2023
nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources Piotr Nawrot AI4CE 17 5 0 05 Sep 2023
Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes Sunjun Kweon Junu Kim Jiyoun Kim Sujeong Im Eunbyeol Cho ... Seungjin Baek Chang Hoon Han Yoon Bin Jung Yohan Jo Edward Choi LM&MA ELM 15 37 0 01 Sep 2023
PointLLM: Empowering Large Language Models to Understand Point Clouds Runsen Xu Xiaolong Wang Tai Wang Yilun Chen Jiangmiao Pang Dahua Lin MLLM 61 151 0 31 Aug 2023
SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills Amey Agrawal Ashish Panwar Jayashree Mohan Nipun Kwatra Bhargav S. Gulavani Ramachandran Ramjee AI4TS LRM 39 93 0 31 Aug 2023
A General-Purpose Self-Supervised Model for Computational Pathology Richard J. Chen Tong Ding Ming Y. Lu Drew F. K. Williamson Guillaume Jaume ... Judy J. Wang Walt Williams L. Le Georg Gerber Faisal Mahmood MedIm 28 42 0 29 Aug 2023
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models Qingyue Wang Y. Fu Yanan Cao Zhiliang Tian Shi Wang Dacheng Tao LLMAG KELM RALM 67 24 0 29 Aug 2023
Multiscale Contextual Learning for Speech Emotion Recognition in Emergency Call Center Conversations Théo Deschamps-Berger L. Lamel Laurence Devillers 19 2 0 28 Aug 2023