FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

27 May 2022

Papers citing "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness"

50 / 1,436 papers shown

Title
MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records Scott L. Fleming Alejandro Lozano W. Haberkorn Jenelle A. Jindal E. Reis ... Jonathan H. Chen Keith Morse Emma Brunskill Jason Alan Fries N. Shah LM&MA 30 54 0 27 Aug 2023
Aligning Language Models with Offline Learning from Human Feedback Jian Hu Li Tao J. Yang Chandler Zhou ALM OffRL 27 7 0 23 Aug 2023
From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning Ming Li Yong Zhang Zhitao Li Jiuhai Chen Lichang Chen Ning Cheng Jianzong Wang Dinesh Manocha Jing Xiao 48 176 0 23 Aug 2023
How Much Temporal Long-Term Context is Needed for Action Segmentation? Emad Bahrami Rad Gianpiero Francesca Juergen Gall ViT 24 25 0 22 Aug 2023
Instruction Tuning for Large Language Models: A Survey Shengyu Zhang Linfeng Dong Xiaoya Li Sen Zhang Xiaofei Sun ... Jiwei Li Runyi Hu Tianwei Zhang Fei Wu Guoyin Wang LM&MA 24 546 0 21 Aug 2023
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models Neel Guha Julian Nyarko Daniel E. Ho Christopher Ré Adam Chilton ... Spencer Williams Sunny G. Gandhi Tomer Zur Varun J. Iyer Zehua Li AILaw LRM ELM 17 147 0 20 Aug 2023
LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models Yixuan Weng Zhiqi Wang Huanxuan Liao Shizhu He Shengping Liu Kang Liu Jun Zhao 45 3 0 20 Aug 2023
MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation Junru Lu Siyu An Mingbao Lin Gabriele Pergola Yulan He Di Yin Xing Sun Yunsheng Wu 49 32 0 16 Aug 2023
OctoPack: Instruction Tuning Code Large Language Models Niklas Muennighoff Qian Liu A. Zebaze Qinkai Zheng Binyuan Hui Terry Yue Zhuo Swayam Singh Xiangru Tang Leandro von Werra Shayne Longpre VLM ALM 71 119 0 14 Aug 2023
Pairing interacting protein sequences using masked language modeling Umberto Lupo Damiano Sgarbossa Anne-Florence Bitbol 27 12 0 14 Aug 2023
Large Language Models for Telecom: Forthcoming Impact on the Industry Ali Maatouk Nicola Piovesan Fadhel Ayed Antonio De Domenico Merouane Debbah 22 50 0 11 Aug 2023
Encode-Store-Retrieve: Enhancing Memory Augmentation through Language-Encoded Egocentric Perception Junxiao Shen John J. Dudley Per Ola Kristensson RALM 25 0 0 10 Aug 2023
Accelerating LLM Inference with Staged Speculative Decoding Benjamin Spector Christal Re 17 102 0 08 Aug 2023
Continual Pre-Training of Large Language Models: How to (re)warm your model? Kshitij Gupta Benjamin Thérien Adam Ibrahim Mats L. Richter Quentin G. Anthony Eugene Belilovsky Irina Rish Timothée Lesort KELM 35 99 0 08 Aug 2023
DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis Zhongjie Duan Lizhou You Chengyu Wang Cen Chen Ziheng Wu Weining Qian Jun Huang DiffM 34 8 0 07 Aug 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 28 3 0 07 Aug 2023
LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning Longteng Zhang Lin Zhang S. Shi X. Chu Bo-wen Li AI4CE 18 92 0 07 Aug 2023
Unfolding Once is Enough: A Deployment-Friendly Transformer Unit for Super-Resolution Yong Liu Hang Dong Boyang Liang Song Liu Qingji Dong Kai Chen Fangmin Chen Lean Fu Fei Wang ViT 21 12 0 05 Aug 2023
JIANG: Chinese Open Foundation Language Model Qinhua Duan Wenchao Gu Yujia Chen Wenxin Mao Zewen Tian Huibing Cao ALM 24 0 0 01 Aug 2023
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer Zhen Qin Dong Li Weigao Sun Weixuan Sun Xuyang Shen ... Yunshen Wei Baohong Lv Xiao Luo Yu Qiao Yiran Zhong 43 15 0 27 Jul 2023
Multilingual Code Co-Evolution Using Large Language Models Jiyang Zhang Pengyu Nie Junyi Jessy Li Miloš Gligorić 32 20 0 27 Jul 2023
Global k-Space Interpolation for Dynamic MRI Reconstruction using Masked Image Modeling Jia-Yu Pan Suprosanna Shit Özgün Turgut Wenqi Huang Hongwei Bran Li Nil Stolt Ansó Thomas Kustner Kerstin Hammernik Daniel Rueckert 33 9 0 24 Jul 2023
L-Eval: Instituting Standardized Evaluation for Long Context Language Models Chen An Shansan Gong Ming Zhong Xingjian Zhao Mukai Li Jun Zhang Lingpeng Kong Xipeng Qiu ELM ALM 40 132 0 20 Jul 2023
FABRIC: Personalizing Diffusion Models with Iterative Feedback Dimitri von Rütte Elisabetta Fedele Jonathan Thomm Lukas Wolf 27 13 0 19 Jul 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 30 1,140 0 17 Jul 2023
Retentive Network: A Successor to Transformer for Large Language Models Yutao Sun Li Dong Shaohan Huang Shuming Ma Yuqing Xia Jilong Xue Jianyong Wang Furu Wei LRM 78 304 0 17 Jul 2023
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation Yi Wang Yinan He Yizhuo Li Kunchang Li Jiashuo Yu ... Ping Luo Ziwei Liu Yali Wang Limin Wang Yu Qiao VLM VGen 33 248 0 13 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 22 41 0 12 Jul 2023
A Comprehensive Overview of Large Language Models Humza Naveed Asad Ullah Khan Shi Qiu Muhammad Saqib Saeed Anwar Muhammad Usman Naveed Akhtar Nick Barnes Ajmal Mian OffRL 70 529 0 12 Jul 2023
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution Mostafa Dehghani Basil Mustafa Josip Djolonga Jonathan Heek Matthias Minderer ... Avital Oliver Piotr Padlewski A. Gritsenko Mario Luvcić N. Houlsby ViT 26 105 0 12 Jul 2023
ReLoRA: High-Rank Training Through Low-Rank Updates Vladislav Lialin Namrata Shivagunde Sherin Muckatira Anna Rumshisky BDL 37 94 0 11 Jul 2023
Large Language Models as General Pattern Machines Suvir Mirchandani F. Xia Peter R. Florence Brian Ichter Danny Driess Montse Gonzalez Arenas Kanishka Rao Dorsa Sadigh Andy Zeng LLMAG 57 185 0 10 Jul 2023
Large Language Models as Batteries-Included Zero-Shot ESCO Skills Matchers Benjamin Clavié Guillaume Soulié 26 11 0 07 Jul 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 40 1,424 0 06 Jul 2023
Focused Transformer: Contrastive Training for Context Scaling Szymon Tworkowski Konrad Staniszewski Mikolaj Pacek Yuhuai Wu Henryk Michalewski Piotr Milo's 29 136 0 06 Jul 2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens Jiayu Ding Shuming Ma Li Dong Xingxing Zhang Shaohan Huang Wenhui Wang Nanning Zheng Furu Wei CLL 41 151 0 05 Jul 2023
SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding Vasilisa Bashlovkina Riley Matthews Zhaobin Kuang Simon Baumgartner Michael Bendersky 40 4 0 30 Jun 2023
RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark Federico Berto Chuanbo Hua Junyoung Park Laurin Luttmann Yining Ma ... Guojie Song Changhyun Kwon Kevin Tierney Lin Xie Jinkyoo Park OffRL 32 27 0 29 Jun 2023
FLuRKA: Fast and accurate unified Low-Rank & Kernel Attention Ahan Gupta Hao Guo Yueming Yuan Yan-Quan Zhou Charith Mendis 21 2 0 27 Jun 2023
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution Eric N. D. Nguyen Michael Poli Marjan Faizi A. Thomas Callum Birch-Sykes ... Stefano Massaroli Yoshua Bengio Stefano Ermon S. Baccus Christopher Ré MedIm 19 222 0 27 Jun 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 17 494 0 27 Jun 2023
When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions Weiming Zhuang Chen Chen Lingjuan Lyu Cheng Chen Yaochu Jin Lingjuan Lyu AIFin AI4CE 99 85 0 27 Jun 2023
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome Zhihan Zhou Yanrong Ji Weijian Li Pratik Dutta R. Davuluri Han Liu 16 173 0 26 Jun 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu (Allen) Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 61 255 0 24 Jun 2023
Bring Your Own Data! Self-Supervised Evaluation for Large Language Models Neel Jain Khalid Saifullah Yuxin Wen John Kirchenbauer Manli Shu Aniruddha Saha Micah Goldblum Jonas Geiping Tom Goldstein ALM ELM 33 23 0 23 Jun 2023
LightGlue: Local Feature Matching at Light Speed Philipp Lindenberger Paul-Edouard Sarlin Marc Pollefeys 3DV VLM 31 396 0 23 Jun 2023
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models Shizhe Diao Rui Pan Hanze Dong Kashun Shum Jipeng Zhang Wei Xiong Tong Zhang ALM 22 63 0 21 Jun 2023
Textbooks Are All You Need Suriya Gunasekar Yi Zhang J. Aneja C. C. T. Mendes Allison Del Giorno ... Sébastien Bubeck Ronen Eldan Adam Tauman Kalai Y. Lee Yuan-Fang Li AI4CE ALM SyDa 27 392 0 20 Jun 2023
Sparse Modular Activation for Efficient Sequence Modeling Liliang Ren Yang Liu Shuohang Wang Yichong Xu Chenguang Zhu Chengxiang Zhai 53 13 0 19 Jun 2023
Anticipatory Music Transformer John Thickstun David Leo Wright Hall Chris Donahue Percy Liang 23 15 0 14 Jun 2023