Papers citing "Scaling Vision Transformers to 22 Billion Parameters"

50 / 138 papers shown

Title
Rethinking Positive Pairs in Contrastive Learning Jiantao Wu Shentong Mo Zhenhua Feng Sara Atito Josef Kitler Muhammad Awais SSL VLM 136 3 0 23 Oct 2024
TIPS: Text-Image Pretraining with Spatial awareness Kevis-Kokitsi Maninis Kaifeng Chen Soham Ghosh Arjun Karpur Koert Chen ... Jan Dlabal Dan Gnanapragasam Mojtaba Seyedhosseini Howard Zhou Andre Araujo VLM 145 3 0 21 Oct 2024
FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model ZiDong Wang Zeyu Lu Di Huang Cai Zhou Wanli Ouyang and Lei Bai 126 6 0 17 Oct 2024
big.LITTLE Vision Transformer for Efficient Visual Recognition He Guo Yulong Wang Zixuan Ye Jifeng Dai Yuwen Xiong ViT 94 0 0 14 Oct 2024
Spatio-Temporal Control for Masked Motion Synthesis Ekkasit Pinyoanuntapong Muhammad Usama Saleem Korrawe Karunratanakul Pu Wang Hongfei Xue Chong Chen Chuan Guo Junli Cao J. Ren Sergey Tulyakov VGen 99 7 0 14 Oct 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Zou Tatsunori Hashimoto VLM 283 7 0 14 Oct 2024
SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning Hojoon Lee Dongyoon Hwang Donghu Kim Hyunseung Kim Jun Jet Tai K. Subramanian Peter R. Wurman Jaegul Choo Peter Stone Takuma Seno OffRL 192 17 0 13 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 167 34 0 10 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 641 1 0 07 Oct 2024
Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Prompts Minh Le Chau Nguyen Huy Nguyen Quyen Tran Trung Le Nhat Ho 137 8 0 03 Oct 2024
DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation Changdae Oh Yixuan Li Kyungwoo Song Sangdoo Yun Dongyoon Han OOD MoMe 91 10 0 03 Oct 2024
Selective Attention Improves Transformer Yaniv Leviathan Matan Kalman Yossi Matias 130 12 0 03 Oct 2024
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 154 8 0 30 Sep 2024
SOAP: Improving and Stabilizing Shampoo using Adam Nikhil Vyas Depen Morwani Rosie Zhao Itai Shapira David Brandfonbrener Lucas Janson Sham Kakade Sham Kakade 169 38 0 17 Sep 2024
Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation Weiqi Feng Yangrui Chen Shaoyu Wang Size Zheng H. Lin Minlan Yu MLLM AI4CE 150 4 0 07 Aug 2024
Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration Luciana Ferrer Daniel Ramos UQCV 68 4 0 05 Aug 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 134 10 0 24 Jul 2024
dMel: Speech Tokenization made Simple Richard He Bai Tatiana Likhomanenko Ruixiang Zhang Zijin Gu Zakaria Aldeneh Navdeep Jaitly 113 6 0 22 Jul 2024
Deconstructing What Makes a Good Optimizer for Language Models Rosie Zhao Depen Morwani David Brandfonbrener Nikhil Vyas Sham Kakade 152 25 0 10 Jul 2024
Controlling Space and Time with Diffusion Models Daniel Watson Saurabh Saxena Lala Li Andrea Tagliasacchi David J. Fleet VGen 168 32 0 10 Jul 2024
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 177 26 0 27 Jun 2024
Is AI fun? HumorDB: a curated dataset and benchmark to investigate graphical humor Veedant Jain Felipe dos Santos Alves Feitosa Gabriel Kreiman VLM 103 2 0 19 Jun 2024
GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement Yifan Yang Zheshu Song Jianheng Zhuo Mingyu Cui Jinpeng Li ... Shuai Fan Kai Yu Wei Zhang Guoguo Chen Xie Chen 142 12 0 17 Jun 2024
LRM-Zero: Training Large Reconstruction Models with Synthesized Data Desai Xie Sai Bi Zhixin Shu Kai Zhang Zexiang Xu Yi Zhou Soren Pirk Arie E. Kaufman Xin Sun Hao Tan SyDa 109 17 0 13 Jun 2024
Scaling White-Box Transformers for Vision Jinrui Yang Xianhang Li Druv Pai Yuyin Zhou Yi-An Ma Yaodong Yu Cihang Xie ViT 118 9 0 30 May 2024
From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks Jacob Russin Sam Whitman McGrath Danielle J. Williams Lotem Elber-Dorozko AI4CE 198 4 0 24 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 337 54 0 23 May 2024
How to set AdamW's weight decay as you scale model and dataset size Xi Wang Laurence Aitchison 161 11 0 22 May 2024
Chameleon: Mixed-Modal Early-Fusion Foundation Models Chameleon Team MLLM 217 338 0 16 May 2024
Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers Peng Gao Le Zhuo Ziyi Lin Ruoyi Du Xu Luo ... Weicai Ye He Tong Jingwen He Yu Qiao Hongsheng Li VGen 107 91 0 09 May 2024
What Foundation Models can Bring for Robot Learning in Manipulation : A Survey Dingzhe Li Yixiang Jin A. Yong Yong A Hongze Yu ... Huaping Liu Gang Hua F. Sun Jianwei Zhang Bin Fang AI4CE LM&Ro 226 15 0 28 Apr 2024
CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data Sachin Mehta Maxwell Horton Fartash Faghri Mohammad Hossein Sekhavat Mahyar Najibi Mehrdad Farajtabar Oncel Tuzel Mohammad Rastegari VLM CLIP 69 7 0 24 Apr 2024
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability Xiao Wang A. Tsaris Siyan Liu Jong Youl Choi Ming Fan Wei Zhang Ju Yin M. Ashfaq Dan Lu Prasanna Balaprakash 86 7 0 23 Apr 2024
Anatomy of Industrial Scale Multilingual ASR Francis McCann Ramirez Luka Chkhetiani Andrew Ehrenberg R. McHardy Rami Botros ... Ahmed Efty Daniel McCrystal Sam Flamini Domenic Donato Takuya Yoshioka 104 12 0 15 Apr 2024
Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models Jordan Dotzel Yash Akhauri Ahmed S. AbouElhamayed Carly Jiang Mohamed S. Abdelfattah Zhiru Zhang MoE 30 2 0 07 Apr 2024
Enhancing Efficiency in Vision Transformer Networks: Design Techniques and Insights Moein Heidari Reza Azad Sina Ghorbani Kolahi René Arimond Leon Niggemeier ... Afshin Bozorgpour Ehsan Khodapanah Aghdam Amirhossein Kazerouni Ilker Hacihaliloglu Dorit Merhof 99 7 0 28 Mar 2024
Tiny Models are the Computational Saver for Large Models Qingyuan Wang B. Cardiff Antoine Frappé Benoît Larras Deepu John 150 2 0 26 Mar 2024
ADAPT to Robustify Prompt Tuning Vision Transformers Masih Eskandar Tooba Imtiaz Zifeng Wang Jennifer Dy VPVLM VLM AAML 92 0 0 19 Mar 2024
Disentangling the Causes of Plasticity Loss in Neural Networks Clare Lyle Zeyu Zheng Khimya Khetarpal H. V. Hasselt Razvan Pascanu James Martens Will Dabney AI4CE 133 38 0 29 Feb 2024
Pretrained Visual Uncertainties Michael Kirchhof Mark Collier Seong Joon Oh Enkelejda Kasneci UQCV 555 8 1 26 Feb 2024
For Better or For Worse? Learning Minimum Variance Features With Label Augmentation Muthuraman Chidambaram Rong Ge AAML 116 0 0 10 Feb 2024
SpecFormer: Guarding Vision Transformer Robustness via Maximum Singular Value Penalization Xixu Hu Runkai Zheng Jindong Wang Cheuk Hang Leung Qi Wu Xing Xie 57 1 0 02 Jan 2024
Neither hype nor gloom do DNNs justice Gaurav Malhotra Christian Tsvetkov B. D. Evans 92 125 0 08 Dec 2023
SAMBA: A Trainable Segmentation Web-App with Smart Labelling Ronan Docherty Isaac Squires Antonis Vamvakeros Samuel J. Cooper 27 4 0 07 Dec 2023
Critical Influence of Overparameterization on Sharpness-aware Minimization Sungbin Shin Dongyeop Lee Maksym Andriushchenko Namhoon Lee AAML 166 2 0 29 Nov 2023
MatFormer: Nested Transformer for Elastic Inference Devvrit Sneha Kudugunta Aditya Kusupati Tim Dettmers Kaifeng Chen ... Yulia Tsvetkov Hannaneh Hajishirzi Sham Kakade Ali Farhadi Prateek Jain 114 31 0 11 Oct 2023
GAIA-1: A Generative World Model for Autonomous Driving Masane Fuchi Lloyd Russell Hudson Yeo Zak Murez Hiroto Minami Alex Kendall Tomohiro Takagi Gianluca Corrado VGen 136 253 0 29 Sep 2023
Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping Martin Pelikan Sheikh Shams Azam Vitaly Feldman Jan Honza Silovsky Kunal Talwar Christopher G. Brinton Tatiana Likhomanenko 113 8 0 29 Sep 2023
Small-scale proxies for large-scale Transformer training instabilities Mitchell Wortsman Peter J. Liu Lechao Xiao Katie Everett A. Alemi ... Jascha Narain Sohl-Dickstein Kelvin Xu Jaehoon Lee Justin Gilmer Simon Kornblith 140 99 0 25 Sep 2023
ClusterFormer: Clustering As A Universal Visual Learner James Liang Yiming Cui Qifan Wang Tong Geng Wenguan Wang Dongfang Liu VLM 104 10 0 22 Sep 2023