Exploring Learngene via Stage-wise Weight Sharing for Initializing
Variable-sized Models

Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models

25 April 2024

ArXiv (abs)PDF HTML

Papers citing "Exploring Learngene via Stage-wise Weight Sharing for Initializing Variable-sized Models"

14 / 14 papers shown

Title
Transferring Core Knowledge via Learngenes Fu Feng Jing Wang Xin Geng 78 7 0 16 Jan 2024
Building Variable-sized Models via Learngene Pool Boyu Shi Shiyu Xia Xu Yang Haokun Chen Zhi Kou Xin Geng 85 2 0 10 Dec 2023
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks Wenhui Wang Hangbo Bao Li Dong Johan Bjorck Zhiliang Peng ... Kriti Aggarwal O. Mohammed Saksham Singhal Subhojit Som Furu Wei MLLM VLM ViT 141 642 0 22 Aug 2022
MiniViT: Compressing Vision Transformers with Weight Multiplexing Jinnian Zhang Houwen Peng Kan Wu Mengchen Liu Bin Xiao Jianlong Fu Lu Yuan ViT 70 124 0 14 Apr 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 465 7,757 0 11 Nov 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 274 2,826 0 15 Jun 2021
Lessons on Parameter Sharing across Layers in Transformers Sho Takase Shun Kiyono 53 87 0 13 Apr 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 450 21,439 0 25 Mar 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 387 6,768 0 23 Dec 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 511 42,449 0 03 Dec 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 368 6,455 0 26 Sep 2019
How to Initialize your Network? Robust Initialization for WeightNorm & ResNets Devansh Arpit Victor Campos Yoshua Bengio 52 59 0 05 Jun 2019
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 413 10,494 0 21 Jul 2016
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 326 18,625 0 06 Feb 2015

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.