Large scale distributed neural network training through online distillation

9 April 2018

Papers citing "Large scale distributed neural network training through online distillation"

50 / 68 papers shown

Title
Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation Qianren Mao Qili Zhang Hanwen Hao Zhentao Han Runhua Xu ... Bo Li Y. Song Jin Dong Jianxin Li Philip S. Yu 71 0 0 27 Apr 2025
Beyond Model Scale Limits: End-Edge-Cloud Federated Learning with Self-Rectified Knowledge Agglomeration Zhiyuan Wu Sheng Sun Yuwei Wang Min Liu Ke Xu Quyang Pan Bo Gao Tian Wen FedML 30 0 0 03 Jan 2025
GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings Raghuveer Thirukovalluru Bhuwan Dhingra 31 2 0 18 Oct 2024
A Lightweight Target-Driven Network of Stereo Matching for Inland Waterways Jing Su Yiqing Zhou Yu Zhang Chao Wang Yi Wei 3DV 28 0 0 10 Oct 2024
The Curse of Diversity in Ensemble-Based Exploration Zhixuan Lin P. DÓro Evgenii Nikishin Aaron C. Courville 40 1 0 07 May 2024
Practical Insights into Knowledge Distillation for Pre-Trained Models Norah Alballa Marco Canini 37 2 0 22 Feb 2024
Predictive Churn with the Set of Good Models J. Watson-Daniels Flavio du Pin Calmon Alexander DÁmour Carol Xuan Long David C. Parkes Berk Ustun 81 7 0 12 Feb 2024
Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning Joonyoung Kim Kangwook Lee Haebin Shin Hurnjoo Lee Sechun Kang Byunguk Choi Dong Shin Joohyung Lee 18 0 0 15 Jul 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 22 12 0 28 Jan 2023
Knowledge Distillation in Federated Edge Learning: A Survey Zhiyuan Wu Sheng Sun Yuwei Wang Min Liu Xue Jiang Runhan Li Bo Gao FedML 27 4 0 14 Jan 2023
FedICT: Federated Multi-task Distillation for Multi-access Edge Computing Zhiyuan Wu Sheng Sun Yuwei Wang Min Liu Quyang Pan Xue Jiang Bo Gao 35 31 0 01 Jan 2023
Co-training $2^L$ Submodels for Visual Recognition Hugo Touvron Matthieu Cord Maxime Oquab Piotr Bojanowski Jakob Verbeek Hervé Jégou VLM 27 9 0 09 Dec 2022
Scalable Collaborative Learning via Representation Sharing Frédéric Berdoz Abhishek Singh Martin Jaggi Ramesh Raskar FedML 22 3 0 20 Nov 2022
Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challenges Enrique Tomás Martínez Beltrán Mario Quiles Pérez Pedro Miguel Sánchez Sánchez Sergio López Bernal Gérome Bovet M. Pérez Gregorio Martínez Pérez Alberto Huertas Celdrán FedML 18 221 0 15 Nov 2022
Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision Jiongyu Guo Defang Chen Can Wang 14 3 0 25 Oct 2022
lo-fi: distributed fine-tuning without communication Mitchell Wortsman Suchin Gururangan Shen Li Ali Farhadi Ludwig Schmidt Michael G. Rabbat Ari S. Morcos 19 24 0 19 Oct 2022
On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models Rohan Anil S. Gadanho Danya Huang Nijith Jacob Zhuoshu Li ... Cristina Pop Kevin Regan G. Shamir Rakesh Shivanna Qiqi Yan 3DV 16 41 0 12 Sep 2022
ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval Nicola Messina Matteo Stefanini Marcella Cornia Lorenzo Baraldi Fabrizio Falchi Giuseppe Amato Rita Cucchiara VLM 11 21 0 29 Jul 2022
Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources Ji Liu Daxiang Dong Xi Wang An Qin Xingjian Li P. Valduriez Dejing Dou Dianhai Yu 18 6 0 14 Jul 2022
PrUE: Distilling Knowledge from Sparse Teacher Networks Shaopu Wang Xiaojun Chen Mengzhen Kou Jinqiao Shi 8 2 0 03 Jul 2022
Teach me how to Interpolate a Myriad of Embeddings Shashanka Venkataramanan Ewa Kijak Laurent Amsaleg Yannis Avrithis 26 2 0 29 Jun 2022
Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt Sören Mindermann J. Brauner Muhammed Razzak Mrinank Sharma Andreas Kirsch ... Benedikt Höltgen Aidan N. Gomez Adrien Morisot Sebastian Farquhar Y. Gal 39 148 0 14 Jun 2022
Federated Distillation based Indoor Localization for IoT Networks Yaya Etiabi Marwa Chafii El-Mehdi Amhoud FedML 28 15 0 23 May 2022
Reducing Model Jitter: Stable Re-training of Semantic Parsers in Production Environments Christopher Hidey Fei Liu Rahul Goel 19 4 0 10 Apr 2022
Learning to Collide: Recommendation System Model Compression with Learned Hash Functions Benjamin Ghaemmaghami Mustafa Ozdal Rakesh Komuravelli D. Korchev Dheevatsa Mudigere Krishnakumar Nair Maxim Naumov 18 6 0 28 Mar 2022
Deep Class Incremental Learning from Decentralized Data Xiaohan Zhang Songlin Dong Jinjie Chen Qiaoling Tian Yihong Gong Xiaopeng Hong CLL 23 11 0 11 Mar 2022
CaMEL: Mean Teacher Learning for Image Captioning Manuele Barraco Matteo Stefanini Marcella Cornia S. Cascianelli Lorenzo Baraldi Rita Cucchiara ViT VLM 25 27 0 21 Feb 2022
Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning Xiaohang Bian Bo Qin Xiaozhe Xin Jianwu Li Xuefeng Su Yanfeng Wang 30 49 0 07 Dec 2021
Efficient Federated Learning for AIoT Applications Using Knowledge Distillation Tian Liu Xian Wei Jun Xia Xin Fu Ting Wang Mingsong Chen 6 15 0 29 Nov 2021
Learning Distilled Collaboration Graph for Multi-Agent Perception Yiming Li Shunli Ren Pengxiang Wu Siheng Chen Chen Feng Wenjun Zhang 27 237 0 01 Nov 2021
Towards Model Agnostic Federated Learning Using Knowledge Distillation A. Afonin Sai Praneeth Karimireddy FedML 30 44 0 28 Oct 2021
Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better Xuanyang Zhang X. Zhang Jian-jun Sun 23 1 0 26 Sep 2021
Recent Advances of Continual Learning in Computer Vision: An Overview Haoxuan Qu Hossein Rahmani Li Xu Bryan M. Williams Jun Liu VLM CLL 23 73 0 23 Sep 2021
Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer Yae Jee Cho Jianyu Wang Tarun Chiruvolu Gauri Joshi FedML 27 30 0 16 Sep 2021
Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIs Qiongkai Xu Xuanli He Lingjuan Lyu Lizhen Qu Gholamreza Haffari MLAU 30 21 0 29 Aug 2021
Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation Maria Tzelepi Anastasios Tefas 11 6 0 26 Aug 2021
Large-Scale Differentially Private BERT Rohan Anil Badih Ghazi Vineet Gupta Ravi Kumar Pasin Manurangsi 24 131 0 03 Aug 2021
DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs J. Nistal Stefan Lattner G. Richard 19 8 0 03 Aug 2021
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation Junnan Li Ramprasaath R. Selvaraju Akhilesh Deepak Gotmare Shafiq R. Joty Caiming Xiong S. Hoi FaML 51 1,884 0 16 Jul 2021
Collaborative Training of Acoustic Encoders for Speech Recognition Varun K. Nagaraja Yangyang Shi Ganesh Venkatesh Ozlem Kalinli M. Seltzer Vikas Chandra 35 11 0 16 Jun 2021
Black-Box Dissector: Towards Erasing-based Hard-Label Model Stealing Attack Yixu Wang Jie Li Hong Liu Yan Wang Yongjian Wu Feiyue Huang Rongrong Ji AAML 19 34 0 03 May 2021
Collaborative Label Correction via Entropy Thresholding Hao Wu Jiangchao Yao Jiajie Wang Yinru Chen Ya-Qin Zhang Yanfeng Wang NoLa 14 4 0 31 Mar 2021
Distilling Object Detectors via Decoupled Features Jianyuan Guo Kai Han Yunhe Wang Han Wu Xinghao Chen Chunjing Xu Chang Xu 24 199 0 26 Mar 2021
Compacting Deep Neural Networks for Internet of Things: Methods and Applications Ke Zhang Hanbo Ying Hongning Dai Lin Li Yuangyuang Peng Keyi Guo Hongfang Yu 16 38 0 20 Mar 2021
Locally Adaptive Label Smoothing for Predictive Churn Dara Bahri Heinrich Jiang NoLa 29 8 0 09 Feb 2021
Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation Kang Li Shujun Wang Lequan Yu Pheng-Ann Heng 60 28 0 07 Jan 2021
Cross-Layer Distillation with Semantic Calibration Defang Chen Jian-Ping Mei Yuan Zhang Can Wang Yan Feng Chun-Yen Chen FedML 34 286 0 06 Dec 2020
Federated Knowledge Distillation Hyowoon Seo Jihong Park Seungeun Oh M. Bennis Seong-Lyun Kim FedML 12 90 0 04 Nov 2020
Anti-Distillation: Improving reproducibility of deep networks G. Shamir Lorenzo Coviello 34 20 0 19 Oct 2020
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications Jihong Park S. Samarakoon Anis Elgabli Joongheon Kim M. Bennis Seong-Lyun Kim Mérouane Debbah 23 161 0 06 Aug 2020