Learning to Maximize Mutual Information for Chain-of-Thought Distillation

5 March 2024

Papers citing "Learning to Maximize Mutual Information for Chain-of-Thought Distillation"

46 / 46 papers shown

Title
Predicting Through Generation: Why Generation Is Better for Prediction Md. Kowsher Nusrat Jahan Prottasha Prakash Bhat Chun-Nam Yu Mojtaba Soltanalian Ivan Garibay O. Garibay Chen Chen Niloofar Yousefi AI4TS 162 0 0 25 Feb 2025
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions Ting-Yao Hsu Chieh-Yang Huang Ryan Rossi Sungchul Kim C. Lee Giles ‘Kenneth’ Huang 97 13 0 23 Oct 2023
Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA Yuhan Ma Haiqi Jiang Chenyou Fan LRM 52 14 0 09 Aug 2023
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step Liunian Harold Li Jack Hessel Youngjae Yu Xiang Ren Kai-Wei Chang Yejin Choi LRM AI4CE ReLM 48 136 0 24 Jun 2023
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Lokesh Nagalapatti Chun-Liang Li Chih-Kuan Yeh Hootan Nakhost Yasuhisa Fujii Alexander Ratner Ranjay Krishna Chen-Yu Lee Tomas Pfister ALM 258 526 0 03 May 2023
SCOTT: Self-Consistent Chain-of-Thought Distillation Jamie Yap Zhengyang Wang Zheng Li K. Lynch Bing Yin Xiang Ren LRM 94 94 0 03 May 2023
Learning to Maximize Mutual Information for Dynamic Feature Selection Ian Covert Wei Qiu Mingyu Lu Nayoon Kim Nathan White Su-In Lee 39 29 0 02 Jan 2023
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters Boshi Wang Sewon Min Xiang Deng Jiaming Shen You Wu Luke Zettlemoyer Huan Sun LRM ReLM 57 236 0 20 Dec 2022
Teaching Small Language Models to Reason Lucie Charlotte Magister Jonathan Mallinson Jakub Adamek Eric Malmi Aliaksei Severyn LRM AI4CE ReLM 148 261 0 16 Dec 2022
Do Text-to-Text Multi-Task Learners Suffer from Task Conflict? David Mueller Nicholas Andrews Mark Dredze 69 6 0 13 Dec 2022
A Close Look into the Calibration of Pre-trained Language Models Yangyi Chen Lifan Yuan Ganqu Cui Zhiyuan Liu Heng Ji 99 50 0 31 Oct 2022
Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Model Dongkyu Lee Zhiliang Tian Ying Zhao Ka Chun Cheung N. Zhang 49 3 0 22 Oct 2022
Improving the Adversarial Robustness of NLP Models by Information Bottleneck Ce Zhang Xiang Zhou Yixin Wan Xiaoqing Zheng Kai-Wei Chang Cho-Jui Hsieh 39 26 0 11 Jun 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 364 6,132 0 05 Apr 2022
Cross-Task Knowledge Distillation in Multi-Task Recommendation Chenxiao Yang Junwei Pan Xiaofeng Gao Tingyu Jiang Dapeng Liu Guihai Chen 77 45 0 20 Feb 2022
Towards Understanding Knowledge Distillation Mary Phuong Christoph H. Lampert 61 316 0 27 May 2021
Distilling Knowledge via Knowledge Review Pengguang Chen Shu Liu Hengshuang Zhao Jiaya Jia 181 433 0 19 Apr 2021
Farewell to Mutual Information: Variational Distillation for Cross-Modal Person Re-Identification Xudong Tian Zhizhong Zhang Shaohui Lin Yanyun Qu Yuan Xie Lizhuang Ma 47 110 0 07 Apr 2021
Are NLP Models really able to Solve Simple Math Word Problems? Arkil Patel S. Bhattamishra Navin Goyal ReLM LRM 65 801 0 12 Mar 2021
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning Zeyuan Allen-Zhu Yuanzhi Li FedML 114 362 0 17 Dec 2020
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering Zhengbao Jiang Jun Araki Haibo Ding Graham Neubig UQCV 56 428 0 02 Dec 2020
Learning Variational Word Masks to Improve the Interpretability of Neural Text Classifiers Hanjie Chen Yangfeng Ji AAML VLM 67 63 0 01 Oct 2020
Multi-task learning for natural language processing in the 2020s: where are we going? Joseph Worsham Jugal Kalita AIMat 38 78 0 22 Jul 2020
Learning to Learn with Variational Information Bottleneck for Domain Generalization Yingjun Du Jun Xu Huan Xiong Qiang Qiu Xiantong Zhen Cees G. M. Snoek Ling Shao BDL OOD 72 165 0 15 Jul 2020
Knowledge Distillation for Multi-task Learning Weihong Li Hakan Bilen MoMe 34 72 0 14 Jul 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 58 2,907 0 09 Jun 2020
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction Bhargavi Paranjape Mandar Joshi John Thickstun Hannaneh Hajishirzi Luke Zettlemoyer 57 100 0 01 May 2020
Pareto Multi-Task Learning Xi Lin Hui-Ling Zhen Zhenhua Li Qingfu Zhang Sam Kwong 71 346 0 30 Dec 2019
Adversarial NLI: A New Benchmark for Natural Language Understanding Yixin Nie Adina Williams Emily Dinan Joey Tianyi Zhou Jason Weston Douwe Kiela 101 991 0 31 Oct 2019
Contrastive Representation Distillation Yonglong Tian Dilip Krishnan Phillip Isola 133 1,042 0 23 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 341 19,824 0 23 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 173 7,437 0 02 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 78 1,847 0 23 Sep 2019
On Mutual Information Maximization for Representation Learning Michael Tschannen Josip Djolonga Paul Kishan Rubenstein Sylvain Gelly Mario Lucic SSL 149 490 0 31 Jul 2019
Learning Representations by Maximizing Mutual Information Across Views Philip Bachman R. Devon Hjelm William Buchwalter SSL 179 1,463 0 03 Jun 2019
On Variational Bounds of Mutual Information Ben Poole Sherjil Ozair Aaron van den Oord Alexander A. Alemi George Tucker SSL 74 802 0 16 May 2019
Multi-Task Deep Neural Networks for Natural Language Understanding Xiaodong Liu Pengcheng He Weizhu Chen Jianfeng Gao AI4CE 119 1,269 0 31 Jan 2019
e-SNLI: Natural Language Inference with Natural Language Explanations Oana-Maria Camburu Tim Rocktaschel Thomas Lukasiewicz Phil Blunsom LRM 396 634 0 04 Dec 2018
Formal Limitations on the Measurement of Mutual Information David A. McAllester K. Stratos SSL 61 275 0 10 Nov 2018
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge Alon Talmor Jonathan Herzig Nicholas Lourie Jonathan Berant RALM 125 1,677 0 02 Nov 2018
MINE: Mutual Information Neural Estimation Mohamed Ishmael Belghazi A. Baratin Sai Rajeswar Sherjil Ozair Yoshua Bengio Aaron Courville R. Devon Hjelm DRL 160 1,264 0 12 Jan 2018
A Survey on Multi-Task Learning Yu Zhang Qiang Yang AIMat 447 2,196 0 25 Jul 2017
Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics Alex Kendall Y. Gal R. Cipolla 3DH 244 3,101 0 19 May 2017
Deep Variational Information Bottleneck Alexander A. Alemi Ian S. Fischer Joshua V. Dillon Kevin Patrick Murphy 94 1,697 0 01 Dec 2016
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 296 19,523 0 09 Mar 2015
Deep Learning and the Information Bottleneck Principle Naftali Tishby Noga Zaslavsky DRL 156 1,570 0 09 Mar 2015