Does Pretraining for Summarization Require Knowledge Transfer?

10 September 2021

Papers citing "Does Pretraining for Summarization Require Knowledge Transfer?"

31 / 31 papers shown

Title
Transfer of Structural Knowledge from Synthetic Languages Mikhail Budnikov Ivan Yamshchikov 7 0 0 21 May 2025
General Reasoning Requires Learning to Reason from the Get-go Seungwook Han Jyothish Pari Samuel J. Gershman Pulkit Agrawal LRM 187 1 0 26 Feb 2025
Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates A. Narayan Mayee F. Chen Kush S. Bhatia Christopher Ré SyDa 41 3 0 07 Oct 2024
Does VLN Pretraining Work with Nonsensical or Irrelevant Instructions? Wang Zhu Ishika Singh Yuan Huang Robin Jia Jesse Thomason 36 2 0 28 Nov 2023
Responsible AI Considerations in Text Summarization Research: A Review of Current Practices Yu Lu Liu Meng Cao Su Lin Blodgett Jackie Chi Kit Cheung Alexandra Olteanu Adam Trischler 36 1 0 18 Nov 2023
ExPT: Synthetic Pretraining for Few-Shot Experimental Design Tung Nguyen Sudhanshu Agrawal Aditya Grover 27 15 0 30 Oct 2023
Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? Ahmed Alajrami Katerina Margatina Nikolaos Aletras AAML 19 1 0 26 Oct 2023
SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation Matthias Lindemann Alexander Koller Ivan Titov AI4CE 19 2 0 01 Oct 2023
Pre-training with Synthetic Data Helps Offline Reinforcement Learning Zecheng Wang Che Wang Zixuan Dong Keith Ross OffRL 36 5 0 01 Oct 2023
SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design Carl Edwards Aakanksha Naik Tushar Khot Martin D. Burke Heng Ji Tom Hope 33 15 0 19 Jun 2023
Injecting structural hints: Using language models to study inductive biases in language learning Isabel Papadimitriou Dan Jurafsky 20 13 0 25 Apr 2023
Revisiting Hidden Representations in Transfer Learning for Medical Imaging Dovile Juodelyte Amelia Jiménez-Sánchez V. Cheplygina OOD 19 1 0 16 Feb 2023
A Survey of Deep Learning for Mathematical Reasoning Pan Lu Liang Qiu Wenhao Yu Sean Welleck Kai-Wei Chang ReLM LRM 50 140 0 20 Dec 2022
On the Role of Parallel Data in Cross-lingual Transfer Learning Machel Reid Mikel Artetxe 21 10 0 20 Dec 2022
Synthetic Pre-Training Tasks for Neural Machine Translation Zexue He Graeme W. Blackwood Yikang Shen Julian McAuley Rogerio Feris 29 3 0 19 Dec 2022
Procedural Image Programs for Representation Learning Manel Baradad Chun-Fu Jonas Wulff Tongzhou Wang Rogerio Feris Antonio Torralba Phillip Isola 31 18 0 29 Nov 2022
Neural networks trained with SGD learn distributions of increasing complexity Maria Refinetti Alessandro Ingrosso Sebastian Goldt UQCV 38 41 0 21 Nov 2022
Where to start? Analyzing the potential value of intermediate models Leshem Choshen Elad Venezian Shachar Don-Yehiya Noam Slonim Yoav Katz MoMe 24 27 0 31 Oct 2022
Downstream Datasets Make Surprisingly Good Pretraining Corpora Kundan Krishna Saurabh Garg Jeffrey P. Bigham Zachary Chase Lipton 50 30 0 28 Sep 2022
MonoByte: A Pool of Monolingual Byte-level Language Models Hugo Queiroz Abonizio Leandro Rodrigues de Souza R. Lotufo Rodrigo Nogueira 46 1 0 22 Sep 2022
Linguistically inspired roadmap for building biologically reliable protein language models Mai Ha Vu Rahmad Akbar Philippe A. Robert B. Swiatczak Victor Greiff G. K. Sandve Dag Trygve Tryslew Haug 49 35 0 03 Jul 2022
Insights into Pre-training via Simpler Synthetic Tasks Yuhuai Wu Felix Li Percy Liang AIMat 28 20 0 21 Jun 2022
E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation Qihuang Zhong Liang Ding Juhua Liu Bo Du Dacheng Tao 54 27 0 30 May 2022
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment Matthew Finlayson Kyle Richardson Ashish Sabharwal Peter Clark 30 12 0 19 Apr 2022
Fusing finetuned models for better pretraining Leshem Choshen Elad Venezian Noam Slonim Yoav Katz FedML AI4CE MoMe 54 87 0 06 Apr 2022
Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training Ramon Sanabria Wei-Ning Hsu Alexei Baevski Michael Auli 21 7 0 01 Mar 2022
Should You Mask 15% in Masked Language Modeling? Alexander Wettig Tianyu Gao Zexuan Zhong Danqi Chen CVBM 29 162 0 16 Feb 2022
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? Alaaeldin El-Nouby Gautier Izacard Hugo Touvron Ivan Laptev Hervé Jégou Edouard Grave SSL 27 149 0 20 Dec 2021
Compositional generalization in semantic parsing with pretrained transformers A. Orhan 22 6 0 30 Sep 2021
ScisummNet: A Large Annotated Corpus and Content-Impact Models for Scientific Paper Summarization with Citation Networks Michihiro Yasunaga Jungo Kasai Rui Zhang Alexander R. Fabbri Irene Z Li Dan Friedman Dragomir R. Radev 73 206 0 04 Sep 2019
Text Summarization with Pretrained Encoders Yang Liu Mirella Lapata MILM 258 1,435 0 22 Aug 2019