Contrastive Decoding: Open-ended Text Generation as Optimization

27 October 2022

Xiang Lisa Li

Ari Holtzman

Daniel Fried

Percy Liang

Jason Eisner

Tatsunori Hashimoto

Luke Zettlemoyer

M. Lewis

ArXiv PDF HTML

Papers citing "Contrastive Decoding: Open-ended Text Generation as Optimization"

50 / 264 papers shown

Title
Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learning Yue Yu Jiaming Shen Tianqi Liu Zhen Qin Jing Nathan Yan Jialu Liu Chao Zhang Michael Bendersky 54 6 0 13 Nov 2023
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions Lei Huang Weijiang Yu Weitao Ma Weihong Zhong Zhangyin Feng ... Qianglong Chen Weihua Peng Xiaocheng Feng Bing Qin Ting Liu LRM HILM 41 722 0 09 Nov 2023
Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions Taehyeon Kim Joonkee Kim Gihun Lee Se-Young Yun 33 11 0 01 Nov 2023
Evaluating Large Language Models on Controlled Generation Tasks Jiao Sun Yufei Tian Wangchunshu Zhou Nan Xu Qian Hu Rahul Gupta John Wieting Nanyun Peng Xuezhe Ma LRM ELM 40 61 0 23 Oct 2023
An Emulator for Fine-Tuning Large Language Models using Small Language Models Eric Mitchell Rafael Rafailov Archit Sharma Chelsea Finn Christopher D. Manning ALM 41 51 0 19 Oct 2023
Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning Huiming Wang Zhaodonghui Li Liying Cheng De Wen Soh Lidong Bing 31 2 0 17 Oct 2023
Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective Huayang Li Tian Lan Z. Fu Deng Cai Lemao Liu Nigel Collier Taro Watanabe Yixuan Su 42 12 0 16 Oct 2023
VLIS: Unimodal Language Models Guide Multimodal Language Generation Jiwan Chung Youngjae Yu VLM 30 1 0 15 Oct 2023
The Consensus Game: Language Model Generation via Equilibrium Search Athul Paul Jacob Yikang Shen Gabriele Farina Jacob Andreas 39 19 0 13 Oct 2023
Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity Cunxiang Wang Xiaoze Liu Yuanhao Yue Xiangru Tang Tianhang Zhang ... Linyi Yang Jindong Wang Xing Xie Zheng-Wei Zhang Yue Zhang HILM KELM 51 184 0 11 Oct 2023
NEFTune: Noisy Embeddings Improve Instruction Finetuning Neel Jain Ping Yeh-Chiang Yuxin Wen John Kirchenbauer Hong-Min Chu ... Avi Schwarzschild Aniruddha Saha Micah Goldblum Jonas Geiping Tom Goldstein 28 75 0 09 Oct 2023
Amortizing intractable inference in large language models Marvin Schmitt Moksh Jain Daniel Habermann Younesse Kaddar Ullrich Kothe Stefan T. Radev Nikolay Malkin AIFin BDL 29 46 0 06 Oct 2023
Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoning Murong Yue Jie Zhao Min Zhang Liang Du Ziyu Yao LRM 32 55 0 04 Oct 2023
Closing the Curious Case of Neural Text Degeneration Matthew Finlayson John Hewitt Alexander Koller Swabha Swayamdipta Ashish Sabharwal 40 16 0 02 Oct 2023
Language Model Decoding as Direct Metrics Optimization Haozhe Ji Pei Ke Hongning Wang Minlie Huang 13 7 0 02 Oct 2023
Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration Qiushi Sun Zhangyue Yin Xiang Li Zhiyong Wu Xipeng Qiu Lingpeng Kong LRM LLMAG 28 44 0 30 Sep 2023
Self-Specialization: Uncovering Latent Expertise within Large Language Models Junmo Kang Hongyin Luo Yada Zhu Jacob A. Hansen James R. Glass David D. Cox Alan Ritter Rogerio Feris Leonid Karlinsky ALM MoMe 27 4 0 29 Sep 2023
Jointly Training Large Autoregressive Multimodal Models Emanuele Aiello L. Yu Yixin Nie Armen Aghajanyan Barlas Oğuz 19 29 0 27 Sep 2023
Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future Zheng Chu Jingchang Chen Qianglong Chen Weijiang Yu Tao He Haotian Wang Weihua Peng Ming-Yu Liu Bing Qin Ting Liu LRM AI4CE 31 151 0 27 Sep 2023
Contrastive Decoding Improves Reasoning in Large Language Models Sean O'Brien Mike Lewis SyDa LRM ReLM 26 31 0 17 Sep 2023
Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decoding Rico Sennrich Jannis Vamvas Alireza Mohammadshahi HILM 32 38 0 13 Sep 2023
Unsupervised Contrast-Consistent Ranking with Language Models Niklas Stoehr Pengxiang Cheng Jing Wang Daniel Preotiuc-Pietro Rajarshi Bhowmik ALM 31 11 0 13 Sep 2023
Does Writing with Language Models Reduce Content Diversity? Vishakh Padmakumar He He 33 81 0 11 Sep 2023
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models Yung-Sung Chuang Yujia Xie Hongyin Luo Yoon Kim James R. Glass Pengcheng He HILM 33 148 0 07 Sep 2023
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning L. Yu Bowen Shi Ramakanth Pasunuru Benjamin Muller O. Yu. Golovneva ... Yaniv Taigman Maryam Fazel-Zarandi Asli Celikyilmaz Luke Zettlemoyer Armen Aghajanyan MLLM 38 135 0 05 Sep 2023
Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generation Charles OÑeill Y. Ting 丁 I. Ciucă Jack Miller Thang Bui SyDa 37 1 0 15 Aug 2023
Lightweight reranking for language model generations Siddhartha Jain Xiaofei Ma Anoop Deoras Bing Xiang 36 4 0 11 Jul 2023
On the Efficacy of Sampling Adapters Clara Meister Tiago Pimentel Luca Malagutti Ethan Gotlieb Wilcox Ryan Cotterell 35 12 0 07 Jul 2023
PREADD: Prefix-Adaptive Decoding for Controlled Text Generation Jonathan Pei Kevin Kaichuang Yang Dan Klein 38 21 0 06 Jul 2023
Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation Jian-Yu Guan Minlie Huang 29 0 0 04 Jul 2023
Stay on topic with Classifier-Free Guidance Guillaume Sanchez Honglu Fan Alexander Spangher Elad Levi Pawan Sasanka Ammanamanchi Stella Biderman 3DV 30 46 0 30 Jun 2023
GPT-FinRE: In-context Learning for Financial Relation Extraction using Large Language Models P. Rajpoot Ankur P. Parikh 24 14 0 30 Jun 2023
Open-Domain Text Evaluation via Contrastive Distribution Methods Sidi Lu Hongyi Liu Asli Celikyilmaz Tianlu Wang Nanyun Peng 23 0 0 20 Jun 2023
On the Reliability of Watermarks for Large Language Models John Kirchenbauer Jonas Geiping Yuxin Wen Manli Shu Khalid Saifullah Kezhi Kong Kasun Fernando Aniruddha Saha Micah Goldblum Tom Goldstein WaLM 16 113 0 07 Jun 2023
EEL: Efficiently Encoding Lattices for Reranking Prasann Singhal Jiacheng Xu Xi Ye Greg Durrett 22 3 0 01 Jun 2023
Less Likely Brainstorming: Using Language Models to Generate Alternative Hypotheses Liyan Tang Yifan Peng Yanshan Wang Ying Ding Greg Durrett Justin F. Rousseau 32 9 0 30 May 2023
Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective Khanh Nguyen LRM 26 8 0 28 May 2023
Robust Natural Language Understanding with Residual Attention Debiasing Fei Wang James Y. Huang Tianyi Yan Wenxuan Zhou Muhao Chen 34 10 0 28 May 2023
MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies Shiyue Zhang Shijie Wu Ozan Irsoy Steven Lu Joey Tianyi Zhou Mark Dredze David S. Rosenberg 23 9 0 26 May 2023
Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning Ximing Lu Faeze Brahman Peter West Jaehun Jang Khyathi Raghavi Chandu ... Bill Yuchen Lin Skyler Hallinan Xiang Ren Sean Welleck Yejin Choi 25 26 0 24 May 2023
From Shortcuts to Triggers: Backdoor Defense with Denoised PoE Qin Liu Fei Wang Chaowei Xiao Muhao Chen AAML 37 21 0 24 May 2023
David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs Xiaochuang Han Sachin Kumar Yulia Tsvetkov Marjan Ghazvininejad DiffM 34 3 0 24 May 2023
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding Weijia Shi Xiaochuang Han M. Lewis Yulia Tsvetkov Luke Zettlemoyer Scott Yih HILM 21 189 0 24 May 2023
Look-back Decoding for Open-Ended Text Generation Nan Xu Chunting Zhou Asli Celikyilmaz Xuezhe Ma 31 9 0 22 May 2023
A Frustratingly Simple Decoding Method for Neural Text Generation Haoran Yang Deng Cai Huayang Li Wei Bi Wai Lam Shuming Shi 46 11 0 22 May 2023
FACE: Evaluating Natural Language Generation with Fourier Analysis of Cross-Entropy Zuhao Yang Yingfang Yuan Yang Xu Shuo Zhan Huajun Bai Kefan Chen CVBM 22 4 0 17 May 2023
Surfacing Biases in Large Language Models using Contrastive Input Decoding G. Yona Or Honovich Itay Laish Roee Aharoni 27 11 0 12 May 2023
HistAlign: Improving Context Dependency in Language Generation by Aligning with History David Wan Shiyue Zhang Joey Tianyi Zhou AI4TS 37 5 0 08 May 2023
SCOTT: Self-Consistent Chain-of-Thought Distillation Jamie Yap Zhengyang Wang Zheng Li K. Lynch Bing Yin Xiang Ren LRM 64 93 0 03 May 2023
The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers Ariel Gera Roni Friedman Ofir Arviv Chulaka Gunasekara Benjamin Sznajder Noam Slonim Eyal Shnarch 46 19 0 02 May 2023