Representation Degeneration Problem in Training Natural Language Generation Models

28 July 2019

Xu Tan

Papers citing "Representation Degeneration Problem in Training Natural Language Generation Models"

50 / 50 papers shown

Title
Accurate and Efficient Multivariate Time Series Forecasting via Offline Clustering Yiming Niu Jinliang Deng L. Zhang Zimu Zhou Yongxin Tong AI4TS 33 0 0 09 May 2025
llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length Issa Sugiura Kouta Nakayama Yusuke Oda 34 1 0 22 Apr 2025
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 68 9 0 20 Feb 2025
DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learning Jiaxin Guo Cheng Chen Shuzhen Li Tianze Zhang 63 0 0 01 Feb 2025
Improving Long-Text Alignment for Text-to-Image Diffusion Models Luping Liu Chao Du Tianyu Pang Zehan Wang Chongxuan Li Dong Xu VLM 53 5 0 15 Oct 2024
Understanding and Minimising Outlier Features in Neural Network Training Bobby He Lorenzo Noci Daniele Paliotta Imanol Schlag Thomas Hofmann 42 3 0 29 May 2024
Event-enhanced Retrieval in Real-time Search Yanan Zhang Xiaoling Bai Tianhua Zhou 49 1 0 09 Apr 2024
The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models Anton Razzhigaev Matvey Mikhalchuk Elizaveta Goncharova Ivan Oseledets Denis Dimitrov Andrey Kuznetsov 35 7 0 10 Nov 2023
Audio Contrastive based Fine-tuning Yang Wang Qibin Liang Chenghao Xiao Yizhi Li Noura Al Moubayed Chenghua Lin 32 0 0 21 Sep 2023
Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks Andreas Roth Thomas Liebig 51 13 0 31 Aug 2023
How Good Are LLMs at Out-of-Distribution Detection? Bo Liu Li-Ming Zhan Zexin Lu Yu Feng Lei Xue Xiao-Ming Wu OODD 40 8 0 20 Aug 2023
Generative Models as a Complex Systems Science: How can we make sense of large language model behavior? Ari Holtzman Peter West Luke Zettlemoyer AI4CE 34 14 0 31 Jul 2023
Make Text Unlearnable: Exploiting Effective Patterns to Protect Personal Data Xinzhe Li Ming Liu Shang Gao MU 45 8 0 02 Jul 2023
Addressing the Rank Degeneration in Sequential Recommendation via Singular Spectrum Smoothing Ziwei Fan Zhiwei Liu Hao Peng Philip S. Yu 43 1 0 21 Jun 2023
Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity Katharina Hämmerl Alina Fastowski Jindrich Libovický Alexander Fraser 30 6 0 01 Jun 2023
Event-Centric Query Expansion in Web Search Yanan Zhang Weijie Cui Yangfan Zhang Xiaoling Bai Zhe Zhang Jin Ma Xinyu Chen Tianhua Zhou 12 2 0 30 May 2023
Exploring Representational Disparities Between Multilingual and Bilingual Translation Models Neha Verma Kenton W. Murray Kevin Duh 21 0 0 23 May 2023
Investigating the Role of Feed-Forward Networks in Transformers Using Parallel Attention and Feed-Forward Net Design Shashank Sonkar Richard G. Baraniuk 19 3 0 22 May 2023
Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation Wen Lai Alexandra Chronopoulou Alexander Fraser 37 5 0 22 May 2023
Unsupervised Sentence Representation Learning with Frequency-induced Adversarial Tuning and Incomplete Sentence Filtering Bing Wang Ximing Li Zhiyao Yang Yuanyuan Guan Jiayin Li Sheng-sheng Wang 35 6 0 15 May 2023
Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning Hao Zhang Nianwen Si Yaqi Chen Wenlin Zhang Xukui Yang Dan Qu Weiqiang Zhang 35 9 0 20 Apr 2023
Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study Mingxu Tao Yansong Feng Dongyan Zhao CLL KELM 32 10 0 02 Mar 2023
Weighted Sampling for Masked Language Modeling Linhan Zhang Qian Chen Wen Wang Chong Deng Xin Cao Kongzhang Hao Yuxin Jiang Wen Wang 32 2 0 28 Feb 2023
Byte Pair Encoding for Symbolic Music Nathan Fradet Nicolas Gutowski F. Chhel Jean-Pierre Briot 29 16 0 27 Jan 2023
Empowering Diffusion Models on the Embedding Space for Text Generation Zhujin Gao Junliang Guo Xuejiao Tan Yongxin Zhu Fang Zhang Jiang Bian Linli Xu DiffM 35 15 0 19 Dec 2022
Reliable Measures of Spread in High Dimensional Latent Spaces Anna C. Marbut Katy McKinney-Bock Travis J. Wheeler 30 2 0 15 Dec 2022
Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention Wenhao Li Xiaoyuan Yi Jinyi Hu Maosong Sun Xing Xie 44 0 0 14 Nov 2022
Reconciliation of Pre-trained Models and Prototypical Neural Networks in Few-shot Named Entity Recognition Youcheng Huang Wenqiang Lei Jie Fu Jiancheng Lv 24 3 0 07 Nov 2022
Prompt Combines Paraphrase: Teaching Pre-trained Models to Understand Rare Biomedical Words Hao Wang Chi-Liang Liu Nuwa Xi Sendong Zhao Meizhi Ju Shiwei Zhang Ziheng Zhang Yefeng Zheng Bing Qin Ting Liu VLM AAML LM&MA 41 6 0 14 Sep 2022
Analyzing Transformers in Embedding Space Guy Dar Mor Geva Ankit Gupta Jonathan Berant 29 84 0 06 Sep 2022
Addressing Token Uniformity in Transformers via Singular Value Transformation Hanqi Yan Lin Gui Wenjie Li Yulan He 32 14 0 24 Aug 2022
Mere Contrastive Learning for Cross-Domain Sentiment Analysis Yun Luo Fang Guo Zihan Liu Yue Zhang 39 15 0 18 Aug 2022
Outliers Dimensions that Disrupt Transformers Are Driven by Frequency Giovanni Puccetti Anna Rogers Aleksandr Drozd F. Dell’Orletta 81 42 0 23 May 2022
CoCoSoDa: Effective Contrastive Learning for Code Search Ensheng Shi Yanlin Wang Wenchao Gu Lun Du Hongyu Zhang Shi Han Dongmei Zhang Hongbin Sun 43 33 0 07 Apr 2022
The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy Tianlong Chen Zhenyu Zhang Yu Cheng Ahmed Hassan Awadallah Zhangyang Wang ViT 41 37 0 12 Mar 2022
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning Weixin Liang Yuhui Zhang Yongchan Kwon Serena Yeung James Zou VLM 52 394 0 03 Mar 2022
PromptBERT: Improving BERT Sentence Embeddings with Prompts Ting Jiang Jian Jiao Shaohan Huang Zi-qiang Zhang Deqing Wang Fuzhen Zhuang Furu Wei Haizhen Huang Liangjie Zhang Qi Zhang 33 120 0 12 Jan 2022
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 77 332 0 11 Nov 2021
Text analysis and deep learning: A network approach Ingo Marquart 25 0 0 08 Oct 2021
Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings Sangwon Yu Jongyoon Song Heeseung Kim SeongEun Lee Woo-Jong Ryu Sung-Hoon Yoon 22 31 0 07 Sep 2021
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer Yuanmeng Yan Rumei Li Sirui Wang Fuzheng Zhang Wei Wu Weiran Xu SSL 52 546 0 25 May 2021
Vision Transformers with Patch Diversification Chengyue Gong Dilin Wang Meng Li Vikas Chandra Qiang Liu ViT 45 62 0 26 Apr 2021
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining Yu Meng Chenyan Xiong Payal Bajaj Saurabh Tiwary Paul N. Bennett Jiawei Han Xia Song 125 203 0 16 Feb 2021
Positional Artefacts Propagate Through Masked Language Model Embeddings Ziyang Luo Artur Kulmizev Xiaoxi Mao 29 41 0 09 Nov 2020
On the Sentence Embeddings from Pre-trained Language Models Bohan Li Hao Zhou Junxian He Mingxuan Wang Yiming Yang Lei Li 30 213 0 02 Nov 2020
A Discrete Variational Recurrent Topic Model without the Reparametrization Trick Mehdi Rezaee Francis Ferraro BDL DRL 17 27 0 22 Oct 2020
Improving Low Compute Language Modeling with In-Domain Embedding Initialisation Charles F Welch Rada Mihalcea Jonathan K. Kummerfeld AI4CE 19 4 0 29 Sep 2020
IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization Wenxuan Zhou Bill Yuchen Lin Xiang Ren 14 24 0 02 May 2020
Improving Neural Language Modeling via Adversarial Training Dilin Wang Chengyue Gong Qiang Liu AAML 43 115 0 10 Jun 2019
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 718 6,748 0 26 Sep 2016