NLPerturbator: Studying the Robustness of Code LLMs to Natural Language
Variations

NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations

28 June 2024

ArXiv (abs)PDF HTML

Papers citing "NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations"

16 / 16 papers shown

Title
CodeCrash: Stress Testing LLM Reasoning under Structural and Semantic Perturbations Man Ho Lam Chaozheng Wang Jen-tse Huang Michael R. Lyu LRM 72 1 0 19 Apr 2025
RobuNFR: Evaluating the Robustness of Large Language Models on Non-Functional Requirements Aware Code Generation Feng Lin Dong Jae Kim Zhiyu Li Jinqiu Yang Tse-Husn Chen AAML 95 1 0 28 Mar 2025
Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation Li Zhong Zilong Wang ELM SILM 85 74 0 20 Aug 2023
Exploring the Robustness of Large Language Models for Solving Programming Problems Atsushi Shirafuji Yutaka Watanobe Takumi Ito Makoto Morishita Yuki Nakamura Yusuke Oda Jun Suzuki ELM 76 21 0 26 Jun 2023
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin Daxin Jiang ELM SyDa ALM 98 687 0 14 Jun 2023
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X Qinkai Zheng Xiao Xia Xu Zou Yuxiao Dong Shanshan Wang ... Andi Wang Yang Li Teng Su Zhilin Yang Jie Tang ELM ALM SyDa 121 341 0 30 Mar 2023
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective Jindong Wang Xixu Hu Wenxin Hou Hao Chen Runkai Zheng ... Weirong Ye Xiubo Geng Binxing Jiao Yue Zhang Xingxu Xie AI4MH 124 235 0 22 Feb 2023
On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex Terry Yue Zhuo Zhuang Li Yujin Huang Fatemeh Shiri Weiqing Wang Gholamreza Haffari Yuan-Fang Li AAML 98 56 0 30 Jan 2023
DocPrompting: Generating Code by Retrieving the Docs Shuyan Zhou Uri Alon Frank F. Xu Zhiruo Wang Zhengbao Jiang Graham Neubig LLMAG 78 138 0 13 Jul 2022
InCoder: A Generative Model for Code Infilling and Synthesis Daniel Fried Armen Aghajanyan Jessy Lin Sida I. Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer M. Lewis SyDa 69 652 0 12 Apr 2022
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models Wei Ping Chejian Xu Shuohang Wang Zhe Gan Yu Cheng Jianfeng Gao Ahmed Hassan Awadallah Yangqiu Song VLM ELM AAML 67 225 0 04 Nov 2021
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation Yue Wang Weishi Wang Shafiq Joty Guosheng Lin 289 1,560 0 02 Sep 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 200 2,004 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 233 5,635 0 07 Jul 2021
TreeGen: A Tree-Based Transformer Architecture for Code Generation Zeyu Sun Qihao Zhu Yingfei Xiong Yican Sun Lili Mou Lu Zhang 60 174 0 22 Nov 2019
Synthetic and Natural Noise Both Break Neural Machine Translation Yonatan Belinkov Yonatan Bisk 114 743 0 06 Nov 2017

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.