v1v2v3 (latest)

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation

2 May 2023

ArXiv (abs)PDF HTML Github (1469★)

Papers citing "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation"

50 / 101 papers shown

Title
Learning to Reason without External Rewards Xuandong Zhao Zhewei Kang Aosong Feng Sergey Levine Dawn Song OffRL ReLM LRM 111 8 0 26 May 2025
CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation Guang Yang Yu Zhou Xiang Chen Wei-Shi Zheng Xing Hu Xin Zhou David Lo Taolue Chen ALM LRM 72 0 0 26 May 2025
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models Hao Chen Haoze Li Zhiqing Xiao Lirong Gao Qi Zhang Xiaomeng Hu Ningtao Wang Xing Fu Junbo Zhao 187 0 0 24 May 2025
L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models Xiaohao Liu Xiaobo Xia Weixiang Zhao Manyi Zhang Xianzhi Yu Xiu Su Shuo Yang See-Kiong Ng Tat-Seng Chua KELM LRM 89 0 0 23 May 2025
Adaptive Plan-Execute Framework for Smart Contract Security Auditing Zhiyuan Wei Jing Sun Zijian Zhang Zhe Hou Zixiao Zhao 177 0 0 21 May 2025
CLEVER: A Curated Benchmark for Formally Verified Code Generation Amitayush Thakur Jasper Lee George Tsoukalas Meghana Sistla Matthew Zhao Stefan Zetzsche Greg Durrett Yisong Yue Swarat Chaudhuri ALM 120 0 0 20 May 2025
Think Only When You Need with Large Hybrid-Reasoning Models Lingjie Jiang Xun Wu Shaohan Huang Qingxiu Dong Zewen Chi Li Dong Xingxing Zhang Tengchao Lv Lei Cui Furu Wei OffRL LRM 151 5 0 20 May 2025
Shadow-FT: Tuning Instruct via Base Taiqiang Wu Runming Yang Jiayi Li Pengfei Hu Ngai Wong Yujiu Yang 223 0 0 19 May 2025
Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings Safal Shrestha Minwu Kim Aadim Nepal Anubhav Shrestha Keith Ross OffRL ReLM LRM 79 0 0 19 May 2025
OSS-Bench: Benchmark Generator for Coding LLMs Yuancheng Jiang Roland Yap Zhenkai Liang 68 0 0 18 May 2025
xGen-small Technical Report Erik Nijkamp Bo Pang Egor Pakhomov Akash Gokul Jin Qu Silvio Savarese Yingbo Zhou Caiming Xiong LLMAG 149 0 0 10 May 2025
Scalable Chain of Thoughts via Elastic Reasoning Yuhui Xu Hanze Dong Lei Wang Doyen Sahoo Junnan Li Caiming Xiong OffRL LRM 112 8 0 08 May 2025
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code Kazuki Fujii Yukito Tajima Sakae Mizuki Hinari Shimada Taihei Shiotani ... Kakeru Hattori Youmi Ma Hiroya Takamura Rio Yokota Naoaki Okazaki SyDa 131 1 0 05 May 2025
CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation Anirudh Khatry Robert Zhang Jia Pan Ziteng Wang Qiaochu Chen Greg Durrett Isil Dillig 80 0 0 21 Apr 2025
Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators Yilun Zhou Austin Xu Peifeng Wang Caiming Xiong Shafiq Joty ELM ALM LRM 156 5 0 21 Apr 2025
CodeCrash: Stress Testing LLM Reasoning under Structural and Semantic Perturbations Man Ho Lam Chaozheng Wang Jen-tse Huang Michael R. Lyu LRM 80 1 0 19 Apr 2025
LLM Sensitivity Evaluation Framework for Clinical Diagnosis Chenwei Yan Xiangling Fu Yuxuan Xiong Tianyi Wang Siu Cheung Hui Ji Wu Xien Liu LM&MA ELM 77 2 0 18 Apr 2025
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? Yang Yue Zhiqi Chen Rui Lu Andrew Zhao Zhaokai Wang Yang Yue Shiji Song Gao Huang ReLM LRM 216 128 0 18 Apr 2025
Generating Planning Feedback for Open-Ended Programming Exercises with LLMs Mehmet Arif Demirtaş Claire Zheng Max Fowler Kathryn Cunningham LRM 67 2 0 11 Apr 2025
How Accurately Do Large Language Models Understand Code? Sabaat Haroon Ahmad Faraz Khan Ahmad Humayun Waris Gill Abdul Haddi Amjad A. R. Butt Mohammad Taha Khan Muhammad Ali Gulzar ELM LRM 102 2 0 06 Apr 2025
Cognitive Debiasing Large Language Models for Decision-Making Yougang Lyu Shijie Ren Yue Feng Zihan Wang Zhongfu Chen Zhaochun Ren Maarten de Rijke 204 0 0 05 Apr 2025
Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems? Kai Yan Yufei Xu Zhengyin Du Xuesong Yao Ziyi Wang Xiaowen Guo Jiecao Chen ReLM ELM LRM 189 5 0 01 Apr 2025
Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute Yingwei Ma Binhua Li Yihong Dong Xue Jiang Rongyu Cao Jingshu Chen Fei Huang Yongqian Li LLMAG LRM 126 7 0 31 Mar 2025
Effective Skill Unlearning through Intervention and Abstention Yongce Li Chung-En Sun Tsui-Wei Weng MU 431 1 0 27 Mar 2025
XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants Adam Štorek Mukur Gupta Noopur Bhatt Aditya Gupta Janie Kim Prashast Srivastava Suman Jana AAML 195 0 0 18 Mar 2025
Can Reasoning Models Reason about Hardware? An Agentic HLS Perspective Luca Collini Andrew Hennessee Ramesh Karri Siddharth Garg ELM LRM 77 2 0 17 Mar 2025
Unveiling Pitfalls: Understanding Why AI-driven Code Agents Fail at GitHub Issue Resolution Zhi Chen Wei Ma Lingxiao Jiang LLMAG 109 0 0 16 Mar 2025
Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models Averi Bates Ryan Vavricka Shane Carleton Ruosi Shao Chongle Pan 101 0 0 15 Mar 2025
PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computing Cheng Deng Luoyang Sun Jiwen Jiang Yongcheng Zeng Xinjian Wu ... Haoyang Li Lei Chen Lionel M. Ni Jun Wang Jun Wang 411 0 0 15 Mar 2025
ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness Ce Guo Tong Zhao 126 1 0 11 Mar 2025
Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models Anastasiia Grishina Vadim Liventsev Aki Härmä Leon Moonen ELM 153 0 0 10 Mar 2025
From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper Sargam Yadav Asifa Mehmood Qureshi Abhishek Kaushik Shubham Sharma Roisin Loughran ... . Nikhil Singh Padraic O'Hara Pranay Jaiswal Roshan Chandru David Lillis 126 1 0 10 Mar 2025
Robust Learning of Diverse Code Edits Tushar Aggarwal Swayam Singh Abhijeet Awasthi Aditya Kanade Nagarajan Natarajan SyDa 527 0 0 05 Mar 2025
Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference Thanh Le-Cong Bach Le Toby Murray LRM 113 1 0 22 Feb 2025
How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark Ruizhong Qiu Weiliang Will Zeng Hanghang Tong James Ezick Christopher Lott 178 22 0 20 Feb 2025
LeDex: Training LLMs to Better Self-Debug and Explain Code Nan Jiang Xiaopeng Li Shiqi Wang Qiang Zhou Soneya Binta Hossain Baishakhi Ray Varun Kumar Xiaofei Ma Anoop Deoras LRM 160 18 0 17 Feb 2025
Flaming-hot Initiation with Regular Execution Sampling for Large Language Models Weizhe Chen Zhicheng Zhang Guanlin Liu Renjie Zheng Wenlei Shi Chen Dun Zheng Wu Xing Jin Lin Yan ALM LRM 159 3 0 17 Feb 2025
GiFT: Gibbs Fine-Tuning for Code Generation Haochen Li Wanjin Feng Xin Zhou Zhiqi Shen SyDa 161 1 0 17 Feb 2025
SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors Bohan Lyu Siqiao Huang Zichen Liang Qi-An Sun Jiaming Zhang ELM LRM 122 0 0 16 Feb 2025
Typhoon T1: An Open Thai Reasoning Model Pittawat Taveekitworachai Potsawee Manakul Kasima Tharnpipitchai Kunat Pipatanakul OffRL LRM 233 0 0 13 Feb 2025
BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models Xu Huang Wenhao Zhu Hanxu Hu Zeang Sheng Lei Li Shujian Huang Fei Yuan ELM 144 4 0 11 Feb 2025
MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations Kaixuan Huang Jiacheng Guo Zihao Li X. Ji Jiawei Ge ... Yangsibo Huang Chi Jin Xinyun Chen Chiyuan Zhang Mengdi Wang AAML LRM 207 17 0 10 Feb 2025
Cardiverse: Harnessing LLMs for Novel Card Game Prototyping Danrui Li Sen Zhang Sam S. Sohn Kaidong Hu Muhammad Usman Mubbasir Kapadia 170 0 0 10 Feb 2025
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories Yijia Xiao Runhui Wang Luyang Kong Davor Golac Wei Wang LLMAG 457 3 0 10 Feb 2025
CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging Md. Ashraful Islam Mohammed Eunus Ali Md. Rizwan Parvez LLMAG 150 4 0 08 Feb 2025
Learning to Generate Unit Tests for Automated Debugging Archiki Prasad Elias Stengel-Eskin Justin Chih-Yao Chen Zaid Khan Joey Tianyi Zhou ELM 158 4 0 03 Feb 2025
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics Kai He Rui Mao Qika Lin Yucheng Ruan Xiang Lan Mengling Feng Min Zhang LM&MA AILaw 226 175 0 28 Jan 2025
mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation Nishat Raihan Antonios Anastasopoulos Marcos Zampieri ELM 118 8 0 28 Jan 2025
Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code Shahin Honarvar Mark van der Wilk Alastair Donaldson 200 8 0 28 Jan 2025
Treefix: Enabling Execution with a Tree of Prefixes Beatriz Souza Michael Pradel 99 1 0 21 Jan 2025