v1v2v3 (latest)

Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks

7 March 2024

ArXiv (abs)PDF HTML Github (32★)

Papers citing "Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks"

23 / 23 papers shown

Title
From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation? Wasi Uddin Ahmad Somshubra Majumdar Boris Ginsburg 151 0 0 24 May 2025
Decoding Secret Memorization in Code LLMs Through Token-Level Characterization Yuqing Nie Chong Wang Kaidi Wang Guoai Xu Guosheng Xu Haoyu Wang OffRL 415 3 0 11 Oct 2024
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples Shuo Yang Wei-Lin Chiang Lianmin Zheng Joseph E. Gonzalez Ion Stoica ALM 55 126 0 08 Nov 2023
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin Daxin Jiang ELM SyDa ALM 95 687 0 14 Jun 2023
Gorilla: Large Language Model Connected with Massive APIs Shishir G. Patil Tianjun Zhang Xin Wang Joseph E. Gonzalez ELM CLL ALM SyDa 86 556 0 24 May 2023
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X Qinkai Zheng Xiao Xia Xu Zou Yuxiao Dong Shanshan Wang ... Andi Wang Yang Li Teng Su Zhilin Yang Jie Tang ELM ALM SyDa 118 338 0 30 Mar 2023
CoCoMIC: Code Completion By Jointly Modeling In-file and Cross-file Context Yangruibo Ding Zijian Wang Wasi Uddin Ahmad M. K. Ramanathan Ramesh Nallapati Parminder Bhatia Dan Roth Bing Xiang 67 71 0 20 Dec 2022
Natural Language to Code Generation in Interactive Data Science Notebooks Pengcheng Yin Wen-Ding Li Kefan Xiao Abhishek Rao Yeming Wen ... Paige Bailey Michele Catasta Henryk Michalewski Oleksandr Polozov Charles Sutton 59 64 0 19 Dec 2022
Multi-lingual Evaluation of Code Generation Models Ben Athiwaratkun Sanjay Krishna Gouda Zijian Wang Xiaopeng Li Yuchen Tian ... Baishakhi Ray Parminder Bhatia Sudipta Sengupta Dan Roth Bing Xiang ELM 154 175 0 26 Oct 2022
Efficient Training of Language Models to Fill in the Middle Mohammad Bavarian Heewoo Jun Nikolas Tezak John Schulman C. McLeavey Jerry Tworek Mark Chen 76 195 0 28 Jul 2022
Repository-Level Prompt Generation for Large Language Models of Code Disha Shrivastava Hugo Larochelle Daniel Tarlow 79 141 0 26 Jun 2022
InCoder: A Generative Model for Code Infilling and Synthesis Daniel Fried Armen Aghajanyan Jessy Lin Sida I. Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer M. Lewis SyDa 69 652 0 12 Apr 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 880 13,148 0 04 Mar 2022
Competition-Level Code Generation with AlphaCode Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser ... Esme Sutherland Robson Pushmeet Kohli Nando de Koray Kavukcuoglu Oriol Vinyals 143 1,403 0 08 Feb 2022
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation Yue Wang Weishi Wang Shafiq Joty Guosheng Lin 289 1,560 0 02 Sep 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 200 1,986 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 233 5,635 0 07 Jul 2021
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He Basel Alomair Jacob Steinhardt ELM AIMat ALM 257 700 0 20 May 2021
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Zhengxiao Du Yujie Qian Xiao Liu Ming Ding J. Qiu Zhilin Yang Jie Tang BDL AI4CE 137 1,550 0 18 Mar 2021
Unified Pre-training for Program Understanding and Generation Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray Kai-Wei Chang 135 769 0 10 Mar 2021
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan M. Zhou Ambrosio Blanco Shuai Ma ELM 112 542 0 22 Sep 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 445 20,298 0 23 Oct 2019
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation R. Agashe R. Campello Arthur Zimek 73 84 0 05 Oct 2019