Execution-Based Evaluation for Open-Domain Code Generation

Execution-Based Evaluation for Open-Domain Code Generation

20 December 2022

Daniel Fried

Graham Neubig

Papers citing "Execution-Based Evaluation for Open-Domain Code Generation"

9 / 59 papers shown

Title
xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval Mohammad Abdullah Matin Khan M Saiful Bari Xuan Long Do Weishi Wang Md. Rizwan Parvez Shafiq R. Joty ALM ELM 34 14 0 06 Mar 2023
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code Shuyan Zhou Uri Alon Sumit Agarwal Graham Neubig ELM ALM 34 98 0 10 Feb 2023
Grounding Language Models to Images for Multimodal Inputs and Outputs Jing Yu Koh Ruslan Salakhutdinov Daniel Fried MLLM 31 117 0 31 Jan 2023
Large Language Models Meet NL2Code: A Survey Daoguang Zan B. Chen Fengji Zhang Di Lu Bingchao Wu Bei Guan Yongji Wang Jian-Guang Lou ELM ALM 31 170 0 19 Dec 2022
DocPrompting: Generating Code by Retrieving the Docs Shuyan Zhou Uri Alon Frank F. Xu Zhiruo Wang Zhengbao Jiang Graham Neubig LLMAG 24 129 0 13 Jul 2022
Training and Evaluating a Jupyter Notebook Data Science Assistant Shubham Chandel Colin B. Clement Guillermo Serrato Neel Sundaresan 40 43 0 30 Jan 2022
Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions Naihao Deng Shuaichen Chang Peng Shi Tao Yu Rui Zhang LMTD 38 4 0 15 Sep 2021
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He D. Song Jacob Steinhardt ELM AIMat ALM 208 624 0 20 May 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 261 1,996 0 31 Dec 2020