Benchmarks and Metrics for Evaluations of Code Generation: A Critical
Review

Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review

18 June 2024

Debalina Ghosh Paul

ArXiv (abs)PDF HTML

Papers citing "Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review"

10 / 10 papers shown

Title
CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation Guang Yang Yu Zhou Xiang Chen Wei-Shi Zheng Xing Hu Xin Zhou David Lo Taolue Chen ALM LRM 68 0 0 26 May 2025
SantaCoder: don't reach for the stars! Loubna Ben Allal Raymond Li Denis Kocetkov Chenghao Mou Christopher Akiki ... Sean M. Hughes Daniel Fried Arjun Guha H. D. Vries Leandro von Werra 154 195 0 09 Jan 2023
Out of the BLEU: how should we assess quality of the Code Generation models? Mikhail Evtikhiev Egor Bogomolov Yaroslav Sokolov T. Bryksin ALM 78 93 0 05 Aug 2022
InCoder: A Generative Model for Code Infilling and Synthesis Daniel Fried Armen Aghajanyan Jessy Lin Sida I. Wang Eric Wallace Freda Shi Ruiqi Zhong Wen-tau Yih Luke Zettlemoyer M. Lewis SyDa 69 647 0 12 Apr 2022
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 198 1,986 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 231 5,539 0 07 Jul 2021
Measuring Coding Challenge Competence With APPS Dan Hendrycks Steven Basart Saurav Kadavath Mantas Mazeika Akul Arora ... Collin Burns Samir Puranik Horace He Basel Alomair Jacob Steinhardt ELM AIMat ALM 254 681 0 20 May 2021
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis Shuo Ren Daya Guo Shuai Lu Long Zhou Shujie Liu Duyu Tang Neel Sundaresan M. Zhou Ambrosio Blanco Shuai Ma ELM 112 534 0 22 Sep 2020
JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation R. Agashe R. Campello Arthur Zimek 73 83 0 05 Oct 2019
SPoC: Search-based Pseudocode to Code Sumith Kulal Panupong Pasupat Kartik Chandra Mina Lee Oded Padon A. Aiken Percy Liang 62 225 0 12 Jun 2019

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.