Title
MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering Rushi Qiang Yuchen Zhuang Yinghao Li D. Kilman Rongzhi Zhang ... Ian Shu-Hei Wong Sherry Yang Percy Liang Chao Zhang Bo Dai ELM 41 0 0 12 May 2025
Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks Kai Xu YiWei Mao XinYi Guan ZiLong Feng 43 0 0 12 May 2025
Enhancing Code Generation via Bidirectional Comment-Level Mutual Grounding Yifeng Di Tianyi Zhang 26 0 0 12 May 2025
HiBayES: A Hierarchical Bayesian Modeling Framework for AI Evaluation Statistics Lennart Luettgau Harry Coppock Magda Dubois Christopher Summerfield Cozmin Ududec 31 0 0 08 May 2025
CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation Sizhe Wang Zhilin Wang Dongsheng Ma Yongan Yu Rui Ling Zehan Li Feiyu Xiong Wenqi Zhang LRM 60 0 0 30 Apr 2025
Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges Yunseo Lee John Youngeun Song Dongsun Kim Jindae Kim Mijung Kim Jaechang Nam HILM LRM 42 0 0 29 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong Zhaoxin Fan Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Tianwei Zhang ALM ELM 86 2 0 26 Apr 2025
Circinus: Efficient Query Planner for Compound ML Serving Banruo Liu Wei-Yu Lin Minghao Fang Yihan Jiang Fan Lai LRM 36 0 0 23 Apr 2025
A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs Musfiqur Rahman SayedHassan Khatoonabadi Emad Shihab ALM 39 0 0 22 Apr 2025
Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo João Loula Benjamin LeBrun Li Du Ben Lipkin Clemente Pasti ... Ryan Cotterel Vikash K. Mansinghka Alexander K. Lew Tim Vieira Timothy J. O'Donnell 34 2 0 17 Apr 2025
ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines Tengjun Jin Yuxuan Zhu Daniel Kang LMTD ELM 47 0 0 07 Apr 2025
ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness Ce Guo Tong Zhao 61 1 0 11 Mar 2025
Transferable Foundation Models for Geometric Tasks on Point Cloud Representations: Geometric Neural Operators Blaine Quackenbush P. Atzberger 3DPC AI4CE 65 2 0 06 Mar 2025
Selective Prompt Anchoring for Code Generation Yuan Tian Tianyi Zhang 94 3 0 24 Feb 2025
An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science Qiuhai Zeng Claire Jin Xinyue Wang Yuhan Zheng Qunhua Li 48 0 0 23 Feb 2025
Pragmatic Reasoning improves LLM Code Generation Zhuchen Cao Sven Apel Adish Singla Vera Demberg LRM 42 0 0 20 Feb 2025
How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark Ruizhong Qiu Weiliang Will Zeng Hanghang Tong James Ezick Christopher Lott 88 15 0 20 Feb 2025
DataSciBench: An LLM Agent Benchmark for Data Science Dan Zhang Sining Zhoubian Min Cai Fengzu Li L. Yang Wei Wang Tianjiao Dong Ziniu Hu J. Tang Yisong Yue ALM ELM 46 2 0 20 Feb 2025
SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors Bohan Lyu Siqiao Huang Zichen Liang Qi-An Sun Jiaming Zhang ELM LRM 60 0 0 16 Feb 2025
KernelBench: Can LLMs Write Efficient GPU Kernels? Anne Ouyang Simon Guo Simran Arora Alex L. Zhang William Hu Christopher Ré Azalia Mirhoseini ALM 52 2 0 14 Feb 2025
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories Yijia Xiao Runhui Wang Luyang Kong Davor Golac Wei Wang LLMAG 162 0 0 10 Feb 2025
LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation Ziyao Zhang Yanlin Wang Chong Wang Jiachi Chen Zibin Zheng 128 14 0 20 Jan 2025
WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models Huawen Feng Pu Zhao Qingfeng Sun Can Xu Fangkai Yang ... Qianli Ma Qingwei Lin Saravan Rajmohan Dongmei Zhang Qi Zhang AAML ALM 62 0 0 23 Dec 2024
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows Fangyu Lei Jixuan Chen Yuxiao Ye Ruisheng Cao Dongchan Shin ... Caiming Xiong Ruoxi Sun Qian Liu Sida I. Wang Tao Yu LMTD 82 21 0 12 Nov 2024
OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models Siming Huang Tianhao Cheng J.K. Liu Jiaran Hao L. Song ... Ge Zhang Zili Wang Yuan Qi Yinghui Xu Wei Chu ALM 80 17 0 07 Nov 2024
GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models Nizar Islah Justine Gehring Diganta Misra Eilif B. Muller Irina Rish Terry Yue Zhuo Massimo Caccia SyDa 42 1 0 05 Nov 2024
Mastering the Craft of Data Synthesis for CodeLLMs Meng Chen Philip Arthur Qianyu Feng Cong Duy Vu Hoang Yu-Heng Hong ... Mark Johnson Kemal Kurniawan Don Dharmasiri Long Duong Yuan-Fang Li SyDa 60 1 0 16 Oct 2024
CursorCore: Assist Programming through Aligning Anything Hao Jiang Qi Liu Rui Li Shengyu Ye Shijin Wang 53 1 0 09 Oct 2024
ScriptSmith: A Unified LLM Framework for Enhancing IT Operations via Automated Bash Script Generation, Assessment, and Refinement Oishik Chatterjee Pooja Aggarwal Suranjana Samanta Ting Dai P. Mohapatra ... Ruchi Mahindru Steve Barbieri Eugen Postea Brad Blancett Arthur De Magalhaes 28 1 0 12 Sep 2024
What can Large Language Models Capture about Code Functional Equivalence? Nickil Maveli Antonio Vergari Shay B. Cohen 44 2 0 20 Aug 2024
AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation Mengkang Hu Yixiao Wang Can Xu Lingfeng Sun Chensheng Peng T. Hannagan Nicola Poerio Saravan Rajmohan LM&Ro LLMAG 69 15 0 01 Aug 2024
CodeUpdateArena: Benchmarking Knowledge Editing on API Updates Zeyu Leo Liu Shrey Pandit Xi Ye Eunsol Choi Greg Durrett KELM ALM 81 4 0 08 Jul 2024
InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation Gaurav Sahu Abhay Puri Juan A. Rodriguez Alexandre Drouin Perouz Taslakian ... Christopher Pal Nicolas Chapados I. Laradji Sai Rajeswar Mudumba Issam Hadj Laradji ELM 48 4 0 08 Jul 2024
ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages Mehant Kammakomati Sameer Pimparkhede Srikanth G. Tamilselvam Prince Kumar Pushpak Bhattacharyya ALM 40 0 0 03 Jul 2024
Agentless: Demystifying LLM-based Software Engineering Agents Chunqiu Steven Xia Yinlin Deng Soren Dunn Lingming Zhang LLMAG 43 85 0 01 Jul 2024
BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions Terry Yue Zhuo Minh Chien Vu Jenny Chim Han Hu Wenhao Yu ... David Lo Daniel Fried Xiaoning Du H. D. Vries Leandro von Werra 77 134 0 22 Jun 2024
Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models Sanjay Vishwakarma Francis Harkins Siddharth Golecha Vishal Sharathchandra Bajpe Nicolas Dupuis Luca Buratti David Kremer Ismael Faro Ruchir Puri Juan Cruz-Benito ELM 50 3 0 20 Jun 2024
CodeRAG-Bench: Can Retrieval Augment Code Generation? Zora Zhiruo Wang Akari Asai Xinyan Velocity Yu Frank F. Xu Yiqing Xie Graham Neubig Daniel Fried RALM 80 30 0 20 Jun 2024
ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation Cheng Yang Chufan Shi Yaxin Liu Bo Shui Junjie Wang ... Yuxiang Zhang Gongye Liu Xiaomei Nie Deng Cai Yujiu Yang MLLM LRM 51 22 0 14 Jun 2024
DCA-Bench: A Benchmark for Dataset Curation Agents Benhao Huang Yingzhuo Yu Jin Huang Xingjian Zhang Jiaqi Ma 36 1 0 11 Jun 2024
Robo-Instruct: Simulator-Augmented Instruction Alignment For Finetuning Code LLMs Zichao Hu Junyi Jessy Li Arjun Guha Joydeep Biswas SyDa ALM 51 1 0 30 May 2024
Kotlin ML Pack: Technical Report Sergey Titov Mikhail Evtikhiev Anton Shapkin Oleg Smirnov Sergei Boytsov ... Dariia Karaeva Maksim Sheptyakov Mikhail Arkhipov T. Bryksin Egor Bogomolov 32 0 0 29 May 2024
ReflectionCoder: Learning from Reflection Sequence for Enhanced One-off Code Generation Houxing Ren Mingjie Zhan Zhongyuan Wu Aojun Zhou Junting Pan Hongsheng Li SyDa 42 7 0 27 May 2024
Granite Code Models: A Family of Open Foundation Models for Code Intelligence Mayank Mishra Matt Stallone Gaoyuan Zhang Yikang Shen Aditya Prasad ... Amith Singhee Nirmit Desai David D. Cox Ruchir Puri Rameswar Panda AI4TS 56 55 0 07 May 2024
PECC: Problem Extraction and Coding Challenges Patrick Haller Jonas Golde Alan Akbik ReLM 40 5 0 29 Apr 2024
CSEPrompts: A Benchmark of Introductory Computer Science Prompts Md. Nishat Raihan Dhiman Goswami Sadiya Sayara Chowdhury Puspo Christian D. Newman Tharindu Ranasinghe Marcos Zampieri ELM 44 2 0 03 Apr 2024
Exploring and Evaluating Hallucinations in LLM-Powered Code Generation Fang Liu Yang Liu Lin Shi Houkun Huang Ruifeng Wang Zhen Yang Li Zhang Zhongqi Li Yuchi Ma 52 108 0 01 Apr 2024
MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization Zhiyu Yang Zihan Zhou Shuo Wang Xin Cong Xu Han ... Pengyuan Liu Dong Yu Zhiyuan Liu Xiaodong Shi Maosong Sun LLMAG 39 26 0 18 Feb 2024
Are LLMs Ready for Real-World Materials Discovery? Santiago Miret N. M. A. Krishnan 40 27 0 07 Feb 2024
CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules Hung Le Hailin Chen Amrita Saha Akash Gokul Doyen Sahoo Shafiq R. Joty LRM 28 42 0 13 Oct 2023