mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation

28 January 2025

Nishat Raihan

Antonios Anastasopoulos

Marcos Zampieri

ELM

ArXiv (abs)PDF HTML

Papers citing "mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation"

25 / 25 papers shown

Title
When Models Reason in Your Language: Controlling Thinking Trace Language Comes at the Cost of Accuracy Jirui Qi Shan Chen Zidi Xiong Raquel Fernández Danielle S. Bitterman Arianna Bisazza LRM 70 0 0 28 May 2025
How Accurately Do Large Language Models Understand Code? Sabaat Haroon Ahmad Faraz Khan Ahmad Humayun Waris Gill Abdul Haddi Amjad A. R. Butt Mohammad Taha Khan Muhammad Ali Gulzar ELM LRM 85 2 0 06 Apr 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yize Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Zheng Zhang Yan Huang Liang Wang Tieniu Tan 399 4 0 18 Mar 2025
TigerLLM -- A Family of Bangla Large Language Models Nishat Raihan Marcos Zampieri 90 0 0 14 Mar 2025
Code LLMs: A Taxonomy-based Survey Nishat Raihan Christian D. Newman Marcos Zampieri 127 1 0 11 Dec 2024
MojoBench: Language Modeling and Benchmarks for Mojo Nishat Raihan Joanna C. S. Santos Marcos Zampieri 82 2 0 23 Oct 2024
CSEPrompts: A Benchmark of Introductory Computer Science Prompts Md. Nishat Raihan Dhiman Goswami Sadiya Sayara Chowdhury Puspo Christian D. Newman Tharindu Ranasinghe Marcos Zampieri ELM 51 2 0 03 Apr 2024
HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization Qiwei Peng Yekun Chai Xuhong Li ELM LM&MA 80 45 0 26 Feb 2024
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model Ahmet Üstün Viraat Aryabumi Zheng-Xin Yong Wei-Yin Ko Daniel D'souza ... Shayne Longpre Niklas Muennighoff Marzieh Fadaee Julia Kreutzer Sara Hooker ALM ELM SyDa LRM 91 228 0 12 Feb 2024
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models Damai Dai Chengqi Deng Chenggang Zhao R. X. Xu Huazuo Gao ... Panpan Huang Fuli Luo Chong Ruan Zhifang Sui W. Liang MoE 102 309 0 11 Jan 2024
xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection Nuno M. Guerreiro Ricardo Rei Daan van Stigt Luísa Coheur Pierre Colombo André F.T. Martins 108 138 0 16 Oct 2023
Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task Ricardo Rei Nuno M. Guerreiro José P. Pombal Daan van Stigt Marcos Vinícius Treviso Luísa Coheur José G. C. de Souza André F. T. Martins 80 63 0 21 Sep 2023
WizardCoder: Empowering Code Large Language Models with Evol-Instruct Ziyang Luo Can Xu Pu Zhao Qingfeng Sun Xiubo Geng Wenxiang Hu Chongyang Tao Jing Ma Qingwei Lin Daxin Jiang ELM SyDa ALM 113 687 0 14 Jun 2023
StarCoder: may the source be with you! Raymond Li Loubna Ben Allal Yangtian Zi Niklas Muennighoff Denis Kocetkov ... Sean M. Hughes Thomas Wolf Arjun Guha Leandro von Werra H. D. Vries 110 777 0 09 May 2023
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation Jiawei Liu Chun Xia Yuyao Wang Lingming Zhang ELM ALM 253 951 0 02 May 2023
Measuring The Impact Of Programming Language Distribution Gabriel Orlanski Kefan Xiao Xavier Garcia Jeffrey Hui Joshua Howland J. Malmaud Jacob Austin Rishah Singh Michele Catasta 136 31 0 03 Feb 2023
No Language Left Behind: Scaling Human-Centered Machine Translation Nllb team Marta R. Costa-jussá James Cross Onur cCelebi Maha Elbayad ... Alexandre Mourachko C. Ropers Safiyyah Saleem Holger Schwenk Jeff Wang MoE 230 1,266 0 11 Jul 2022
Systematic Inequalities in Language Technology Performance across the World's Languages Damián E. Blasi Antonios Anastasopoulos Graham Neubig 159 139 0 13 Oct 2021
Program Synthesis with Large Language Models Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski ... Ellen Jiang Carrie J. Cai Michael Terry Quoc V. Le Charles Sutton ELM AIMat ReCod ALM 216 2,004 0 16 Aug 2021
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 236 5,647 0 07 Jul 2021
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 877 42,379 0 28 May 2020
The State and Fate of Linguistic Diversity and Inclusion in the NLP World Pratik M. Joshi Sebastin Santy A. Budhiraja Kalika Bali Monojit Choudhury LMTD 122 856 0 20 Apr 2020
Unsupervised Cross-lingual Representation Learning at Scale Alexis Conneau Kartikay Khandelwal Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov 228 6,587 0 05 Nov 2019
BERTScore: Evaluating Text Generation with BERT Tianyi Zhang Varsha Kishore Felix Wu Kilian Q. Weinberger Yoav Artzi 352 5,860 0 21 Apr 2019
Mapping Language to Code in Programmatic Context Srinivasan Iyer Ioannis Konstas Alvin Cheung Luke Zettlemoyer 69 238 0 29 Aug 2018