Moving Beyond Medical Exam Questions: A Clinician-Annotated Dataset of Real-World Tasks and Ambiguity in Mental Healthcare

22 February 2025

Monika Drummond Roots

Papers citing "Moving Beyond Medical Exam Questions: A Clinician-Annotated Dataset of Real-World Tasks and Ambiguity in Mental Healthcare"

38 / 38 papers shown

Title
XtraGPT: LLMs for Human-AI Collaboration on Controllable Academic Paper Revision Nuo Chen Andre Lin HuiKai Jiaying Wu Junyi Hou Zining Zhang Qian Wang Xiaobei Wang Bingsheng He 22 1 0 16 May 2025
Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers Jared Moore Declan Grabb William Agnew Kevin Klyman Stevie Chancellor Desmond C. Ong Nick Haber AI4MH 63 1 0 25 Apr 2025
DICE: A Framework for Dimensional and Contextual Evaluation of Language Models Aryan Shrivastava Paula Akemi Aoyagui 42 0 0 14 Apr 2025
BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices Anka Reuel Amelia F. Hardy Chandler Smith Max Lamparth Malcolm Hardy Mykel J. Kochenderfer ELM 127 19 0 20 Nov 2024
Measuring Free-Form Decision-Making Inconsistency of Language Models in Military Crisis Simulations Aryan Shrivastava Jessica Hullman Max Lamparth 56 6 0 17 Oct 2024
Can AI Relate: Testing Large Language Model Response for Mental Health Support Saadia Gabriel Isha Puri Xuhai Xu Matteo Malgaroli Marzyeh Ghassemi LM&MA AI4MH 68 12 0 20 May 2024
Capabilities of Gemini Models in Medicine Khaled Saab Tao Tu Wei-Hung Weng Ryutaro Tanno David Stutz ... Christopher Semturs S. S. Mahdavi Juraj Gottweis Alan Karthikesalingam Vivek Natarajan ELM AI4MH LM&MA 41 164 0 29 Apr 2024
Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementation Declan Grabb Max Lamparth N. Vasan 55 15 0 02 Apr 2024
Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations Max Lamparth Anthony Corso Jacob Ganz O. Mastro Jacquelyn G. Schneider Harold Trinkunas 73 8 0 06 Mar 2024
Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: A Benchmark Study Prottay Kumar Adhikary Aseem Srivastava Shivani Kumar Salam Michael Singh Puneet Manuja Jini K. Gopinath Vijay Krishnan Swati Kedia K. Deb Tanmoy Chakraborty AI4MH 72 10 0 29 Feb 2024
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence Timothy R. McIntosh Teo Susnjak Tong Liu Paul Watters Malka N. Halgamuge ALM ELM 71 53 0 15 Feb 2024
Towards Conversational Diagnostic AI Tao Tu Anil Palepu M. Schaekermann Khaled Saab Jan Freyberg ... Katherine Chou Greg S. Corrado Yossi Matias Alan Karthikesalingam Vivek Natarajan AI4MH LM&MA 38 99 0 11 Jan 2024
MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models Xin Liu Yichen Zhu Jindong Gu Yunshi Lan Chao Yang Yu Qiao 59 98 0 29 Nov 2023
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models Zeming Chen Alejandro Hernández Cano Angelika Romanou Antoine Bonnet Kyle Matoba ... Axel Marmet Syrielle Montariol Mary-Anne Hartley Martin Jaggi Antoine Bosselut LM&MA AI4MH MedIm 83 190 0 27 Nov 2023
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue Yuansheng Ni Kai Zhang Tianyu Zheng Ruoqi Liu ... Yibo Liu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen OSLM ELM VLM 140 833 0 27 Nov 2023
GPQA: A Graduate-Level Google-Proof Q&A Benchmark David Rein Betty Li Hou Asa Cooper Stickland Jackson Petty Richard Yuanzhe Pang Julien Dirani Julian Michael Samuel R. Bowman AI4MH ELM 64 584 0 20 Nov 2023
MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models Kailai Yang Tianlin Zhang Zi-Zhou Kuang Qianqian Xie Jimin Huang Sophia Ananiadou AI4MH 48 51 0 24 Sep 2023
SafetyBench: Evaluating the Safety of Large Language Models Zhexin Zhang Leqi Lei Lindong Wu Rui Sun Yongkang Huang Chong Long Xiao Liu Xuanyu Lei Jie Tang Minlie Huang LRM LM&MA ELM 52 95 0 13 Sep 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 197 11,484 0 18 Jul 2023
ECG-QA: A Comprehensive Question Answering Dataset Combined With Electrocardiogram Jungwoo Oh Gyubok Lee Seongsu Bae Joon-Myoung Kwon Edward Choi 43 17 0 21 Jun 2023
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models Wei Ping Weixin Chen Hengzhi Pei Chulin Xie Mintong Kang ... Zinan Lin Yuk-Kit Cheng Sanmi Koyejo D. Song Yue Liu 46 405 0 20 Jun 2023
PMC-LLaMA: Towards Building Open-source Language Models for Medicine Chaoyi Wu Weixiong Lin Xiaoman Zhang Ya Zhang Yanfeng Wang Weidi Xie LM&MA AI4MH 111 77 0 27 Apr 2023
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation Lorenz Kuhn Y. Gal Sebastian Farquhar UQLM 106 282 0 19 Feb 2023
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them Mirac Suzgun Nathan Scales Nathanael Scharli Sebastian Gehrmann Yi Tay ... Aakanksha Chowdhery Quoc V. Le Ed H. Chi Denny Zhou Jason W. Wei ALM ELM LRM ReLM 149 1,068 0 17 Oct 2022
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned Deep Ganguli Liane Lovitt John Kernion Amanda Askell Yuntao Bai ... Nicholas Joseph Sam McCandlish C. Olah Jared Kaplan Jack Clark 257 458 0 23 Aug 2022
Language Models (Mostly) Know What They Know Saurav Kadavath Tom Conerly Amanda Askell T. Henighan Dawn Drain ... Nicholas Joseph Benjamin Mann Sam McCandlish C. Olah Jared Kaplan ELM 92 758 0 11 Jul 2022
Teaching Models to Express Their Uncertainty in Words Stephanie C. Lin Jacob Hilton Owain Evans OOD 48 388 0 28 May 2022
Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support Ashish Sharma Inna Wanyin Lin Adam S. Miner David C. Atkins Tim Althoff AI4MH 53 197 0 28 Mar 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 662 12,525 0 04 Mar 2022
Evaluating Large Language Models Trained on Code Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Pondé ... Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba ELM ALM 141 5,328 0 07 Jul 2021
Pathological Visual Question Answering Xuehai He Zhuo Cai Wenlan Wei Yichen Zhang Luntian Mou Eric Xing P. Xie 105 24 0 06 Oct 2020
What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams Di Jin Eileen Pan Nassim Oufattole W. Weng Hanyi Fang Peter Szolovits FaML ELM LM&MA 72 749 0 28 Sep 2020
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika D. Song Jacob Steinhardt ELM RALM 127 4,222 0 07 Sep 2020
Learning to summarize from human feedback Nisan Stiennon Long Ouyang Jeff Wu Daniel M. Ziegler Ryan J. Lowe Chelsea Voss Alec Radford Dario Amodei Paul Christiano ALM 164 2,071 0 02 Sep 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 95 2,682 0 05 Jun 2020
Scalable agent alignment via reward modeling: a research direction Jan Leike David M. Krueger Tom Everitt Miljan Martic Vishal Maini Shane Legg 65 402 0 19 Nov 2018
Deep reinforcement learning from human preferences Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg Dario Amodei 86 3,197 0 12 Jun 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 366 4,444 0 18 Apr 2017