CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

24 October 2023

Taiwei Shi

Nancy F. Chen

Diyi Yang

ArXiv (abs)PDF HTML Github (21★)

Papers citing "CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation"

23 / 23 papers shown

Title
Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics Hamed Mahdavi Alireza Hashemi Majid Daliri Pegah Mohammadipour Alireza Farhadi Samira Malek Yekta Yazdanifard Amir Khasahmadi V. Honavar ELM LRM 141 4 0 01 Apr 2025
Few-shot LLM Synthetic Data with Distribution Matching Jiyuan Ren Zhaocheng Du Zhihao Wen Qinglin Jia Sunhao Dai Chuhan Wu Zhenhua Dong SyDa 191 0 0 09 Feb 2025
Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback Lester James V. Miranda Yizhong Wang Yanai Elazar Sachin Kumar Valentina Pyatkin Faeze Brahman Noah A. Smith Hannaneh Hajishirzi Pradeep Dasigi 124 12 0 24 Oct 2024
Mitigating Selection Bias with Node Pruning and Auxiliary Options Hyeong Kyu Choi Weijie Xu Chi Xue Stephanie Eckman Chandan K. Reddy 90 2 0 27 Sep 2024
ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model Lifan Jiang Zhihui Wang Siqi Yin Guangxiao Ma Peng Zhang Boxi Wu DiffM 141 0 0 28 Aug 2024
Can Unconfident LLM Annotations Be Used for Confident Conclusions? Kristina Gligorić Tijana Zrnic Cinoo Lee Emmanuel J. Candès Dan Jurafsky 153 12 0 27 Aug 2024
Real-time Speech Summarization for Medical Conversations Khai-Nguyen Nguyen Khai Le-Duc Long Vo-Dang Truong-Son Hy MedIm 173 2 0 22 Jun 2024
Is ChatGPT a Good NLG Evaluator? A Preliminary Study Jiaan Wang Yunlong Liang Fandong Meng Zengkui Sun Haoxiang Shi Zhixu Li Jinan Xu Jianfeng Qu Jie Zhou LM&MA ELM ALM AI4MH 138 472 0 07 Mar 2023
ChatGPT: Jack of all trades, master of none Jan Kocoñ Igor Cichecki Oliwier Kaszyca Mateusz Kochanek Dominika Szydło ... Maciej Piasecki Lukasz Radliñski Konrad Wojtasik Stanislaw Wo'zniak Przemyslaw Kazienko AI4MH 130 553 0 21 Feb 2023
Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT Qihuang Zhong Liang Ding Juhua Liu Bo Du Dacheng Tao AI4MH 121 245 0 19 Feb 2023
Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech Fan Huang Haewoon Kwak Jisun An AI4MH 73 267 0 11 Feb 2023
TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media Daniel Loureiro Aminette D'Souza Areej Muhajab Isabella A. White Gabriel Wong Luis Espinosa Anke Leonardo Neves Francesco Barbieri Jose Camacho-Collados 71 26 0 15 Sep 2022
Language Models (Mostly) Know What They Know Saurav Kadavath Tom Conerly Amanda Askell T. Henighan Dawn Drain ... Nicholas Joseph Benjamin Mann Sam McCandlish C. Olah Jared Kaplan ELM 133 833 0 11 Jul 2022
Teaching Models to Express Their Uncertainty in Words Stephanie C. Lin Jacob Hilton Owain Evans OOD 96 425 0 28 May 2022
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation Alisa Liu Swabha Swayamdipta Noah A. Smith Yejin Choi 173 221 0 16 Jan 2022
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets Ann Yuan Daphne Ippolito Vitaly Nikolaev Chris Callison-Burch Andy Coenen Sebastian Gehrmann SyDa 174 23 0 11 Nov 2021
Creating Training Sets via Weak Indirect Supervision Jieyu Zhang Bohan Wang Xiangchen Song Yujing Wang Yaming Yang Jing Bai Alexander Ratner OffRL 136 17 0 07 Oct 2021
Want To Reduce Labeling Cost? GPT-3 Can Help Shuohang Wang Yang Liu Yichong Xu Chenguang Zhu Michael Zeng 75 257 0 30 Aug 2021
Pareto-Optimal Quantized ResNet Is Mostly 4-bit AmirAli Abdolrashidi Lisa Wang Shivani Agrawal J. Malmaud Oleg Rybakov Chas Leichner Lukasz Lew MQ 69 36 0 07 May 2021
Named Entity Recognition without Labelled Data: A Weak Supervision Approach Pierre Lison A. Hubin Jeremy Barnes Samia Touileb 68 114 0 30 Apr 2020
Learning from Rules Generalizing Labeled Exemplars Abhijeet Awasthi Sabyasachi Ghosh Rasna Goyal Sunita Sarawagi 91 86 0 13 Apr 2020
Conversations Gone Awry: Detecting Early Signs of Conversational Failure Justine Zhang Cristian Danescu-Niculescu-Mizil Cristian Danescu-Niculescu-Mizil Lucas Dixon Yiqing Hua Nithum Thain Dario Taraborelli 66 190 0 14 May 2018
Snorkel: Rapid Training Data Creation with Weak Supervision Alexander Ratner Stephen H. Bach Henry R. Ehrenberg Jason Alan Fries Sen Wu Christopher Ré 83 1,032 0 28 Nov 2017