The Effect of Natural Distribution Shift on Question Answering Models

29 April 2020

Benjamin Recht

Papers citing "The Effect of Natural Distribution Shift on Question Answering Models"

42 / 42 papers shown

Title
Efficient Learning Under Density Shift in Incremental Settings Using Cramér-Rao-Based Regularization Behraj Khan Behroz Mirza Nouman Durrani T. Syed 60 0 0 18 Feb 2025
SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains Ran Xu Hui Liu Sreyashi Nag Zhenwei Dai Yaochen Xie ... Chen Luo Yang Li Joyce C. Ho Carl Yang Qi He RALM 78 8 0 28 Jan 2025
Evaluating Model Performance Under Worst-case Subpopulations Mike Li Hongseok Namkoong Shangzhou Xia 48 17 0 01 Jul 2024
Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering Jie Ma Min Hu Pinghui Wang Wangchun Sun Lingyun Song Hongbin Pei Jun Liu Youtian Du 39 4 0 18 Apr 2024
SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials Mael Jullien Marco Valentino André Freitas LM&MA 41 41 0 07 Apr 2024
Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study Chenguang Wang Ruoxi Jia Xin Liu Dawn Song VLM 29 7 0 15 Mar 2024
Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data YongKyung Oh Dongyoung Lim Sungil Kim AI4TS 43 13 0 22 Feb 2024
Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP Hyuhng Joon Kim Hyunsoo Cho Sang-Woo Lee Junyeob Kim Choonghyun Park Sang-goo Lee Kang Min Yoo Taeuk Kim VLM OOD 48 1 0 23 Oct 2023
Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learning Lucas Weber Elia Bruni Dieuwke Hupkes 32 25 0 20 Oct 2023
CLIFT: Analysing Natural Distribution Shift on Question Answering Models in Clinical Domain Ankit Pal 19 2 0 19 Oct 2023
AGent: A Novel Pipeline for Automatically Creating Unanswerable Questions Son Quoc Tran Gia-Huy Do Phong Nguyen-Thuan Do Matt Kretchmar Xinya Du 29 0 0 10 Sep 2023
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use Yonatan Bitton Hritik Bansal Jack Hessel Rulin Shao Wanrong Zhu Anas Awadalla Josh Gardner Rohan Taori L. Schimdt VLM 31 77 0 12 Aug 2023
Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations Lifan Yuan Yangyi Chen Ganqu Cui Hongcheng Gao Fangyuan Zou Xingyi Cheng Heng Ji Zhiyuan Liu Maosong Sun 41 73 0 07 Jun 2023
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models Shuo Chen Jindong Gu Zhen Han Yunpu Ma Philip Torr Volker Tresp VPVLM VLM 45 17 0 03 Jun 2023
Measuring the Robustness of NLP Models to Domain Shifts Nitay Calderon Naveh Porat Eyal Ben-David Alexander Chapanin Zorik Gekhman Nadav Oved Vitaly Shalumov Roi Reichart 21 7 0 31 May 2023
A Practical Toolkit for Multilingual Question and Answer Generation Asahi Ushio Fernando Alva-Manchego Jose Camacho-Collados SyDa 34 13 0 27 May 2023
An Empirical Comparison of LM-based Question and Answer Generation Methods Asahi Ushio Fernando Alva-Manchego Jose Camacho-Collados 25 20 0 26 May 2023
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark Minje Choi Jiaxin Pei Sagar Kumar Chang Shu David Jurgens ALM LLMAG 35 69 0 24 May 2023
On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Training Hao Fei Tat-Seng Chua Chenliang Li Donghong Ji Meishan Zhang Yafeng Ren 35 39 0 19 Apr 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 40 20 0 07 Mar 2023
Diagnosing Model Performance Under Distribution Shift Tiffany Cai Hongseok Namkoong Steve Yadlowsky 37 27 0 03 Mar 2023
Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems Wang Zhu Jesse Thomason Robin Jia VLM OOD NAI LRM 34 6 0 26 Oct 2022
Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization Daniel LeJeune Jiayu Liu Reinhard Heckel 26 0 0 20 Oct 2022
Are Sample-Efficient NLP Models More Robust? Nelson F. Liu Ananya Kumar Percy Liang Robin Jia VLM OOD 30 6 0 12 Oct 2022
Generative Language Models for Paragraph-Level Question Generation Asahi Ushio Fernando Alva-Manchego Jose Camacho-Collados ELM 13 45 0 08 Oct 2022
A Survey on Measuring and Mitigating Reasoning Shortcuts in Machine Reading Comprehension Xanh Ho Johannes Mario Meissner Saku Sugawara Akiko Aizawa OffRL 35 4 0 05 Sep 2022
Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP Thao Nguyen Gabriel Ilharco Mitchell Wortsman Sewoong Oh Ludwig Schmidt CLIP VLM 50 99 0 10 Aug 2022
Robustness Analysis of Video-Language Models Against Visual and Language Perturbations Madeline Chantry Schiappa Shruti Vyas Hamid Palangi Yogesh S Rawat Vibhav Vineet VLM 123 17 0 05 Jul 2022
Training a Tokenizer for Free with Private Federated Learning Eugene Bagdasaryan Congzheng Song Rogier van Dalen M. Seigel Áine Cahill FedML 22 5 0 15 Mar 2022
Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings Neeraj Varshney Swaroop Mishra Chitta Baral 27 55 0 01 Mar 2022
Measure and Improve Robustness in NLP Models: A Survey Xuezhi Wang Haohan Wang Diyi Yang 139 130 0 15 Dec 2021
Robust fine-tuning of zero-shot models Mitchell Wortsman Gabriel Ilharco Jong Wook Kim Mike Li Simon Kornblith ... Raphael Gontijo-Lopes Hannaneh Hajishirzi Ali Farhadi Hongseok Namkoong Ludwig Schmidt VLM 64 691 0 04 Sep 2021
Predicting with Confidence on Unseen Distributions Devin Guillory Vaishaal Shankar Sayna Ebrahimi Trevor Darrell Ludwig Schmidt UQCV OOD 20 116 0 07 Jul 2021
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning Anders Andreassen Yasaman Bahri Behnam Neyshabur Rebecca Roelofs OOD OODD 30 78 0 30 Jun 2021
Reliability Testing for Natural Language Processing Systems Samson Tan Chenyu You K. Baxter Araz Taeihagh G. Bennett Min-Yen Kan 15 38 0 06 May 2021
MLDemon: Deployment Monitoring for Machine Learning Systems Antonio A. Ginart Martin Jinye Zhang James Zou 54 18 0 28 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 183 27,846 0 26 Feb 2021
Why do classifier accuracies show linear trends under distribution shift? Horia Mania S. Sra OOD 37 19 0 31 Dec 2020
WILDS: A Benchmark of in-the-Wild Distribution Shifts Pang Wei Koh Shiori Sagawa Henrik Marklund Sang Michael Xie Marvin Zhang ... A. Kundaje Emma Pierson Sergey Levine Chelsea Finn Percy Liang OOD 95 1,377 0 14 Dec 2020
Utility is in the Eye of the User: A Critique of NLP Leaderboards Kawin Ethayarajh Dan Jurafsky ELM 24 51 0 29 Sep 2020
Measuring Robustness to Natural Distribution Shifts in Image Classification Rohan Taori Achal Dave Vaishaal Shankar Nicholas Carlini Benjamin Recht Ludwig Schmidt OOD 27 533 0 01 Jul 2020
A Mutual Information Maximization Perspective of Language Representation Learning Lingpeng Kong Cyprien de Masson dÁutume Wang Ling Lei Yu Zihang Dai Dani Yogatama SSL 226 165 0 18 Oct 2019