On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law

19 May 2020

Papers citing "On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law"

50 / 80 papers shown

Title
QIRL: Boosting Visual Question Answering via Optimized Question-Image Relation Learning Quanxing Xu Ling Zhou Xian Zhong Feifei Zhang Rubing Huang Chia-Wen Lin 39 0 0 04 Apr 2025
Benchmark Data Repositories for Better Benchmarking Rachel Longjohn Markelle Kelly Sameer Singh Padhraic Smyth 51 0 0 31 Oct 2024
On the Role of Visual Grounding in VQA Daniel Reich Tanja Schultz 21 1 0 26 Jun 2024
GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation Baiqi Li Zhiqiu Lin Deepak Pathak Jiayao Li Yixin Fei ... Tiffany Ling Xide Xia Pengchuan Zhang Graham Neubig Deva Ramanan EGVM 52 25 0 19 Jun 2024
Holistic Safety and Responsibility Evaluations of Advanced AI Models Laura Weidinger Joslyn Barnhart Jenny Brennan Christina Butterfield Susie Young ... Sebastian Farquhar Lewis Ho Iason Gabriel Allan Dafoe William S. Isaac ELM 45 8 0 22 Apr 2024
Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learning Yuhang Zheng Zhen Wang Long Chen 24 2 0 28 Jan 2024
Uncovering the Full Potential of Visual Grounding Methods in VQA Daniel Reich Tanja Schultz 32 5 0 15 Jan 2024
Object Attribute Matters in Visual Question Answering Peize Li Q. Si Peng Fu Zheng Lin Yan Wang 35 0 0 20 Dec 2023
Holistic chemical evaluation reveals pitfalls in reaction prediction models Victor Sabanza Gil Andres M Bran Malte Franke Remi Schlama J. Luterbacher Philippe Schwaller ELM 33 1 0 14 Dec 2023
Attribute Diversity Determines the Systematicity Gap in VQA Ian Berlot-Attwell Kumar Krishna Agrawal A. M. Carrell Yash Sharma Naomi Saphra 31 1 0 15 Nov 2023
One-Shot Strategic Classification Under Unknown Costs Elan Rosenfeld Nir Rosenfeld OffRL 33 8 0 05 Nov 2023
An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI Ross Gruetzemacher Alan Chan Kevin Frazier Christy Manning Stepán Los ... Clíodhna Ní Ghuidhir Mark M. Bailey Daniel Eth Toby D. Pilditch Kyle A. Kilian 24 5 0 22 Oct 2023
Robust Collaborative Filtering to Popularity Distribution Shift An Zhang Wenchang Ma Jingnan Zheng Xiang Wang Tat-Seng Chua 29 20 0 16 Oct 2023
Grounded Image Text Matching with Mismatched Relation Reasoning Yu Wu Yan-Tao Wei Haozhe Jasper Wang Yongfei Liu Sibei Yang Xuming He 36 6 0 02 Aug 2023
Robust Visual Question Answering: Datasets, Methods, and Future Challenges Jie Ma Pinghui Wang Dechen Kong Zewei Wang Jun Liu Hongbin Pei Junzhou Zhao OOD 32 18 0 21 Jul 2023
Classical Out-of-Distribution Detection Methods Benchmark in Text Classification Tasks M. Baran Joanna Baran Mateusz Wójcik Maciej Ziȩba Adam Gonczarek OODD 49 4 0 13 Jul 2023
Beyond AUROC & co. for evaluating out-of-distribution detection performance Galadrielle Humblot-Renaux Sergio Escalera T. Moeslund OODD 22 4 0 26 Jun 2023
Experts' cognition-driven ensemble deep learning for external validation of predicting pathological complete response to neoadjuvant chemotherapy from histological images in breast cancer Yongquan Yang Fengling Li Yani Wei Yuanyuan Zhao Jing Fu Xiuli Xiao Hong Bu 38 3 0 19 Jun 2023
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA A. Vosoughi Shijian Deng Songyang Zhang Yapeng Tian Chenliang Xu Jiebo Luo CML 53 3 0 31 May 2023
Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup Damien Teney Jindong Wang Ehsan Abbasnejad 35 6 0 26 May 2023
A Symbolic Framework for Evaluating Mathematical Reasoning and Generalisation with Transformers Jordan Meadows Marco Valentino Damien Teney André Freitas 41 8 0 21 May 2023
An Empirical Study on the Language Modal in Visual Question Answering Daowan Peng Wei Wei Xian-Ling Mao Yuanyuan Fu Dangyang Chen 42 4 0 17 May 2023
HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models Kenta Takemoto Moyuru Yamada Tomotake Sasaki H. Akima 37 0 0 17 May 2023
Distribution-aware Fairness Test Generation Sai Sathiesh Rajan E. Soremekun Yves Le Traon Sudipta Chattopadhyay 37 0 0 08 May 2023
Adaptive loose optimization for robust question answering Jie Ma Pinghui Wang Ze-you Wang Dechen Kong Min Hu Tingxu Han Jun Liu OOD 38 4 0 06 May 2023
Divide and Conquer: Answering Questions with Object Factorization and Compositional Reasoning Shi Chen Qi Zhao 47 5 0 18 Mar 2023
Accounting for multiplicity in machine learning benchmark performance Kajsa Møllersen Einar J. Holsbø 11 2 0 10 Mar 2023
Meta Input: How to Leverage Off-the-Shelf Deep Neural Networks Minsu Kim Youngjoon Yu Sungjune Park Y. Ro OOD 18 0 0 21 Oct 2022
Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA Q. Si Fandong Meng Mingyu Zheng Zheng Lin Yuanxin Liu Peng Fu Yanan Cao Weiping Wang Jie Zhou 32 20 0 10 Oct 2022
Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning Q. Si Yuanxin Liu Fandong Meng Zheng Lin Peng Fu Yanan Cao Weiping Wang Jie Zhou 37 23 0 10 Oct 2022
Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering Jingjing Jiang Zi-yi Liu Nanning Zheng 26 8 0 14 Sep 2022
ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets Damien Teney Yong Lin Seong Joon Oh Ehsan Abbasnejad OOD 391 47 0 01 Sep 2022
Generative Bias for Robust Visual Question Answering Jae-Won Cho Dong-Jin Kim H. Ryu In So Kweon OOD CML 33 19 0 01 Aug 2022
Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem Yudong Han Liqiang Nie Jianhua Yin Jianlong Wu Yan Yan 26 13 0 24 Jul 2022
Rethinking Data Augmentation for Robust Visual Question Answering Long Chen Yuhang Zheng Jun Xiao OOD 35 42 0 18 Jul 2022
Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning Damien Teney Maxime Peyrard Ehsan Abbasnejad 38 29 0 06 Jul 2022
EBMs vs. CL: Exploring Self-Supervised Visual Pretraining for Visual Question Answering Violetta Shevchenko Ehsan Abbasnejad A. Dick Anton Van Den Hengel Damien Teney 49 0 0 29 Jun 2022
VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives Zhuofan Ying Peter Hase Joey Tianyi Zhou LRM 33 13 0 22 Jun 2022
Methods for Estimating and Improving Robustness of Language Models Michal Stefánik 13 1 0 16 Jun 2022
Guiding Visual Question Answering with Attention Priors T. Le Vuong Le Sunil R. Gupta Svetha Venkatesh T. Tran 27 6 0 25 May 2022
Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization Aishwarya Agrawal Ivana Kajić Emanuele Bugliarello Elnaz Davoodi Anita Gergely Phil Blunsom Aida Nematzadeh OOD 40 17 0 24 May 2022
Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective Edoardo Manino Julia Rozanova Danilo S. Carvalho André Freitas Lucas C. Cordeiro 30 7 0 26 Apr 2022
The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning Jessica Hullman Sayash Kapoor Priyanka Nanayakkara Andrew Gelman Arvind Narayanan 33 39 0 12 Mar 2022
General Greedy De-bias Learning Xinzhe Han Shuhui Wang Chi Su Qingming Huang Qi Tian 11 7 0 20 Dec 2021
Weakly-Supervised Video Object Grounding via Causal Intervention Wei Wang Junyu Gao Changsheng Xu CML 30 20 0 01 Dec 2021
Understanding and Testing Generalization of Deep Networks on Out-of-Distribution Data Rui Hu Jitao Sang Jinqiang Wang Rui Hu Chaoquan Jiang CML OOD 27 7 0 17 Nov 2021
Language bias in Visual Question Answering: A Survey and Taxonomy Desen Yuan 30 12 0 16 Nov 2021
Introspective Distillation for Robust Question Answering Yulei Niu Hanwang Zhang 27 59 0 01 Nov 2021
Perceptual Score: What Data Modalities Does Your Model Perceive? Itai Gat Idan Schwartz A. Schwing 33 30 0 27 Oct 2021
Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering Long Chen Yuhang Zheng Yulei Niu Hanwang Zhang Jun Xiao AAML OOD 21 36 0 03 Oct 2021