Revisiting Visual Question Answering Baselines

Visual question answering (VQA) is an interesting learning setting for evaluating the abilities and shortcomings of current systems for image understanding. Many of the recently proposed VQA systems include attention or memory mechanisms designed to perform "reasoning". Furthermore, for the task of multiple-choice VQA, nearly all of these systems train a multi-class classifier on image and question features to predict the answers. This paper questions the value of these common practices and develops a simple alternative model based on binary classification. Instead of treating answers as competing choices, our model receives the answer as input and predicts whether or not an image-question-answer triplet is correct. We evaluate our model on the Visual7W Telling and the VQA Real Multiple Choice tasks, and find that even simple versions of our model perform competitively. Our best model achieves state-of-the-art performance of 65.8% on the Visual7W Telling task and competes surprisingly well with the most complex systems proposed for the VQA Real Multiple Choice task. Additionally, we explore variants of our model and study the transferability of our model between both datasets. We also present an error analysis of our best model, the results of which suggests that a key problem of current VQA systems lies in the lack of visual grounding and localization of concepts that occur in the questions and answers.