A Novel Approach on Visual Question Answering by Parameter Prediction using Faster Region Based Convolutional Neural Network

TitleA Novel Approach on Visual Question Answering by Parameter Prediction using Faster Region Based Convolutional Neural Network
Publication TypeJournal Article
Year of PublicationIn Press
AuthorsJha, S., A. Dey, R. Kumar, and V. Kumar-Solanki
JournalInternational Journal of Interactive Multimedia and Artificial Intelligence
ISSN1989-1660
IssueIn Press
VolumeIn Press
NumberIn Press
Date Published08/2018
Pagination1-8
Abstract

Visual Question Answering (VQA) is a stimulating process in the field of Natural Language Processing (NLP) and Computer Vision (CV). In this process machine can find an answer to a natural language question which is related to an image. Question can be open-ended or multiple choice. Datasets of VQA contain mainly three components; questions, images and answers. Researchers overcome the VQA problem with deep learning based architecture that jointly combines both of two networks i.e. Convolution Neural Network (CNN) for visual (image) representation and Recurrent Neural Network (RNN) with Long Short Time Memory (LSTM) for textual (question) representation and trained the combined network end to end to generate the answer. Those models are able to answer the common and simple questions that are directly related to the image’s content. But different types of questions need different level of understanding to produce correct answers. To solve this problem, we use faster Region based-CNN (R-CNN) for extracting image features with an extra fully connected layer whose weights are dynamically obtained by LSTMs cell according to the question. We claim in this paper that a single R-CNN architecture can solve the problems related to VQA by modifying weights in the parameter prediction layer. Authors trained the network end to end by Stochastic Gradient Descent (SGD) using pretrained faster R-CNN and LSTM and tested it on benchmark datasets of VQA.

KeywordsComputer vision, Long Short Time Memory, Natural Language Processing, Neural Network, Stochastic Gradient Descent.
DOI10.9781/ijimai.2018.08.004
URLhttp://www.ijimai.org/journal/sites/default/files/files/2018/08/ip_08_04_pdf_11336.pdf
AttachmentSize
ip_08_04.pdf1.8 MB