{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/104918"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/104918","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Visual question answering using external knowledge","abstract":"Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction, a novel `fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, keyword matching techniques have been employed to successively reduce the large set of facts and were shown to yield compelling results despite being vulnerable to misconceptions due to synonyms and homographs. To overcome these shortcomings, we introduce two new approaches in this work. We develop a learning-based approach which goes straight to the facts via a learned embedding space. We demonstrate state-of-the-art results on the challenging recently introduced factbased visual question answering dataset, outperforming competing methods by more than 5%. Upon further analysis, we observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. To counter this, in our second approach we develop an entity graph and use a graph convolutional network to `reason' about the correct answer by jointly considering all entities. We show on the FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state-of-the-art.","abstract_html":"Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction, a novel `fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, keyword matching techniques have been employed to successively reduce the large set of facts and were shown to yield compelling results despite being vulnerable to misconceptions due to synonyms and homographs. To overcome these shortcomings, we introduce two new approaches in this work. We develop a learning-based approach which goes straight to the facts via a learned embedding space. We demonstrate state-of-the-art results on the challenging recently introduced factbased visual question answering dataset, outperforming competing methods by more than 5%. Upon further analysis, we observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. To counter this, in our second approach we develop an entity graph and use a graph convolutional network to `reason' about the correct answer by jointly considering all entities. We show on the FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state-of-the-art.","abstract_has_math":false,"creators":["Gulganjalli Narasimhan, Medhini"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Schwing, Alexander G.","Lazebnik, Svetlana"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2019,"date_issued":"2019-08-23T20:02:06Z","date_published":"2019-08-23T20:02:06Z","updated_at":"2026-07-22T22:24:42Z","subjects":["Visual question answering, knowledge bases, graph convolution networks"],"languages":["en"],"rights":["Copyright 2019 Medhini Gulganjalli Narasimhan"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"http://hdl.handle.net/2142/104918","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Schwing, Alexander G.","Lazebnik, Svetlana"]},{"key":"dc:creator","label":"Author","values":["Gulganjalli Narasimhan, Medhini"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2019-08-23T20:02:06Z","2019-04-25","2019-05"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Visual question answering, knowledge bases, graph convolution networks"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2019 Medhini Gulganjalli Narasimhan"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["http://hdl.handle.net/2142/104918"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction, a novel `fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, keyword matching techniques have been employed to successively reduce the large set of facts and were shown to yield compelling results despite being vulnerable to misconceptions due to synonyms and homographs. To overcome these shortcomings, we introduce two new approaches in this work. We develop a learning-based approach which goes straight to the facts via a learned embedding space. We demonstrate state-of-the-art results on the challenging recently introduced factbased visual question answering dataset, outperforming competing methods by more than 5%. Upon further analysis, we observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. To counter this, in our second approach we develop an entity graph and use a graph convolutional network to `reason' about the correct answer by jointly considering all entities. We show on the FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state-of-the-art.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2019-08-22 without embargo terms","The student, Medhini Gulganjalli Narasimhan, accepted the attached license on 2019-04-23 at 23:16.","The student, Medhini Gulganjalli Narasimhan, submitted this Thesis for approval on 2019-04-23 at 23:28.","This Thesis was approved for publication on 2019-04-25 at 11:06.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13860 on 2019-08-22 at 14:46:23","Made available in DSpace on 2019-08-23T20:02:06Z (GMT). No. of bitstreams: 2 GULGANJALLINARASIMHAN-THESIS-2019.pdf: 13445615 bytes, checksum: 40781f42299896e7f5c97b8e0fa4fcc1 (MD5) LICENSE.txt: 4227 bytes, checksum: daaa7f5cee44a45b2582190aff7b782e (MD5) Previous issue date: 2019-04-25"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Visual question answering using external knowledge"]}]}],"canonical_facts":{"dc:contributor":["Schwing, Alexander G.","Lazebnik, Svetlana"],"dc:creator":["Gulganjalli Narasimhan, Medhini"],"dc:date":["2019-08-23T20:02:06Z","2019-04-25","2019-05"],"dc:description":["Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction, a novel `fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, keyword matching techniques have been employed to successively reduce the large set of facts and were shown to yield compelling results despite being vulnerable to misconceptions due to synonyms and homographs. To overcome these shortcomings, we introduce two new approaches in this work. We develop a learning-based approach which goes straight to the facts via a learned embedding space. We demonstrate state-of-the-art results on the challenging recently introduced factbased visual question answering dataset, outperforming competing methods by more than 5%. Upon further analysis, we observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. To counter this, in our second approach we develop an entity graph and use a graph convolutional network to `reason' about the correct answer by jointly considering all entities. We show on the FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state-of-the-art.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2019-08-22 without embargo terms","The student, Medhini Gulganjalli Narasimhan, accepted the attached license on 2019-04-23 at 23:16.","The student, Medhini Gulganjalli Narasimhan, submitted this Thesis for approval on 2019-04-23 at 23:28.","This Thesis was approved for publication on 2019-04-25 at 11:06.","DSpace SAF Submission Ingestion Package generated from Vireo submission #13860 on 2019-08-22 at 14:46:23","Made available in DSpace on 2019-08-23T20:02:06Z (GMT). No. of bitstreams: 2 GULGANJALLINARASIMHAN-THESIS-2019.pdf: 13445615 bytes, checksum: 40781f42299896e7f5c97b8e0fa4fcc1 (MD5) LICENSE.txt: 4227 bytes, checksum: daaa7f5cee44a45b2582190aff7b782e (MD5) Previous issue date: 2019-04-25"],"dc:format":["application/pdf"],"dc:identifier":["http://hdl.handle.net/2142/104918"],"dc:language":["en"],"dc:rights":["Copyright 2019 Medhini Gulganjalli Narasimhan"],"dc:subject":["Visual question answering, knowledge bases, graph convolution networks"],"dc:title":["Visual question answering using external knowledge"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:42Z"}