Connecting Text and Charts Using Large Vision-Language Models

Data visualizations are essential for presenting complex dataset, but the disconnect between charts and accompanying textual descriptions often leads to misinterpretation and increased cognitive effort—especially for users with limited data literacy. While prior methods attempt to bridge this gap, many depend on manual annotations or fixed chart structures, limiting scalability across diverse documents. In this thesis, we propose two large vision-language model (LVLM)-based frameworks—a single-agent baseline and a multi-agent architecture—for automatically linking textual descriptions with their corresponding chart data. Both frameworks extract structured data from chart images and use lexical, syntactic, and arithmetic reasoning to perform sentence-to-data alignment. We evaluate the performance of these frameworks on a curated dataset of Pew Research charts. Finally, we develop a browser extension that integrates this approach into Pew Research articles, enabling interactive text–chart linking for enhanced reading experiences.

Author dc:creator

• Chowdhury, Nafis Tahmid

Advisor dc:contributor.advisor

• Prince, Enamul Hoque

• Computer science

Statement dc:rights

• Author owns copyright, except where explicitly noted. Please contact the author directly with licensing requests.

Language dc:language

en

Handle dc:identifier.uri

https://hdl.handle.net/10315/43845

OAI record oai:identifier

https://yorkspace.library.yorku.ca/oai/request?verb=GetRecord&metadataPrefix=dim&identifier=oai%3Ayorkspace.library.yorku.ca%3A10315%2F43845