Exploring screen summarization with large language and multimodal models

Mobile UIs are inherently multimodal. They can be represented by both visual representations, (screenshots) and structural metadata (view hierarchies). The image modality is rich and can contain information such as images, colors and positional information, while the view-hierarchies represent a tree of UI Elements, with data such as element types, element text and bounding boxes. Recently, pre-trained large language models (LLMs) and large multimodal models (LMMs) have been shown to have strong performance on various downstream tasks such as sentiment analysis and text classification. Further, LLMs have been shown to have a competitive performance when applied to screen summarization tasks with effective prompting strategies. First, we seek to understand whether the LLMs benefit from the hierarchical representation of a view-hierarchy, and explore different heuristics to represent the view-heirarchy in text, and observe whether this can affect performance in screen summarization. We study the impact of in-context learning, and note a visible gain in performance. We also explore the use of LMMs to effectively use the multimodal nature of mobile UIs. To this end, we experiment with the performance of state-of-the-art LMMs, LLaVA and GPT-4V when provided with both screenshots as well as view hierarchies. We investigate whether LMMs are able to achieve good summarization capability with a single modality, or if multimodality can help create more complete summaries. Finally, we conduct a deep analysis to study the characteristics of LLM and LMM generated screen summaries. We show that our multimodal approach not only yields competitive performance but also offers generalizability and user-friendly application. We achieve a CLAIR score of 0.79, and show qualitative examples to highlight that both modalities hold information that are useful to create informative and complete screen summaries. We are able to achieve this performance without requiring the creation of dedicated datasets and expensive pre-training or fine-tuning. This study contributes to the broader understanding of how mobile UIs can be represented textually and how their multimodal nature can be effectively leveraged in practical applications.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Adrakatti, Vivek

Contributors dc:contributor

• Kumar, Ranjitha

• Screen Understanding
• Mobile Ui Understanding
• Screen Summarization
• Mobile Ui Summarization
• Artificial Intelligence
• Machine Learning
• Human-computer Interaction
• Large Language Models
• Vision Language Models
• Large Multimodal Models

Statement dc:rights

• Copyright 2024 Vivek Adrakatti

Language dc:language

eng, en

Handle dc:identifier

https://hdl.handle.net/2142/124410

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/124410