{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/124410"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/124410","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Exploring screen summarization with large language and multimodal models","abstract":"Mobile UIs are inherently multimodal. They can be represented by both visual representations, (screenshots) and structural metadata (view hierarchies). The image modality is rich and can contain information such as images, colors and positional information, while the view-hierarchies represent a tree of UI Elements, with data such as element types, element text and bounding boxes. Recently, pre-trained large language models (LLMs) and large multimodal models (LMMs) have been shown to have strong performance on various downstream tasks such as sentiment analysis and text classification. Further, LLMs have been shown to have a competitive performance when applied to screen summarization tasks with effective prompting strategies. First, we seek to understand whether the LLMs benefit from the hierarchical representation of a view-hierarchy, and explore different heuristics to represent the view-heirarchy in text, and observe whether this can affect performance in screen summarization. We study the impact of in-context learning, and note a visible gain in performance. We also explore the use of LMMs to effectively use the multimodal nature of mobile UIs. To this end, we experiment with the performance of state-of-the-art LMMs, LLaVA and GPT-4V when provided with both screenshots as well as view hierarchies. We investigate whether LMMs are able to achieve good summarization capability with a single modality, or if multimodality can help create more complete summaries. Finally, we conduct a deep analysis to study the characteristics of LLM and LMM generated screen summaries. We show that our multimodal approach not only yields competitive performance but also offers generalizability and user-friendly application. We achieve a CLAIR score of 0.79, and show qualitative examples to highlight that both modalities hold information that are useful to create informative and complete screen summaries. We are able to achieve this performance without requiring the creation of dedicated datasets and expensive pre-training or fine-tuning. This study contributes to the broader understanding of how mobile UIs can be represented textually and how their multimodal nature can be effectively leveraged in practical applications.","abstract_html":"Mobile UIs are inherently multimodal. They can be represented by both visual representations, (screenshots) and structural metadata (view hierarchies). The image modality is rich and can contain information such as images, colors and positional information, while the view-hierarchies represent a tree of UI Elements, with data such as element types, element text and bounding boxes. Recently, pre-trained large language models (LLMs) and large multimodal models (LMMs) have been shown to have strong performance on various downstream tasks such as sentiment analysis and text classification. Further, LLMs have been shown to have a competitive performance when applied to screen summarization tasks with effective prompting strategies. First, we seek to understand whether the LLMs benefit from the hierarchical representation of a view-hierarchy, and explore different heuristics to represent the view-heirarchy in text, and observe whether this can affect performance in screen summarization. We study the impact of in-context learning, and note a visible gain in performance. We also explore the use of LMMs to effectively use the multimodal nature of mobile UIs. To this end, we experiment with the performance of state-of-the-art LMMs, LLaVA and GPT-4V when provided with both screenshots as well as view hierarchies. We investigate whether LMMs are able to achieve good summarization capability with a single modality, or if multimodality can help create more complete summaries. Finally, we conduct a deep analysis to study the characteristics of LLM and LMM generated screen summaries. We show that our multimodal approach not only yields competitive performance but also offers generalizability and user-friendly application. We achieve a CLAIR score of 0.79, and show qualitative examples to highlight that both modalities hold information that are useful to create informative and complete screen summaries. We are able to achieve this performance without requiring the creation of dedicated datasets and expensive pre-training or fine-tuning. This study contributes to the broader understanding of how mobile UIs can be represented textually and how their multimodal nature can be effectively leveraged in practical applications.","abstract_has_math":false,"creators":["Adrakatti, Vivek"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"M.S.","degree_level":"Thesis","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Kumar, Ranjitha"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-04-29","date_published":"2024-04-29","updated_at":"2026-07-22T22:25:00Z","subjects":["Screen Understanding","Mobile Ui Understanding","Screen Summarization","Mobile Ui Summarization","Artificial Intelligence","Machine Learning","Human-computer Interaction","Large Language Models","Vision Language Models","Large Multimodal Models"],"languages":["eng","en"],"rights":["Copyright 2024 Vivek Adrakatti"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/124410","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Kumar, Ranjitha"]},{"key":"dc:creator","label":"Author","values":["Adrakatti, Vivek"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-04-29","2024-05"]},{"key":"dc:type","label":"Dc Type","values":["Text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Thesis"]},{"key":"thesis:degree_name","label":"Degree Name","values":["M.S."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Screen Understanding","Mobile Ui Understanding","Screen Summarization","Mobile Ui Summarization","Artificial Intelligence","Machine Learning","Human-computer Interaction","Large Language Models","Vision Language Models","Large Multimodal Models"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["eng","en"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Vivek Adrakatti"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/124410"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Mobile UIs are inherently multimodal. They can be represented by both visual representations, (screenshots) and structural metadata (view hierarchies). The image modality is rich and can contain information such as images, colors and positional information, while the view-hierarchies represent a tree of UI Elements, with data such as element types, element text and bounding boxes. Recently, pre-trained large language models (LLMs) and large multimodal models (LMMs) have been shown to have strong performance on various downstream tasks such as sentiment analysis and text classification. Further, LLMs have been shown to have a competitive performance when applied to screen summarization tasks with effective prompting strategies. First, we seek to understand whether the LLMs benefit from the hierarchical representation of a view-hierarchy, and explore different heuristics to represent the view-heirarchy in text, and observe whether this can affect performance in screen summarization. We study the impact of in-context learning, and note a visible gain in performance. We also explore the use of LMMs to effectively use the multimodal nature of mobile UIs. To this end, we experiment with the performance of state-of-the-art LMMs, LLaVA and GPT-4V when provided with both screenshots as well as view hierarchies. We investigate whether LMMs are able to achieve good summarization capability with a single modality, or if multimodality can help create more complete summaries. Finally, we conduct a deep analysis to study the characteristics of LLM and LMM generated screen summaries. We show that our multimodal approach not only yields competitive performance but also offers generalizability and user-friendly application. We achieve a CLAIR score of 0.79, and show qualitative examples to highlight that both modalities hold information that are useful to create informative and complete screen summaries. We are able to achieve this performance without requiring the creation of dedicated datasets and expensive pre-training or fine-tuning. This study contributes to the broader understanding of how mobile UIs can be represented textually and how their multimodal nature can be effectively leveraged in practical applications.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Vivek Adrakatti, accepted the attached license on 2024-04-25 at 15:37.","The student, Vivek Adrakatti, submitted this Thesis for approval on 2024-04-25 at 16:03.","This Thesis was approved for publication on 2024-04-29 at 16:22.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20648 on 2024-09-16 at 00:37:04"]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Exploring screen summarization with large language and multimodal models"]}]}],"canonical_facts":{"dc:contributor":["Kumar, Ranjitha"],"dc:creator":["Adrakatti, Vivek"],"dc:date":["2024-04-29","2024-05"],"dc:description":["Mobile UIs are inherently multimodal. They can be represented by both visual representations, (screenshots) and structural metadata (view hierarchies). The image modality is rich and can contain information such as images, colors and positional information, while the view-hierarchies represent a tree of UI Elements, with data such as element types, element text and bounding boxes. Recently, pre-trained large language models (LLMs) and large multimodal models (LMMs) have been shown to have strong performance on various downstream tasks such as sentiment analysis and text classification. Further, LLMs have been shown to have a competitive performance when applied to screen summarization tasks with effective prompting strategies. First, we seek to understand whether the LLMs benefit from the hierarchical representation of a view-hierarchy, and explore different heuristics to represent the view-heirarchy in text, and observe whether this can affect performance in screen summarization. We study the impact of in-context learning, and note a visible gain in performance. We also explore the use of LMMs to effectively use the multimodal nature of mobile UIs. To this end, we experiment with the performance of state-of-the-art LMMs, LLaVA and GPT-4V when provided with both screenshots as well as view hierarchies. We investigate whether LMMs are able to achieve good summarization capability with a single modality, or if multimodality can help create more complete summaries. Finally, we conduct a deep analysis to study the characteristics of LLM and LMM generated screen summaries. We show that our multimodal approach not only yields competitive performance but also offers generalizability and user-friendly application. We achieve a CLAIR score of 0.79, and show qualitative examples to highlight that both modalities hold information that are useful to create informative and complete screen summaries. We are able to achieve this performance without requiring the creation of dedicated datasets and expensive pre-training or fine-tuning. This study contributes to the broader understanding of how mobile UIs can be represented textually and how their multimodal nature can be effectively leveraged in practical applications.","Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2024-09-16 without embargo terms","The student, Vivek Adrakatti, accepted the attached license on 2024-04-25 at 15:37.","The student, Vivek Adrakatti, submitted this Thesis for approval on 2024-04-25 at 16:03.","This Thesis was approved for publication on 2024-04-29 at 16:22.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20648 on 2024-09-16 at 00:37:04"],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/124410"],"dc:language":["eng","en"],"dc:rights":["Copyright 2024 Vivek Adrakatti"],"dc:subject":["Screen Understanding","Mobile Ui Understanding","Screen Summarization","Mobile Ui Summarization","Artificial Intelligence","Machine Learning","Human-computer Interaction","Large Language Models","Vision Language Models","Large Multimodal Models"],"dc:title":["Exploring screen summarization with large language and multimodal models"],"dc:type":["Text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Thesis"],"thesis:degree_name":["M.S."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:00Z"}