Mobile design semantics

Given the growing number of mobile apps and their increasing impact on modern life, researchers have developed black-box approaches to mine mobile app design and interaction data. Although the data captured during interaction mining is descriptive, it does not expose the design semantics of UIs: what elements on the screen mean and how they are used. This thesis introduces an automatic approach for semantically annotating the elements comprising a UI given the data captured during interaction mining. Through an iterative open coding of 73k UI elements and 720 screens, we first created a lexical database of 24 types of UI components, 197 text button concepts, and 135 icon classes shared across apps. Using the labeled data created during this process, we learned code-based patterns to detect components, and trained a convolutional neural network which distinguishes between 99 icon classes with 94% accuracy. With this automated approach, we computed semantic annotations for the 72k unique UIs comprising the Rico dataset, assigning labels for 78% of the total visible, non-redundant elements.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2018

Author dc:creator

• Liu, Thomas F

Contributors dc:contributor

• Kumar, Ranjitha

• design, mobile

Statement dc:rights

• Copyright 2018 Thomas Liu

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/101246

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/101246