{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/125801"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/125801","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Shifting paradigms in the UX evaluation of human-AI interaction: From dyadic to monadic designs","abstract":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2026-08-01","abstract_html":"Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2026-08-01","abstract_has_math":false,"creators":["Zheng, Qingxiao"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Information Sciences","degree_department":null,"school":null,"contributors":["Huang, Yun","Yao, Mike","Wang, Yang","Bosch, Nigel"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2024,"date_issued":"2024-07-10","date_published":"2024-07-10","updated_at":"2026-07-22T22:25:02Z","subjects":["Human--ai Interaction","Evaluation","Generative Ai","User Experience"],"languages":["en","eng"],"rights":["Copyright 2024 Qingxiao Zheng"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/125801","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Huang, Yun","Yao, Mike","Wang, Yang","Bosch, Nigel"]},{"key":"dc:creator","label":"Author","values":["Zheng, Qingxiao"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2024-07-10","2024-08"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Information Sciences"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Human--ai Interaction","Evaluation","Generative Ai","User Experience"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2024 Qingxiao Zheng"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/125801"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2026-08-01","The student, Qingxiao Zheng, accepted the attached license on 2024-07-09 at 09:03.","The student, Qingxiao Zheng, submitted this Dissertation for approval on 2024-07-09 at 09:20.","This Dissertation was approved for publication on 2024-07-10 at 16:23.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20999 on 2025-02-04 at 21:25:42","With generative AI, we are entering a new era where individuals without coding backgrounds can become creators of AI-based solutions for delivering different services. For example, business owners can define AI chatbots to answer customers’ inquiries, and legal professionals can tailor AI to draft, review, and manage legal documentation. However, there is a lack of a systematic framework to evaluate the UX of human--AI collaboration in these interactions. This thesis identifies a paradigm shift in the UX evaluation of human--AI interaction. Prior to generative AI, UX research began with examining dyadic interactions between end users and AI, progressively expanding to polyadic interactions, where AI mediates between end users adopting multi-stakeholder perspectives (e.g., service providers and service consumers). The evaluation of these interaction modalities often considers user satisfaction, linguistic features, and task completion rate, and the challenges that mainly arise as a result of the uncertainty and complexity of AI. With the advent of generative AI, individuals with minimal AI literacy can become creators, introducing a new modality---monadic interaction---which emphasizes the unity and feedback loop between AI systems and their Creator-Users. These creator-users actively participate in defining and refining the AI's functions, allowing both to adapt and evolve. Although humans and AI systems can leverage their respective strengths to achieve better outcomes than either could independently, assessing how well AI aligns with creators’ values and intentions poses significant challenges, particularly when creators' norms deviate from broader societal standards. This dissertation further introduces a UX evaluation framework called EvalignUX (evaluating alignment of UX), designed to guide the evaluation of the three interaction modalities while maintaining the alignment focus. First, I synthesized the literature on UX research in conversational human-AI interaction before the popularity of generative AI and proposed four dimensions for evaluating dyadic and polyadic interactions: linguistics and prosodics quality, task performance, socio-emotional intelligence, and ethics and risks. Each dimension includes several evaluation metrics, such as word variation, sense of control, perceived warmth and competence, and security perception. These evaluation metrics can serve as a design material to enhance UX effectively. I then extended the EvalignUX framework to include monadic interactions through two case studies. The first case demonstrates the creator-AI interaction within a multistakeholder service, where public service providers customize their AI to address community members’ queries. New evaluation metrics such as \"pronominal comprehension\" and \"identity-based emotive expression\" prove crucial when AI is designed to play the role of human creators as service providers. The second case illustrates the creator-AI interaction for self-defined services, where creators possess diverse domain expertise and require AI to provide customized documentation for individual use. In this context, a new evaluation dimension, \"normative fluidity,\" is particularly significant in assessing AI's ability to adapt to varying normative expectations. The term \"fluidity\" underscores the smoothness and ease of transition, when creators’ task definitions differ from the existing AI models. The proposed EvalignUX framework can assist UX researchers and design tool makers in addressing the challenges of evaluating AI systems in a responsive fashion, from dyadic to monadic interactions. The thesis concludes with future research towards more aligned AI with boundary awareness."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Shifting paradigms in the UX evaluation of human-AI interaction: From dyadic to monadic designs"]}]}],"canonical_facts":{"dc:contributor":["Huang, Yun","Yao, Mike","Wang, Yang","Bosch, Nigel"],"dc:creator":["Zheng, Qingxiao"],"dc:date":["2024-07-10","2024-08"],"dc:description":["Submission published under a 24 month embargo labeled 'Closed Access', the embargo will last until 2026-08-01","The student, Qingxiao Zheng, accepted the attached license on 2024-07-09 at 09:03.","The student, Qingxiao Zheng, submitted this Dissertation for approval on 2024-07-09 at 09:20.","This Dissertation was approved for publication on 2024-07-10 at 16:23.","DSpace SAF Submission Ingestion Package generated from Vireo submission #20999 on 2025-02-04 at 21:25:42","With generative AI, we are entering a new era where individuals without coding backgrounds can become creators of AI-based solutions for delivering different services. For example, business owners can define AI chatbots to answer customers’ inquiries, and legal professionals can tailor AI to draft, review, and manage legal documentation. However, there is a lack of a systematic framework to evaluate the UX of human--AI collaboration in these interactions. This thesis identifies a paradigm shift in the UX evaluation of human--AI interaction. Prior to generative AI, UX research began with examining dyadic interactions between end users and AI, progressively expanding to polyadic interactions, where AI mediates between end users adopting multi-stakeholder perspectives (e.g., service providers and service consumers). The evaluation of these interaction modalities often considers user satisfaction, linguistic features, and task completion rate, and the challenges that mainly arise as a result of the uncertainty and complexity of AI. With the advent of generative AI, individuals with minimal AI literacy can become creators, introducing a new modality---monadic interaction---which emphasizes the unity and feedback loop between AI systems and their Creator-Users. These creator-users actively participate in defining and refining the AI's functions, allowing both to adapt and evolve. Although humans and AI systems can leverage their respective strengths to achieve better outcomes than either could independently, assessing how well AI aligns with creators’ values and intentions poses significant challenges, particularly when creators' norms deviate from broader societal standards. This dissertation further introduces a UX evaluation framework called EvalignUX (evaluating alignment of UX), designed to guide the evaluation of the three interaction modalities while maintaining the alignment focus. First, I synthesized the literature on UX research in conversational human-AI interaction before the popularity of generative AI and proposed four dimensions for evaluating dyadic and polyadic interactions: linguistics and prosodics quality, task performance, socio-emotional intelligence, and ethics and risks. Each dimension includes several evaluation metrics, such as word variation, sense of control, perceived warmth and competence, and security perception. These evaluation metrics can serve as a design material to enhance UX effectively. I then extended the EvalignUX framework to include monadic interactions through two case studies. The first case demonstrates the creator-AI interaction within a multistakeholder service, where public service providers customize their AI to address community members’ queries. New evaluation metrics such as \"pronominal comprehension\" and \"identity-based emotive expression\" prove crucial when AI is designed to play the role of human creators as service providers. The second case illustrates the creator-AI interaction for self-defined services, where creators possess diverse domain expertise and require AI to provide customized documentation for individual use. In this context, a new evaluation dimension, \"normative fluidity,\" is particularly significant in assessing AI's ability to adapt to varying normative expectations. The term \"fluidity\" underscores the smoothness and ease of transition, when creators’ task definitions differ from the existing AI models. The proposed EvalignUX framework can assist UX researchers and design tool makers in addressing the challenges of evaluating AI systems in a responsive fashion, from dyadic to monadic interactions. The thesis concludes with future research towards more aligned AI with boundary awareness."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/125801"],"dc:language":["en","eng"],"dc:rights":["Copyright 2024 Qingxiao Zheng"],"dc:subject":["Human--ai Interaction","Evaluation","Generative Ai","User Experience"],"dc:title":["Shifting paradigms in the UX evaluation of human-AI interaction: From dyadic to monadic designs"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Information Sciences"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:02Z"}