{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/130030"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/130030","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"A data-augmented approach to evaluating teamwork using log data from digital collaboration tools","abstract":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-08-01","abstract_html":"Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-08-01","abstract_has_math":false,"creators":["Shi, Wenxuan"],"institution":"University of Illinois Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Computer Science","degree_department":null,"school":null,"contributors":["Bailey, Brian","Sundaram, Hari","Mercier, Emma","Sterman, Sarah","Wang, Dakuo"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2025,"date_issued":"2025-07-10","date_published":"2025-07-10","updated_at":"2026-07-22T22:25:06Z","subjects":["Human-computer Interaction","Collaboration","Teamwork","Team Assessment","Peer Evaluations","Log Data","Data-driven Assessment","Interface Design"],"languages":["en","eng"],"rights":["Copyright 2025 Wenxuan Shi"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/130030","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Bailey, Brian","Sundaram, Hari","Mercier, Emma","Sterman, Sarah","Wang, Dakuo"]},{"key":"dc:creator","label":"Author","values":["Shi, Wenxuan"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2025-07-10","2025-08"]},{"key":"dc:type","label":"Dc Type","values":["text"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Computer Science"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Human-computer Interaction","Collaboration","Teamwork","Team Assessment","Peer Evaluations","Log Data","Data-driven Assessment","Interface Design"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["Copyright 2025 Wenxuan Shi"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/130030"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-08-01","The student, Wenxuan Shi, accepted the attached license on 2025-07-10 at 08:32.","The student, Wenxuan Shi, submitted this Dissertation for approval on 2025-07-10 at 08:48.","This Dissertation was approved for publication on 2025-07-10 at 12:08.","DSpace SAF Submission Ingestion Package generated from Vireo submission #22473 on 2025-10-21 at 10:05:45","Teamwork is a critical skill in both educational and professional settings, yet evaluating individual contributions to collaborative work remains a persistent challenge. Instructors often rely on peer evaluations to assess teamwork, but these evaluations are prone to social and cognitive biases, including memory limitations, inconsistent standards, and interpersonal pressures. At the same time, the increasing use of digital collaboration tools such as Google Docs, GitHub, and Slack presents a new opportunity: these tools generate rich log data that can reveal how teams interact, coordinate, and contribute over time. This dissertation presents the design and evaluation of a data-augmented peer evaluation system that extracts, analyzes, and visualizes log data from digital collaboration tools to support students' evaluations of teamwork. By reviewing the visualization prior to rating their teammates, students can ground their evaluations in concrete evidence of each team member's behaviors. My approach centers students as active participants in interpreting and annotating their own data, thus also contributing knowledge of how students understand and value contributions within teams. This system is the culmination of a multi-stage investigation into how peer evaluations of teamwork can be augmented with log data from digital collaboration tools. I begin with a mixed-methods study examining current and potential uses of log data to support students' and instructors' evaluation practices. Through interviews and surveys, I find that while both groups already refer to log data to validate or challenge peer evaluations, this usage is often informal, retrospective, and shaped by uncertainty about how to interpret the data. Next, I conduct a between-subjects experiment to test whether reviewing log data during the peer evaluation process improves evaluation quality. The results show that log data increased consistency and perceived accuracy of the evaluations, motivating the design of a system that incorporates log data into the peer evaluation process. Grounded in the empirical findings of these two studies, I design and implement the data-augmented peer evaluation system through an iterative user-centered design process. The system captures multiple components of the teamwork: content, technical, and communication contributions in Google Drive, GitHub, and Slack. In the interface, the student reflects on an interactive timeline visualization of the team’s log data prior to evaluating their teammates, allowing them to ground their evaluations in the data. The student can also contextualize the data by marking contributions as valuable and tagging behaviors displayed by their teammates. I evaluate this system in a semester-long field study in a user interface design course. Results show that the data-augmented peer evaluation system helped students recall specific contributions, reflect on work patterns, and justify their ratings. At the same time, students expressed concerns about the representativeness of the data, with some even demonstrating resistance towards the system. Together, these contributions demonstrate the value of integrating log data into peer evaluations and advance knowledge of how data can be used not just to monitor teams, but to support fair and meaningful assessments of teamwork that can guide learning. This work also contributes design principles for creating data-driven evaluation systems that balance fairness, student agency, and meaningful reflection. This dissertation progresses toward a future in which every person working in a team can compose and receive meaningful feedback that helps them to feel valued for their contributions and learn effective teamwork behaviors."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["A data-augmented approach to evaluating teamwork using log data from digital collaboration tools"]}]}],"canonical_facts":{"dc:contributor":["Bailey, Brian","Sundaram, Hari","Mercier, Emma","Sterman, Sarah","Wang, Dakuo"],"dc:creator":["Shi, Wenxuan"],"dc:date":["2025-07-10","2025-08"],"dc:description":["Submission published under a 24 month embargo labeled 'U of I Access', the embargo will last until 2027-08-01","The student, Wenxuan Shi, accepted the attached license on 2025-07-10 at 08:32.","The student, Wenxuan Shi, submitted this Dissertation for approval on 2025-07-10 at 08:48.","This Dissertation was approved for publication on 2025-07-10 at 12:08.","DSpace SAF Submission Ingestion Package generated from Vireo submission #22473 on 2025-10-21 at 10:05:45","Teamwork is a critical skill in both educational and professional settings, yet evaluating individual contributions to collaborative work remains a persistent challenge. Instructors often rely on peer evaluations to assess teamwork, but these evaluations are prone to social and cognitive biases, including memory limitations, inconsistent standards, and interpersonal pressures. At the same time, the increasing use of digital collaboration tools such as Google Docs, GitHub, and Slack presents a new opportunity: these tools generate rich log data that can reveal how teams interact, coordinate, and contribute over time. This dissertation presents the design and evaluation of a data-augmented peer evaluation system that extracts, analyzes, and visualizes log data from digital collaboration tools to support students' evaluations of teamwork. By reviewing the visualization prior to rating their teammates, students can ground their evaluations in concrete evidence of each team member's behaviors. My approach centers students as active participants in interpreting and annotating their own data, thus also contributing knowledge of how students understand and value contributions within teams. This system is the culmination of a multi-stage investigation into how peer evaluations of teamwork can be augmented with log data from digital collaboration tools. I begin with a mixed-methods study examining current and potential uses of log data to support students' and instructors' evaluation practices. Through interviews and surveys, I find that while both groups already refer to log data to validate or challenge peer evaluations, this usage is often informal, retrospective, and shaped by uncertainty about how to interpret the data. Next, I conduct a between-subjects experiment to test whether reviewing log data during the peer evaluation process improves evaluation quality. The results show that log data increased consistency and perceived accuracy of the evaluations, motivating the design of a system that incorporates log data into the peer evaluation process. Grounded in the empirical findings of these two studies, I design and implement the data-augmented peer evaluation system through an iterative user-centered design process. The system captures multiple components of the teamwork: content, technical, and communication contributions in Google Drive, GitHub, and Slack. In the interface, the student reflects on an interactive timeline visualization of the team’s log data prior to evaluating their teammates, allowing them to ground their evaluations in the data. The student can also contextualize the data by marking contributions as valuable and tagging behaviors displayed by their teammates. I evaluate this system in a semester-long field study in a user interface design course. Results show that the data-augmented peer evaluation system helped students recall specific contributions, reflect on work patterns, and justify their ratings. At the same time, students expressed concerns about the representativeness of the data, with some even demonstrating resistance towards the system. Together, these contributions demonstrate the value of integrating log data into peer evaluations and advance knowledge of how data can be used not just to monitor teams, but to support fair and meaningful assessments of teamwork that can guide learning. This work also contributes design principles for creating data-driven evaluation systems that balance fairness, student agency, and meaningful reflection. This dissertation progresses toward a future in which every person working in a team can compose and receive meaningful feedback that helps them to feel valued for their contributions and learn effective teamwork behaviors."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/130030"],"dc:language":["en","eng"],"dc:rights":["Copyright 2025 Wenxuan Shi"],"dc:subject":["Human-computer Interaction","Collaboration","Teamwork","Team Assessment","Peer Evaluations","Log Data","Data-driven Assessment","Interface Design"],"dc:title":["A data-augmented approach to evaluating teamwork using log data from digital collaboration tools"],"dc:type":["text"],"thesis:degree_discipline":["Computer Science"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois Urbana-Champaign"]},"updated_at":"2026-07-22T22:25:06Z"}