Human-AI Handshaking: Supporting Extreme Sensemaking through Trustworthy Shared Perception

Extreme sensemaking occurs when teams of people must build situational awareness in high-stakes, dynamic environments such as search and rescue, military, or security operations. These contexts are marked by uncertainty, fragmented information, and time-critical decisions that stretch human cognitive and physical limits. Artificial intelligence (AI) offers potential assistance, but distributed AI systems that rely on object detection, such as drone swarms, autonomous vehicles, and large-scale sensor networks, face their own challenges, including fluctuating accuracy, false detections, and fragile resilience under real-world dynamics. This dissertation introduces the "Human-AI Handshake", a novel human-AI interaction technique that incorporates human-in-the-loop (HITL) and crowd-in-the-loop (CITL) components to enhance object detection accuracy and shared perception in distributed systems. The Human-AI Handshake addresses core challenges by combining human feedback with AI model performance to mitigate uncertainties and improve trustworthiness. The foundation of the proposed concept comprises four key components: (1) evaluating HITL concepts for improving computer vision accuracy, (2) enhancing shared perception among distributed AI systems to enable better situational awareness, (3) ensuring AI trustworthiness through a synchronized perception trust framework, and (4) interpreting contextual awareness to help AI systems adapt to diverse, real-time scenarios. The technique is tested in extreme sensemaking applications, including augmented reality (AR)-assisted search and rescue, where accurate and reliable object detection is essential. Overall, the Human-AI Handshake provides an assured, scalable solution for improving AI assurance in distributed, dynamic environments, ensuring greater reliability, trust, and performance in critical operations.

Name thesis:degree_name

Doctor of Philosophy

Level thesis:degree_level

doctoral

Discipline thesis:degree_discipline

Computer Science & Applications

Department dc:contributor.department

Computer Science and#38; Applications

Grantor dc:publisher

Virginia Tech

Year dc:date.issued

2025

Author dc:creator

• Wilchek, Matthew Ryan

Chairs dc:contributor.committeechair

• Luther, Kurt
• Batarseh, Feras A.

Committee members dc:contributor.committeemember

• Freeman, Laura June
• Caba, Wilson
• Tilevich, Eli
• Bowman, Douglas Andrew

• Human-Computer Interaction
• Human-AI Collaboration
• Computer Vision
• Trustworthy AI
• Object Detection
• AI for Surface Water
• Search and Rescue

Statement dc:rights

• Creative Commons Attribution-NonCommercial 4.0 International

Licence dc:rights.uri

• http://creativecommons.org/licenses/by-nc/4.0/

Language dc:language.iso

en

Dc Identifier Other

vt_gsexam:45081

OAI identifier oai:identifier

oai:vtechworks.lib.vt.edu:10919/140538