Quake-OVUDA: Component-Level AI-Based Post-Earthquake Building Inspections with Vision–Language Guided Unsupervised Domain Adaptation

The semantic segmentation of building components and their damage states is a critical step toward automating structural inspections in the aftermath of disasters. A key challenge towards training deep neural networks for these tasks is the manual development of large, annotated datasets for the real world. Training Unsupervised Domain Adaptation (UDA) models, using synthetic, automatically generated imagery, has shown promise in reducing reliance on manual labeling and improving the adaptation of features learned from the synthetic domain to the real-world domain of images. However, existing UDA methods can still struggle to generalize to certain real-world representations of objects that are stylistically different between synthetic and real domains or are generally underrepresented in the synthetic dataset. Domain gaps like these are especially relevant in synthetic structural imagery, where real-world building components and damage can adopt a wide variety of patterns. To address these limitations, this thesis incorporates features learned from open-vocabulary (OV) semantic segmentation into the UDA framework, DAFormer, thereby enabling improved contextual understanding and transferring of features from the synthetic to real domains. Specifically, the Simple Encoder Decoder (SED) OV model is fine-tuned on the synthetic dataset, QuakeCity, and is used to generate pseudo-labels for unlabeled real-world images that are used in DAFormer’s self-supervised learning step. To further improve classification performance, particularly of nuanced damage classes, additional refinement and classification of instances predicted by Quake-OVUDA are performed by leveraging the vast context provided by vision language models (VLMs). The proposed framework, termed Quake-OVUDA, and its additional supplementation with VLM expertise, outperform direct segmentation, open-vocabulary segmentation, and baseline UDA methods for building component and component damage state segmentation tasks. Evaluation of the proposed methods shows improvements of up to 5.49 mean Intersection-over-Union (mIoU) for the component segmentation task and up to 11.02 mIoU for the damage state segmentation task over the baseline UDA approach. This work demonstrates an effective combination of OV segmentation and VLM-based reasoning to guide UDA and reduce reliance on manually annotated real-world data for visual building assessment.

Name thesis:degree_name

Master of Science

Discipline thesis:degree_discipline

Engineering Data Science

Grantor

University of Houston

Year dc:date.issued

2025

Author dc:creator

• Abdo, Charles 2002-

Advisor dc:contributor.advisor

• Hoskere, Vedhus

Committee members dc:contributor.committeemember

• Kulkarni, Yashashree
• Ballarini, Roberto

• Computer science

Language dc:language.iso

English

Handle dc:identifier.uri

https://hdl.handle.net/10657/20674

OAI identifier oai:identifier

oai:uh-ir.tdl.org:10657/20674