Automated field compliance checking for improved construction safety using natural language processing and computer vision

A large portion of construction-jobsite fatalities and injuries, including those caused by fall, can be attributed to field noncompliances with safety regulations, particularly the Occupational Safety and Health Administration (OSHA) regulations, which include a large number of fall protection requirements for different equipment, facilities, and operations. Current jobsite safety checking practices are time consuming and error prone, because they rely heavily on manual inspection to identify violations. To reduce the time, cost, and errors of such a manual and exhaustive process, there is a need for automated field compliance checking, which seeks to automate the process of extracting safety requirements from OSHA, capturing site condition information, and identifying violations. To address this need, this thesis proposes a new deep learning-based method that utilizes natural language processing (NLP) and computer vision techniques to automatically detect field noncompliances with OSHA. The proposed method includes three main components. First, a set of computational models and NLP methods to automatically extract requirements from safety regulations, including entity extraction, relation extraction, and knowledge graph-based representation. Second, a set of computational models and computer vision methods to automatically detect site condition information from site images, including object detection, attribute recognition, visual relation detection, and scene graph representation. Third, a knowledge graph-based classification and reasoning method to retrieve applicable clauses, detect noncompliances, and generate documentation. The research methodology included seven primary research tasks. First, conducting a comprehensive literature review on related topics such as construction safety, automated compliance checking, machine learning, information extraction, object detection, attribute recognition, visual relation detection, knowledge graphs, and generative models. Second, developing a machine learning-based method to extract entities that describe fall protection requirements from OSHA. Third, developing a machine learning-based method to extract relations that describe fall protection requirements from OSHA and represent the extracted requirements as query graphs. Fourth, developing a machine learning-based method to identify fall-related site objects, with multiple attributes, to provide intricate site condition information. Fifth, developing a machine learning-based method to detect visual relations among fall-related site objects and generate scene graphs to represent the detected site information in a similar structure to the query graphs. Sixth, developing a machine learning-based method to retrieve applicable safety clauses according to the site scenes and conduct knowledge graph-based compliance reasoning. And seventh, integrating the developed computational models and methods into a prototype system and comparing the proposed approach with a general end-to-end approach that utilizes multimodal foundation models. The aforementioned computational models and methods were individually tested, each achieving over 80.0% in precision and recall. The final integrated automated field compliance checking prototype system was evaluated using a set of fall-related site images, as well as fall-related requirements from OSHA. The prototype showed effective performance, achieving an average precision of 71.2%, recall of 73.9%, and F-1 measure of 72.5%, respectively, for noncompliance detection. The generated documentation showed reasonable chain-of-thoughts, precise summarization, and proper suggestions of corrective measures. These experimental results demonstrate the good potential of the proposed automated field compliance checking approach.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Civil Engineering

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Wang, Xiyu

Contributors dc:contributor

• El-Gohary, Nora
• Golparvar-Fard, Mani
• El-Rayes, Khaled
• Zhai, ChengXiang

• Construction Safety
• Automated Field Compliance Checking
• Construction Safety Regulations
• Natural Language Processing
• Information Extraction
• Computer Vision
• Visual Information Detection
• Knowledge Graphs
• Automated Reasoning
• Large Language Models.

Statement dc:rights

• Copyright 2024 Xiyu Wang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124709

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124709