Video representation learning for vision-driven activity analysis in construction

Labor and equipment are two of the most important and costly resources that directly impact productivity and profits in a construction project. Many controllable factors such as the site layout and sequence of work operations can influence the "direct-work" or "tool time" of workers and equipment. As such, construction managers require a comprehensive and continuous understanding of the on-site activities to identify productivity inhibitors that affect the operation of field resources, devise a plan to reduce these issues, and continuously measure improvements resulting from these changes. Activity analysis --the continuous process of measuring and improving the amount of time craft workers and equipment spend on actual construction-- can offer a detailed description of such field activities to the management. However, the measurement of direct-work rates has traditionally involved trained observers walking along the randomly selected routes on the site to classify and record each observed construction worker and equipment into different activity categories by visual observations. Despite the effectiveness of the results, the large size of manual observations needed to guarantee the statistical accuracy of workface assessment and the cost associated with the process can easily discourage project teams from conducting activity analysis. Other factors such as the distance limits to construction workers and equipment, the observers' bias and fatigue, as well as the observers' visual judgments due to overproductiveness phenomenon caused by construction workers under observations or instantaneous reaction of the observer to categorize work activities can lead to erroneous data and thus negatively impact the baselines for productivity improvements. Over the past two decades, a large number of time-lapse and video cameras have been installed on construction sites to offer remote progress monitoring and safety analysis. The ubiquitous nature of these cameras and the improved quality of their video feeds provide an opportunity to develop systems that can automatically offer activity analysis to the management and allow them to focus on planning and implementing productivity improvements to work sequences. To do so, several computer vision techniques have been developed in recent years to detect, track, and analyze motion trajectories and activities of the workers and equipment from these video feeds. While computer vision-based detection and tracking techniques have significantly matured, analyzing worker and equipment activities has remained largely an unsolved problem. Interpretation, identification, and joint-optimization of work activities in conjunction with ergonomics, require approaches that address the following questions: (a) what are the most appropriate atomic primitive representations for worker and equipment activities, (b) how to combine these primitive representations, particularly in the context of equipment-worker-tool interactions and site-layout context to produce complex activities,(c) what are the required invariances for machine learning-based inference algorithms, (d) how to build computational models and multi-objective optimizations for each of these inferences to enable joint productivity-ergonomic assessments, and lastly (e) how a significant amount of ground truth data can be generated to efficiently and effectively train these machine learning models. This dissertation provides a broad overview and in-depth discussion of these issues. Specifically, it introduces a new collection of deep learning methodologies and optimization algorithms to leverage real-world and synthetic visual data to offer activity analysis for both equipment and worker operations. In these algorithms, the spatio-temporal characterization of construction activities is described in terms of appearance, motion, pose reconstructions, and agent-tool-environment interactions. A substantial body of experimental results is presented, demonstrating how these distinguishable data representations allow for temporally coherent interpretations of the worker and equipment activities to produce reliable predictions for workface assessment and productivity improvements. In addition, these results illustrate that reasoning about atomic activities, ergonomic assessment, and accumulated worker-joint movements allows for finding optimal activity execution sequences per worker. It is shown that the actionable insight from these assessments can offer management an opportunity to review and revise work sequences and site layout such that productivity is maximized without negatively impacting worker health and safety in site operations. The merits and drawbacks of these strategies are examined in-depth concerning the practice of activity analysis to demonstrate their potential for minimizing the time needed for workface assessment and thus, allowing professionals to focus their time on the more important task of root-cause analysis and investigating alternatives for performance improvement in field operations.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Civil Engineering

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Torres Calderon, Wilfredo

Contributors dc:contributor

• Golparvar-Fard, Mani
• Hoiem, Derek
• El-Rayes, Khaled
• Liu, Liang
• El-Gohary, Nora

• Construction Management
• Computer Vision
• Productivity Analysis
• Deep Learning

Statement dc:rights

• Copyright 2022 Wilfredo Torres Calderon

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/115763

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F115763