Performance of automatic speech recognition (ASR) systems utilizing only acoustic information degrades significantly in noisy environments such as a car cabins. Incorporating audio and visual information together can improve performance in these situations. This work proposes a lip detection and tracking algorithm to serve as a visual front end to an audio-visual automatic speech recognition (AVASR) system.
Several color spaces are examined that are effective for segmenting lips from skin pixels. These color components and several features are used to characterize lips and to train cascaded lip detectors. Pre- and post-processing techniques are employed to maximize detector accuracy. The trained lip detector is incorporated into an adaptive mean-shift tracking algorithm for tracking lips in a car cabin environment. The resulting detector achieves 96.8% accuracy, and the tracker is shown to recover and adapt in scenarios where mean-shift alone fails.
","abstract_html":"<p>Performance of automatic speech recognition (ASR) systems utilizing only acoustic information degrades significantly in noisy environments such as a car cabins. Incorporating audio and visual information together can improve performance in these situations. This work proposes a lip detection and tracking algorithm to serve as a visual front end to an audio-visual automatic speech recognition (AVASR) system.</p> <p>Several color spaces are examined that are effective for segmenting lips from skin pixels. These color components and several features are used to characterize lips and to train cascaded lip detectors. Pre- and post-processing techniques are employed to maximize detector accuracy. The trained lip detector is incorporated into an adaptive mean-shift tracking algorithm for tracking lips in a car cabin environment. The resulting detector achieves 96.8% accuracy, and the tracker is shown to recover and adapt in scenarios where mean-shift alone fails.</p>","abstract_has_math":false,"creators":["Wang, Benjamin"],"institution":null,"degree_name":"MS in Electrical Engineering","degree_level":null,"degree_discipline":"Electrical Engineering","degree_department":null,"school":null,"contributors":["Jane Zhang"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2017,"date_issued":"2017-01-01T08:00:00Z","date_published":"2017-01-01T08:00:00Z","updated_at":"2026-07-24T01:31:49Z","subjects":["Lip Detection","Haar-like Features","Histograms of Oriented Gradients","Local Binary Patterns","Mean-Shift Tracking","Signal Processing"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[{"key":"dc:identifier","label":"Identifier","values":["10.15368/theses.2017.1"],"render_values":[{"text":"10.15368/theses.2017.1","href":"https://doi.org/10.15368/theses.2017.1","code":true}]}]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/1695","outbound_label":"Repository record","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Jane Zhang"]},{"key":"dc:creator","label":"Author","values":["Wang, Benjamin"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2017-01-16T08:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Electrical Engineering"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Electrical Engineering"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Lip Detection","Haar-like Features","Histograms of Oriented Gradients","Local Binary Patterns","Mean-Shift Tracking","Signal Processing"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/1695","10.15368/theses.2017.1"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["Performance of automatic speech recognition (ASR) systems utilizing only acoustic information degrades significantly in noisy environments such as a car cabins. Incorporating audio and visual information together can improve performance in these situations. This work proposes a lip detection and tracking algorithm to serve as a visual front end to an audio-visual automatic speech recognition (AVASR) system.
Several color spaces are examined that are effective for segmenting lips from skin pixels. These color components and several features are used to characterize lips and to train cascaded lip detectors. Pre- and post-processing techniques are employed to maximize detector accuracy. The trained lip detector is incorporated into an adaptive mean-shift tracking algorithm for tracking lips in a car cabin environment. The resulting detector achieves 96.8% accuracy, and the tracker is shown to recover and adapt in scenarios where mean-shift alone fails.
"]},{"key":"dc:title","label":"Title","values":["Lip Detection and Adaptive Tracking"]}]}],"canonical_facts":{"dc:contributor":["Jane Zhang"],"dc:creator":["Wang, Benjamin"],"dc:date.available":["2017-01-16T08:00:00Z"],"dc:description.abstract":["Performance of automatic speech recognition (ASR) systems utilizing only acoustic information degrades significantly in noisy environments such as a car cabins. Incorporating audio and visual information together can improve performance in these situations. This work proposes a lip detection and tracking algorithm to serve as a visual front end to an audio-visual automatic speech recognition (AVASR) system.
Several color spaces are examined that are effective for segmenting lips from skin pixels. These color components and several features are used to characterize lips and to train cascaded lip detectors. Pre- and post-processing techniques are employed to maximize detector accuracy. The trained lip detector is incorporated into an adaptive mean-shift tracking algorithm for tracking lips in a car cabin environment. The resulting detector achieves 96.8% accuracy, and the tracker is shown to recover and adapt in scenarios where mean-shift alone fails.
"],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/1695","10.15368/theses.2017.1"],"dc:subject":["Lip Detection","Haar-like Features","Histograms of Oriented Gradients","Local Binary Patterns","Mean-Shift Tracking","Signal Processing"],"dc:title":["Lip Detection and Adaptive Tracking"],"thesis:degree_discipline":["Electrical Engineering"],"thesis:degree_name":["MS in Electrical Engineering"]},"updated_at":"2026-07-24T01:31:49Z"}