Robust appearance based sign language recognition

In this work, we introduce a robust appearance-based sign language recognition system which is derived from a large vocabulary speech recognition system. The system employs a large variety of methods known from automatic speech recognition research for the modeling of temporal and language specific issues. The feature extraction part of the system is based on recent developments in image processing which model different aspects of the signs and accounts for visual variabilities in appearance. Different issues of appearance-based sign language recognition such as datasets, appearance-based features, geometric features, training, and recognition parts are investigated and analyzed. We discuss the state of the art in sign language and gesture recognition. In contrast to the proposed system, most of the existing approaches use special data acquisition tools to collect the data of the signings. The systems which use this kind of data capturing tools are not useful in practical environments. Furthermore, the datasets created within their own group are not publicly available which makes it difficult to compare the results. To overcome these shortcomings and the problems of the existing approaches, our system is built to use video data only and evaluated on publicly available data. First, to overcome the scarceness of publicly available data and to remove the dependency on impractical data capturing devices, we use normal video files publicly available and create appropriate transcriptions of these files. Then, appearance-based features are extracted directly from the videos. To cope with the visual variability of the signs occurring in the image frames, pronunciation clustering, invariant distances, and different reduction methods are investigated. Furthermore, geometric features capturing the configuration of the signers' hand are investigated improving the accuracy of the recognition system. The geometric features represent the position, the orientation and the configuration of the signers' dominant hand which plays a major role to convey the meaning of the signs. Finally, it is described how to employ the introduced methods and how to combine the features to construct a robust sign language recognition system.

Grantor dc:publisher

Publikationsserver der RWTH Aachen University

Year dc:date

2007

Author dc:creator

• Zahedi, Morteza

Contributors dc:contributor

• Ney, Hermann

• info:eu-repo/classification/ddc/004
• American sign language
• Deutsche Gebärdensprache
• Gebärdensprache
• British sign language
• Informatik
• American sign language recognition
• appearance-based
• gesture recognition

Statement dc:rights

• info:eu-repo/semantics/openAccess

Language dc:language

eng

Identifier

https://publications.rwth-aachen.de/search?p=id:%22RWTH-CONV-124047%22

OAI identifier oai:identifier

oai:publications.rwth-aachen.de:62481