{"id":{"repo_id":"uiuc","oai_identifier":"oai:www.ideals.illinois.edu:2142/117779"},"canonical_url":"https://search.dev.ndltd.org/etd/uiuc/oai:www.ideals.illinois.edu:2142/117779","repository":{"repo_id":"uiuc","name":"University of Illinois - Urbana-Champaign","base_url":"https://www.ideals.illinois.edu/oai-pmh"},"display":{"title":"Hand gesture recognition and hand tracking for medical applications","abstract":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-04-12 without embargo terms","abstract_html":"Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-04-12 without embargo terms","abstract_has_math":false,"creators":["Sharif, Hajar"],"institution":"University of Illinois at Urbana-Champaign","degree_name":"Ph.D.","degree_level":"Dissertation","degree_discipline":"Mechanical Engineering","degree_department":null,"school":null,"contributors":["Kesavadas, Thenkurussi K","Salapaka, Srinivasa M","Sreenivas, Ramavarapu S","Phillips, Heidi"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":2022,"date_issued":"2022-12","date_published":"2022-12","updated_at":"2026-07-22T22:24:56Z","subjects":["Hand Gesture Recognition","Activities Of Daily Living","Hand Grasps Classification","Surgical Tasks Classification","Metric Development","Surgical Proficiency Assessment","Quantitative Evaluation Of Rehabilitation","Video Processing Of Surgical Tasks","Leap Motion Controller","Azure Kinect Dk","Myo Armband","Artificial Intelligence","Evaluating Suturing And Knot Tying"],"languages":["en","eng"],"rights":["© 2022 Hajar Sharif"],"rights_urls":[],"identifier_entries":[]},"links":{"outbound_url":"https://hdl.handle.net/2142/117779","outbound_label":"Handle","outbound_source":"dc:identifier"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Kesavadas, Thenkurussi K","Salapaka, Srinivasa M","Sreenivas, Ramavarapu S","Phillips, Heidi"]},{"key":"dc:creator","label":"Author","values":["Sharif, Hajar"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date","label":"Dc Date","values":["2022-12","2022-12-02"]},{"key":"dc:type","label":"Dc Type","values":["text","Thesis"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Mechanical Engineering"]},{"key":"thesis:degree_level","label":"Degree Level","values":["Dissertation"]},{"key":"thesis:degree_name","label":"Degree Name","values":["Ph.D."]},{"key":"thesis:institution_name","label":"Thesis Institution Name","values":["University of Illinois at Urbana-Champaign"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Hand Gesture Recognition","Activities Of Daily Living","Hand Grasps Classification","Surgical Tasks Classification","Metric Development","Surgical Proficiency Assessment","Quantitative Evaluation Of Rehabilitation","Video Processing Of Surgical Tasks","Leap Motion Controller","Azure Kinect Dk","Myo Armband","Artificial Intelligence","Evaluating Suturing And Knot Tying"]}]},{"id":"language_rights","label":"Language and Rights","entries":[{"key":"dc:language","label":"Dc Language","values":["en","eng"]},{"key":"dc:rights","label":"Dc Rights","values":["© 2022 Hajar Sharif"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://hdl.handle.net/2142/117779"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description","label":"Description","values":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-04-12 without embargo terms","The student, Hajar Sharif, accepted the attached license on 2022-11-24 at 09:15.","The student, Hajar Sharif, submitted this Dissertation for approval on 2022-11-29 at 11:53.","This Dissertation was approved for publication on 2022-12-02 at 10:54.","DSpace SAF Submission Ingestion Package generated from Vireo submission #18631 on 2023-04-12 at 07:32:32","Hand gestures are a mean of communication and a prevalent type of body language that conveys messages through different shapes constructed by palm and fingers. Hand gesture recognition (HGR) has been of interest in many research fields such as sign language translation, musical creation, and virtual environment control. There are also several studies on HGR for robotics, prosthetic, and rehabilitation applications. In this dissertation, the application of HGR for addressing two challenges in the medical field is presented. The first challenge is to develop a quantitative metric to improve rehabilitation of neurological conditions, with a focus on improvement in performing activities of daily living (ADL), while the second challenge is to develop ATTENTIVE, an automated and quantitative assessment system, to enhance a better evaluation of surgical skills proficiency. Many neurological conditions lead to motor impairment of upper extremity that includes muscle weakness, altered muscle tone, joint laxity, and impaired motor control. As a result, common activities such as reaching, picking up objects, and holding onto them are compromised. Therefore, such patients will experience disability in performing ADL such as eating, writing, performing housework, and so on. Several evaluation methods are commonly used to assess problems in performing ADL. Despite the wide application of these methods, all of them are subjective techniques, i.e. they are either questionnaires or qualitative scores assigned by a medical professional. We hypothesize that providing a more quantitative metric can enhance evaluation of the rehabilitation progress, and lead to a more efficient rehabilitation regimen tailored to the specific needs of each individual patient. Since the first step of developing a metric is to distinguish different ADL activities using hand gesture data, in this dissertation the focus is on classification of ADL tasks using hand gestures. Data analysis pipelines were developed to take in data, collected by the leap motion controller as well as the electromyography and inertial measurement unit sensors, from the lower arm during completion of certain ADL tasks. These pipelines output classification accuracies to distinguish the ADL tasks. Different preprocessing, feature extraction, and classification methods were tested on the data from healthy adults to detect the best structure and parameters for the proposed pipelines. The developed pipelines can be trained and their parameters can be tuned based on data from an intact-adult population. Then, The tuned pipelines can be set as the references. Subsequently, hand motion data from a neurological patient completing the same tasks in the same data collection setup can be fed into the reference pipelines to obtain the classification accuracies. The achieved accuracies indicate how close a patient’s hand motions and muscle activation are to the hand motions and muscle activation of the healthy population. This method enhances assessment of the overall performance of a patient in a quantitative fashion. In addition, the acquired confusion matrices provide insight into the patient’s performance in completing each individual task. The second section of this dissertation includes the design of ATTENTIVE; an evidence-supported, automated, robust, real-time, comprehensive, quantitative assessment system for evaluating proficiency in basic surgical skills. Since ATTENTIVE provides quantitative feedback, it can have a variety of applications in teaching surgical skills either in traditional settings or within incorporation of the augmented reality systems. As of now, the presence of an automated and quantitative assessment system to provide feedback on surgical tasks performance is lacking, and expert surgeon’s involvement is necessary to provide feedback to the surgical trainees. As a result, a trainee’s opportunities to receive feedback on one’s performance is restricted to the availability of an expert surgeon, which is limited due to pre-existing high workload of the expert surgeons. ATTENTIVE can eliminate such restriction that in turn may result in surgical trainees’ performance improvement and superior surgical outcomes over the long run. In this work, the idea and pipeline for developing ATTENTIVE are presented. Next, the apparatus and experimental setup and protocol to investigate the feasibility of ATTENTIVE were designed and built. Afterwards, data was collected from 65 participants completing four basic surgical tasks. The participants were students, residents, and expert surgeons in the fields of veterinary and human medicine. To benefit from both sensor-based and vision-based HGR methods for solving the problem in hand, Azure Kinect DK, Leap Motion Controller, and Myo armband were used to collect data from the lower arm of the participants. ATTENTIVE’s workflow consists of three major steps including separating the main task part from the preparation and cleanup after the task completion, classifying the input surgical task, and assigning a performance score to the input task. In this dissertation, many parts of these three steps are completed, and algorithms to complete the rest are determined and implemented to a vast extend. The details of the data analysis steps are beyond the scope of the abstract and are presented in the second section’s chapters. The last chapter of the current dissertation contains preliminary work on design and fabrication of a wearable device, named iBand, to collect biosignals and kinematic data from the lower arm. Different components of iBand have been selected and calibrated to read synchronized data from the lower arm, transfer them to a computer via Bluetooth, and save them as separate files. An easy-to-work user interface has been developed for iBand to enable user to save the data in the desired folder and with the desired file name. In addition, the user interface enhances a real-time data observation in which the user can choose the sensor from which the collected signal is displayed. Upon completion, the iBand can replace the discontinued Myo armband for research and daily life applications."]},{"key":"dc:format","label":"Dc Format","values":["application/pdf"]},{"key":"dc:title","label":"Title","values":["Hand gesture recognition and hand tracking for medical applications"]}]}],"canonical_facts":{"dc:contributor":["Kesavadas, Thenkurussi K","Salapaka, Srinivasa M","Sreenivas, Ramavarapu S","Phillips, Heidi"],"dc:creator":["Sharif, Hajar"],"dc:date":["2022-12","2022-12-02"],"dc:description":["Submission original under an indefinite embargo labeled 'Open Access'. The submission was exported from vireo on 2023-04-12 without embargo terms","The student, Hajar Sharif, accepted the attached license on 2022-11-24 at 09:15.","The student, Hajar Sharif, submitted this Dissertation for approval on 2022-11-29 at 11:53.","This Dissertation was approved for publication on 2022-12-02 at 10:54.","DSpace SAF Submission Ingestion Package generated from Vireo submission #18631 on 2023-04-12 at 07:32:32","Hand gestures are a mean of communication and a prevalent type of body language that conveys messages through different shapes constructed by palm and fingers. Hand gesture recognition (HGR) has been of interest in many research fields such as sign language translation, musical creation, and virtual environment control. There are also several studies on HGR for robotics, prosthetic, and rehabilitation applications. In this dissertation, the application of HGR for addressing two challenges in the medical field is presented. The first challenge is to develop a quantitative metric to improve rehabilitation of neurological conditions, with a focus on improvement in performing activities of daily living (ADL), while the second challenge is to develop ATTENTIVE, an automated and quantitative assessment system, to enhance a better evaluation of surgical skills proficiency. Many neurological conditions lead to motor impairment of upper extremity that includes muscle weakness, altered muscle tone, joint laxity, and impaired motor control. As a result, common activities such as reaching, picking up objects, and holding onto them are compromised. Therefore, such patients will experience disability in performing ADL such as eating, writing, performing housework, and so on. Several evaluation methods are commonly used to assess problems in performing ADL. Despite the wide application of these methods, all of them are subjective techniques, i.e. they are either questionnaires or qualitative scores assigned by a medical professional. We hypothesize that providing a more quantitative metric can enhance evaluation of the rehabilitation progress, and lead to a more efficient rehabilitation regimen tailored to the specific needs of each individual patient. Since the first step of developing a metric is to distinguish different ADL activities using hand gesture data, in this dissertation the focus is on classification of ADL tasks using hand gestures. Data analysis pipelines were developed to take in data, collected by the leap motion controller as well as the electromyography and inertial measurement unit sensors, from the lower arm during completion of certain ADL tasks. These pipelines output classification accuracies to distinguish the ADL tasks. Different preprocessing, feature extraction, and classification methods were tested on the data from healthy adults to detect the best structure and parameters for the proposed pipelines. The developed pipelines can be trained and their parameters can be tuned based on data from an intact-adult population. Then, The tuned pipelines can be set as the references. Subsequently, hand motion data from a neurological patient completing the same tasks in the same data collection setup can be fed into the reference pipelines to obtain the classification accuracies. The achieved accuracies indicate how close a patient’s hand motions and muscle activation are to the hand motions and muscle activation of the healthy population. This method enhances assessment of the overall performance of a patient in a quantitative fashion. In addition, the acquired confusion matrices provide insight into the patient’s performance in completing each individual task. The second section of this dissertation includes the design of ATTENTIVE; an evidence-supported, automated, robust, real-time, comprehensive, quantitative assessment system for evaluating proficiency in basic surgical skills. Since ATTENTIVE provides quantitative feedback, it can have a variety of applications in teaching surgical skills either in traditional settings or within incorporation of the augmented reality systems. As of now, the presence of an automated and quantitative assessment system to provide feedback on surgical tasks performance is lacking, and expert surgeon’s involvement is necessary to provide feedback to the surgical trainees. As a result, a trainee’s opportunities to receive feedback on one’s performance is restricted to the availability of an expert surgeon, which is limited due to pre-existing high workload of the expert surgeons. ATTENTIVE can eliminate such restriction that in turn may result in surgical trainees’ performance improvement and superior surgical outcomes over the long run. In this work, the idea and pipeline for developing ATTENTIVE are presented. Next, the apparatus and experimental setup and protocol to investigate the feasibility of ATTENTIVE were designed and built. Afterwards, data was collected from 65 participants completing four basic surgical tasks. The participants were students, residents, and expert surgeons in the fields of veterinary and human medicine. To benefit from both sensor-based and vision-based HGR methods for solving the problem in hand, Azure Kinect DK, Leap Motion Controller, and Myo armband were used to collect data from the lower arm of the participants. ATTENTIVE’s workflow consists of three major steps including separating the main task part from the preparation and cleanup after the task completion, classifying the input surgical task, and assigning a performance score to the input task. In this dissertation, many parts of these three steps are completed, and algorithms to complete the rest are determined and implemented to a vast extend. The details of the data analysis steps are beyond the scope of the abstract and are presented in the second section’s chapters. The last chapter of the current dissertation contains preliminary work on design and fabrication of a wearable device, named iBand, to collect biosignals and kinematic data from the lower arm. Different components of iBand have been selected and calibrated to read synchronized data from the lower arm, transfer them to a computer via Bluetooth, and save them as separate files. An easy-to-work user interface has been developed for iBand to enable user to save the data in the desired folder and with the desired file name. In addition, the user interface enhances a real-time data observation in which the user can choose the sensor from which the collected signal is displayed. Upon completion, the iBand can replace the discontinued Myo armband for research and daily life applications."],"dc:format":["application/pdf"],"dc:identifier":["https://hdl.handle.net/2142/117779"],"dc:language":["en","eng"],"dc:rights":["© 2022 Hajar Sharif"],"dc:subject":["Hand Gesture Recognition","Activities Of Daily Living","Hand Grasps Classification","Surgical Tasks Classification","Metric Development","Surgical Proficiency Assessment","Quantitative Evaluation Of Rehabilitation","Video Processing Of Surgical Tasks","Leap Motion Controller","Azure Kinect Dk","Myo Armband","Artificial Intelligence","Evaluating Suturing And Knot Tying"],"dc:title":["Hand gesture recognition and hand tracking for medical applications"],"dc:type":["text","Thesis"],"thesis:degree_discipline":["Mechanical Engineering"],"thesis:degree_level":["Dissertation"],"thesis:degree_name":["Ph.D."],"thesis:institution_name":["University of Illinois at Urbana-Champaign"]},"updated_at":"2026-07-22T22:24:56Z"}