Enhancing Neural Network Performance through SHAP-based Latent Class Integration

Deep learning models have become powerful tools for modeling complex, nonlinear relationships in biomedical data, yet they often operate under the implicit assumption that all features contribute uniformly across all observations. This assumption is particularly limiting in clinical settings like oncology, where latent subpopulations may exist that differ meaningfully in how features affect outcomes. Standard predictive models rarely capture this heterogeneity, resulting in suboptimal performance and limited interpretability. This dissertation presents two novel deep learning architectures—FORCE (Feature-Oriented Representation with Clustering and Explanation) and JEDI-net (Joint Embedding with Dynamic Integration Network)—that integrate SHAP (SHapley Additive exPlanations)-based feature importance into the model training process to uncover and leverage latent substructures in data. Rather than clustering in the raw feature space, both architectures rely on absolute SHAP values to group observations by similarity in feature relevance, providing outcome-aware latent structure discovery. FORCE introduces a two-stage pipeline that calculates SHAP values using a baseline model (gradient boosting classifier), clusters the resulting SHAP vectors using kernel k-means, and then incorporates the resulting cluster labels and SHAP values into a downstream neural network. SHAP values are used both to guide an attention mechanism and as latent embeddings via clustered group membership. Across multiple benchmark datasets, FORCE demonstrated substantial gains in F1 score, AUC, and accuracy compared to traditional architectures, confirming the added value of integrating feature relevance into network learning. To address the architectural complexity and external dependencies of FORCE, JEDI-net builds a fully end-to-end trainable model that computes SHAP values internally and performs dynamic clustering during training. Using k-means and the Hungarian algorithm to update and realign cluster identities over time, JEDI-net embeds evolving subgroup membership directly into the learning loop, allowing the network to adapt as it uncovers latent structure. Evaluated on the same datasets, JEDI-net performed comparably or better than FORCE, while significantly reducing computational overhead. The utility of JEDI-net was further demonstrated in a real-world application involving survival prediction in colorectal cancer patients using data from The Cancer Genome Atlas (TCGA). A set of routinely collected clinical features were used (e.g., age, stage, histology, race) which resulted in JEDI-net identifying clinically plausible patient subgroups with distinct SHAP attribution profiles and survival outcomes. For instance, one cluster featured younger patients with advanced-stage tumors and disproportionately higher mortality—aligning with known disparities in early-onset CRC and healthcare access. Another cluster with older patients and high SHAP attribution to polyps and histology suggested potential missed screening or surveillance failures. By embedding model explanations into the training pipeline, both FORCE and JEDI-net bridge the gap between predictive performance and interpretability. This work contributes to the growing field of explanation-aware deep learning and demonstrates that SHAP-based latent class integration offers a principled and scalable approach to improving model performance, uncovering hidden patient subgroups, and enhancing clinical insight.

Grantor dc:publisher

University of Kansas

Year dc:date.issued

2025

Author dc:creator

• MUKHERJEE, RISHAV

Advisor dc:contributor.advisor

• Thompson, Jeffrey Ahearn

• Biostatistics
• Artificial Intelligence
• Data Science
• Latent Classes
• Neural Networks
• Shapley Additive Explanations
• xAI

Statement dc:rights

• This item is protected by copyright and unless otherwise specified the copyright of this thesis/dissertation is held by the author.

Language dc:language.iso

en

Dc Identifier Other

https://www.proquest.com/LegacyDocView/DISSNUM/32240775

OAI identifier oai:identifier

oai:kuscholarworks.ku.edu:1808/37834