Explainable Anomaly Detection with Few Labeled Data
Anomaly detection is the process of identifying unusual/rare patterns that deviate from normal behaviour. While the collected data form extra-high dimensional datasets, challenges on the predictive models’ scalability regarding both data size and dimension arises. However, extensive labeled training data for anomaly detection is enormously expensive and often unavailable in data-sensitive applications due to privacy constraints. Unsupervised approaches are promising due to their ability to learn from unlabeled data. Nonetheless, these methods offer inadequate anomaly explanations compared to supervised methods. This motivates the development of semi-supervised solutions that can provide reasonable detection accuracy and interpretation of the anomalies. This thesis breaks the problem down, explores related work, and presents possible solutions. We first survey existing anomaly detection methods of different levels of supervision, with their advantages and limitations. Afterwards, we review feature selection approaches and provide discussion on their adaptability with semi-supervised anomaly detectors. Noticing most of these methods suffer when very limited labels are provided, we propose solutions for both feature selection and anomaly detection tasks. We propose a histogram-based label-spreading approach that maximizes the use of labeled data, which propagates the small amount of class information to the local neighbourhoods. We show that this mechanism enables information-gain-based calculations, which generally require the full ground truth to be known. We extend this work as a filter-based feature selection method that is computationally light and adaptable with many statistical measurements on distributions. Furthermore, we propose a tree-based anomaly detection method that utilizes this mechanism. We show empirically that this method is effective with very limited labels and provides feature-wise anomaly explanations that showing consistencies to fully supervised models such as Random Forest.
Networked Digital Library of Theses and Dissertations