Developing explainable machine-learning model using augmented concept activation vector

Abstract

Machine-learning models use high-dimensional feature spaces to map their inputs to the corresponding class labels; however, these features often do not have a one-to-one correspondence with the physical concepts that are understandable by humans. This hinders the ability to provide meaningful explanations for the decisions that are made by these models. We propose a method for measuring the correlation between high-level concepts and the decisions that are made by machine-learning models. Our method can isolate the impact of a given high-level concept and accurately measure it quantitatively. Additionally, this study aims to determine the prevalence of frequent patterns in machine-learning models that often occur in imbalanced data sets. We successfully applied the proposed method to fundus images and managed to quantitatively measure the impacts of the radiomic patterns on the model’s decisions.