Interpretable Deep Learning Classifier by Detection of Prototypical Parts
on Kidney Stones Images

Urolithiasis disease refers to the formation of kidney stones (KS). Several industrialized countries present a high incidence of kidney stone episodes (around 10$\%$ of the population is affected [21, 14]). The stone formation is a multi-factorial process [4, 6], where the diet is one of the most important factors [10, 19] but several complementary factors can produce it (e.g., hereditary-family history, chronic diseases, and sedentary lifestyle). Early identification of the type of kidney stone aids the urologist to have an accurate diagnosis, enabling them to prescribe the appropriate treatment (e.g., diet adaptation or surgery), but most importantly, to reduce eventual relapses [10].

The kidney stone type of an ex-vivo sample (extracted during endoscopic surgery) can be identified using a two-step procedure, a method known as Morpho-Constitutional Analysis (MCA) [5, 6]. First, a microscopic morphological examination of the visual characteristics of the stone (e.g., size, form, color, texture, and appearance of the surface and section view) is strongly correlated with the molecular study [4] and it enables to preserve important diagnostic information. On the other hand, an Infrared-spectrophotometry analysis leads to a more precise identification of the crystalline composition of the stone [8].

Although the MCA efficiently establishes the type of ex-vivo kidney stones, it is very difficult to provide a reliable diagnosis during an endoscopic intervention (the results of the MCA may take days). Also, it is time-consuming and tedious (fragments extraction can take up to one hour of surgery) [4] and very difficult to train specialists on. Recently, it has been suggested that an endoscopic (intra-operative) stone recognition (ESR) CAD tool could help to obtain a faster diagnosis based solely on the video signal information provided by the endoscope, in tandem with the visual aid on the screen [9]. Also, the operations are quicker to perform and less traumatic, due to dusting can fragment and destroy the kidney stones inside the urinary tract.

Several methods have been proposed in recent years for performing automated ESR, based both on traditional and deep learning techniques with very encouraging results [16]. On the one hand, shallow Machine Learning (ML) models have shown that the efficient extraction of features (e.g., color and texture) in kidney stone views (surface and section) can have a significant impact on the classification (with accuracy $\geq 90\%$) on in-vivo endoscopic images (strongly correlated to the morphological analysis of kidney stones) [12, 11, 16]. However, in visualizations such as UMAP [13], the clusters are not tight enough, which could mean that they are not the best features that could be used in the classifier. On the other hand, Deep Learning (DL) based models [11, 9, 2, 16] have shown excellent results (high accuracy $\geq 95\%$) for extracting features relevant to the classifier (and tight clusters in UMAP). However, DL models lack interpretation of the features they extract, making these models not very useful in clinical settings.

As a matter of fact, current ML and DL models are unable to describe the inner workings that led to a given prediction beyond the class label. Therefore, these types of models cannot provide useful information to the specialist to understand how the input image was used to perform a diagnosis (i.e., classify the kidney stone type).

In order to pave the way for AI-based ESR using deep learning techniques, in this work, we leverage recent strides in explainability that seek to base image classifiers decisions on case-based reasoning to make them more interpretable [3, 15, 17, 1]. Additionally, we provide both visual explanations and quantitative information about visual characteristics deemed important by the network. It must be emphasized that our approach follows the reasoning processes of urologists in detecting morphological relevant features of each image (i.e., MCA). Overall, this work is aimed at facilitating human-machine collaboration in the context of CAD tools for urolithiasis prevention.

Evaluation metrics of our model, its backbone model, and AlexNet (as reference), are reported in Table 1. The performance of ProtoPNet is comparable with its corresponding uninterpretable backbone model ($\leq 3\%$ difference). In contrast to black-box classifiers, our proposal provides explanations for input images, by showing the activation area of PPs, the corresponding representative image to each activated PPs, and their descriptors (Fig. 1(b)).

We plot PPs and their descriptors activations for input images on a UMAP visualization of the three most discriminant dimensions (umap1 to umap3). This UMAP allows seeing class separability for each output class of the ProtoPNet, as shown in Fig. 3. In this way, additional global insight is gained in the case of a new classification observed surrounded by samples of the same class, which provides confidence of a correct classification (also described in Fig. 3). However, it was observed that multiple PPs end up indicating the same training patch as their explanation, a behavior similar to mode collapse on Generative Adversarial Networks (GANs), which limits the variety of possible explanations provided for an output. The use of descriptors mitigates the cases for visually similar PPs by providing details on the characteristics most relevant for each PP [15].

We showed that by training of PPs and extracting their descriptors we convert an uninterpretable VGG19 into an interpretable model. This can facilitate the use of these models for ESR by a urologist. However, mode collapse of the learned PPs is a limitation on the current implementations of ProtoPNets. To prevent this, better initialization procedures and loss function adjustments will be explored. Finally, we found indications of better class separability by use of PPs and their descriptors, to the point UMAP visualizations could be used to provide global context of the certainty of the output classification of a particular image.

The authors wish to thank the AI Hub and the CIIOT at ITESM for their support for carrying the experiments reported in this paper in their NVIDIA’s DGX computer.