SimCalib: Graph Neural Network Calibration based on Similarity between Nodes

Herein we consider the problem of calibrating GNNs on semi-supervised node classification tasks. Specifically, given an undirected graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, consisting of nodes $\mathcal{V}$ and edges $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$, each node $v_{i}\in\mathcal{V}$ is associated with a feature vector ${\bm{x}}_{i}\in\mathbb{R}^{d}$. Moreover, a proper subset of nodes, denoted as $\mathcal{L}\subset\mathcal{V}$, is further associated with labels $\{y_{i}\}_{i:v_{i}\in\mathcal{L}}$, where $y_{i}\in\mathcal{Y}=\{1,\dots,K\}$ is the ground-truth label for $v_{i}$. And the goal of semi-supervised node classification is to infer the labels for the unlabeled nodes $\mathcal{U}=\mathcal{V}\setminus\mathcal{L}$. A graph neural network approaches the problem by taking into account both nodewise features and structural information, i.e. adjacency matrix $\mathbf{A}$, and it predicts a probability distribution $\hat{p}_{i}$ over all the classes for each node $v_{i}$. The value on the $j$-th position of the distribution, i.e. $\hat{p}_{i}^{(j)}$, describes the estimated probability of $v_{i}$ being in class $j$.

For each node, $\hat{p}_{i}$ induces the corresponding label prediction $\hat{y}_{i}:={\tt\mathop{argmax}}_{j}\ \hat{p}_{i}^{(j)}$ and confidence $\hat{c}_{i}:={\tt\mathop{max}}_{j}\ \hat{p}_{i}^{(j)}$. Perfect calibration is defined as (Wang et al. 2021):

$$\forall\ c\in[0,1],\quad{\tt P}(y_{i}=\hat{y}_{i}|\hat{c}_{i}=c)=c.$$

(1)

In practice, perfect calibration cannot be estimated with a finite number of samples, therefore calibration quality is often quantified by expected calibration error (ECE) instead (Naeini, Cooper, and Hauskrecht 2015; Guo et al. 2017):

$\displaystyle{\tt ECE}:={\mathbb{E}}_{p}\Bigg{[}\Bigg{|}{\mathbb{E}}[Y=k|\tt\hat{p}(Y=k|{\bm{x}})=p]-p\Bigg{|}\Bigg{]},$

(2)

where ${\tt\hat{p}}(Y=k|{\bm{x}})$ is the predicted probability of ${\bm{x}}$ being in class $k$, the inner expectation represents the ground-truth probability of ${\bm{x}}$ belonging to $k$, and the outer expectation iterates over all $p\in(0,1)$.

To preserve the nodewise predictions, CaGCN calibrates logits by scaling them with nodewise temperatures, i.e.

$$\hat{z}_{i}^{\prime}=\frac{\hat{z}_{i}}{T_{i}},$$

(3)

where $\hat{z}_{i}$ is the nodewise logits for $v_{i}$ produced by the pretrained classifier and $T_{i}>0$ is the temperature for $v_{i}$ estimated by CaGCN. A noticeable property of such a mechanism is

$$\mathop{{\tt argmax}}_{j\in\mathcal{Y}}\ \hat{z}_{i}^{\prime}=\mathop{{\tt argmax}}_{j\in\mathcal{Y}}\ \hat{z}_{i},$$

(4)

which maintains the prediction accuracy of GNNs after calibration. In this work, we follow the practice and calibrate models in the same manner.

The model calibration task was first proposed in 2017 (Guo et al. 2017). About this problem, works can be roughly classified as post-hoc and training based methods. Post-hoc methods calibrate pretrained nodewise classifiers in ways that preserve predictions, as featured by temperature scaling (TS) (Guo et al. 2017), ensemble temperature scaling (ETS) (Zhang, Kailkhura, and Han 2020), multi-class isotonic regression (IRM) (Zhang, Kailkhura, and Han 2020), spline calibration (Gupta et al. 2020), Dirichlet calibration (Kull et al. 2019), etc. In contrast, instead of transforming logits from a pretrained classifier, training based methods modify either the model architecture or the training process itself. A plethora of methods based on evidential theory (Sensoy, Kaplan, and Kandemir 2018), model ensembling (Lakshminarayanan, Pritzel, and Blundell 2017), adversarial calibration (Tomani and Buettner 2021) and Bayesian approach (Hernández-Lobato and Adams 2015; Wen et al. 2018) belongs to the category. Whereas training based methods provide more flexibility compared to post-hoc ones, a limitation is that they hardly promise accuracy-preserving, thereby requiring careful trade-off between accuracy and calibration performance.

Comparatively, GNN calibration is currently less explored. Teixeira et al.(2019) empirically evaluate the post-hoc model calibration techniques developed for the standard i.i.d. setting on GNN calibration, and show that such a paradigm fails in the task due to an oversight of graph structural information. Afterwards, CaGCN (Wang et al. 2021) produces nodewise temperatures by processing nodewise logits via a graph convolutional network, to account for the graph structure. Additionally, GATS (Hsu et al. 2022) experimentally points out influential factors of GNN calibration and produces nodewise temperatures with an attention-based architecture. Furthermore, Hsu et al.(2022) propose edgewise calibration metrics. Recently, uncertainty quantification is also considered via conformal prediction (Huang et al. 2023; Zargarbashi, Antonelli, and Bojchevski 2023). However, our post-hoc calibration strategy differs from all the previous works with theoretical foundation and similarity-oriented mechanisms.

An intuitive form of feature similarity is raw feature similarity. However, we find with experiments that it aligns badly with GNN predictions, and also the subsequent calibration process, thereby performing suboptimally in GNN calibration. Thus we calculate similarity based on intermediate features from the pretrained GNN classifier $G_{pre}$, which is to be calibrated, because the intermediate features are better clustered and better aligned with GNN predictions (Kipf and Welling 2016). Hereafter we denote the intermediate feature map from the $l$-th layer of $G_{pre}$ as $X_{pre}^{(l)}$.

A naive solution would be to feed $X_{pre}^{(l)}$ directly into another GNN $G_{feat}$ for feature processing, in the hope that $G_{feat}$ implicitly takes feature similarity into account and produces nodewise temperatures $T$:

$$T=G_{feat}(X^{(l)}_{pre},\mathbf{A})\in\mathbb{R}^{|V|}.$$

(14)

Unfortunately, this renders the number of parameters for $G_{feat}$ highly dependent on the number of $X_{pre}^{(l)}$’s dimensions. Specifically, if, for constants $h\ ,h^{\prime}\in\mathbb{N}$, the first layer of $G_{feat}$ projects $X_{pre}^{(l)}$ from $\mathbb{R}^{h}$ to $\mathbb{R}^{h^{\prime}}$, then $G_{feat}$ will contain at least $h\times h^{\prime}$ parameters, which can easily result in overfitting when $h$ gets large. Thus, we desire a feature similarity mechanism that does not rely on input feature dimension or feature semantics.

Motivated by prototypical learning (Nassar et al. 2023; Snell, Swersky, and Zemel 2017), for each class $k$, we first take the average feature as a classwise template,

$$\forall k\in\mathcal{Y},\ \hat{\mu}_{k}:=\frac{1}{|\mathcal{L}_{k}|}\mathop{\sum}_{i:v_{i}\in\mathcal{L}_{k}}x_{i}^{(l)},$$

(15)

where $\mathcal{L}_{k}:=\{v_{i}\in\mathcal{L}|y_{i}=k\}$, and $x_{i}^{(l)}$ is the intermediate feature for $v_{i}$ in $X_{pre}^{(l)}$. Then, we define the similarity between $x_{i}^{(l)}$ and templates with the assistance of a distance measure $d(\cdot,\cdot)$:

$$s_{i}:={\tt sim}(x_{i}^{(l)},\{\hat{\mu}_{k}\}):=\zeta(\{d(x_{i}^{(l)},\hat{\mu}_{k})\}),$$

(16)

where $\zeta(x)=\frac{x}{||x||_{2}}$ normalizes the input vector. Naturally, we consider $s_{i}$, the feature similarity between $x_{i}^{(l)}$ and template features, as a proxy of the similarity between $x_{i}^{(l)}$ and intermediate features of all the nodes from a class. Following Lee et al.(2018), we first compute variance matrix,

$$\hat{\Sigma}:=\frac{1}{|\mathcal{L}|}\mathop{\sum}_{k}\mathop{\sum}_{i\in\mathcal{L}_{k}}(x_{i}^{(l)}-\hat{\mu}_{k})(x_{i}^{(l)}-\hat{\mu}_{k})^{T},$$

(17)

and then induce the Mahalanobis distance (Mahalanobis 2018) accordingly:

$$d(x_{i}^{(l)},\hat{\mu}_{k}):=(x_{i}^{(l)}-\hat{\mu}_{k})^{T}\hat{\Sigma}^{-1}(x_{i}^{(l)}-\hat{\mu}_{k}).$$

(18)

Finally, we feed $\{s_{i}\}$ and $\mathbf{A}$ to $G_{feat}$ to propagate feature-level similarities along the graph structure, i.e.,

$$T_{feat}:=G_{feat}(\{s_{i}\},\mathbf{A})\in\mathbb{R}^{|\mathcal{V}|}$$

(19)

where $T_{feat}$ is the nodewise temperature estimate by feature-level similarity.

Recently, Yan et al.(2022) quantify nodewise representation movement dynamics of GNNs in node classification task, whose three cases of representation movement serve as the foundation of our representation movement similarity-aware mechanism. Hereafter, we provide the necessary backgrounds.

We denote the node degree of $v_{i}$ as $d_{i}$, and the homophily of $v_{i}$ is defined as: $h_{i}:=\mathbb{P}(y_{i}=y_{j}|v_{j}\in\mathcal{N}_{i})$, in which $\mathcal{N}_{i}$ is $v_{i}$’s neighbor set. Finally, the expected relative degree of node $v_{i}$ is $\overline{r_{i}}:=\mathbb{E}_{\mathbf{A}|d_{i}}(\frac{1}{d_{i}}\mathop{\sum}_{j\in\mathcal{N}_{i}}r_{ij}|d_{i})$, where $r_{ij}:=\sqrt{\frac{d_{i}+1}{d_{j}+1}}$. Then, Yan claims that node representation dynamics can be grouped as three cases:

• •

  Case 1: when $h_{i}$ is low, node representations move closer to the representations of the other class, whatever value $\overline{r_{i}}$ takes.

• •

  Case 2: when $h_{i}$ is high but relative degree $\overline{r_{i}}$ is low, node representations still move closer to the other class but not as much as in the first case.

• •

  Case 3: only when both $h_{i}$ and $\overline{r_{i}}$ are high, node representations tend to move away from the other class.

We provide Fig. 2 for an illustration. The existence of such dynamics can easily obscure information of logits and features, e.g. $v_{i}$ of case 1 from class 1 may end up having its logits similar to that of $v_{j}$, which is of case 2 from another class. Therefore, we desire our GNN calibrator to be able to decompose effects from similar nodewise representation movement behavior and nodewise input information, producing better calibrated results. Nonetheless, one difficulty of applying the theorem is the absence of ground-truth labels during training, making $h_{i}$ unaccessible. Worse still, when applied to GNNs, both $h_{i}$ and $\overline{r_{i}}$ are unable to differentiate messages from different neighbors, limiting the calibrator’s model capacity. To circumvent these problems, we approximate homophily by $\hat{z}_{i}\cdot\hat{z}_{j}$. The relative degree information is considered by $\frac{d_{i}+1}{d_{j}+1}$. Furthermore, aware of correlation between ECE and distances to training nodes (Hsu et al. 2022), we estimate nodewise homophily and apply it to graph attention:

$$\alpha_{i,j}:=\mathop{{\tt softmax}}_{j\in\mathcal{N}_{i}}(\sigma(\frac{1}{\eta_{i}\eta_{j}}\hat{z}_{i}\ \cdot\ \hat{z}_{j})),$$

(20)

where $\eta_{i}$ is the distance from $v_{i}$ to the nearest training node, and $\sigma:=LeakyReLU$ (Xu et al. 2015). Then messages from neighbors are weighted and summed:

$$T_{move}={\tt softplus}(\mathop{\sum}_{j\in\mathcal{N}_{i}}\alpha_{i,j}\eta_{j}(\frac{d_{i}+1}{d_{j}+1})^{t}\tilde{z}_{j}^{T}W),$$

(21)

where $t$ is a hyperparameter, $\tilde{z}$ sorts the logits (Rahimi et al. 2020) and $W\in\mathbb{R}^{|\mathcal{Y}|}$ is a trainable parameter modeling the message from $v_{j}$.

It is worth mentioning that representation movement dynamics was initially proven to relate to the oversmoothing problem(Yan et al. 2022), which refers to the problem that node features of GNNs converge towards the same values with the increase of model depth (Rusch, Bronstein, and Mishra 2023). Thus, the proceeding mechanism implies a relationship between oversmoothing and GNN calibration.

We formalize our GNN calibrator, SimCalib, as:

$$\begin{split}\forall v_{i}\in\mathcal{V},\ \hat{p}_{i}^{\prime}&=\omega\cdot{\tt softmax}(\frac{\hat{z}_{i}}{T_{feat}})\\ &\ \ +(1-\omega)\cdot{\tt softmax}(\frac{\hat{z}_{i}}{T_{move}})\end{split}$$

(22)

where $\omega\in(0,1)$ is a hyperparameter balancing feature and representation movement similarities. It is obvious that SimCalib is the composition of order-preserving functions and thus accuracy-preserving. We provide Fig. 1 for illustration.

In the experiments, we apply the commonly used equal-width binning scheme from Guo et al.(2017): for any node subset $\mathcal{N}\subset\mathcal{V}$, samples are regrouped into $M$ equally spaced intervals according to their confidences, formally, $B_{m}:=\{v_{i}\in\mathcal{N}|\frac{m-1}{M}<\hat{c}_{i}\leq\frac{m}{M}\}$, to compute the expected calibration error (ECE) of the GNN:

$${\tt ECE}=\mathop{\sum}^{M}_{m=1}\frac{|B_{m}|}{|\mathcal{N}|}|{\tt acc}(B_{m})-{\tt conf}(B_{m})|,$$

(23)

where $acc(B_{m})$ and $conf(B_{m})$ are defined as:

$$\begin{split}{\tt acc}(B_{m})&=\frac{1}{|B_{m}|}\mathop{\sum}_{i:v_{i}\in B_{m}}\mathbf{1}(y_{i}=\hat{y}_{i}),\\ {\tt conf}(B_{m})&=\frac{1}{|B_{m}|}\mathop{\sum}_{i:v_{i}\in B_{m}}\hat{c}_{i}.\end{split}$$

(24)

To make a fair comparison, the evaluation protocol is mainly adopted from GATS. Specifically, we first train a series of GCNs (Kipf and Welling 2016) and GATs (Veličković et al. 2017) with node classification on eight widely used datasets: Cora (McCallum et al. 2000), Citeseer (Giles, Bollacker, and Lawrence 1998), Pubmed (Sen et al. 2008), CoraFull (Bojchevski and Günnemann 2017), and the four Amazon datasets (Shchur et al. 2018). Then we train calibrators on top of the pretrained nodewise classifiers to evaluate its calibration performance. After training, we evaluate models by ECE with $M=15$ equally sized bins. To reduce the influence of randomness, we randomly assign 15% of nodes as $\mathcal{L}$, and the rest as $\mathcal{U}$, and we repeat this assignment process with randomness five times for each dataset. Once $\mathcal{L}$ has been sampled, we use three-fold cross-validation on it. Also, in each fold we randomly initialize our models five times. Therefore, this results in a total of 75 runs for experiment, the mean and standard deviation of which are finally reported. Full implementation details are presented in the Appendix.

We benchmark SimCalib against a variety of baselines on GNN calibration tasks:

• •

  Temperature scaling(TS) applies a global temperature to scale every nodewise logits.

• •

  Vector scaling(VS) scales and adds a bias to each class in a class-wise manner.

• •

  Ensemble temperature scaling(ETS) softens probabilistic outputs by learning an ensemble of uncalibrated, TS-calibrated and uniform distribution.

• •

  Graph convolution network as a calibration function (CaGCN) uses a GCN to process logits, producing nodewise temperatures.

• •

  Graph attention temperature scaling (GATS) identifies factors that influence GNN calibration, and addresses them with graph attention mechanism.

We also report the ECEs for uncalibrated predictions as a reference. Among the baselines, TS, VS and ETS are designed for standard i.i.d. multi-class classification. CaGCN and GATS propagate information along the graph structure, and produce separate nodewise temperatures.

For all the experiments, the pretrained GNN classifiers will be frozen, and its first-layer feature map, together with the logits, will be fed into our calibration model as inputs. We train calibrators on validation sets by minimizing NLL loss, and validate it on the training set, following the common practice (Wang et al. 2021). We provide details of comparison settings and hyperparameters in Appendix. The calibration results are summarized in Table 1. We also provide the comparisons on adaptive calibration error (ACE) in Table 2.

Overall, SimCalib consistently produces well calibrated results for all the GNN backbones on every dataset. It sets a new SOTA for all experiments, with two exceptions of GAT on Citeseer(2nd best) and GAT on CS(2nd best), on which SimCalib performs worse than GATS by at most 3%. In contrast, SimCalib’s improvements are more statistically significant, reducing average ECE by 10.4% compared to GATS.

With Wilcoxon signed test (Wilcoxon 1992) backed by scipy (Virtanen et al. 2020), we claim that our model is superior to the previous SOTA model, namely, GATS, with confidence 99.9% and $p=7.63\times 10^{-4}$.

Furthermore, we conduct experiments on CoraFull to assess the correlation between calibration improvement and nodewise similarity. We opt for CoraFull because it is the most complicated dataset of the eight, with 19,793 nodes, 126,842 edges, 70 classes and 8710 features, which makes it well representative of the real-world scenario.

In Fig. 4, we visually examine the correlation between feature similarity and calibration improvement by comparing the calibration performance of GATS, an uncalibrated GNN backbone and SimCalib. We group nodes by the Mahalanobis distances to the nearest template representation, and display the group-level ECEs as bars and ECE improvements as dashed lines. The figure shows that the miscalibration issue worsens with the decrease of feature similarity, while our calibration strategy calibrates the nodewise confidence in a consistent way. We believe that with the decrease of feature-level similarity, the samples become more outlying in the feature space, indistinguishable from samples from other classes, and thus it gets harder for the uncalibrated model to accurately tune its confidence in congruence with ground-truth prediction accuracy. In contrast, our model successfully overcomes such ambiguity in feature space by taking feature-level similarity into account, thereby consistenly calibrating samples of various feature similarity with similar expected calibration error. Therefore, this results in stronger improvement for SimCalib when feature similarity gets weaker. Although we discover a similar pattern for GATS, its ECE improvement is weaker than SimCalib when features become dissimilar. We attribute the performance gap to the feature-similarity-aware mechanism. The observation aligns with our theoretical hypothesis in that it suggests that feature similarity indeed plays a critical role in GNN calibration.

In addition, we analyze the data-efficiency and expressivity of SimCalib for GNN calibration. For this, we reuse the GCN classifier pretrained on CoraFull, and compare the expected calibration errors between baselines and SimCalib, with different amounts of calibration data. The results are shown in Fig. 3. From the figure, we draw that SimCalib is both data-efficient and expressive. SimCalib does not require a lot of labels to perform decently, consistenly outperforming all the baselines under all label rates. Moreover, SimCalib also expresses robustness to label rates.

To understand the effects of the two similarity-oriented mechanisms, we conduct a thorough ablation study in this section. The results are shown in Table 3 and Table 4. Overall, each mechanism plays a critical role in GNN calibration and removing any will in general decrease performance while increasing variances.

Effect of feature similarity In order to decompose the effects from number of trainable parameters and feature-similarity mechanism, we investigate the performance of two different models: SimCalib with only representation movement similarity($\text{SimCalib}_{m}$) and SimCalib with $s_{i}$ replaced by $G_{pre}$’s output logits($\text{SimCalib}_{l}$). Comparing $\text{SimCalib}_{m}$ with $\text{SimCalib}_{l}$, we see that the calibration performance slightly improves with more trainable parameters. However, the improvements are rather moderate, unable to match the performance of SimCalib. We hypothesize that since logits information has already been integrated into the calibration process in the nodewise representation movement similarity, adding an extra branch of logits propagation only helps GNN calibration by introducing more parameters.

Effect of representation movement similarity Our representation movement similarity mechanism consists of two aspects of network designs, i.e. homophily term $\hat{z}_{i}\cdot\hat{z}_{j}$ and relative degree $(\frac{d_{j}+1}{d_{i}+1})^{t}$, therefore we design three models to analyze the effects of various components in the representation movement similarity mechanism. Particularly, we test the calibration performance of $\text{SimCalib}_{h}$ that disables relative degree, $\text{SimCalib}_{r}$ that disables homophily, and $\text{SimCalib}_{n}$ in which neither takes effects. We can easily draw from the experiments that the integrity of representation movement similarity mechanism is important to calibration performance and removing of any results in a worsened GNN calibrator. We attribute the performance drop to the inability of the calibrator to decompose effects from nodewise input information and effects from representation movement.