Baseline validation of a bias-mitigated loan screening model based on the European Banking Authority’s trust elements of Big Data & Advanced Analytics applications using Artificial Intelligence

The European Banking Authority (EBA) published in 2020 a report on big data and advanced analytics (BD&AA) where it observed a fast-growing interest in the use BD&AA solutions, with two out of three EU credit institutions already deploying such solutions 1.
EBA identified: i) the four functions of BD&AA applications; ii) the four key pillars of BD&AA applications, and; iii) the eleven fundamental trust elements of BD&AA applications based on Artificial Intelligence.

The European Commission High-Level Expert Group on Artificial Intelligence (AI HLEG) has also prepared a document which elaborates the Guidelines on trustworthy AI used. Machine Learning is a subcategory of AI.

The definitions of non-performing loans (NPLs) and their subcategories adopted by the Bank of Italy have been harmonized within the Single Supervisory Mechanism (SSM) and meet the European Banking Authority (EBA) standards 6, 7,8, 9, 10, 11, 12, 13, 14.
NPLs are exposures to debtors who are no longer able to meet all or part of their contractual obligations because their economic and financial circumstances have deteriorated.
The 3 subcategories of NPLs are:

1. 1.

   Bad loans are exposures to debtors that are insolvent or in substantially similar circumstances.

2. 2.

   Unlikely-to-pay exposures (aside from those included among bad loans) are those in respect of which banks believe the debtors are unlikely to meet their contractual obligations in full unless action such as the enforcement of guarantees is taken.

3. 3.

   Overdrawn and/or past-due exposures (aside from those classified among bad loans and unlikely-to-pay exposures) are those that are overdrawn and/or past-due by more than 90 days and for above a predefined amount.

The goal of our research was to test if 5D, a machine-learning-based loan screening application, could detect a sub-subset of bad loans: 1) in a small subset of non-performing loans (bad loans, unlikely-to-pay, overdrawn and/or past-due exposures), and; 2) in a larger set of performing and non-performing loans subject to the following constraints: a) achieve a sensitivity and specificity not lower than 0.9: b) utilize a maximally relevant, minimally redundant number of publicly available features unrelated to the credit history, gender, race or ethnicity of the borrower (BiMOPT features); 3) comply with the European Banking Authority’s and EU AI HLEG’s Commission’s principles on Big Data & Advanced Analytics based on Artificial Intelligence.

The 5D research project is divided into 4 Phases as shown in Figure 1:

1. 1.

   in Phase 0 we selected a subset of 10 BiMOPT features out of a total of 84 features;

2. 2.

   in Phase I we trained 5D to detect bad loans in a historical dataset extracted from a mandatory report to the Bank of Italy consisting of 7,289 non-performing loans (NPLs) closed in the period 2010-2021;

3. 3.

   in Phase II we assessed the baseline performance of 5D on a distinct validation dataset consisting of an active portolio of 63,763 outstanding loans (performing and non-performing) for a total financed value of over EUR 11.5 billion as of December 31, 2021;

4. 4.

   in Phase III we will monitor the baseline performance for a period of 5 years (2023-27) to assess the prospective real-world bias-mitigation and performance of the 5D system and its utility in credit and fintech institutions.

The timeline of the 5D research project is shown in Figure 2:

The 5D model has been trained and tested on the historical portfolio extracted from a mandatory report to the Bank of Italy and validated on the active portfolio:

• •

  The historical portfolio extracted from a mandatory report to the Bank of Italy consists only of Non-performing Loans (NPLs) closed in the period 2010-2021 as shown in Figure 3: i) NPLs are exposures to debtors who are no longer able to meet all or part of their contractual obligations because their economic and financial circumstances have deteriorated; ii) Bad loans is the most severe subcategory of NPLs consisting of exposures to debtors that are insolvent or in substantially similar circumstances;

• •

  The active portfolio consists of both Performing and Non-performing Loans (NPLs) as of December 31, 2021 as shown in Figure 4.

The historical portfolio (Train-Test Dataset) was balanced versus a highly imbalanced active portfolio (Baseline Validation Dataset) as shown in Figure 5 and Table 1.

All datasets have been anonymized and pseudoanonymized. We removed any references to an identifiable person/entity from all data sets, thus turning personal information into non-personal information. The remaining data have been pseudonymized by masking and noising, so that no person/entity can be identified from the information without reference to additional information. The additional information has been kept separately from the pseudonymized person/entity information and is subject to technical and organizational safeguards (such as access controls) to keep it secure 15, 16.

We performed data-based bias mitigation by 17, 18, 19, 20, 21 removing features related to the credit history, gender, race or ethnicity of the borrower 22, 23 .
We improved the performance of 5D by selecting maximally relevant, minimally redundant features 24, 25, 26.
We referred to these new features as Bias-Mitigated Minimal-Optimal Features (BiMOPT) as shown in Figure 6 and Table 2 .

We trained a scalable tree-based extreme gradient boosting meta-algorithm 27 and assessed its performance and skill on the Train-Test Dataset by calibrating the predicted probabilities 28 and by measuring the Brier Loss 29 and Brier Skill Score (Figure 7) 30.

1. 1.

   The Expected Calibration Error (ECE) curve (reliability diagram) compares how well the probabilistic predictions of the 5D binary classifier are calibrated. It plots the true frequency of the positive label (Bad Loans) against its predicted probability, for binned predictions 31, 32, 28;
   

2. 2.

   The Brier Loss Score (BS) measures the mean squared difference between the predicted probability and the actual outcome. The smaller the Brier score loss, the better, hence the naming with loss 1.

   $$\begin{array}[]{l}BS=\frac{1}{N}\displaystyle\sum_{t=1}^{N}(f_{t}-o_{t})^{2}~~~~~~(0\leq BS\leq 1)\end{array}$$

   (1)

   The Brier Score always takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1). The Brier score is appropriate for binary and categorical outcomes 29;
   

3. 3.

   The quality of the predictions and thus the reliability of probability predictions can be evaluated using the Brier Skill Score (BSS) 2.

   $$\begin{array}[]{l}BSS=1-\frac{BS}{BS_{ref}}~~~~~~(BSS\leq 1)\end{array}$$

   (2)

   Where BS is the Brier Score, BS_ref is the No-skill Brier Score, or the Brier Score resulting by just predicting the positive class (bad loans) based on prevalence, in this case 0.0253.
   The BSS indicates the degree of improvement of the classifier over a classifier with no skill. A skill score value less than zero means that the performance is even worse than that of the baseline or reference predictions 3.