Detection of False-Reading Attacks in the AMI Net-Metering System

Various solutions have been proposed in the literature to detect false-reading attacks in the consumption metering system. While some of these solutions use shallow detectors [1, 7, 6], other solutions use deep learning-based detectors [8, 9, 10].

A few works in the literature have addressed detecting false-reading attacks in the FIT system [2, 3]. Krishna et. al. [2] have proposed different approaches to design customer-specific anomaly detectors based on the auto-regressive integrated moving average (ARIMA) and the Kullback-Leibler divergence (KLD) to detect false-reading attacks in the generation domain of the FIT system. The proposed detectors have been trained only on the benign samples of various datasets including the Ausgrid dataset [11]. Krishna et. al. [2] have proposed a set of attacks, that can maximize the financial profit of the attacker by reporting false readings, to evaluate the performance of the proposed detectors. The lower the financial profit of the attacker, the more robust the detector against false-reading attacks. Unlike [2] that trains customer-specific detectors, Ismail et. al. [3] have trained a general detector using a synthetic dataset. A set of attacks has been proposed to generate malicious samples from the benign samples of the synthetic dataset. Unlike the detectors in [2] which are trained on benign samples only, the detector in [3] is trained on both benign and malicious samples. The proposed detector has a deep learning architecture, and the results indicate that the proposed detector achieves a higher performance than the detectors in [2].

As discussed in the previous two subsections, all the existing works consider only the consumption metering and FIT systems, and detecting false-reading attacks in the net-metering system has not been addressed. The net-metering system is a practical system that is currently adopted in many countries including USA, Italy, and Brazil [4]. Thus, this paper tries to fill the research gap by investigating a deep learning-based detector to detect false-reading attacks in the net-metering system.

Detection of false-reading attacks in the net-metering system is different from the other metering systems for the following reasons. In case of the consumption metering system, the detector can be trained on the consumption pattern of the customer, which depends on his lifestyle, to detect the false readings. Similarly, in the FIT system, the detector can be trained on the generation pattern of the customer’s solar panels to detect the false readings. However, the problem is more complicated in case of the net-metering system because the net meter readings depend on the lifestyle, the solar irradiance, and the generation capacity of the solar panels, i.e., the readings simultaneously include consumption and generation patterns. This means that a new detection approach that considers both the consumption and the generation patterns is needed to be able to detect the false-reading attacks in the net-metering system. What makes the problem more difficult is that to devise a general detector that can be applied for all customers, it needs to be trained on data from different customers who have different lifestyle, type of solar panels, and generation capacity.

Furthermore, new attacks tailored for the net-metering system should be investigated for the following reasons. The attacks against the consumption metering system target reducing the readings while trying to mimic the consumption pattern, and the attacks against the FIT system target increasing the generation readings while trying to mimic the generation pattern. However, in the net-metering system, the attacker needs to consider both the consumption and generation patterns in computing the false readings while achieving financial gains. Moreover, the majority of the research works in the consumption metering system have proposed simple attacks such as reporting zero readings [1, 7, 6, 8, 9, 10] or continuously reporting the same reading during the successive periods [1]. Also, the research works in the FIT system have proposed simple attacks such as reporting readings higher than the generation capacity of the solar panels [2, 3] or reporting generation readings higher than zero after the sunset [3]. Such attacks can be easily detected even without using machine learning techniques. However, in this paper, we avoid these limitations by proposing more sophisticated attacks that consider the generation capacity of the solar panels and try to mimic the consumption and generation patterns.

Fig. 1 shows the different entities in our net-metering system and the interactions between them. At the customer side, there are solar panels installed on the rooftop of the customer’s house. These solar panels convert the energy collected from the sun to direct current (DC) electricity which cannot be used directly to power the house’s appliances or injected to the grid. Therefore, an inverter is used to convert the DC electricity to alternating current (AC) electricity. Then, the AC electricity flows from the inverter to the main service panel that controls feeding the electricity to the house’s appliances. In the net-metering system, the customer’s solar generation system is connected to the utility’s power grid as shown in Fig. 1 so that the excess power generated can be injected directly to the grid through the net meter, and thus, the customer does not need to purchase an expensive solar battery. On the other hand, if the generated power is insufficient, the customer can satisfy his consumption needs by drawing electricity from the power grid through the net meter. Thus, the net meter acts as an interface between the customer and the utility, and it is a bidirectional meter that allows the electricity flow in either direction.

The net meter periodically records readings, which represent the difference between the power consumed by the house appliances and the power generated by the solar panels. Therefore, the reading is positive if the consumed power is more than the generated power, the reading is negative if the generated power is more than the consumed power, and the reading is zero if the consumed power is similar to the generated power. These fine-grained readings are communicated to the utility’s control center through the AMI network via wired communications such as power line communication or wireless communication such as cellular communication [15]. These readings are used by the utility for billing purposes and for demand side management (i.e, achieving a balance between the energy demand and supply).

Given the high installation costs of the solar generation system, customers may be keen to get as high profits as they can from the system even by illegal ways to shorten the time taken to compensate the paid costs. In doing so, malicious customers may launch false-reading attacks by compromising their net meters to report false readings to the utility to achieve financial gains illegally. Specifically, malicious customers can report lower readings when the consumed power is more than the generated power (in case of positive readings) and report higher readings when the generated power is more than the consumed power (in case of negative readings).

SMs can be compromised to report false readings by programming a malicious firmware and installing it in the SM that is accessible through the ANSI optical port [2]. This port is usually secured via weak passwords and there are some tools, such as Terminator, that are used to launch brute force attack to guess the passwords and gain access to the SM [2]. Moreover, recent studies have shown that the AMI communication networks have vulnerabilities, which can be exploited by malicious customers to launch the false-reading attacks [16]. The false reported readings not only cause financial losses to the utility but also can result in wrong decisions regarding energy management. Thus, in this paper, we propose a deep learning-based detector to analyze the readings reported by the customers’ net meters to detect the false-reading attacks.

Deep learning model is a neural network which has multiple hidden layers. Generally, the neural network composes of input, output, and hidden layers [17]. Deep learning is a promising technique to many applications like face recognition [18] and using voice for age identification [19] because of its high accuracy. In this paper, we use different deep learning models to detect false-reading attacks in the net-metering system. This is a classification problem which needs one of the supervised learning methods that use a labeled dataset to train a model. There are various types of supervised learning models including the MLP [20], CNN [21], and recurrent neural network (RNN) [22].

The aim of the training process of a model is to obtain good values for all the model weights and biases. This can be done by defining an objective function and using an optimizer and labeled data samples. First, the input data goes from the first layer in the model through the intermediate layers for a predefined number of iterations. Then, the model’s weights and biases are updated in each iteration in the direction of minimizing the objective function $\Theta$ using feed-forward and back-propagation [23]. The most widely used objective function in the classification problems is categorical cross-entropy $C(y,\hat{y})$, and it measures the loss between the true distribution $y$ and the predicted distribution $\hat{y}$, for $N$ classes as follows:

In our paper, the MLP, CNN, and GRU are used to train the false-reading attacks detector.

FFN is also called MLP [20], and it consists of three types of layers as can be seen in Fig. 2 as follows.

• •

  Input Layer: This is the first layer of an FFN, and it passes the input data to the following layers through nodes, called neurons.

• •

  Output Layer: This is the last layer that is responsible for determining the output (or classification) of the model.

• •

  Hidden Layers: These are the intermediate layers between the input and output layers. Each hidden layer composes of a number of neurons, where each neuron uses an activation function to transform its input values into the output value of that neuron. Every neuron is fully connected to the neurons of the previous layer through a number of connections.

CNN is widely used in image and natural language processing applications [21] because of its capability to extract the important features and capture complex patterns in the input data. As shown in Fig. 3, a CNN model’s architecture includes input, convolution, pooling, fully connected, and output layers. The convolution layer has a number of filters that are used to extract features from the input, and the pooling layer reduces the dimensions of the convolution layer’s output. Convolution and pooling layers are usually followed by one or more fully connected layers that process the features extracted to be used for prediction.

GRU is a type of RNN, which consists of hidden states and connections between the internal units to construct a directed graph as shown in Fig. 4. In each time step $t$, a transition function takes the current time information $X_{t}$ and the previous hidden state ${H}_{t-1}$ to update the current hidden state ${H}_{t}$ as follows.

where ${F}$ is a nonlinear AF, e.g., Tanh. As can be observed in Eq. 2, ${H}_{t-1}$ can be considered as a memory for previous inputs, and thus, GRU can memorize long sequences of input patterns. In GRU, reset and update gates are used to learn which information is important to keep and which information can be discarded. Therefore, GRU has the ability to capture the correlations between the inputs. GRU is widely used in the text generation and speech recognition and synthesis applications [24, 25].

The AF is an important component in the machine learning models since it has a major impact on the model accuracy and convergence speed. Non-linear AFs are usually used because they enable the model to create complex mappings between the inputs and outputs. In the following, we explain two of the AFs used in this paper [26].

• •

  Rectified Linear Unit (ReLU): It uses a simple max function to determine the output of a given input $x$ as follows.

  $$\text{ReLU}(x)=max(0,x)$$

  (3)

• •

  Softmax: It is commonly used in the output layer for classification problems. For a given input vector $\mathbf{z}=[z_{1},\ldots,z_{N}]\in\mathbb{R}^{N}$, the Softmax function is defined as follows.

  $$\text{Softmax}(z_{i})={\frac{e^{z_{i}}}{\sum_{j=1}^{N}e^{z_{j}}}}\ \ for\ \ i=\{1,\dots,N\},$$

  (4)

  where $N$ is the number of classes.

In this paper, we use a publicly available dataset released by Ausgrid, the largest distributor of electricity on Australia’s east coast [11] to prepare our dataset. The Ausgrid dataset contains real power consumption and generation readings for a group of customers who are located in Sydney and regional New South Wales, and have solar panels installed on the rooftops of their homes. These readings are recorded for the period from 1-July-2010 to 30-June-2013. Each customer has two SMs; one SM is used for measuring the power consumption and the other SM is used for measuring the generated power from the solar panels. The Ausgrid dataset contains information about the generation capacity that indicates the maximum amount of electricity generated from the solar panels of each customer per hour ($C_{max}$). The dataset also contains the location of each customer, the category that indicates whether an SM reading is consumption or generation, the date, and the SMs readings at half-hour granularity.

Given the Ausgrid dataset, we apply the following operations to create our benign dataset.

• •

  First, we follow the same methodology of [27] to remove the anomalous measurements from the Ausgrid dataset and produce a clean dataset. This is because some factors such as the failure of the solar generation system can cause anomalies in the dataset.

• •

  Second, for each customer, we subtract the readings of the generation SM from the readings of the consumption SM to obtain net readings. These readings are equivalent to the readings that would be recorded if the two SMs were replaced by a single net meter of the net metering system because the amount of drawn/injected power from/to the utility at any time is equal to the difference between the power consumed and the power generated by the customer at this time.

• •

  Third, from the half-hour granularity dataset, we have created a dataset at one-hour granularity (i.e., 24 readings per day) by aggregating the readings. The reason we selected one-hour granularity is that, the lower the sampling rate, the less likely private information about the customer can be revealed [1]. We will also show later that our detector can detect false-reading attacks with high detection rate at this reduced sampling rate.

Using these operations, we have created a clean dataset for 31 customers with net meter readings at one-hour granularity for 1096 days from 1-July-2010 to 30-June-2013. To get better insights from this dataset, we visualize it. Data visualization is a means that can be used to better understand the dataset by displaying the data in a visual context so that patterns and correlations within the data can be explored.

The net meter readings of four randomly selected customers from the dataset are visualized in Fig. 5. We can observe from the figure that the net meter readings can be positive or negative depending on the direction of the electricity flow between the customer and the utility. More importantly, we can observe that the pattern of the readings of each customer has a periodicity. For example, the shapes of the pattern of each customer over the months (from 9 to 1) are almost the same regardless of the year. We have the same observation over the months (from 1 to 5) and the months (from 5 to 9). It can also be observed that the readings are different between the days within any period. This indicates that the readings depend on the day and the season of the year because both the consumption pattern and the amount of power generated by the solar panels depend on the season and the power consumption also depends on the day.

Moreover, to perceive the inner relation between the consecutive readings of the time series data representing the net meter readings of a customer, we use the autocorrelation function (ACF). The ACF gives the autocorrelation, i.e., the correlation between a time series data and itself at different time lags. Fig. 6 shows the ACFs of the readings of the same customers selected in Fig. 5. The shaded blue areas of Fig. 6 are the 95% confidence intervals used to determine the significance of the autocorrelation at certain time lag. We can see that at least the autocorrelation values at time lags of 1 and 2 for all customers are located outside the blue shaded area, which indicates that there is a significant correlation between each reading and its subsequent two readings within the time series. Furthermore, we can observe from Fig. 6 that the shape of ACF over one day is nearly similar to its shape over the consecutive days for all the customers, which indicates that the net meter readings of any customer have a daily pattern. This pattern and the time correlations between the readings can be learned by the detector to identify the false-reading attacks because deviation from the pattern can be detected as anomaly.

Due to the unavailability of real malicious samples for the net metering system, we propose a set of attacks to mimic the behavior of malicious customers. These attacks are given in Table I. For attackers to achieve financial gains in the net-metering system, they should reduce their reported net readings when the power consumed is more than the power generated (i.e., the readings are positive), and increase the reported readings when the power consumed is less than the power generated (i.e., the readings are negative). Further, the proposed attacks can be classified into intermittent and continuous. In intermittent attacks, the attacker reports false readings at some time slots, and reports the true readings at other time slots aiming to confuse the detector, and in continuous attacks, the attacker reports false readings all the time aiming to achieve high profit.

Under the intermittent attacks, we propose attack #1 in which the attacker cheats during a random time interval starts at $t_{s}$ and ends at $t_{e}$, and otherwise he reports the true readings. During the cheating interval, the attacker reports a scaled-down version of the current true reading ($\text{TR}_{t}$) by a time-dependant factor $b_{t}$ at the time slots of positive readings, while reports the higher value between a large percentage ($p_{t}$) of the maximum solar generation capacity ($C_{max}$) and the absolute value of the current true reading ($\mid\!\text{TR}_{t}\!\mid$) at the time slots of negative readings.

Under the continuous attacks, we propose three attacks that are either scaling-based or history-based. In the scaling-based attacks, the attacker scales positive readings down and scales negative readings up without considering the values of previous readings. However, in the history-based attack, the attacker uses the previous readings to compute the false reading. In attack #2, the attacker cheats by always reporting a scaled-down version of $\text{TR}_{t}$ by $\alpha$ when readings are positive, while reporting a scaled-up version of $\text{TR}_{t}$ by $\beta$ when readings are negative, where $0\leq\alpha<1$ and $\beta>1$. Note that attack #2 considers that the attacker’s reported reading does not exceed $C_{max}$ and this is denoted by $-\min(\mid\!\!\beta*\text{TR}_{t}\!\!\mid,C_{max})$ in Table I. Attack #3 is also a scaling-based attack, but unlike attack #2, both the scaling down and the scaling up parameters ($\alpha$ and $\beta$) are time-dependent.

Finally, attack #4 is history-based in which the attacker cheats by reporting the minimum value between $\text{TR}_{t}$ and last reported positive reading (PR) when readings are positive, and reporting the maximum value between $\text{TR}_{t}$ and last reported negative reading (NR) when readings are negative. Note that the factors $M_{1_{t}}$ and $M_{2_{t}}$ in attack #4 are not scaling factors but they act as masks to avoid reporting the same exact reading in multiple time slots to confuse the detector; where the value of $M_{1_{t}}$ is a little bit less than one, while the value of $M_{2_{t}}$ is a little bit more than one.

According to [28], the amount of power generated from a solar panel depends on both the solar irradiance and the temperature. Moreover, $C_{max}$ of each customer can be calculated by the utility based on the number of solar panels and their characteristics recorded in the contract between the customer and the utility. Therefore, in addition to using the readings reported by the net meter to detect the false-reading attacks, relevant data from trustworthy sources, including the solar irradiance, temperature, and $C_{max}$, can be used by the detector because they give indication about the power generation pattern that can be used to verify the net meter readings. The rationale here is that although an attacker can compromise his net meter to report false readings, he cannot manipulate the solar irradiance, temperature, and $C_{max}$ because they are beyond his control. Thus, the true data from the trustworthy sources can help the detector to identify the false-reading attacks.

The Ausgrid dataset contains $C_{max}$ of each customer but it does not contain information about the solar irradiance and temperature. However, they can be obtained from SOLCAST [12] using the locations of customers given in Ausgrid. SOLCAST is a website that can provide the weather information including the solar irradiance and temperature of any location in the world at any date given the longitude and the latitude of this location. Thus, given the customers’ locations, we have found the longitudes and latitudes of these locations to obtain the solar irradiance and temperature at these locations during the period from 1-July-2010 to 30-June-2013 via SOLCAST.

Given the time series data representing the net meter readings of a certain customer and the time series data representing the corresponding values of the solar irradiance, the correlation between the two time series data is visualized in Fig. 7 for the same customers selected in Figs. 5 and 6. The scatter plots of Fig. 7 indicate a negative correlation between the net meter readings and the solar irradiance. This is because the higher the irradiance, the higher the generated power by the solar panels and thus the lower the reported net reading. Furthermore, the correlation values given in each subplot of Fig. 7 indicate the significance of this correlation for all customers. Similarly, we have used a similar approach but with using the temperature and found also a negative correlation between the net meter readings and the temperature. Therefore, if the readings reported by the net meter are true, there should be correlations between the readings and the values of the irradiance and temperature, otherwise, the detector can decide that the reported readings are false.

Given the dataset of benign net meter readings of 31 customers for 1096 days, the readings of each day are treated as a benign sample for a total of 33,976 (31*1096) benign samples. Then, the four proposed attacks are used to create four malicious samples from each benign sample. After that, our dataset of benign and malicious samples is extended by including more data. Specifically, each sample in the extended dataset consists of 75 values representing 24 fine-grained net meter readings, the corresponding 24 fine-grained irradiance values, the corresponding 24 fine-grained temperature values, $C_{max}$, the day, and the season. The dataset that is further divided into training and test sets with a ratio of 2:1, respectively. After that, the training and test sets are normalized to bring all the features’ values to a common scale to guarantee the fair contributions of all the features towards the classification of the detector. This step is also useful to help the detectors that use the Gradient Descent optimization to converge faster [29]. Finally, given that the number of malicious samples is four times the number of benign samples, the adaptive synthetic (ADASYN) sampling approach [30] is used to balance the training set by over-sampling the minority class of benign samples to avoid biasing the trained model towards the majority class of malicious samples.

Machine learning. Among the existing detectors of false-reading attacks in the AMI network, machine learning-based detectors outperform the other detectors that are based on state estimation and game theory [31].
Deep learning. The machine learning-based detectors can use either shallow classifiers such as DT, RF and SVM or deep learning architectures such as the CNN and the RNN [8]. However, the results of recent studies have indicated that the deep learning architectures can accurately detect false-reading attacks in the AMI network better than the shallow classifiers [8, 9, 10, 3].
General detector. The detector can be either customer-specific, where a customized detector is trained for each customer or a general detector that can be used for all customers. Unlike general detectors, customer-specific detectors require collecting historical metering readings for each customer to train them, and thus, they cannot be used to detect false-reading attacks launched by new customers until enough readings are collected. Besides, customer-specific detectors are vulnerable to contamination attacks, where a new customer reports false readings from the beginning. If the detector is trained on this data, the customer can continue reporting false data without being detected [31].
Data correlation. Based on the data analysis provided in Section V-A, there are time correlations between the consecutive readings within the benign samples of any customer. Thus, it is important for the detector to be able to learn these correlations so that it can identify false-reading attacks if these correlations are not found in the tested sample.
Multi-source data. Based on the data analysis provided in Section V-C, there are correlations between the time-series data representing the true net meter readings and the time-series data obtained from trustworthy sources such as the solar irradiance and temperature that are always true because they are beyond the control of customers. Thus, it is good for the detector to learn these correlations so that it can identify false-reading attacks if these correlations are not found in the tested sample of net meter readings and the corresponding values of the solar irradiance and temperature. Moreover, we have shown in Section V that data such as the day, the season, and $C_{max}$ are also important to be considered because the net meter readings depend on the day and the season, and $C_{max}$ can help the detector to perceive the limits of the reported readings.

Based on the above discussion, our detector shall have the following characteristics. It shall be a general deep learning-based detector trained on the data collected from all customers to be used for any customer in the system including the new customers. Moreover, the deep learning architecture of the detector needs to capture the correlation within the net meter readings. Finally, instead of designing a single-data-source detector that detects false-reading attacks based solely on the readings reported by the net meter, our detector shall be a multi-data-source detector that is trained on relevant data from various trustworthy sources in addition to the net meter readings to enhance the detection performance.

Our detector is designed to have six different types of data (i.e., data from six different sources) as shown in Fig. 8. The first one is the fine-grained net meter readings of one day. The second and third input data are the fine-grained irradiance and temperature values in the same day, respectively. The rest of the input data are the values of $C_{max}$, the day , and the season. Moreover, we have designed our detector in three stages as shown in Fig. 8 to enhance the detection performance of our detector by considering more input data at each stage. These stages are described as follows.

• •

  Stage 1 considers only one type of input data, which is the net meter readings, and it has a hybrid CNN & GRU architecture. Since the time-series data can be considered as 1-dimensional (1-D) data, the 1-D CNN architecture can be used due to its capability to extract the features from the 1-D data and hence leading to a high detection performance. Moreover, since the time-series data representing the net meter readings has correlations between the consecutive readings, the GRU architecture can be used due to its capability to capture the time correlation in the data. Thus, we have chosen a hybrid CNN & GRU architecture for Stage 1 so that the CNN layers can extract the features from the input net meter readings while the GRU layers can capture the correlation between the extracted features.

• •

  Stage 2 considers three input data by feeding it with the output of Stage 1, the irradiance, and the temperature. Stage 2 has a set of GRU layers to help the detector to capture the correlation between the net meter readings and the corresponding values of the irradiance and the temperature.

• •

  Stage 3 takes all the six input data into consideration by feeding it with the output of Stage 2, $C_{max}$, the day and the season. Stage 3 has a set of dense layers to help the detector to take complex decisions regarding the input samples by considering the effect of important features like $C_{max}$, the day and the season on the output of Stage 2 that represents the net meter reading, the irradiance, and the temperature.

In order to assess the impact of considering more input data on the detection performance, in the evaluation we get an output from each stage as shown in Fig. 8, but when the detector is used by the utility, the output should be obtained from Stage 3 only.

In this subsection, we describe the details of our experimental environment in terms of software and hardware, and also the evaluation metrics used to assess the performance of our detector.

Various Python 3 libraries are used in our work as follows. Specifically, Pandas and Numpy are used in data preparation, while Matplotlib [32], Statsmodels [33], and Seaborn [34] are used in data visualization. To train the detectors and optimize the hyper-parameters, Keras Functional API [35] and Hyperopt [36] are used, respectively. Finally, Sklearn [37] is used for evaluating the performance of the detector. All the experiments are run on the high-performance cluster of the Tennessee Technological University using two NVIDIA Tesla K80 GPUs.

The following metrics are considered to evaluate the performance of our detector.

• •

  Accuracy ($ACC$). It measures the percentage of the correctly classified samples to the total number of tested samples, and it is calculated as follows.

  $$ACC(\%)=\frac{TP+TN}{TP+TN+FP+FN}\times 100,$$

  where, $TP$ is the number of true positive samples (i.e, the correctly classified malicious samples), $TN$ is the number of true negative samples (i.e, the correctly classified benign samples), $FP$ is the number of false positive samples (i.e, the misclassified benign samples) and $FN$ is the number of false negative samples (i.e, the misclassified malicious samples).

• •

  Precision ($PR$). It measures the percentage of the correctly classified positive samples to the total number of samples classified as positive and it is calculated as follows.

  $$PR(\%)=\frac{TP}{TP+FP}\times 100$$

• •

  Detection rate ($DR$). It measures the percentage of the correctly classified positive samples to the total number of real positive samples and it is calculated as follows.

  $$DR(\%)=\frac{TP}{TP+FN}\times 100$$

• •

  False Alarm ($FA$). It measures the percentage of misclassified negative samples to the total number of real negative samples and it is calculated as follows.

  $$FA(\%)=\frac{FP}{FP+TN}\times 100$$

• •

  Highest difference ($HD$). It measures the difference between $DR$ and $FA$.

  $$HD(\%)=DR(\%)-FA(\%)$$

• •

  F1-score ($F1$). It is the harmonic mean between $PR$ and $DR$ and it is calculated as follows.

  $$F1(\%)=\frac{2*PR*DR}{PR+DR}\times 100,$$

• •

  Receiver operating characteristic (ROC) curve. It is a graphical representation of the relation between $TP$ rate and $FP$ rate at different decision thresholds. The higher the area under the ROC curve (AUC-ROC), the higher the detector performance.

• •

  Precision-recall (P-R) curve. It is a graphical representation of the relation between $PR$ and recall at different decision thresholds. The higher the area under the P-R curve (AUC-P-R), the higher the detector performance.

In this subsection, we investigate four possible deep learning architectures, namely, MLP, GRU, CNN, and hybrid CNN & GRU, to determine the one that provides the best performance by considering only the readings reported by the customers’ net meters. The MLP is firstly investigated since it is the simplest deep learning architecture, and then iw will be used as a baseline for assessing the performance of the other deep learning architectures. To do a fair investigation, the dataset prepared in Section V is used to train four different detectors with the aforementioned architectures, and Hyperopt is used to optimize the hyper-parameters including the number of layers, the number of units per layer, and the AF used in each layer. The optimal hyper-parameters of these detectors are given in Tables II-V.

Results and Discussion. Table VI gives a comparison between the performance of the four detectors in terms of $ACC$, $PR$, $DR$, $FA$, $HD$, and $F1$. First, we can see that while the MLP-based detector has the lowest computational complexity among all detectors, it achieves the lowest performance. Second, it can be observed that the GRU-based detector provides a better performance compared to the MLP-based detector, which makes sense because the GRU layers are capable of capturing the correlation between the inputs and it has been proved in Section V-A that there is a temporal correlation between the consecutive net meter readings. Third, it can be observed that the CNN-based detector provides a better performance compared to the MLP-based detector, which makes sense because the CNN layers provide the detector with a better feature extraction capability. Overall, the hybrid CNN & GRU-based detector achieves the best performance among all detectors because the CNN layers can extract the features from the input readings while the GRU layers can capture the correlation between the extracted features. Thus, the hyprid CNN & GRU architecture is chosen to design our multi-data-source detector.

In this subsection, we provide a comparison between the outputs of Stages 1, 2, and 3 of our detector shown in Fig. 8 to evaluate the benefit of considering multiple data sources. To train our detector, the dataset prepared in Section V is used as follows. Stage 1 is trained using only the net meter readings, While Stage 2 is trained using the readings besides the solar irradiance and temperature. Finally, Stage 3 is trained using all data included in our dataset. In addition, Hyperopt is used to optimize the hyper-parameters of the different stages of our detector, and the optimal hyper-parameters are given in Table VII.

Results and Discussion. Table VIII gives a comparison between the performance of the three stages in terms of $ACC$, $PR$, $DR$, $FA$, $HD$ and $F1$. In addition, Figs. 9 and 10 visualize the difference in the performance between the three stages using ROC curves and P-R curves, respectively. It can be clearly concluded from the results given in Table VIII that Stage 3 achieves the best performance among all stages. This can also be observed in Figs. 9 and 10 because Stage 3 has the biggest AUC-ROC and AUC-P-R, respectively among all stages.

The superiority of Stage 2 over Stage 1 is due to the capability of Stage 2 to successfully capture the correlations between the net meter readings and the corresponding values of the solar irradiance and temperature. We have discussed in Section V-C that these correlations are significant when the reported readings are true. Thus, if a customer maliciously manipulates his readings, this can affect the correlations between the net meter readings and the corresponding values of the irradiance and temperature. For a malicious customer to fully preserve these correlations, he has to manipulate the solar irradiance and temperature values the same way he manipulates the net meter readings. However, this is impossible because the irradiance and temperature values are not reported by him. This means that Stage 2 has additional features that help in differentiating between the benign and malicious samples by checking the correlations between the net meter readings and the corresponding values of the irradiance and temperature, which results in higher $DR$, lower $FA$, and higher $HD$.

The superiority of Stage 3 over Stages 1 and 2 is because Stage 3 takes into consideration additional important features, which allows the detector to make a more complex classification boundary between the benign and malicious samples. This results in further increase in $DR$, decrease in $FA$, and increase in $HD$. Finally, we can observe from Table VIII that the use of multiple data sources improves the $HD$ from $91.83\%$ to $95.66\%$ (i.e., about $4\%$ more increase in the $HD$ compared to using only the net meter readings).