A big data-based ensemble for fault prediction in electrical secondary distribution network

Abstract

The introduction of smart meters, sensors and integrated electronic devices in the electrical secondary distribution network (ESDN) has led to the collection of massive amounts of data. Accurate prediction of faults from this data can help to improve the reliability, safety and operational efficiency of ESDN. Due to its complexity, ESDN big data are hard to process and manage using traditional technologies and tools. The difficulties posed by dataset complexity arise from issues including high dimensionality, imbalance and variability, and one current challenge is to address them simultaneously. Currently, the capability of fault prediction techniques to address this challenge remains limited. New approaches are needed to overcome it. To this purpose, this article presents a big data-based ensemble for fault prediction in ESDN (BDEFP-ESDN) on Apache Spark with gradient-boosted trees, random forest, decision tree and binomial logistic regression base models. BDEFP-ESDN is optimized for the complexity of the ESDN dataset by dimension reduction, bootstrap sampling and hyperparameter optimization approaches in the training process and a weighted voting approach in the prediction process. Our experimental results illustrate the efficiency of BDEFP-ESDN against traditional classifiers like ANN, SVM, RF and XGB, achieving an accuracy of 99.6% for both binary and multiclass classification.

Keywords:

• Ensemble algorithm
• classification algorithms
• fault prediction
• electrical secondary distribution network
• big data

Reviewing Editor:

• Qingsong Ai, Senior Editor, Wuhan University of Technology, CHINA

Subjects:

• Systems & Computer Architecture
• Parallel Systems
• Electrical & Electronic Engineering
• Intelligent Systems
• Systems & Controls
• Systems & Control Engineering
• Machine Learning

Introduction

Generally, the electrical secondary distribution network (ESDN) is an electrical power distribution system configuration designed to deliver power to final consumers (Chen et al., 2012). One of the dominant challenges utility companies face for the management and control functions in ESDN is the process of fault detection, identification and clearance. Some of the leading causes of this challenge include highly expanding distribution network due to increasing power demand and aging facilities (Amin, 2011). As such, accurately predicting faults before they occur in ESDN is one of the important measures in addressing this challenge.

Fault prediction analyzes power system historical data to discover abnormal operating condition patterns to forecast future failures so that appropriate measures are taken to optimize the lifespan of power system equipment and facilitate system recovery (Zhang et al., 2018). The advantages of fault prediction in power distribution systems are numerous such as reducing the maintenance time and cost, prolonged life of power system assets and increased safety and reliability. Therefore, there is an increasing interest amongst researchers and practitioners to effectively use fault prediction mechanisms in the management and control functions in ESDN.

Many machine learning classification algorithms such as support vector machine (SVM), artificial neural networks (ANNs) and random forests (RF) have been employed for fault prediction in power distribution systems as single models. Nevertheless, some studies have demonstrated that combining a collection of individual models in an ensemble algorithm can outperform single models (Dietterich, 2000; Sagi & Rokach, 2018; Zhou, 2021). While ensemble algorithms are accepted to yield the best results, the question of how to weigh each model, as well as the inherent complexity of ESDN datasets, create issues in ESDN fault prediction systems.

First, ESDN datasets may be high-dimensional. With high-dimensional datasets, the performance of some algorithms that perform well with low-dimension datasets tend to deteriorate with the increase of dimensions in the datasets (Thudumu et al., 2020). Second, ESDN datasets are typically imbalanced. This state occurs when one or more classes of the training datasets have a minority number of data points. This situation affects the algorithm’s accuracy because a data point from the minority class may be classified as a data point from the majority class (Weiss & Provost, 2001).

Another issue is caused by the introduction of smart meters, sensors, smart appliances and integrated electronic devices (IEDs) resulting in ESDN big data. This data are hard to process and manage within a tolerable operating time and hardware resources using traditional technologies and tools due to its high volume, high velocity, complexity and variability (Jacobs, 2009; Kezunovic et al., 2013). In addition, the base learners provide default hyperparameter values that are sub-optimal for all data values that can be used for predictions in general cases. In some instances such as with ESDN datasets, this may lead to reducing the prediction performance, increasing the complexity of the learning model and increasing the risk of overfitting (Probst et al., 2019).

The issues associated with ESDN dataset complexity hamper the performance and accuracy of machine learning fault prediction algorithms. To effectively and accurately predict faults in ESDN, all issues associated with the complexity of ESDN datasets need to be addressed simultaneously (Della Giustina et al., 2014). Several studies (see next section for a review) have proposed various methods to perform fault prediction in the power distribution system. These methods, however, have limited capability in simultaneously addressing all issues associated with ESDN datasets.

This article proposes a machine learning approach for creating an efficient supervised classification ensemble algorithm to simultaneously deal with the aforementioned issues. The proposed method combines the individual predictions of the base classifiers using a weighted voting strategy based on their performances. To handle the complexity of ESDN datasets, such as high dimensionality, imbalance and variability, the classification ensemble algorithm is enhanced with dimension reduction, bootstrap sampling and hyperparameter optimization. To implement the classification ensemble algorithm, a big data processing framework Apache Spark is applied for parallel and distributed processing. To evaluate the proposed ensemble algorithm, numerical experiments on the ESDN dataset have been conducted to demonstrate its performance.

Related work

Several methods have been put forth by researchers to perform fault prediction in the power distribution system. More specifically, various methodologies have been proposed based on classification algorithms aiming at monitoring the network’s health condition and the different assets to predict abnormalities in the power distribution system. This section presents a brief overview of some valuable outcomes of the proposed methodologies.

Roland and Eseosa (2015) proposed a predictive model for predicting incipient transformer faults in the distribution network. Their predictive model applies ANN on dissolved gas analysis (DGA) data to predict incipient transformer faults. Along that line, Sayar and Yüksel (2020) proposed an algorithm for predicting power outages in the electrical distribution network. The proposed algorithm was trained using ANN on hourly outages and meteorological data. The model was able to derive insights on how power system failures and network health conditions are affected by meteorological conditions with 70.59% accuracy. However, both studies have not addressed the issues of high dimensionality, imbalance and variability inherent in the ESDN dataset.

Huang et al. (2020) focused on the prediction of faults in the distribution network by having a fair amount of historical datasets available from distribution network management and monitoring systems. Their proposed model deals with different types of faults in the network by utilizing SVM. Based on their numerical experiments, it was established that the proposed model achieved a higher performance compared to ANN and C 4.5 decision tree models. Wang et al. (2021) extended the previous work and proposed a prediction system for cascade failures in the smart grid. Their extensive numerical experiments on real-time data concluded that the proposed method can efficiently predict cascade failures with accuracy close to 100%. However, SVM may suffer bottlenecks when applied to raw big data in ESDN (Qiu et al., 2016).

Lin et al. (2019) utilized a voted RF (VRF) classifier and developed an efficient fault prediction model for a smart grid distribution system. Their numerical experiments on distribution network faults logs and meteorological data demonstrated that robust classification ensemble models could be developed by modifying the implementation of available ensemble algorithms by adapting more efficient voting methodologies. Furthermore, their results show that the proposed model outperformed ANN, RF and SVM. Though better performance has been reported, VRF still cannot handle the high dimensionality, imbalance and variability of ESDN dataset (Qiu et al., 2016).

Cai et al. (2020) recommended a predictive system for feeder fault prediction in a power distribution network based on XGBoost. During the numerical experimentation process on datasets collected over 17 years from Fujian Electric Power in China, the proposed method attained an AUC of 0.8899, indicating that the proposed system is valid and efficient. However, although XGBoost can handle imbalanced data well (Le et al., 2022), its capability on data sets with high dimensionality and variability remains limited (Ma et al., 2021).

Stefenon et al. (2021) evaluated the classification efficacy of a hybrid time series model for fault prediction in distribution insulators. The proposed model was implemented using the wavelet technique and long short-term memory (LSTM). The authors presented the results illustrating that wavelet LSTM outperformed non-linear auto-regressive (NAR) and NAR with eXogenous input (NARX).

Hou et al. (2021) focused on a case study from typhoon ‘Mijiage’ of 2015 to predict power outage faults caused by typhoon disasters. They built their model based on RF using a dataset with 14 features from multiple sources, including geographical data, power grid data and meteorological data. Their numerical experiments illustrated the proposed model’s efficiency for predicting power faults achieving an accuracy of up to 92.44%. However, RF cannot perform well with high dimensional, imbalanced and variable ESDN dataset (Lin et al., 2019; Qiu et al., 2016).

Yang (2021) considered the problem of multi-label fault prediction for the distribution system by proposing a predictive model based on RF and multi-classification SVM (MSVM). RF is used to calculate and sort feature weights to obtain the optimal fault feature set, and then, MSVM is used to predict the fault level. The numerical experiments on original data from the distribution network coupled with meteorological data indicated a practical application value in the proposed method, with accuracy reaching 95%. However, while the study tackles the issue of high dimensionality, it does not address the imbalance and variability issues of ESDN dataset.

Materials and methods

The objective of this study was to propose an approach for making reliable fault predictions in ESDN using an ensemble algorithm on the parallel and distributed big data processing framework Apache Spark. This study describes the results obtained from aggregated ESDN dataset comprising data recorded using automatic meter reading (AMR) and the corresponding weather data of temperature and rainfall. The dataset contains 105118 instances with a resolution of 20 min collected from January 2015 to December 2018. Table 1 displays the description of the ESDN dataset.

Table 1. Description of ESDN dataset.

S/N	Attribute	Description
1	Timestamp	Timestamp measurements were recorded
2	VA	Voltage measurement of Phase A
3	VB	Voltage measurement of Phase B
4	VC	Voltage measurement of Phase C
5	CA	Current measurement of Phase A
6	CB	Current measurement of Phase B
7	CC	Current measurement of Phase C
8	KVAh	Apparent power measurement
9	KVArh	Reactive power measurement
10	KWh	Active power measurement
11	Temperature	Temperature measurement
12	Precipitation	Rainfall measurement
13	Fault	Target variable

Figure 1 depicts the main process of the construction of BDEFP-ESDN that uses four base predictors, namely, gradient boosted trees (GBT), RF, decision tree (DT) and binomial logistic regression (BLR) from Spark MLlib (Assefi et al., 2017). Van der Laan et al. (2007) argue that combining individual classification algorithms leads to optimal predictions.

Figure 1. Illustration of BDEFP-ESDN construction.

Data preparation and feature normalization

First, the collected SDN measurement data files in text format and weather data files in Excel format were imported into the Jupyter Lab notebook. Next, the data were extracted from the data files into dataFrames. Then we aggregated current, voltage, power, temperature and rainfall dataFrames into a single dataFrame. The missing rainfall and temperature data points were replaced using the method of linear interpolation.

Thereafter, the datasets were labeled for training the prediction model. The labels were added to the datasets using the sliding window approach based on whether or not a fault would appear in the next twenty minutes. The datasets were labeled as ‘1’ if a fault is likely to occur in the next twenty minutes and as ‘0’ if a fault is not likely to occur in the next twenty minutes. Figure 2 illustrates the sliding window approach used in the dataset labeling process.

Figure 2. Sliding window labeling.

After data preparation, feature normalization was performed to standardize the dataset so that the values of all features fall within a particular range to avoid the features with large value range to dominate those with smaller ones. The standard score Z is calculated using Z-Score normalization presented in Equation (1)(1) $$Z=\frac{x-\mu }{\sigma }$$(1) . (1) $$Z=\frac{x-\mu }{\sigma }$$(1) where $x$ is the scaled feature sample, $\mu$ is the mean of the training feature samples and $\sigma$ is the standard deviation of the training feature samples.

Dimension reduction

Dimension reduction is performed by selecting the most important input feature variables by measuring the importance of each input feature variable using its gain ratio. First, the information entropy of the target feature variable in the subset $S_i \left(i=1,2,\dots ,n\right)$ of the training dataset $S$ is calculated using Equation (2)(2) $$\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)=\sum _{a=1}^b-p_a{\mathrm{\text{log}}}_2{p}_a$$(2) . (2) $$\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)=\sum _{a=1}^b-p_a{\mathrm{\text{log}}}_2{p}_a$$(2) where $b$ is the number of values in the range of the target variable in $S_i$ and $p_a$ is the probability of each type of value $a$ for all types in $S_i.$

Then, the information entropy of each input feature variable $y_{\mathit{\text{ij}}}$ in the subset $S_i$ apart from the target variable is calculated using Equation (3)(3) $$\mathrm{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)=\sum _{v\in V\left(y_{\mathit{\text{ij}}}\right)}^b\frac{\left|S_{\left(v,i\right)}\right|}{\left|S_i\right|}\mathrm{\text{Entropy}}\left(v\left(y_{\mathit{\text{ij}}}\right)\right)$$(3) . (3) $$\mathrm{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)=\sum _{v\in V\left(y_{\mathit{\text{ij}}}\right)}^b\frac{\left|S_{\left(v,i\right)}\right|}{\left|S_i\right|}\mathrm{\text{Entropy}}\left(v\left(y_{\mathit{\text{ij}}}\right)\right)$$(3) where $y_{\mathit{\text{ij}}}$ is the j-th input feature variable ($j=1,2,\dots ,M$) of the subset $S_i,$ $b$ is the number of different values of $y_{\mathit{\text{ij}}},$ $V\left(y_{\mathit{\text{ij}}}\right)$ is the set of all possible values of $y_{\mathit{\text{ij}}},$ $S_{\left(v,i\right)}$ is a subset containing sampled data points from $S_i$ where the value of $y_j$ is $v,$ $\left|S_{\left(v,i\right)}\right|$ is the number of samples in $S_{\left(v,i\right)}$ and $\left|S_i\right|$ is the number of samples in the subset $S_i.$

At this point, the information gain $G\left(y_{\mathit{\text{ij}}}\right)$ for $y_{\mathit{\text{ij}}}$ in the subset $S_i$ of the training dataset $S$ can be calculated using Equation (4)(4) $$\begin{array}{c}G\left(y_{\mathit{\text{ij}}}\right)=\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)‐\mathrm{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)\\ =\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)-\sum _{v\in V\left(y_{\mathit{\text{ij}}}\right)}^b\frac{\left|S_{\left(v,i\right)}\right|}{\left|S_i\right|}\mathrm{\text{Entropy}}\left(v\left(y_{\mathit{\text{ij}}}\right)\right)\end{array}$$(4) . (4) $$\begin{array}{c}G\left(y_{\mathit{\text{ij}}}\right)=\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)‐\mathrm{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)\\ =\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)-\sum _{v\in V\left(y_{\mathit{\text{ij}}}\right)}^b\frac{\left|S_{\left(v,i\right)}\right|}{\left|S_i\right|}\mathrm{\text{Entropy}}\left(v\left(y_{\mathit{\text{ij}}}\right)\right)\end{array}$$(4)

Next, the split information $I\left(y_{\mathit{\text{ij}}}\right)$ of each input feature variable in the subset $S_i$ of the training dataset $S$ is computed in Equation (5)(5) $$I\mathrm{ }\left(y_{\mathit{\text{ij}}}\right)=\sum _{a=1}^b-p_{\left(a,j\right)}{\mathrm{\text{log}}}_2\left(p_{\left(a,j\right)}\right)$$(5) . (5) $$I\mathrm{ }\left(y_{\mathit{\text{ij}}}\right)=\sum _{a=1}^b-p_{\left(a,j\right)}{\mathrm{\text{log}}}_2\left(p_{\left(a,j\right)}\right)$$(5) where $b$ is the number of values in the range of the input variable in $y_{\mathit{\text{ij}}}$ and $p_{\left(a,j\right)}$ is the probability of each type of value $a$ for all types in $y_{\mathit{\text{ij}}}.$

After that, the gain ratio $\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)$ of each input feature variable in the subset $S_i$ of the training dataset $S$ is obtained using Equation (6)(6) $$\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)=\frac{G\left(y_{\mathit{\text{ij}}}\right)}{I\left(y_{\mathit{\text{ij}}}\right)}$$(6) . (6) $$\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)=\frac{G\left(y_{\mathit{\text{ij}}}\right)}{I\left(y_{\mathit{\text{ij}}}\right)}$$(6)

Finally, the importance of the input feature variable $\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right)$ is computed using Equation (7)(7) $$\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right)=\frac{\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)}{{\sum }_{a=1}^M\mathit{\text{GR}}\left(y_{\left(i,a\right)}\right)}$$(7) . It measures a score for each input feature variable $y_{\mathit{\text{ij}}}$ in the subset $S_i$ of the training dataset $S$ that indicates the relative importance or relevance of each input feature variable. (7) $$\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right)=\frac{\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)}{{\sum }_{a=1}^M\mathit{\text{GR}}\left(y_{\left(i,a\right)}\right)}$$(7)

The algorithm then sorts the input feature variables in descending order by their importance values. Then, the top $p$ features are selected as the most important ones $(p<m).$ Further $(m-p)$ input feature variables are selected randomly, resulting in $m$ selected input feature variables in total. As a result, the number of features will be reduced from $M$ to $m.$ Algorithm 1 presents the steps of the dimension reduction process.

Algorithm 1:

Dimension Reduction

Input:

$S_i;$

$p;$

$m;$

Output:

$\mathit{\text{FI}};$ // $\mathit{\text{FI}}$ is the set of $m$ features of $S_i$

BEGIN

1. Initialize $F_i,$ ${\mathit{\text{GR}}}_i,$ ${\mathit{\text{GRT}}}_i,$ $V_i;$ //${\mathit{\text{GR}}}_i$ is the set of gain ratio for all input feature variables in $S_i,$ ${\mathit{\text{GRT}}}_i$ is the total gain ratio (${\sum }_{a=1}^M\mathit{\text{GR}}\left(y_{\left(i,a\right)}\right)$) for all input feature variables in $S_i,$ $V_i$ is the set of pairs of $y_{\mathit{\text{ij}}}$ and their importance score for all input variables in $S_i$

2. Standardize the training subset $S_i;$

3. Calculate $\mathit{\text{Entropy}} \left(S_i\right)$ using Equation (2)(2) $$\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)=\sum _{a=1}^b-p_a{\mathrm{\text{log}}}_2{p}_a$$(2) ;

4. FOREACH $y_{\mathit{\text{ij}} }\in S_i;$

5. Calculate $\mathit{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)$ using Equation (3)(3) $$\mathrm{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)=\sum _{v\in V\left(y_{\mathit{\text{ij}}}\right)}^b\frac{\left|S_{\left(v,i\right)}\right|}{\left|S_i\right|}\mathrm{\text{Entropy}}\left(v\left(y_{\mathit{\text{ij}}}\right)\right)$$(3) ;

6. Calculate $G\left(y_{\mathit{\text{ij}}}\right)$ using Equation (4)(4) $$\begin{array}{c}G\left(y_{\mathit{\text{ij}}}\right)=\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)‐\mathrm{\text{Entropy}}\left(y_{\mathit{\text{ij}}}\right)\\ =\mathrm{\text{Entropy}}\mathrm{ }\left(S_i\right)-\sum _{v\in V\left(y_{\mathit{\text{ij}}}\right)}^b\frac{\left|S_{\left(v,i\right)}\right|}{\left|S_i\right|}\mathrm{\text{Entropy}}\left(v\left(y_{\mathit{\text{ij}}}\right)\right)\end{array}$$(4) ;

7. Calculate $I \left(y_{\mathit{\text{ij}}}\right)$ using Equation (5)(5) $$I\mathrm{ }\left(y_{\mathit{\text{ij}}}\right)=\sum _{a=1}^b-p_{\left(a,j\right)}{\mathrm{\text{log}}}_2\left(p_{\left(a,j\right)}\right)$$(5)

8. Calculate $\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)$ using Equation (6)(6) $$\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)=\frac{G\left(y_{\mathit{\text{ij}}}\right)}{I\left(y_{\mathit{\text{ij}}}\right)}$$(6) ;

9. ${\mathit{\text{GR}}}_i\leftarrow \mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right);$

10. ${\mathit{\text{GRT}}}_i\leftarrow {\mathit{\text{GRT}}}_i+\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right);$

11. END

12. FOREACH $y_{\mathit{\text{ij}} }\in S_i;$

13. Calculate $\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right)$ using Equation (7)(7) $$\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right)=\frac{\mathit{\text{GR}}\left(y_{\mathit{\text{ij}}}\right)}{{\sum }_{a=1}^M\mathit{\text{GR}}\left(y_{\left(i,a\right)}\right)}$$(7) ;

14. $V_i\leftarrow \left(y_{\mathit{\text{ij}}},\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right)\right);$

15. END

16. Sort $V_i$ in descending order by $\mathit{\text{VI}}\left(y_{\mathit{\text{ij}}}\right);$

17. Select top $p$ feature variables from $V_i$ to $\mathit{\text{FI}};$

18. WHILE count($\mathit{\text{FI}}$) < $m$ DO

19. Push $y_{\mathit{\text{ij}}}$ randomly from $V_i\left(M-p\right)$ to $\mathit{\text{FI}};$

20. END

21. RETURN $\mathit{\text{FI}};$

END

Bootstrap sampling

In order to enhance the accuracy of the ensemble on imbalanced ESDN datasets, an improved bootstrap sampling algorithm has been designed based on the stationary block bootstrap resampling procedure proposed by Politis and Romano (1994). The bootstrap sampling algorithm creates the set $B$ of all bootstrap samples $\left\{S_1,S_2,\dots S_b\right\}$ from the training dataset $S$ with $n$ data points as shown in Algorithm 2. Symbol $g$ represents the geometric distribution value that controls the typical size of the blocks sampled and $b$ is the number of bootstrap samples.

Each bootstrap sample $S_{\mathrm{l}}\in B$ is obtained by initially picking the first data point $x_k\in S (k=1)$ for the first sampled data point of a pseudo series value $x_k^{*}\in S_{\mathrm{l}}\mathrm{ }\left(l=1,2,\dots ,b\right).$ Next, a discrete uniform distribution variable $d$ is chosen randomly in the range of 0 and 1. With probability $1-g$ (that is when $d>g$) the choice for the index $i \left(i=1,2,\dots ,n\right)$ of the next sampled data point from $S$ is $k+1$ otherwise an index $e$ of a data point in $S$ is selected at random. Then, $x_i\in S$ is selected to be the next sampled data point for a pseudo series value $x_j^{*}\in S_{\mathrm{l}}\mathrm{ }\left(j=2,3,\dots ,n\right).$

Algorithm 2:

Bootstrap Sampling

Input:

$S;$

$g;$

$b;$

Output:

$B;$

BEGIN

1. FOR $l=1,\dots ,b$ DO

2. $k=1;$ //$k$ is current index of a sampled data point from $S$

3. Initialize $S_l$ with $x_k^{*}=x_k;$

4. FOR $j=2,\dots ,\mathit{\text{COUNT}}(S)$ DO

5.  Choose $d$ at random;

6.  IF $d>g$ THEN

7.   $i=k+1;$

8.  ELSE

9.   Choose $e$ at random;

10. $\hspace{1em}\hspace{1em}\hspace{1em}i=e;$

11.  END

12. $\hspace{1em}\hspace{1em}{x}_j^{*}=x_i;$

13.  $S_l\leftarrow S_l\cup x_j^{*};$

14.  $k=i;$

15. END

16. $B\leftarrow {B\cup S}_l;$

17. END

18. RETURN $B;$

END

Weighted voting

The voting weights for the base predictors are computed based on their local performance. After training, each base predictor $F_{i },$ from the trained model $F$ is evaluated using the testing data set $T,$ each instance of $T$ is predicted by all predictors in the model. Then, the results of each predictor are aggregated by voting a final result for each instance of the training data set. The local performance ${\mathit{\text{PP}}}_{i }$ of each predictor $F_{i }$ is measured in terms of the classification accuracy (CA) as defined in Equation (8)(8) $$\begin{array}{c}{\mathit{\text{PP}}}_{i\mathrm{ }}={\mathit{\text{CA}}}_{i\mathrm{ }}\\ =\frac{I_L\left(F_i\left(T\right)=c\right)}{I_L\left(F_i\left(T\right)=c\right)+\sum I_L\left(F_i\left(T\right)=e\right)}\end{array}$$(8) (8) $$\begin{array}{c}{\mathit{\text{PP}}}_{i\mathrm{ }}={\mathit{\text{CA}}}_{i\mathrm{ }}\\ =\frac{I_L\left(F_i\left(T\right)=c\right)}{I_L\left(F_i\left(T\right)=c\right)+\sum I_L\left(F_i\left(T\right)=e\right)}\end{array}$$(8) where $T$ is the testing dataset, $c\in L$ is the value in the correct class, $e\in L$ is the value in the error class and $I_L$ is the indicator function for the prediction of the classifier $F_{i }$ on instances of $T.$

After obtaining ${\mathit{\text{PP}}}_{i }$a set of trained models and their corresponding prediction performance $\mathit{\text{FE}}\mathrm{ }\left\{(F_1,{\mathit{\text{PP}}}_1),(F_2,{\mathit{\text{PP}}}_2),\dots (F_n,{\mathit{\text{PP}}}_n)\right\}$ is created. Next, the normalized classification weights ${\mathit{\text{nw}}}_i,$ are calculated using Equation (9)(9) $${\mathit{\text{nw}}}_i=\frac{{\mathit{\text{PP}}}_i}{\sum _{k=1}^n{\mathit{\text{PP}}}_k}$$(9) . (9) $${\mathit{\text{nw}}}_i=\frac{{\mathit{\text{PP}}}_i}{\sum _{k=1}^n{\mathit{\text{PP}}}_k}$$(9)

The final classification result of the $n$ base predictors of the ensemble is the majority vote of the classification results. The weighted classification result $P_c(t_i),$ is defined in Equation (10)(10) $$P_c\left(t_i\right)=\sum _{j=1}^n{\mathit{\text{nw}}}_j\times F_j\left(t_i\right)$$(10) . (10) $$P_c\left(t_i\right)=\sum _{j=1}^n{\mathit{\text{nw}}}_j\times F_j\left(t_i\right)$$(10)

In the prediction phase, this weighted voting mechanism increases the influence of the base predictors with better performance compared to those with poor performance. The steps of the weighted voting scheme in the designed algorithm are presented in Algorithm 3.

Algorithm 3:

Weighted Voting

Input:

$T;$

$F;$

Output:

$P(T);$ //prediction result for T

BEGIN

1. Initialize $\mathit{\text{FE}},$ $\mathit{\text{FW}};$ //FE model with evaluation, FW model with weight

2. FOREACH $F_{i\mathrm{ }}\in F$ // $F_i$ is the ith Predictor of $F$

3. Apply $F_i$ on testing data set $T;$

5. Compute ${\mathit{\text{PP}}}_i$ using Equation (8)(8) $$\begin{array}{c}{\mathit{\text{PP}}}_{i\mathrm{ }}={\mathit{\text{CA}}}_{i\mathrm{ }}\\ =\frac{I_L\left(F_i\left(T\right)=c\right)}{I_L\left(F_i\left(T\right)=c\right)+\sum I_L\left(F_i\left(T\right)=e\right)}\end{array}$$(8) ;

9. $\hspace{1em}\mathit{\text{FE}}\leftarrow \left(F_i,{\mathit{\text{PP}}}_i\right);$

10. END

11. Sort $\mathit{\text{FE}}$ in descending order by $\mathit{\text{PP}};$

12. FOREACH $t_{i\mathrm{ }}\in T$

13. FOREACH ${\mathit{\text{FE}}}_{j\mathrm{ }}\in \mathit{\text{FE}}$

14.  Apply $F_j$ on $t_i;$

16.  Compute ${\mathit{\text{nw}}}_j$ using Equation (9)(9) $${\mathit{\text{nw}}}_i=\frac{{\mathit{\text{PP}}}_i}{\sum _{k=1}^n{\mathit{\text{PP}}}_k}$$(9) ;

20. $\hspace{1em}\mathit{\text{FW}}\leftarrow \left(F_j,{\mathit{\text{nw}}}_j\right);$

21. END

23. Vote the final prediction $P_c\left(t_i\right)$ using Equation (10)(10) $$P_c\left(t_i\right)=\sum _{j=1}^n{\mathit{\text{nw}}}_j\times F_j\left(t_i\right)$$(10) ;

24. $\hspace{1em}P(T)\leftarrow P_c(t_i);$

29. END

30. RETURN $P(T);$

END

Hyperparameter optimization

The best combination of the values for the hyperparameters was determined by the use of Bayesian Optimization. The prediction model was trained and evaluated using each combination of the hyperparameters $H\left\{h_1,h_2,h_2,\dots h_n\right\}.$ The best prediction model was identified based on the combination of the hyperparameter values with the best performance. The objective function $f$ to evaluate the combination of hyperparameter values is defined in Equation (11)(11) $${\mathit{\text{PP}}}_{i\mathrm{ }}=f\left(h_i\right)$$(11) . It uses the set of hyperparameters $h_i$ to calculate the score that determines the performance ${\mathit{\text{PP}}}_{i }$ of the model on the testing dataset using $h_i.$ (11) $${\mathit{\text{PP}}}_{i\mathrm{ }}=f\left(h_i\right)$$(11)

The performance ${\mathit{\text{PP}}}_{i }$ of the prediction model $F_{i }$ with hyperparameter values $h_{i }$ and the set of unique classification labels $L$ is determined by the CA as defined in Equation (8)(8) $$\begin{array}{c}{\mathit{\text{PP}}}_{i\mathrm{ }}={\mathit{\text{CA}}}_{i\mathrm{ }}\\ =\frac{I_L\left(F_i\left(T\right)=c\right)}{I_L\left(F_i\left(T\right)=c\right)+\sum I_L\left(F_i\left(T\right)=e\right)}\end{array}$$(8) . Clearly, the best prediction model is the one which produces the highest CA. We then maximize the objective function using Equation (12)(12) $$h^{\mathrm{*}}=\underset{h_i\in H}{\mathit{\text{argmax}}}f\left(h_i\right)$$(12) to obtain the optimal set of hyperparameters $h^{*}.$ (12) $$h^{\mathrm{*}}=\underset{h_i\in H}{\mathit{\text{argmax}}}f\left(h_i\right)$$(12)

Because the number of hyperparameter combinations is usually large, it becomes difficult and costly to calculate the objective function score on each of the hyperparameter combinations. Bayesian optimization aims at reducing the number of times the objective function is computed by applying a surrogate model. The surrogate model calculates the conditional probability of the objective function to propose hyperparameter sets that are likely to improve the performance score. The surrogate model is presented in Equation (13)(13) $$h_i=\underset{h\in H}{\mathit{\text{argmax}}}{A}_i\left(h;S_{i-1}\right)$$(13) . (13) $$h_i=\underset{h\in H}{\mathit{\text{argmax}}}{A}_i\left(h;S_{i-1}\right)$$(13) where $S$ is the Gaussian process (GP) surrogate model and $A$ is the acquisition function based on expected improvement (EI).

The steps of the hyperparameter optimization process are presented in Algorithm 4. First, a surrogate probability model of the objective function is initialized. Next, for each iteration, we find the surrogate model’s best-performing combination of hyperparameters $h_i,$ where $A_i$ is maximized. Then, we obtain the best-performing combination of hyperparameters using the true objective function. After that, the set of hyperparameters combination and their corresponding score is augmented to the set of observation history $\mathit{\text{OH}},$ of other samples. Finally, the surrogate probability model is re-trained using the latest history of samples. After the maximum number of iterations is exhausted, the latest true objective function’s best-performing combination of hyperparameters is the optimal one.

Algorithm 4:

Hyperparameters Optimization

Input:

$H;$

$Q;$ //maximum number of iterations

Output:

$\mathit{\text{OP}};$ //Optimal hyperparameters with performance score

BEGIN

1. Initialize $h_0\in H$ randomly;

2. Evaluate $f\left(h_0\right)$ using Equation (8)(8) $$\begin{array}{c}{\mathit{\text{PP}}}_{i\mathrm{ }}={\mathit{\text{CA}}}_{i\mathrm{ }}\\ =\frac{I_L\left(F_i\left(T\right)=c\right)}{I_L\left(F_i\left(T\right)=c\right)+\sum I_L\left(F_i\left(T\right)=e\right)}\end{array}$$(8) ;

3. Initialize $\mathit{\text{OH}}\leftarrow \left(h_0,f\left(h_0\right)\right);$ //$\mathrm{ }\mathit{\text{OH}}$ is observation history

4. Initialize $h^{*}\leftarrow h_0;$

5. Initialize $f^{*}\leftarrow f\left(h_0\right);$ //$f^{*}$ is performance score of model with $h^{*}$

6. FOR $i=1,\dots ,Q$ DO

7. Select $h_i\in H$ using Equation (13)(13) $$h_i=\underset{h\in H}{\mathit{\text{argmax}}}{A}_i\left(h;S_{i-1}\right)$$(13) ;

8. Evaluate $f\left(h_i\right)$ using Equation (8)(8) $$\begin{array}{c}{\mathit{\text{PP}}}_{i\mathrm{ }}={\mathit{\text{CA}}}_{i\mathrm{ }}\\ =\frac{I_L\left(F_i\left(T\right)=c\right)}{I_L\left(F_i\left(T\right)=c\right)+\sum I_L\left(F_i\left(T\right)=e\right)}\end{array}$$(8) ;

9. IF $\mathit{f}\left({\mathit{h}}_{\mathit{i}}\right)>{\mathit{f}}^{\mathit{*}}$ THEN

10. $f^{*}\leftarrow f\left(h_i\right);$

11. $h^{*}\leftarrow h_i;$

12. END

13. $\mathit{\text{OH}}\leftarrow \mathit{\text{OH}}\cup \left(h_i,f\left(h_i\right)\right);$

14. Fit a new model $S_i$ to $\mathit{\text{OH}};$

15. END

16. $\mathit{\text{OP}}\leftarrow \left(h^{*},f^{*}\right);$

17. RETURN $\mathit{\text{OP}};$

END

Model integration

The integrated BDEFP-ESDN is presented in Algorithm 5. The input dataset is preprocessed and divided into training and testing datasets. Then, a dimension reduction mechanism is performed on the preprocessed data before the bootstrap samples are generated. During the training phase, the ensemble model is constructed using the determined set of optimal hyperparameters and the relevant features obtained during dimension reduction. Finally, the final prediction result is obtained by combining the results of the ensemble predictors using the calculated voting weights.

Algorithm 5:

BDEFP-ESDN

Input:

$D;$ //Historical dataset

$p;$ //Number of selected important features

$m;$ //Number of selected relevant input feature variables

$b;$ //Number of bootstrap samples

$k;$ //Number of nearest neighbors

$n;$ // Minimum number of neighbors of the majority set from the minority set

$E;$// Set of predictors $\left\{E_1,E_2\dots E_n\right\}$ which constitute the ensemble

$H;$ //Hyperparameter search space

$Q;$ //Maximum number of iterations

Output:

$P(T);$ //Prediction result

BEGIN

1. Preprocess the data from $D;$

2. Split $D$ into training dataset $S$ and testing dataset $T;$

3. Select relevant features from $S,$ $\mathit{\text{IF}}\leftarrow \mathit{\text{dimensionReduction}}\left(S,p,m\right);$

4. Create the bootstrap samples from $S,$ $B\leftarrow \mathit{\text{bootstrapSampling}}\left(S,b,k,n\right);$

5. Initialize the empty set $F$ of trained predictors of the ensemble

6. FOREACH $E_{i\mathrm{ }}\in E$

7. Determine the set of optimal hyperparameters, $\mathit{\text{OP}}\leftarrow \mathit{\text{hyperOpt}}\left(H,\mathrm{Q}\right);$

8. Train $E_{i }$ with bootstrap sample $S_{i }$ and a combination of hyperparameters $h^{*};$

9. $F_i=\mathit{\text{Trained}} E_{i };$

10. $F\leftarrow F\cup F_i;$

11. END

12. Perform weighted voting prediction, $P\left(T\right)\leftarrow \mathit{\text{weightedVoting}}\left(T,F\right);$

13. RETURN $P(T);$

END

Performance evaluation

The metrics used in the evaluation of the performance of BDEFP-ESDN are accuracy, recall, precision and F 1-Score, as presented in Equations (14–17). These metrics come from the four fundamental parameters of all the classification results namely, true positive (TP), false positive (FP), true negative (TN) and false negative (FN). TP indicates that the positive label is predicted positive, while TN means the negative label is predicted negative. On the other hand, FP shows that the negative label is predicted positive, and FN shows that the positive label is predicted negative. (14) $$\text{Accuracy =}\frac{\text{TP + TN}}{\text{TP + TN + FP + FN}}$$(14) (15) $$\text{Precision =}\frac{\text{TP}}{\text{TP + FP}}$$(15) (16) $$\text{Recall =}\frac{\text{TP}}{\text{TP + FN}}$$(16) (17) $$\text{F 1‐Score =}\frac{\text{2 x Precision x Recall}}{\text{Precision + Recall}}$$(17)

Other metrics are the area under the precision-recall curve (AUPR) as well as the area under the receiver operator characteristic curve (AUROC). AUROC provides a performance measure that shows the TPs against the FPs. Unlike accuracy, AUROC is threshold independent because it implicitly compares the base error rates between classifiers. Thus, AUROC is considered more reliable in the presence of high false-positive rates (Ling et al., 2003; Sibona & Brickey, 2012). AUPR is another threshold-independent metric obtained from the plot of precision against recall. In the case of the highly imbalanced ESDN dataset, AUPR is said to be more effective and informative than AUROC (Saito & Rehmsmeier, 2015).

Results and discussion

This section presents the discussion on the results produced after applying BDEFP-ESDN with the optimization methods presented in the previous sections on the ESDN dataset using the Apache Spark platform. First, we discuss how BDEFP-ESDN was optimized with dimension reduction, bootstrap sampling and how the optimal hyperparameter sets were obtained. Finally, we discuss the comparison of the classification performance of BDEFP-ESDN against methods used in previous works.

Dimension reduction

The performance analysis of BDEFP-ESDN with dimension reduction is presented in this section. The number of input features in the ESDN dataset is 11, as shown in Table 1. Both binary and multiclass classification models were generated with two to all input feature variables and the models with the highest performance were selected.

For the binary classification problem, the prediction model with 11 features produced the best AUPR (0.991235). This model was selected as the best prediction model for binary classification fault prediction. The performance of the designed algorithm with dimension reduction for binary classification is presented in Figure 3.

Figure 3. Performance of binary classification prediction with different number of features.

For multiclass classification, the best performance in terms of accuracy (0.979934) was produced by the prediction model with both 11 features and a combination of features producing the best performance from the individual base models (DT: 10 features; MLR: 03 features; OVR-GBT: 11; RF: 07 features). Hence, the model with 11 features was selected as the best prediction model for multiclass classification fault prediction. The performance of the designed algorithm with dimension reduction for multiclass classification is presented in Figure 4.

Figure 4. Performance of multiclass classification prediction with different number of features.

Bootstrap sampling

This section presents the performance analysis of BDEFP-ESDN with bootstrap sampling. Using the stationary block bootstrap sampling, the optimal geometric distributions are considered to effectively sample block lengths that greatly influence the balancing of minority class datasets in the generated bootstrap samples. In turn, the reduced imbalance in the generated bootstrap samples enhances the performance of classification models used in fault prediction.

Aimed at finding the optimal geometric distributions for ESDN dataset, each classification model is constructed with geometric distribution from 0.111111 to 0.999999. Figure 5 shows the performance of BDEFP-ESDN with bootstrap sampling for binary classification.

Figure 5. Performance of binary classification prediction with different values of geometric distribution.

According to Figure 4, the prediction model with a geometric distribution value of 0.888888 gets the highest performance (Accuracy = 0.98989) and is selected as the optimal geometric distribution for binary classification of the ESDN dataset.

The performance of BDEFP-ESDN with bootstrap sampling for multiclass classification is illustrated in Figure 6. The best prediction model selected for multiclass classification of the ESDN dataset was the one with a geometric distribution value of 0.222222. This model produced the highest performance in terms of accuracy (0.97963315).

Figure 6. Performance of multiclass classification prediction with different values of geometric distribution.

Hyperparameter optimization

This section presents the findings of the critical hyperparameters for generating optimal ensembles from BDEFP-ESDN for ESDN. The strategy to determine the best set of hyperparameters involved the application of Bayesian Optimization. The best prediction model with highest accuracy for multiclass classification and the highest AUPR for binary classification was selected. Table 2 shows the combinations of hyperparameters used for tuning and optimizing the performance of BDEFP-ESDN for both binary and multiclass classification problems.

Table 2. Classification optimal hyperparameters.

Base model	Hyperparameter	Hyperparameter value
		Binary class	Multi-class
GBT	maxDepth	7	7
	maxBins	29	29
	maxIter	10	10
DT	Impurity	gini	gini
	maxDepth	10	10
	maxBins	28	18
RF	numTrees	5	5
	Impurity	gini	entropy
	maxDepth	7	10
	maxBins	27	25
LR	maxIter	39	10
	regParam	0.0	0. 06314125237829517
	elasticNetParam	0.4289238962347097	0. 159655399819757

Result comparison

In this section, we discuss the comparison in CA of the proposed algorithm against some traditional classifiers used in previous works: ANN, SVM, RF and XGB. Both binary and multiclass classification results are presented to assess the proposed approach’s effectiveness and validate the results obtained from BDEFP-ESDN.

Tables 3 and Table 4 present the performance of the proposed algorithm BDEFP-ESDN against ANN, SVM, RF and XGB for binary and multiclass classification, respectively. The highest classification performance for each metric is highlighted in bold. The results show that BDEFP-ESDN is the most efficient algorithm since it demonstrates the best overall classification performance for both binary and multiclass classification.

Table 3. Binary classification results. The values in bold indicate best performance.

Algorithm	Accuracy	Precision	Recall	F1 Score	AUROC	AUPR
ANN	0.954837	0.956818	0.954837	0.952802	0.896859	0.812532
SVM	0.944326	0.947708	0.944326	0.943935	0.923139	0.788706
RF	0.991326	0.991489	0.991326	0.991294	0.987815	0.967879
XGB	0.992217	0.992201	0.992217	0.992146	0.993134	0.985629
BDEFP- ESDN	0.99617	0.996307	0.99617	0.996136	0.994248	0.990143

Table 4. Multiclass classification results. The values in bold indicate best performance.

Algorithm	Accuracy	Precision	Recall	F1 score
ANN	0.962181	0.960846	0.962181	0.959833
SVM	0.94374	0.943075	0.94374	0.940686
RF	0.978657	0.977438	0.978657	0.977834
XGB	0.990753	0.990627	0.990753	0.990491
BDEFP-ESDN	0.995586	0.995801	0.995586	0.995569

Figures 7 and 8 show the graphical representation of the binary classification results using the ROC curve and PR curve, respectively. With the ROC curve, best-performing algorithms tend to produce curves nearer to the top-left corner and those closer to the dashed diagonal of the ROC space indicate poor performance. Then again, algorithms producing curves nearer to the top-right corner in the PR curve diagram indicate better performance. According to Figures 7 and 8, BDEFP-ESDN outperforms ANN, SVM, RF and XGB.

Figure 7. ROC curves for binary classification performance results.

Figure 8. PR curves for binary classification performance results.

Conclusions and future work

With high dimensional, imbalanced and variable ESDN datasets, the performance of some algorithms tends to deteriorate and increase the complexity and risk of over fitting the learning model. As far as the authors are aware, no comprehensive work was dedicated to addressing these issues. This paper proposes a big data-based ensemble for fault prediction in an ESDN to simultaneously tackle these issues associated with the inherent complexity of ESDN datasets and weighting of base models. The proposed algorithm was implemented on the parallel and distributed big data framework Apache Spark using a weighted vote approach on DT, RF, GBT and LR base models. To address the ESDN dataset complexity issues, the algorithm is optimized through dimension reduction, bootstrap sampling and hyperparameter optimization.

The experimental results on the ESDN dataset collected from AMR showed that the proposed algorithm is more suitable for fault prediction in ESDN than traditional classifiers like ANN, SVM, RF and XGB. For future work, we will focus on extending the algorithm to cater for regression problems in forecasting ESDN measurements associated with the predicted faults. Additionally, it will be of interest to explore the proposed approach in different areas of big data predictive modeling other than ESDN fault prediction.

Acknowledgment

This research was carried out as part of the iGrid-Project at the University of Dar es Salaam (UDSM) under the Swedish International Development Agency (SIDA) sponsorship. The authors also appreciate TANESCO for the collaboration provided during the research.

Disclosure statement

No conflict of interest.

Additional information

Notes on contributors

David T. Makota

David Makota is an assistant lecturer and consultant in the Department of Computer Science, at the Institute of Finance Management (IFM), in Dar es Salaam, Tanzania. He is pursuing his doctoral degree in Computer and IT Systems Engineering at the University of Dar es Salaam. His research interests include machine learning, big data, knowledge sharing, smart grid and database systems.