Flood susceptibility mapping using ANNs: a case study in model generalization and accuracy from Ontario, Canada

Abstract

Accurate flood susceptibility mapping (FSM) is critical for mitigating the environmental, social and economic consequences of floods. The influence of model generalizability onto new watersheds, and the impact of arbitrarily selecting a small subset of flooded and nonflooded locations are current major knowledge gaps in FSM research impacting predictive accuracy. As such, this study conducts an assessment of machine learning models – (i) an Artificial Neural Network - Synthetic Minority Oversampling Technique (ANN-SMOTE) hybrid ensemble with (ii) knowledge-based Analytical Hierarchy Process (AHP) and (iii) diversity-based Shannon Entropy approaches. The ANN-SMOTE, AHP and Entropy models were trained and tested on the Don River watershed in Ontario, Canada, with Overall Accuracy (OA) results of 0.549, 0.404 and 0.452, respectively. ANN-SMOTE’s predictive accuracy remained high when it was tested on four independent watersheds from southern Ontario, indicating strong generalization ability. To simulate the commonly used flood point inventory approach, the number of training samples was reduced by a factor of a 1000, which resulted in a 28% decrease in accuracy. The high performance and generalization potential of the ANN-SMOTE model demonstrate its utility and versatility for future FSM studies, and as a support tool in flood risk management decision making.

Keywords:

• SMOTE
• AHP
• entropy
• floodplain
• GIS

1. Introduction

Floods, the most frequent and severe of global natural disasters, inflict significant social, environmental and fiscal impacts, including the loss of human life, damage to natural habitats and damage to infrastructure (Rincón et al. 2018; Yang et al. 2018). Factors contributing to floods include rapid urbanization, overflowing of river channels, snow melt and the increased frequency of extreme rain events due to global climate change (Hu et al. 2018; Rincón et al. 2018; Cabrera and Lee 2020). Flood susceptibility mapping (FSM) is a process of identifying flood vulnerable areas based on their physical characteristics. As such, FSM facilitates risk management decision making, assists in implementation of flood mitigation and protection strategies, and provides guidance towards landcover/land use development in flood vulnerable areas (Vojtek and Vojteková 2019; Bui et al. 2020; Wang et al. 2020).

Existing FSM models can be broadly categorized as either physically based or empirical (Giovannettone et al. 2018; Mudashiru et al. 2021a). Physically based models predict flood susceptibility by using simplified representations of complex physical and natural processes. To do so, a large number of hydrological, meteorological and geophysical inputs are required to parametrize models that are, then, run under different storm scenarios, resulting in a time consuming and iterative process (Giovannettone et al. 2018; Choubin et al. 2019; Mudashiru et al. 2021a). Additionally, larger scale applications (e.g. regional) suffer from significant computational time, and validating physically based models requires high-resolution datasets which may not always be available (Hong et al. 2018; Liu et al. 2022; Lin et al. 2023).

Empirical models, which rely on observational and data-driven relationships between the input and output, can further be subdivided into two categories: quantitative and qualitative. Quantitative models form mathematical relationships between inputs (commonly referred to as flood conditioning factors, FCFs) and the output (e.g. flood susceptibility) (Mudashiru et al. 2021b). Examples of quantitative methods include: (i) statistical methods such as Logistic Regression (LR) (Al-Juaidi et al. 2018; Ali et al. 2020; Pham et al. 2020), Frequency Ratio (FR) (Tehrany and Kumar 2018; Wang et al. 2021), Weights of Evidence (WoE) (Rahmati et al. 2015; Costache, Pham, et al. 2020) and Shannon Entropy (Khosravi et al. 2016; Mahmoody Vanolya and Jelokhani-Niaraki 2021) amongst others; and (ii) Machine Learning (ML) models such as Artificial Neural Networks (ANN) (Zhao et al. 2018; Jahangir et al. 2019; Janizadeh et al. 2019; Andaryani et al. 2021), Convolutional Neural Networks (CNN) (Wang et al. 2020; Zhao et al. 2020), Support Vector Machines (SVM) (Liu et al. 2022), Random Forests (RF) (Abedi et al. 2021; Farhadi et al. 2021). Conversely, qualitative models rely on expert opinion to weigh an area’s physical characteristics (FCFs) according to the degree of perceived influence on flood susceptibility (Hong et al. 2018). The most commonly used qualitative model for FSM is the knowledge-based Analytical Hierarchy Process (AHP) (Rincón et al. 2018; Hammami et al. 2019; Vojtek and Vojteková 2019).

AHP integrates several competing criteria such as environmental, economical and social factors, as well as has the ability to produce FSMs in areas with scarce flood inventory data (Gudiyangada Nachappa et al. 2020; Vojtek et al. 2021). AHP prioritizes expert knowledge in ranking FCFs, which can vary the most influential FCF from study to study. This uncertainty due to the subjective nature of the decision-making scheme is a common limitation in AHP studies (Hammami et al. 2019; Souissi et al. 2019; Das 2020).

To minimize this uncertainty from the qualitative nature of AHP, a quantitative statistical approach such as Entropy, can be used as an alternative. Entropy models calculate the degree of diversity within each FCF input, and rank more spatially diverse FCFs as more influential to flood susceptibility (Khosravi et al. 2016; Mahmoody Vanolya and Jelokhani-Niaraki 2021). One of the most significant limitations of a diversity approach is the lack of consideration given to the interdependencies between the input and output (Malekinezhad et al. 2021), leading to potential inaccuracies. This limitation is addressed by using data-driven objective ML approaches, where weighted connections between the inputs and output are formed, to predict flood susceptibility. In particular, ANNs have demonstrated ability to handle complex nonlinear relationships, making them a commonly used data-driven approach in FSM.

ANNs are widely used in many hydrological forecasting applications such as streamflow predictions (Cheng et al. 2020; Snieder et al. 2020), rainfall-runoff simulations (Asadi et al. 2019; Kim and Han 2020; Vidyarthi et al. 2020), low impact development design optimization (Latifi et al. 2019; Raei et al. 2019) and FSM (Zhao et al. 2018; Jahangir et al. 2019; Janizadeh et al. 2019; Andaryani et al. 2021). In a study of ANN’s comparative performance against other ML methods including self-organizing map (SOM) and fuzzy adaptive resonance theory (FART), the ANN produced the FSM with the greatest accuracy in a river basin in Iran (Andaryani et al. 2021). Recently, advanced forms of ANNs, known as deep learning neural networks (DLNNs), have risen in popularity in FSM studies (Chen et al. 2021). Chen et al. (2021) compared the performance of these methods, such as CNNs to shallow ANNs, and their results showed that the increased complexity of CNNs yields detrimental performance in comparison to the ANN. The complexity of these advanced models is partially governed by the number of interconnected layers, and may contribute to their lower performance, whereas the shallow ANNs maintain simpler architecture and frequently exhibit performance that is on par with these models (Chakrabortty et al. 2021; Kalantar et al. 2021; Zhang et al. 2021).

Within flood mapping studies, ANN performance is adversely impacted by data imbalances, where there is a much larger representation of nonflooded to flooded training points (Batista et al. 2004; Wang et al. 2019; Priscillia et al. 2021). To remedy this, techniques such as the Synthetic Minority Oversampling Technique (SMOTE) are used to increase the number of flooded points within the training process. ANN performance can also be affected by the number of training points used (Tehrany and Jones 2017). Often FSM studies use an arbitrarily selected number of training points to form a point inventory of known flooded and nonflooded locations from within a watershed (Costache, Țîncu, et al. 2020; Pham et al. 2020; Wang et al. 2020; Ahmadlou et al. 2021; Shahabi et al. 2021). The impact of this approach on ANN performance, in comparison to using a larger number of training points, should be assessed as it is the most prevalent in current practice. A further limitation in existing studies using ANNs for FSM is models are frequently trained and validated on the same watershed, with very few studies testing their performance on independent watersheds (Zhao et al. 2021; Seleem et al. 2022). This is a key indicator of the ANN’s generalization, facilitating flood susceptibility mapping across multiple areas, which is particularly important in data sparse regions. Therefore, the impact of the number of training points on model performance and the model’s generalizability are some of the existing knowledge gaps within ANN-FSM applications.

To address the aforementioned knowledge gaps, and the current limitations of physically based models, qualitative (AHP) and quantitative (Entropy) approaches, an alternative data driven ANN-SMOTE ensemble model is proposed. These three models were selected for comparison due to their widespread use in FSM studies (Siahkamari et al. 2017; Zhao et al. 2018; Hammami et al. 2019; Andaryani et al. 2021; Malekinezhad et al. 2021; Costache et al. 2022), and ease of application (Mudashiru et al. 2021a, 2021b).

The proposed ANN-SMOTE ensemble is compared to the existing commonly used models to establish its baseline performance for the Don River watershed in Southern Ontario, Canada. As few studies assess model generalization, and none using ANN ensembles, we propose testing the ANN-SMOTE’s model performance on four independent test watersheds within Southern Ontario. This novel research aims to demonstrate the generalizability of ANNs in producing accurate FSMs for independent watersheds whose inputs were not part of the ANN calibration, thereby contributing to a better understanding of the predicative ability of ANNs. In doing so, the ANN’s utility and applicability is highlighted not only for data scarce regions but introduces the ANN as an alternative or complementary approach to physically based and qualitative models for updating outdated FSMs. Finally, given the prevalence of the flood point inventory (FPI) approach, we assess the performance of conventional ANN’s developed in this manner to the baseline performance of the ANN-SMOTE model developed using a larger number of training samples. This comparison is missing in existing literature, signifying a necessary comparison to further the understanding and application of the FPI method within FSM studies. The findings of this study highlight the superiority of FSMs produced using the ANN-SMOTE ensemble, and may be easily expanded to advancing generalization frameworks and uncertainty quantification within the FPI approach.

2. Materials and methods

2.1. Study area

Five watersheds from southern Ontario were selected for this research: the Don, Highland, Humber, Etobicoke and Mimico River watersheds, as shown in Figure 1. These watersheds lie within the Toronto and Region Conservation Area (TRCA) jurisdiction – the local government authority in charge of flood protection – who provided the existing modelled floodplain maps used to build the ANNs. These five watersheds were selected out of the ten within the TRCA jurisdiction, as they contain eight out of the top ten Flood Vulnerable Areas (FVAs) identified by the TRCA (IBI Group 2019). These FVAs consist of clusters of roads and structures that have experienced and are expected to be susceptible to riverine flooding under severe and less severe storms, presenting a risk to both human life and economic welfare (IBI Group 2019). Of the five selected watersheds, the Don River watershed was selected as the training watershed as it is the most densely populated, with an area of 360 km², home to 1.4 million residents and has the commercial and financial centre of the City of Toronto within its boundaries. The independent test watersheds are the Highland, Humber River, Etobicoke and Mimico watersheds. The Highland watershed is the most urbanised of all five, with over 89% urban landcover and an area of 102 km². The Humber River watershed is the largest, with an area of 911 km² and 600 waterbodies. It is home to approximately 1 million inhabitants and has 37% urban landcover. Etobicoke has an area of 212 km² and approximately 67% urban landcover. Finally, Mimico is the smallest watershed with an area of 78 km² and 88% urban landcover.

Figure 1. The location of the selected watersheds within Southern Ontario, Canada.

2.2. Flood conditioning factors

The FCFs chosen in this study were selected based on their widespread use within FSM studies, consisting of the most commonly used factors which are considered integral to the field of FSM (Wang et al. 2020; Zhao et al. 2020; Bunmi Mudashiru et al. 2022; Dahri et al. 2022; Seleem et al. 2022). Rincón et al. (2018) used AHP to conduct a flood hazard mapping study for the lower Don River where they found the best performing model used distance to streams (DS), slope, height above nearest drainage (HAND) and curve number (CN). This study builds on these results by introducing effective precipitation (EP) as it incorporates rainfall (commonly used in studies) (Tehrany et al. 2014; Hammami et al. 2019; Andaryani et al. 2021; Vilasan and Kapse 2022) into a surface runoff depth calculation. Topographic wetness index (TWI), which also factors flow accumulation, is frequently used in literature (Vojtek and Vojteková 2019; Das 2020; Vojtek et al. 2021; Vilasan and Kapse 2022).

Detailed information for each FCF is provided below. These FCFs were generated as geospatial layers using: a 30 m-by-30 m Digital Elevation Model (DEM), a 1 to 50,000 m resolution landcover dataset, a 1 to 50,000 m watercourse layer, and finally a precipitation layer obtained from an online IDF tool developed by Simonovic et al. (2015) for gauged and ungauged locations. All the FCFs were converted to raster format with a grid size of 30 m-by-30m using ArcGIS 3.1.2.

Slope was selected because it governs the velocity of surface runoff and its pooling potential (Choubin et al. 2019; Vojtek and Vojteková 2019; Islam et al. 2023). A flatter slope angle leads to slower water velocities and greater water stagnation potential; therefore, flatter regions are at greater risk of flooding. Slope was calculated using the Slope tool in ArcGIS with the DEM layer as an input.

The DS indicates the proximity to rivers and streams, since flood vulnerable areas are generally in closer vicinity to river and streams (Rahmati et al. 2015; Shafizadeh-Moghadam et al. 2018; Wang et al. 2020; Andaryani et al. 2021). The DS was calculated using Euclidean distance from the watercourses layer.

The CN of an area, defined by the Soil Conservation Services (SCS), describes the infiltration capacity and subsequent runoff generation capability of an area, and is thus an indicator of landuse/landcover and soil permeability (Zhao et al. 2018; Rahmati et al. 2020). Values of CN can range from 30 for areas with highly permeable soils with low runoff potential to 98 for impermeable surfaces with high runoff potential. This layer was generated using the SCS CN values published (USDA 1986) as well as the landuse and soil type.

Rainfall data is commonly included in FSM studies as it is directly related to flooding potential (Tehrany et al. 2014; Hammami et al. 2019; Das 2020; Bera et al. 2022). Using the online IDF tool, rainfall depths across the watershed were obtained and interpolated using the inverse distance weighting (IDW) method, to produce a Total Precipitation (TP) layer. The EP layer was then generated using the TP layer. EP, also known as runoff, is the precipitation depth after losses (such as infiltration). The greater the value of EP, the greater the potential for localized flooding as the runoff pools on the surface. It is calculated using TP and CN using the following equation: (1) $$\mathrm{\text{EP}}=\frac{{\left(\mathrm{\text{TP}}-\frac{5080}{\mathrm{\text{CN}}}+50.8\right)}^2}{\left(\mathrm{\text{TP}}+\frac{20,320}{\mathrm{\text{CN}}}-203.2\right)}$$(1) where TP represents total precipitation, CN represents the curve number. This equation is expressed in metric units for mm.

The TWI is a quantitative factor incorporating slope and flow accumulation to identify areas with high water accumulation potential (Khosravi et al. 2016; Tehrany and Kumar 2018; Bui et al. 2020; Costache, Ngo, et al. 2020). Areas having a higher TWI are more susceptible to flooding than areas with a lower TWI value. It is calculated as follows: (2) $$\mathrm{\text{TWI}}=\mathrm{\text{ln}}\hspace{0.17em}\left(\frac{\alpha }{\mathrm{\text{tan}}\hspace{0.17em}\beta }\right)$$(2) where $\alpha$ is the upslope catchment area draining from a point with a slope angle of $\beta .$

Finally, HAND impacts flood susceptibility as low-lying areas adjacent to streams are more susceptible to flooding, in contrast to higher-lying ground (Rincón et al. 2018). This layer was generated using the HAND tool, which is part of the Riparian Topography Toolbox created by Dilts (2015). Simply, HAND is the elevation difference between a grid cell and its nearest drainage cell.

The FCFs layers for the Don River are provided in Figure 2, whereas the input data source for each layer is provided in Table 1.

Figure 2. FCF layers for the Don River.

Table 1. FCFs and their associated data sources.

FCF	Input layers	Source
Slope	Digital elevation model	Provincial Digital Elevation Model (PDEM) (Ontario GeoHub 2019a)
DS	Water courses - TRCA	Watercourses TRCA (TRCA Open Data Portal 2020)
	Water bodies -Ontario	Topographic Data of Canada - CanVec Series - (Open Government Portal 2017)
	Water courses – Ontario
CN	Landcover	Southern Ontario Land Resource Information System (SOLRIS) 3.0 | (Ontario GeoHub 2019c)
	Soils	Soil Survey Complex | Soil Survey Complex | (Ontario GeoHub 2019b)
EP	Rainfall	IDF tool (Simonovic et al. 2015)
	CN layer	User generated
TWI	Slope	User generated
HAND	DEM
	Water courses

To ensure consistency of units across these six FCFs for each of the three proposed methods, ANN-SMOTE, AHP and Entropy, each input value was normalized based on whether larger or smaller values corresponded to greater flood susceptibility (Mahmoody Vanolya and Jelokhani-Niaraki 2021). Datasets in which larger values represent a higher flood susceptibility potential were maximised, such that the highest value in the dataset was assigned a 1 and the lowest value a 0. Conversely, minimum normalization was performed on datasets where lower values represent a higher flood susceptibility and were therefore assigned a value of 1. For example, larger values of CN led to greater flood susceptibility so maximum normalization was applied, whereas smaller values of DS correspond to greater flood susceptibility, so minimum normalization was applied. The maximum and minimum normalization are given by Eqs. (3)(3) $$a_{\mathit{\text{ij}}}=\frac{S_{\mathit{\text{ij}}}‐S_j^{\mathrm{\text{min}}}}{S_j^{\mathrm{\text{max}}}‐S_j^{\mathrm{\text{min}}}}$$(3) and (4)(4) $$a_{\mathit{\text{ij}}}=\frac{\mathrm{ }{S}_j^{\mathrm{\text{max}}}‐S_{\mathit{\text{ij}}}}{S_j^{\mathrm{\text{max}}}‐S_j^{\mathrm{\text{min}}}}$$(4) , respectively. (3) $$a_{\mathit{\text{ij}}}=\frac{S_{\mathit{\text{ij}}}‐S_j^{\mathrm{\text{min}}}}{S_j^{\mathrm{\text{max}}}‐S_j^{\mathrm{\text{min}}}}$$(3) (4) $$a_{\mathit{\text{ij}}}=\frac{\mathrm{ }{S}_j^{\mathrm{\text{max}}}‐S_{\mathit{\text{ij}}}}{S_j^{\mathrm{\text{max}}}‐S_j^{\mathrm{\text{min}}}}$$(4) where $S_{\mathrm{\text{ij}}}$ is the value of the ith pixel in the jth dataset, $S_j^{\mathrm{\text{max}}}$ is the maximum value of the jth dataset, $S_j^{\mathrm{\text{min}}}$ is the minimum value of the jth dataset and $a_{\mathit{\text{ij}}}$ is the standardized pixel value at location i for the jth dataset. Accordingly, CN, TWI and EP were maximised whereas slope, HAND and DS were minimised.

2.2.1. Similarities of FCFs across selected watersheds

As ANN generalizability is being assessed, an investigation of the similarities and differences between the Don River FCFs and the independent test watersheds is necessary. To do this, the probability density functions (pdf) of each FCF for each watershed were plotted and are presented in Subsection 3.1. These plots provide a visual indication of the shared topographic and physical characteristics across the five urban watersheds; allowing a greater understanding of the applicability of ANN generalizability. For example, if the watersheds share similarities across slope, we can surmise that the ANN will have consistent performance across these watersheds as they share these similarities. This is of vital importance as it underscores the ANN as an established accurate approach to FSM within these urban watersheds and may be used in place of its time consuming, data-intensive and subjective physical and qualitative model counterparts.

2.3. Flood inventory

Due to the nature of fluvial flooding, and the increased susceptibility of floodplain adjacent areas to flooding (Zhao et al. 2020), the models were trained on and validated against a floodplain layer obtained from the TRCA. This layer delineates the extent of the floodplain through both hydrological and hydraulic modelling. The hydrological model simulates the flow within rivers and streams based on the rainfall resulting from a 100-year or regional storm. The hydraulic model then shows the inundated areas and extent of the floodplain (How Does TRCA Define Flood Risk? 2023).

2.4. Analytical hierarchy process

AHP is a pairwise comparison framework based on expert knowledge contribution for multicriteria and multi-objective decision making (Gudiyangada Nachappa et al. 2020; Bunmi Mudashiru et al. 2022). Developed by Saaty (1988), the criteria are ranked against each other using an established scale ranging from 1 to 9, where 1 is equal relevance and 9 is extreme relevance. To calculate the AHP rankings, the six FCFS are ranked in a pairwise comparison where variables considered to have equal influence over flood susceptibility are given a ranking of 1. Where one variable is deemed to be more important than the other, a higher ranking is given to that variable. For example, if DS is considered to have moderate importance over CN, the ranking of DS would be 3 when compared to CN and the ranking of CN would be 1/3 when compared to DS. The pairwise comparison matrix is, then, normalized using the eigenvector technique and tested for consistency using the consistency index (CI).

The CI is calculated as follows: (5) $$\mathrm{\text{CI}}=\frac{\lambda -n}{n-1}$$(5) where $\lambda$ is the eigenvector and n is the number of criteria. For the AHP method to be consistent, the CI must be below 0.1.

2.5. Shannon entropy

The Entropy ranking method is based on principle of Shannon Entropy, which measures the amount of diversity within the dataset. If the dataset is more diverse across its values, the greater its Entropy-based rank will be (Mahmoody Vanolya and Jelokhani-Niaraki 2021). The rank of each j dataset can be calculated as follows: (6) $$E_j=-\frac{\sum _{i=1}^m{P}_{\mathit{\text{ij}}}\mathrm{ }\mathit{\text{ln}}\left(P_{\mathit{\text{ij}}}\right)}{\mathit{\text{ln}}\left(m\right)}$$(6) where $E_j$ is the Entropy value, m is the dataset size and $P_{\mathit{\text{ij}}}$ is calculated as follows for each normalized pixel value: (7) $$P_{\mathit{\text{ij}}}=\frac{a_{\mathit{\text{ij}}}}{\sum _{i=1}^m{a}_{\mathit{\text{ij}}}}$$(7) where $a_{\mathit{\text{ij}}}$ is the ith pixel value for the jth FCF. The degree of diversity, (b_j), in the dataset is, then, calculated as: (8) $$b_j=1-E_j$$(8)

The more diverse a dataset is, the higher the value of this criterion. The larger this value, the smaller the value of Entropy will be, and the larger the Entropy-based ranking ($W_{E_j}$) is, which can be calculated as follows: (9) $$W_{E_j}=\frac{b_j}{\sum _{j=1}^n{b}_j}$$(9)

Therefore, this demonstrates that a dataset is considered more important if it is more diverse. If a dataset is completely homogenous, its criterion ranking will be 0 and subsequently it will be removed from the decision-making dataset.

Through this approach, the FCFs are ranked solely on the diversity within that dataset. If there is little variation in the values of the dataset, it will be considered to have little importance to flood susceptibility and its weighting will reflect that. For example, if the slope within the entire watershed is consistently shallow, it will have a smaller ranking than if there was more variation within the topography. This method prioritizes variables with greater diversity, however, that may not be a holistic depiction of its importance to flood susceptibility.

2.6. Artificial neural networks

As demonstrated, shallow ANNs exhibit comparable performance to their deep learning counterparts (Chakrabortty et al. 2021; Chen et al. 2021; Kalantar et al. 2021; Zhang et al. 2021), providing a commonly used, robust and easily implementable model, an important advantage to policymakers and modellers. The ANN framework commonly consists of a set of interconnected layers, namely the input, hidden and output layer. Information is relayed across the layers in a feedforward sequential nature. Each input-hidden node connection has an associated multiplicative weight, w_k, that is randomly initialized at the onset of training, and iteratively adjusted at each training step to minimize the cross-entropy cost function. Similarly, the output-hidden node connection has both randomly initialized weights and biases, b_k, as well as a nonlinear activation function. In developing this ANN, a trial-and-error procedure was used to determine the number of hidden nodes. A single layer with 10 hidden nodes was selected, as using a larger number of hidden nodes had negligible impact on predictive accuracy. Thus, to avoid increasing model complexity and to align with best practises, a small hidden layer was selected (Khan et al. 2018; Snieder et al. 2021). Six inputs nodes were initialized for each FCF, where each input was normalized following the procedure mentioned in Subsection 2.2, and one output node predicted flood susceptibility in the range of [0,1]. The scaled conjugate descendent backpropagation algorithm was used along with a sigmoid output activation function. An early stopping procedure was used in the validation dataset, whereby if the error between the predicted and true output continuously increased for 6 epochs, the training is stopped to minimize the risk of overfitting.

Ensemble modelling has been proven to produce more accurate FSMs, compared to using individual ANN models (Ren et al. 2016; Choubin et al. 2019; Ahmed et al. 2021; Yaseen et al. 2022). Therefore, a 50 ANN ensemble was created where each ANN used a 60 − 20 − 20% datasplit for training, validation and testing, respectively. Each new ANN in the ensemble was generated by randomly resampling the original input dataset with replacement, a process known as bootstrapping. Additionally, each model is trained, validated and tested on a different subset of the original dataset, producing distinctly initialized models and predictive capabilities. The ANNs were trained and validated against the Don River TRCA modelled floodplain layer to predict flood susceptibility. Model performance was then tested on the entire input dataset and on four independent test watersheds not used in the training process. The training process is adversely impacted with the use of imbalanced datasets (Batista et al. 2004; Priscillia et al. 2021), which is the case in this study as the nonflooded points make up approximately 90% of the input dataset.

2.6.1. Data imbalances - SMOTE

To increase the number of underrepresented flooded points, SMOTE is used in conjunction with the ANN ensemble to create a hybrid ANN-SMOTE ensemble approach. This approach increases the number of underrepresented variables in the dataset by finding the K nearest neighbours (KNN) to the minority sample and calculating the Euclidean distance between them. This difference is multiplied by a random number between 0 and 1 and added back to the minority sample, effectively creating a random point along the line segment between the minority sample and its neighbour (Chawla et al. 2002). To remain consistent with common practices, five KNNs were used creating an additional 100,000 input data points for the base Don River model, for a total of approximately 507,000 input points. Within FSM studies, however, it is common practice to develop a FPI consisting of a balanced set of 100–200 flooded and nonflooded data points (Costache, Țîncu, et al. 2020; Pham et al. 2020; Wang et al. 2020; Ahmadlou et al. 2021; Shahabi et al. 2021). The performance implications of using a much smaller training dataset in comparison to the ANN-SMOTE should be explored.

2.6.2. Varying the number of training points

A second ANN ensemble approach is developed in this study to assess the impacts on predicted FSM accuracy. To remain consistent with standard practice, 200 nonflooded and flooded data points were randomly selected from the Don River and floodplain layer, respectively, for a total of 400 points. This approach will be referred to as the ANN-FPI and will be compared with the ANN-SMOTE approach. To predict flood susceptibility across the entire watershed, the complete FCF dataset for each watershed was used as inputs into each respective ANN ensemble approach. Meaning, although the ANN-FPI approach was trained on 240 out of the 400 points, the input dataset to predict flood susceptibility across the Don River consisted of each 30 m-by-30 m pixel, resulting in 407249 input points for each of the 6 FCFs (a 407249 by 6 input matrix). Throughout this study, the ANN-SMOTE is approach is used, except where the comparison to ANN-FPI is conducted.

2.6.3. Combined neural pathway strength

Knowing which FCF has the greatest propensity for inducing flood susceptibility within a region is a critical piece of information for policymakers, affecting their ability to accurately allocate flood preventative measures. As such this information is extracted from the ANN ensemble through using the different neural weights between the input and output, which signify the input’s predicative capability to the output. These weights can be extracted from the ensemble of nets to quantify the relative importance of each input, through a process known as Combined Neural Pathway Strength (CNPS) (Snieder et al. 2020).

The CNPS method calculates the neural pathway strength between the inputs and output based on their weights and ranks the inputs in order of highest pathway strength. This is first done through a matrix multiplication of the input-hidden weights and hidden-output weights. Then, the inputs are ranked based on the consistency of their positive or negative correlation to the output. This is calculated as follows: (10) $${\alpha }_i=\frac{\mathrm{\text{max}}(\sum \left({\mathrm{\text{CNPS}}}_{i\mathrm{ }}>0\right),\sum \left({\mathrm{\text{CNPS}}}_{i\mathrm{ }}<0\right))}{n}$$(10) where ${\mathrm{\text{CNPS}}}_{\text{i}}$ represents matrix multiplication of the weights and n is number of ensemble members. If in an ensemble of 50 ANNs, an input exhibits positive correlation with the output 50 times, it will have a consistency score of 1. If there are multiple inputs with a consistency score of 1, they are ranked on the relative range of their CNPS values as follows: (11) $$S=\frac{\mathrm{\text{min}}\left({\mathrm{\text{CNPS}}}_{i\mathrm{ }}\right)}{\mathrm{\text{max}}\left({\mathrm{\text{CNPS}}}_{i\mathrm{ }}\right)}$$(11)

Inputs with a smaller CNPS range will receive a higher score, with a maximum score of 1, indicating the same stable behaviour across the entire ensemble. Using CNPS in this manner offers an explanatory approach to which inputs are deemed most influential to predicting flood susceptibility.

2.7. Performance metrics

To analyze and validate predictive performance, the area under the Receiver Operating Characteristic curve (AUC-ROC) and the overall accuracy (OA) were used. The AUC-ROC curve is a commonly used metric, constructed by plotting sensitivity on the y-axis and (1 – specificity) on the x-axis (Shafizadeh-Moghadam et al. 2018; Rahmati et al. 2020; Wang et al. 2020). Sensitivity measures the ratio of accurately predicted flood locations to the total number of flood locations, while specificity measures the ratio of accurately predicted nonflooded locations to the total number of nonflooded locations. The AUC-ROC is indicative of predictive capability, and is bounded between 0 and 1, where a larger value indicates better model performance. OA measures the ratio of correct predictions to the total number of data points and is measured as follows: (12) $$\mathrm{\text{OA}}=\frac{\mathrm{\text{TP}}+\mathrm{\text{TN}}}{\mathrm{\text{TP}}+\mathrm{\text{TN}}+\mathrm{\text{FP}}+\mathrm{\text{FN}}}$$(12) where TP is true positive, TN is true negative, both denoting correctly classified flooded and nonflooded locations, respectively. Additionally, FP is false positive, and FN is false negative both denoting inaccurately classified flooded and nonflooded locations, respectively. As both ANN-FPI and ANN-SMOTE predict flood susceptibility and not a binary classification of flood occurrence, the predictions must be translated to a binary classification for the calculation of OA components. This is done through obtaining each unique flood susceptibility value and using that as a threshold for binary reclassification. For example, if the flood susceptibility predictions ranged from 0 to 1, for each threshold, all predictions greater than or equal to this value are classified as 1 and all values less than this value are classified as 0.

Within the ANN ensemble model, the median of all predictions was calculated and then used in the AUC-ROC and OA calculations. For the Don River model, the complete predicted median dataset was used to calculate AUC-ROC and OA, and similarly, the complete dataset of the remaining four independent test watersheds was used to calculate the test performance of the Don River for both ANN approaches.

2.8. Multicollinearity analysis

Multicollinearity occurs when highly correlated factors are introduced in the modelling process, leading to a potentially biased output (Wang et al. 2021). Therefore, this is tested for through the Variance Inflation Factor (VIF), which is calculated as follows: (13) $$\mathrm{\text{VIF}}=\frac{1}{1‐R^2}$$(13) where $R^2$ is the coefficient of determination. There is consensus that VIF values < 10 indicate no multicollinearity problem, whilst values above that are problematic and indicate those variables should be removed from the input data set (Rahmati et al. 2020).

3. Results

3.1. FCF distribution across all watersheds

Figure 3 provides the pdfs of the FCFs across the five watersheds. All five watersheds are relatively flat with typical slope values below three degrees. Since slope is used in the TWI calculation, the TWI pdfs are also similar across the watersheds. The differences are more apparent in the CN, although the watersheds are all highly urbanized, the degree of imperviousness differs. The Don River and Highland watersheds both have large diversity of DS values, while Etobicoke, Humber and Mimico have a larger concentration of regions close to streams. Finally, HAND values are typically concentrated within the lower bounds, indicating low lying regions close to flow accumulating regions. Humber, however, has more regions that are low lying close to the flow accumulation regions compared to the other watersheds. These results suggest that there are similarities within the topography of watersheds, as seen by slope and TWI, and more pronounced variations in DS, HAND and CN.

Figure 3. FCF probability distribution comparison across all watersheds.

3.2. Multicollinearity

The multicollinearity test was conducted on the input dataset to ensure there is no strong correlation between the variables. The VIF values are presented in Table 2 and based on this, none of the FCFs exhibit a VIF value greater than 10, therefore, there is no strong correlation between them, and they are all included in the input dataset for the three methods.

Table 2. Variance inflation factor values for each of the FCFs.

Flood conditioning factor	VIF
DS	1.42
HAND	1.31
Slope	1.39
CN	1.69
EP	1.71
TWI	1.22

3.3. ANN, AHP and entropy comparisons

The AHP rankings were derived using expert knowledge, shown in Table 3. The FCFs were classified into five categories of flood susceptibility: very low, low, medium, high and very high using the natural breaks method. These rankings lead to a CI value of 0.007, which meets the 0.1 threshold for consistency.

Table 3. FCF rankings for AHP.

FCF				Pairwise comparison matrix
	DS	HAND	Slope	CN	EP	TWI
DS	1	2	2	3	3	4
HAND	1/2	1	1	2	2	3
Slope	½	1	1	2	2	3
CN	1/3	1/2	1/2	1	1	2
EP	1/3	1/2	1/2	1	1	2
TWI	1/4	1/3	1/3	1/2	1/2	1
Sum	2.92	5.33	5.33	9.50	9.50	15.00

Table 4 lists the FCF rankings obtained for all three methods, the ANN-SMOTE, the AHP and Entropy weightings. The ANN-SMOTE rankings are obtained through CNPS and Entropy rankings are assigned based on their degree of diversity, as explained in Subsection 2.5. Each method assigns a different ranking to the FCFs, representative of its influence to flood susceptibility. These ranking are highly informative to policymakers as they can be used as explanatory variables, representative of the underlying physical characteristics with the greatest propensity for flooding.

Table 4. Ranking scheme for the FCFs for each FSM method.

FCF	ANN ranking	AHP ranking	Entropy ranking
DS	20	33	12
HAND	20	19	12
Slope	13	19	3
EP	12	11	11
CN	16	11	37
TWI	19	7	25

The resulting flood susceptibility maps of the Don River from the ANN-SMOTE, AHP and Entropy are shown in Figure 4, along with the actual floodplain map. The AHP and Entropy FSM was derived using the weighted overlay method in ArcGIS. Each FCF is overlayed according to the rankings given in Table 4. Each pixel within the watershed is multiplied by the FCF ranking and the final map is produced by the addition of each FCF pixel after it is multiplied by its associated ranking. These maps show five categories of susceptibility; very low, low, medium, high and very high. ANN-SMOTE’s predictions were classified into the same five categories using the equal interval method in ArcGIS, where pixels with a flood susceptibility value below 20% were rated very low, pixels between 20% and 40% were rated low and so forth. With 84% of flood susceptibility pixels classified as very low flood susceptibility, ANN-SMOTE had the highest proportion of very low susceptibility pixels. The very high susceptibility pixels accounted for 3.77% of the total watershed flood susceptibility and were mainly concentrated along the floodplain. Results from AHP and Entropy did not exhibit a similar spatial pattern with more spread-out locations of very high flood susceptibility. Both methods had most of the flood susceptibility points concentrated in the high and medium flood susceptibility categories. The ANN-SMOTE had 60.6% and 31.7% susceptibility in the high and medium categories, respectively, whereas, using Entropy yielded 39.8% and 49.8% flood susceptibility, respectively. Very low flood susceptibility was not recorded in the AHP method, whereas it occupied only 0.01% in the Entropy method, a very large difference from the ANN-SMOTE results.

Figure 4. A comparison of the FSMs resultant from ANN-SMOTE (top right), Entropy (bottom left) and AHP (bottom right) along with the actual floodplain (top left).

Figure 5 provides the AUC-ROC results, with the dashed green line representing the random predictor model performance. Any lines falling below the dashed line indicate worse performance than randomly predicting whether a flood occurs or not at a given location. Table 5 shows the OA and AUC-ROC scores. AUC-ROC was the highest for ANN-SMOTE with a value of 0.902 whereas AHP and Entropy achieved values of 0.650 and 0.472. Higher AUC-ROC values indicate better model performance so, ANN-SMOTE exhibits the best performance. In OA, ANN-SMOTE has the greatest accuracy with 54.9% whereas AHP and Entropy follow behind with 40.4% and 45.2%, respectively. It is worth noting that these values of OA are averaged over a range of threshold values and as such, high OA values are neutralized by low OA values. Nonetheless, these results suggest that across the threshold values for all three methods, ANN-SMOTE demonstrated greater accuracy in classifying flood susceptibility across the watershed than the other two methods.

Figure 5. AUC-ROC performance comparison for the ANN, AHP and Entropy for the Don River.

Table 5. Performance metrics for the ANN-SMOTE, AHP and Entropy methods on the Don River.

Performance metric	ANN-SMOTE	AHP	Entropy
AUC-ROC	0.902	0.650	0.472
OA	0.549	0.404	0.452

Given the lower performance metrics, the subjective nature and potential bias introduction characteristic of AHP as well as the lack of interdependence consideration between the inputs and output in Entropy, the ANN-smote will be the sole model further explored due to its performance, ease of implementation and data-driven nature.

3.4. Testing ANN generalizability

To test ANN-SMOTE’s ability to generalize flood susceptibility, it was applied on four independent watersheds, which were not included in the training of the ANN. The resulting ANN-SMOTE flood susceptibility maps and each watershed’s floodplain are compared in Figure 6, and the performance metrics for each watershed is given in Table 6. The floodplain maps show the extent of flood water in the river valley and potential spill areas where the flood waters were not contained within the river valley.

Figure 6. Test watersheds predicted by the Don River ANN-SMOTE model. From top to bottom: Etobicoke (1st row). Mimico (2nd row). Humber (3rd row). Highland (4th row).

Table 6. Performance metrics for the four test watersheds.

Performance metric	Highland	Humber	Mimico	Etobicoke
AUC-ROC	0.919	0.865	0.735	0.856
OA	0.532	0.573	0.557	0.569

In the Highland model, ANN-SMOTE had the highest AUC-ROC performance metric, but the lowest OA in comparison with the other modelled watersheds, with an OA of 53.2%. Generally, OA values remained in the range of 50% with the Humber, Mimico and Etobicoke models demonstrating OA values of 57.3%, 55.7% and 56.9%, respectively. Given the pdfs demonstrated in Figure 3, the consistent prediction accuracy across the watersheds is expected, as they share topographic similarities between the Don River and independent test watersheds. Additionally, from Figure 6, the predicted FSM maps for these independent test watersheds, align very closely with the modelled TRCA floodplain. The very high susceptibility regions are concentrated along the floodplain for each watershed, with most of the high susceptibility areas also centered around the floodplain. These prediction results and shared similarities demonstrate the ability of ANN-SMOTE to accurately predict flood susceptibility across urban watersheds. This marks a significant finding as these results are particularly important in data-scarce regions where hydrological/hydraulic modelling may be limited due to the large data requirement of these models. Additionally, any outdated FSMs that require updating due to shifting urban patterns could be produced using ANN-SMOTE in this manner with demonstrated prediction accuracy.

ANN-SMOTE’s ability to infer connections between the FCFs and flood susceptibility is clearly demonstrated in the Humber model. As demonstrated in Figure 6, the Humber model exhibits significantly more flood susceptible areas when compared to the floodplain. This is because certain minor tributaries were not included in the hydraulic modelling efforts undertaken by the TRCA. They are however included in the ANN-SMOTE’s flood susceptibility map as those streams were not removed from the input set, demonstrating that the ANN-SMOTE trained on the Don River inferred a strong connection between the proximity and existence of streams to that of flood susceptibility. This is shown through the concentration of predicted very high flood susceptibility around the rivers and tributaries, a finding that aligns with the tied first place ranking of DS by CNPS in Table 4. As such, even though these smaller tributaries were removed from the modelling process for the TRCA modelled floodplain map, their existence within the ANN-SMOTE’s input set guarantees that high flood susceptibility is observed in these areas. Additionally, the Humber watershed is uniquely characterized by a larger number of smaller streams and tributaries, a fact not shared with the other watersheds.

The results for the Mimico model demonstrated the lowest AUC-ROC value, however, OA results were on par with other watersheds. This could be attributed to a greater proportion of FPs, as shown by the streams predicted throughout the Mimico model in Figure 6, which are not found within the TRCA floodplain layer. The FPs decrease the (1 – specificity) value in AUC-ROC, which is not explicitly calculated in OA, explaining the difference in values.

3.5. A comparison of ANN-SMOTE to ANN-FPI

The accuracy of FSM’s generated using a subset of flooded and nonflooded points (ANN-FPI approach) is compared against the FSM generated using the ANN-SMOTE approach. The results for the Don River are shown in Figure 7 (may be compared with the ANN-SMOTE approach in Figure 4) and a comparison of performance metrics for both approaches is provided in Table 7.

Figure 7. FSM for the Don River produced by the ANN-FPI method.

Table 7. Performance metrics comparison for the ANN-SMOTE and ANN-FPI approach.

Performance metric	ANN-SMOTE	ANN-FPI
AUC-ROC	0.902	0.897
OA	0.549	0.265
Mean	0.118	0.265
Median	0.034	0.195

As demonstrated by the results, the AUC-ROC metrics are very similar for both approaches, while the OA score deteriorates from 0.549 for ANN-SMOTE to 0.265 for the ANN-FPI approach. Although both metrics are averaged, the results suggest that a greater portion ANN-FPI predictions demonstrate lower performance than the ANN-SMOTE approach. Graphically, this finding is supported by Figure 8, which provides the components of a confusion matrix (namely TP, TN, FP, FN) for both approaches across each threshold within the ANN-SMOTE predictions. As shown in Figure 8(a), the proportion of TN identified by the ANN-SMOTE is greater than those identified by ANN-FPI, suggesting that ANN-FPI more frequently misclassifies nonflooded areas as flooded. This is reiterated in the plot of FP and FN predictions, where the ANN-FPI demonstrates a greater tendency to incorrectly identify nonflooded areas as flooded. The bottom two plots show the individual components of the AUC-ROC curve, alongside the AUC-ROC curve. This is provided to highlight the differences between the individual components of the AUC-ROC plot, as those differences are lost in translation within the AUC-ROC plot. From Figure 8(c) the false positive rate of ANN-FPI is consistently greater than ANN-SMOTE, which is supported by the findings of TN and FP. The sensitivity is greater for ANN-FPI than ANN-SMOTE until a threshold value of approximately 0.7 when the opposite becomes true. This suggests that the ANN-FPI approach performs better up to a threshold of 0.7, also demonstrated by Figure 8(a,b) where the TP metric of ANN-SMOTE overtakes ANN-FPI. These findings suggest a closer look at the individual components of AUC-ROC are necessary to obtain a more holistic view of performance metrics, as these differences between false positive rate and sensitivity are not as visibly expressed in Figure 8(d).

Figure 8. Confusion matrix components for the ANN-FPI and ANN-SMOTE approach for the Don River. The plots demonstrate (a) TP and TN metrics, (b) FP and FN metrics, (c) False positive rate and sensitivity and (d) AUC-ROC plots.

Additionally, a comparison of the mean and median, provided in Table 7, of both approaches reveals that ANN-FPI overestimates flood susceptibility, indicated by the larger values of both mean and median. For the same input set of FCFs for the Don River, the ANN-FPI approach has a mean flood susceptibility prediction of 26.5% compared to 11.8%. These results reiterate the previous results of Figure 8, where it was demonstrated that ANN-FPI has a higher misclassification rate of FPs. This is also graphically demonstrated by the pdf of flood susceptibility, shown in Figure 9. Both approaches display a right skew, which is expected given that there are more areas with lower flood susceptibility. The ANN-FPI approach has a greater spread of flood susceptibility predictions, indicating greater uncertainty and variability within the predictions, whereas the ANN-SMOTE approach shows a much lower spread, with a much larger concentration of predicted data under 10% flood susceptibility.

Figure 9. Probability distribution of the Don River flood susceptibility values for both ANN-SMOTE and ANN-FPI.

A breakdown percentage of each susceptibility class and its respective area is provided in Table 8. The percent difference in Table 8 is between the ANN-FPI and ANN-SMOTE approach. The ANN-FPI approach overestimates the low, medium and high susceptibility areas and underestimates the very low flood susceptibility areas. This demonstrates a more conservative approach to FSM due to its overestimating tendency.

Table 8. A comparison of the Don River map areas designated by their various susceptibility categories for each FSM approach.

Flood risk category	ANN-SMOTE (%)	ANN-FPI (%)	Difference (%)
Very low	84.8	50.2	−34.6
Low	6.22	23.4	17.2
Medium	2.97	15.2	12.2
High	2.27	8.46	6.19
Very high	3.75	2.75	−1

These findings demonstrate the ANN-FPI approach more frequently misclassifies nonflooded regions as flooded, thereby resulting in a decrease in prediction accuracy. These results are presented to highlight these tendencies, and therefore, bring attention to the conservative nature of these predictions, especially given the wide-spread use of this approach within FSM studies.

The impact of using the ANN-FPI approach on the test watersheds is evaluated next. The resulting FSM maps for the four test watersheds along with the performance metrics table are provided in Figure 10 and Table 9, respectively.

Figure 10. FSM produced by the Don River ANN-FPI. Top left: Etobicoke. Top right: Mimico. Bottom left: Humber. Bottom right: Highland.

The difference between the ANN-SMOTE and ANN-FPI approach performance metrics are also provided in Table 9. A positive difference indicates that the ANN-SMOTE approach had better performance, whereas a negative difference indicates the ANN-FPI approach performed better. Based on these results, there was no substantial change in the AUC-ROC metric. Based on this finding, as well as the findings of Figure 8(c,d), the AUC-ROC metric may not be sensitive enough to portray the differences in its individual components. This highlights the importance of multiple performance metrics and suggests AUC-ROC results should be presented along with their individual components for a more accurate depiction of performance. OA values demonstrated a drop, falling from approximately 55% to 28%. It is worth noting that though OA values are averaged, meaning high performance is neutralized by lower performance, averaging to about 50% accuracy as expected, the drop to 28% signals a decrease in TN predictions, which was shown in the Don River predictions in Figure 8(a). This drop in OA signifies the conservative approach of ANN-FPI, extended onto the independent test watersheds.

Table 9. Performance metrics for the test watersheds based on the Don River ANN-FPI.

Performance metrics	Watershed
	Highland	Humber	Mimico	Etobicoke
AUC - ROC	0.916	0.829	0.798	0.844
OA	0.274	0.492	0.286	0.282
Difference in AUC-ROC	0.003	0.036	−0.063	0.012
Difference in OA	0.258	0.081	0.271	0.287

Flood susceptibility of each class as a proportion of the entire watershed is provided in Table 10. Similar to the findings of the Don River, the ANN-FPI approach here demonstrates a conservative estimate of flood susceptibility across the watersheds, by underestimating the most extreme flood susceptibility designations (very low and very high) and overestimating the remaining three categories. This is indicative of its lower accuracy, relative to ANN-SMOTE, and subsequent performance drop in accurately distinguishing between flooded and nonflooded locations, resulting in a more unimodal probability distribution rather than the more realistic bimodal distribution.

Table 10. A comparison of the proportion of watershed areas categorized into their respective flood susceptibility designation.

Flood risk	ANN-FPI approach				ANN-SMOTE approach
	Highland	Humber	Mimico	Etobicoke	Highland	Humber	Mimico	Etobicoke
Very low	38.6	3.36	39.8	12.5	74.9	23.7	80.39	64.3
Low	28.4	10.6	30.8	26.0	10.5	17.3	7.83	16.4
Medium	17.7	26.7	15.8	34.1	5.23	15.0	4.10	7.50
High	11.1	48.2	9.99	22.0	4.01	16.7	3.12	5.08
Very high	4.11	11.1	3.60	5.33	5.37	27.1	4.55	6.67

For completeness, four other ANNs were trained using the inputs of each independent watershed and were used to predict flood susceptibility across the remaining watersheds, including the Don River. This was done for both ANN-FPI and ANN-SMOTE, and the findings from the Don River trained ANN were found to be consistent across the results from these ANNs, which are presented in Appendix.

4. Discussion

One of the requirements of successful flood susceptibility mitigation and prevention are accurate FSMs (Wang et al. 2020). Of the various FSM approaches, shallow ANNs were used to address the limitations associated with the computational complexity of physically based models (Giovannettone et al. 2018; Choubin et al. 2019), the subjective nature of qualitative models (AHP) (Souissi et al. 2019; Das 2020) and the lack of interdependency consideration in diversity-based quantitative approaches (Entropy) (Malekinezhad et al. 2021). The ANN-SMOTE model developed in this study demonstrated strong potential as an accurate FSM tool with good generalization. It can be used as a complementary or standalone model in future studies where other approaches may not be feasible. Finally, using the FPI approach adversely impacts predictive accuracy when compared to using a larger training set, and given its prevalence, this uncertainty should be expressed as a limitation in future studies.

Based on a comparison of the performance metrics, the ANN-SMOTE exceeded AHP and Entropy across all metrics. These findings are supported by Andaryani et al. (2021), where the ANN outperformed FART and SOM. Dahri et al. (2022) compared the performance of Analytic Network Process (ANP) and ANNs and found the ANN to outperform the ANP for FSM. Finally in landslide susceptibility mapping, Park et al. (2013) found the ANN to have the highest AUC-ROC values out of AHP, RF and FR. Akay (2021) compared multiple FSM approaches, amongst which were Entropy and AHP, and found them to have similar performance, which resonates with this study’s OA findings for both, provided in Table 5. Wang et al. (2021) investigated ANNs against FR and Entropy, and found the ANN to outperform both other approaches, in line with the findings of this research. This research confirms that ML approaches are superior to subjective qualitative approaches (AHP) and diversity-based approaches (Entropy) due to their ability to quantitatively infer connections within the input data that best describe the output.

Ranking variables provides insight into the most critical factors contributing to flood potential, which is beneficial to policymakers (Wang et al. 2020). The FCFs were ranked differently by all models, however there was some similarity within the ANN-SMOTE and AHP ranking schemes, such as DS and HAND being in the top three ranked variables. Shafizadeh-Moghadam et al. (2018) found high flood susceptibility to be concentrated in areas with low DS, and consequently, selected DS as the most important factor in their ML approach to FSM, yielding similar results to our findings. Slope is frequently found as the most influential factor in FSM studies; however, it was ranked 5th in our study. This ranking is offset by TWI’s 3rd ranking, which factors in slope, such that high flood susceptibility was found along regions with larger values of TWI, in line with findings across other studies (Choubin et al. 2019; Vojtek and Vojteková 2019; Islam et al. 2023). The ability to easily select the most influential FCFs highlights ANN-SMOTE’s utility, in comparison to the more time-consuming ad-hoc trial-and-error procedure of qualitative approaches. This is especially beneficial under climate change considerations, where the most influential FCFs may change under different climate patterns (An et al. 2022; Bera et al. 2022), ANNs can be easily retrained whereas their qualitative counterpart will require a more intensive approach.

The generalizability of ANN-SMOTE was tested on four independent watersheds that were not included in the training dataset. The resulting flood susceptibility maps were similar to their respective floodplain maps (which were developed via hydrological and hydraulic modelling), indicating the ANN-SMOTE’s generalization capability to accurately retain the geo-physical relationships between riverine flooding and the given inputs, and apply that on the test watersheds. As shown in Figure 3, the test watersheds displayed some geo-physical similarities with the Don River such as slope and HAND, however, there were more pronounced differences within the levels of imperviousness (CN) and resulting runoff (EP). Wading through these differences and similarities, the ANN-SMOTE maintained consistent performance across both the AUC-ROC and OA indices as it did with the Don River study model, as shown in Table 6. These results suggest that ANN models may effectively replace or complement hydraulic/hydrological models in FSM applications, through being used in data-scarce regions where it is difficult to validate and implement a hydraulic/hydrological model or being used to map certain regions within watersheds experiencing shifting urban patterns and mosaicking this prediction within the broader FSM. These findings are supported by Seleem et al. (2022) who used RF, ANN and CNN to predict FSM on the entirety of Berlin using flooded points from the entire region with a high concentration in the central region. Their results demonstrated that the ANN maintained a high accuracy in regions that were not sampled during the training process (Seleem et al. 2022). Similarly, Guo et al. (2022) used a CNN to predict flood susceptibility across four independent regions to assess model generalizability in different catchments (Guo et al. 2022). Their findings suggest that the CNN maintains a reasonable accuracy in identifying the extent of flooding, even under different landcover conditions such as rural and urban (Guo et al. 2022) which align with our findings.

It is standard practice in FSM studies to accumulate a flooding point inventory from a collection of historical flood events (Mudashiru et al. 2021a). Although common practice, this study found the proposed ANN-SMOTE approach outperforms ANN-FPI across all performance metrics. This is supported by the results of Tehrany and Jones (2017), who varied the FPI values from 1000 to 50, and found an approximate 20% drop in prediction accuracy. The ANN-SMOTE method extends this by facilitating the use of all available data points in the study area in conjunction with a minority oversampling technique to address the data imbalances, and comparing it against the smaller balanced FPI sample. The random selection of flooded events and their location is also addressed by the ANN-SMOTE approach. This arbitrary choice of FPI and number of points in the FPI method is reiterated in the works of Al-Abadi and Pradhan (2020). Their discussion highlights the random nature of point selection without scientific basis, rather an extraction of locations from singular floods in certain cases (Al-Abadi and Pradhan 2020). Similarly, Al-Abadi (2018) used approximately 3500 flood polygons to ensure the inclusion of flood potential extent into their model, which is a practice inherent in our model. Therefore, the limitations discussed associated with the FPI approach have been addressed by the ANN-SMOTE procedure.

Using the FPI approach is demonstrated to overestimate areas of medium and high risk as shown in Table 10 and Figure 8. The number of TN and FP predictions was shown to be greater than its ANN-SMOTE counterpart, showing that the decrease in the number of training points adversely affects the accuracy of prediction. Interestingly, the Humber model had the highest OA performance and demonstrated only an 8% drop in the ANN-FPI approach. The smaller difference in performance, when compared to the other test watersheds, is attributed to an improved TN prediction over the ANN-SMOTE approach for certain thresholds. In Figure 8, the ANN-FPI consistently underperformed in TN predictions across all thresholds, whereas in the Humber model case, ANN-FPI does outperform ANN-SMOTE in TN predictions over a short range of thresholds, likely contributing to the improved performance in comparison to the other watershed models. The visual comparisons of performance metrics for the Humber model predicted by Don River are presented in Appendix.

Given the overestimation of the FPI approach, it is recommended in future studies to incorporate data from across the entire study area coupled with SMOTE to address data imbalances. Using the prevalent FPI approach results in conservative FSM estimates, inflating the degree of susceptibility within regions. This overestimation of high flood susceptibility may adversely impact flood mitigation efforts as there is no clear demarcation of regions requiring the most efforts.

5. Conclusion

Accurate flood susceptibility maps are of critical importance since they can be used as a tool to help policymakers in identifying flood vulnerable zones. Therefore, a comparison of existing methods and their accuracy is the first step to developing reliable maps. Three approaches to FSM were trained on the Don River and validated against four independent watersheds. The highly popular AHP method, Shannon’s Entropy and the proposed ANN-SMOTE model were compared against each other using the common AUC-ROC and OA metrics. The ANN-SMOTE ensemble (AUC-ROC: 0.902, OA: 0.549) outperformed the other two methods across all performance metrics. It was selected as the superior model as it addresses the limitations associated with subjective (AHP) and diversity-based (Entropy) approaches through its quantitative forming of connections in the input dataset that best align with the output. High accuracy was maintained when the ANN-SMOTE ensemble was applied onto four test watersheds with AUC-ROC results ranging from 0.919 to 0.735. These findings are valuable to policymakers requiring accurate FSM, particularly in regions where high-resolution datasets are not available for physically based models, or in regions experiencing rapid changes (e.g. urbanization) where FSMs need to be frequently regenerated. The ANN-SMOTE also substantially outperformed an ANN ensemble trained using the FPI approach. FSMs generated using FPI suffer from flood susceptibility overestimation. Therefore, it is recommended to incorporate data from the entire study area in conjunction with a minority oversampling technique, such as SMOTE, to address data imbalances. Future research should focus on the impact of data resolution, particularly the threshold of resolution at which prediction is adversely impacted. Additionally, coupling machine learning models with optimization algorithms to enhance the FCF ranking should be explored.

Disclosure statement

The authors report there are no competing interests to declare.

Data availability statement

The raw data that were used in this research are publicly available as listed in the article (Table 1). Model data may be obtained from the corresponding author, RK, upon reasonable request.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was supported by the Ontario Ministry of Environment Conservation and Parks under Grant TPON Case No.: 2021-02-1-1569389059; The Natural Sciences and Research Council of Canada under Grant RGPIN-2023-05077; Rahma Khalid was supported by a Natural Sciences and Research Council CGS scholarship.

Appendix

Table A1. Highland model prediction metrics for both ANN-SMOTE and ANN-FPI approaches.

		Highland	Humber	Etobicoke	Mimico
ANN-SMOTE	AUC-ROC	0.964	0.913	0.914	0.879
	OA	0.539	0.535	0.535	0.532
ANN-FPI	AUC-ROC	0.949	0.937	0.917	0.883
	OA	0.139	0.073	0.210	0.280

Table A2. Humber model prediction metrics for both ANN-SMOTE and ANN-FPI approaches.

		Highland	Humber	Etobicoke	Mimico
ANN-SMOTE	AUC-ROC	0.908	0.920	0.905	0.845
	OA	0.572	0.574	0.572	0.561
ANN-FPI	AUC-ROC	0.890	0.909	0.884	0.715
	OA	0.389	0.371	0.418	0.693

Table A3. Etobicoke model prediction metrics for both ANN-SMOTE and ANN-FPI approaches.

		Highland	Humber	Etobicoke	Mimico
ANN-SMOTE	AUC-ROC	0.833	0.882	0.890	0.865
	OA	0.559	0.568	0.569	0.565
ANN-FPI	AUC-ROC	0.834	0.875	0.879	0.846
	OA	0.246	0.152	0.312	0.339

Table A4. Mimico model prediction metrics for both ANN-SMOTE and ANN-FPI approaches.

		Highland	Humber	Etobicoke	Mimico
ANN-SMOTE	AUC-ROC	0.723	0.854	0.848	0.883
	OA	0.538	0.562	0.561	0.567
ANN-FPI	AUC-ROC	0.744	0.769	0.842	0.858
	OA	0.187	0.131	0.289	0.318