A survey on underwater coral image segmentation based on deep learning

ABSTRACT

Image-based coral reef survey technologies have revolutionized the monitoring of coral reefs by offering a cost-effective and noninvasive method for collecting data across large spatial scales and extended periods. Among these technologies, underwater videography has emerged as a well-established and reliable tool for remote sensing in coral research. Automatic segmentation of coral images represents a forward-looking and fundamental research area in underwater remote sensing. It aims to address a major challenge that limits traditional in situ underwater coral survey research: the difficulty of automatically generating accurate and reproducible high-resolution maps of the underlying coral reef ecosystems. Understanding recent achievements and their relevance to coral ecology monitoring needs is crucial for future planning. This paper presents a literature review on underwater coral image segmentation, focusing on the deep learning implementation pipeline. Furthermore, we introduce a new densely annotated dataset specifically designed for the semantic segmentation of underwater coral images. We systematically evaluate State-of-the-Art (SOTA) methodologies and novel techniques not previously applied to coral image semantic segmentation using the proposed dataset. We then discuss their feasibility in this context. Our goal for this review is to spark innovative ideas and directions for future research in underwater coral image segmentation and to provide readers with an accessible overview of some of the most significant advancements in this field over the past decade. By accomplishing these objectives, we hope to advance research in underwater coral image segmentation and support the development of effective monitoring and conservation strategies for coral reef ecosystems.

KEYWORDS:

• Deep learning
• underwater videography
• remote sensing
• semantic segmentation
• coral reef monitoring

1. Introduction

Coral reefs are a vital part of the marine ecosystem, akin to towering structures in a cityscape, offering essential habitats for a wide variety of marine life, as well as delivering significant economic and cultural benefits to coastal communities worldwide (Asner et al. 2022; Bewley et al. 2015; Muruga, Siqueira, and Bellwood 2024). Unfortunately, coral reefs are significantly threatened by human activities, including overfishing, marine pollution, destructive fishing practices, and coastal development, along with the natural threats of global warming and ocean acidification (Horoszowski-Fridman et al. 2024; Hughes et al. 2017). The cumulative impact of these pressures has led to widespread degradation of coral reef ecosystems. Currently, the Status of Coral Reefs of the World report indicates a loss of approximately 14% of coral reefs since 2009, with projected losses reaching 70%-90% if global warming exceeds 1.5°C (Benkwitt et al. 2023; IPCC 2023; Souter et al. 2021). To provide early warnings, assess adverse factors, and ensure the resilience of coral reef ecosystems, continuous monitoring of their benthic communities is imperative. Regular remote sensing ecology monitoring of the status of coral reefs is essential for evaluating the adverse effects of stressors and enabling prompt responses (Asner et al. 2020; Zhong et al. 2023).

Over the past decades, satellite, aerial, and drone photogrammetry and remote sensing have played significant roles in the long-term and large-scale ecology monitoring of coral reefs (Casella et al. 2022; Giles et al. 2023; Liu et al. 2014; Lyons et al. 2024). Rich and effective image data serve as a visual gateway into the intricate dynamics of coral reef ecosystems, allowing scientists to intuitively understand their critical growth patterns, spatial relationships, and biodiversity. Compared to traditional direct observations by divers, remote sensing of coral reefs through these sensors has greatly accelerated the task of monitoring of coral spatial structures and distribution maps, improving the efficiency of data collection. However, this method faces several challenges that are difficult to overcome, primarily including adverse weather conditions and image distortion and occlusion caused by water surface reflection or refraction. These constraints collectively attenuate their capacity to furnish coral reefs with data of precision, accuracy, and detailed (Shihavuddin et al. 2013). Although ship-based remote sensing techniques can generate valuable information like bathymetry, reef percent cover, and rugosity information at higher spatial resolution (Asner et al. 2020; Shihavuddin et al. 2013), their use requires a significant trade-off between area coverage and access, especially in nearshore reef habitats. These beautiful coral reef habitats are full of terrifying undersea obstacles for ships. In the past decade, the use of underwater imaging technology for onsite data recording and archiving has become increasingly popular, making research into direct acquisition of coral in situ data via images or videos underwater more prevalent. Moreover, with advancements in image processing methods, an increasing number of marine scientists have begun to use annotated underwater coral data for statistical analyzes of coral habitat coverage and health status (Beijbom et al. 2012). In recent years, the rapid development of Unmanned Underwater Vehicle (UUV) capabilities for automated operations underwater has made long-term, low-cost remote sensing observations of coral ecosystems more accessible. This technology represents an underwater remote sensing monitoring method for coral habitats that is high-resolution and cost-effective (Qin et al. 2022; Zhang, Grün, and Li 2022).

With the acquisition of a vast amount of underwater coral remote sensing data, the traditional method of manually annotating coral images has resulted in a large volume of image data not being timely processed. Moreover, different people and annotation methods can introduce significant variabilities and biases (Curtis et al. 2024). A survey reported from the National Oceanic and Atmosphere Administration (NOAA) states that among the millions of underwater coral reef images acquired each year, only 1–2% are subsequently analyzed by experts (Pavoni et al. 2021). Meanwhile, valuable undersea scientific observation data from coral reef ecosystems lie dormant in various image libraries we have built, awaiting annotation, classification and mapping. Fortunately, in recent years, with the rapid development of photogrammetric computer vision and machine learning, automatic image segmentation technology has not only become fundamental to image understanding in general computer vision but has also emerged prominently in the task of image interpretation in underwater coral remote sensing monitoring.

1.1. Traditional image segmentation techniques

Image segmentation aims at grouping similar regions or segments of an image under their respective class labels. Traditional machine learning methods have been widely employed in image segmentation tasks, such as Support Vector Machine (SVM) (Wang, Fan, and Wang 2021) and K-nearest neighbor (KNN) (Li et al. 2023). These algorithms typically rely on pre-designed features like texture, color, and shape. During the training phase, they learn the relationship between these features and their corresponding labels using a dataset. Specifically, the SVM algorithm seeks to find a decision boundary to separate different categories, while the KNN algorithm determines categories by measuring the distance between samples, arguably making it the most simple and easy to understand machine learning algorithm. These approaches are capable of effectively recognizing and classifying underwater coral images (Beijbom et al. 2012, 2015; Shihavuddin et al. 2013). Although prior studies employing these algorithms have demonstrated commendable accuracy, their applicability across diverse classes or datasets remains a challenge. This challenge arises from the prerequisite of pre-designed specific features to better describe the target class. Given that each class and dataset possesses distinct salient features, adapting these algorithms to novel contexts proves intricate. Previous research has shown that traditional machine learning models have more classification advantages on small-scale datasets, while deep learning models exhibit better capabilities on large-scale data and recognition accuracy (Wang, Fan, and Wang 2021).

1.2. Deep learning-based techniques

The emergence and widespread application of deep neural networks have revolutionized traditional approaches in machine learning paradigms, significantly enhancing the capability to process complex image data and improving overall segmentation system performance. Hence, the advances in deep learning have provided researchers in a variety of fields with sufficient cause to reexamine traditional methods for image segmentation and to determine if deep learning approaches can indeed improve performance. One such field is coral reef ecology, where many approaches to assessing the ecological state of coral reef ecosystems entail the analysis of underwater image data on the spatial distribution of benthos. This data is commonly obtained from underwater images acquired in situ by human divers or by autonomous or remotely operated UUVs. Deep learning-based methods excel in automatically learning features necessary for segmentation. Through extensive research and practical applications, an increasing number of deep learning-based algorithms now contribute to the efficient identification of objects in coral reefs, improving the accuracy of image segmentation for underwater coral images. These algorithms have shown that modern deep learning architectures are indeed capable of outperforming conventional methods for segmentation in underwater coral reef images (King, Bhandarkar, and Hopkinson 2018; Lütjens and Sternberg 2021). However, challenges persist in the segmentation of target objects due to substantial morphological variability within and among coral species, coupled with diverse regional species pools. The remarkable strides made in recognizing specific objects, object classes, and scenes on benchmark datasets (e.g. Caltech 101 (Li, Fergus, and Perona 2004), Pascal (Everingham et al. 2010)) cannot be straightforwardly transferred to address the new challenges of recognition in underwater coral images. Consequently, the development of a systematic pipeline for accurate underwater coral image segmentation remains a complex challenge, demanding simultaneous consideration of processing speed, cost, and accuracy. Convolutional Neural Networks (CNNs) have been a cornerstone in the field of deep learning, with their prominence surging in the 2000s. The iconic event of AlexNet, designed by Alex Krizhevsky and others, winning the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 with a significant margin showcased the powerful capabilities of CNNs in image recognition tasks (Krizhevsky et al. 2012). This milestone marked a turning point, significantly boosting interest and research in deep learning. This was facilitated by advancements in computing power, the availability of large-scale image datasets, and improvements in neural network training techniques (LeCun, Bengio, and Hinton 2015). According to published references, commonly employed neural networks in coral image segmentation tasks include Residual Network (ResNet), UNet and DeepLab series models (Pavoni et al. 2019, 2020; Zhang, Grün, and Li 2022). These deep neural networks can extract multi-scale context information from the image through convolution operation and demonstrate better generalization compared to shallow networks with the same parameters, which is exactly needed in the coral feature extraction. In addition to this, the Transformer, a novel neural network architecture based on attention mechanism, has revolutionized in computer vision in recent years, but it has not been introduced in underwater coral image recognition and segmentation (Strudel et al. 2021). While coral image classification based on deep learning is a promising trend, it is also limited by key factors such as dataset partitioning, data augmentation, network structure, and generalization ability.

Several articles have reviewed SOTA techniques in deep learning for coral remote sensing (Hedley et al. 2016; Mittal, Srivastava, and Jayanth 2022; Moniruzzaman et al. 2017; Raphael et al. 2020). These reviews often cover a wide range of issues, from aerial to underwater perspectives in remote sensing, with a primary focus on satellite or aerial coral remote sensing. While there is a growing number of studies on the classification and segmentation of underwater coral remote sensing images, few reviews capture the latest research advancements and cross-domain developments in image segmentation. To address this gap, this paper will summarize and analyze existing literature and methods in deep learning for underwater coral image segmentation, discussing the potential application of the latest research progress from other fields. It is worth noting that accurately distinguishing corals from other seabed elements in underwater images and obtaining precise semantic information are among the most pressing issues in coral surveys. Therefore, these coral image semantic segmentation methods mainly utilize datasets with annotations to train segmentation networks. This fully supervised approach to coral image semantic segmentation has shown great potential in improving the accuracy of coral reef ecological monitoring and analysis. Consequently, our discussions will mainly focus on fully supervised segmentation methods, providing a comprehensive review of current deep learning-based methods for underwater coral image segmentation. We will cover the key steps required for implementing deep learning-based underwater coral image segmentation methods, including data annotation, data augmentation, data partitioning, neural network models, loss functions, and evaluation metrics. Figure 1 illustrates the improvements in these steps. We will conduct comprehensive custom experiments to validate representative or newly advanced methods, with the goal of assisting coral ecologists and spatial information remote sensing scientists engaged in this field, thereby promoting research innovation.

Figure 1. The development trend of coral image segmentation. Data annotation: This process involves marking different regions of an image with labels that represent specific categories or objects. Data partitioning: This step includes not only dividing the dataset into training, validation, and test sets but also segmenting large images into smaller patches suitable for network input. Data augmentation: This technique is used to increase the diversity of your dataset by applying a series of transformations to the original image data. Multi-model data preparation: This involves integrating various types of data sources or modalities to improve the segmentation performance of deep learning models. Network design and optimization: These processes involve selecting and refining the architecture of the neural network to improve its performance in distinguishing between different segments of an image. Loss function selection: This is a critical step that can significantly influence the effectiveness and efficiency of a deep learning model in image segmentation tasks.

2. Background

As previously mentioned, deep learning is considered one of the most recent and astonishing advancements in the field of machine learning. Its ability to autonomously learn complex features from large datasets has surpassed traditional machine learning methods, particularly in areas such as image segmentation and recognition. This has made it the leading technology for understanding and processing images. In underwater coral image segmentation, using annotated training images of underwater corals as input for deep learning image segmentation models allows for the extraction of features from various coral objects, ultimately enabling the automatic segmentation of coral images to obtain the semantic maps of coral habitats we need. This has become a crucial task in image-based coral reef ecological monitoring. Currently, based on a review of the literature in this field, this paper categorizes the deep learning-based supervised neural network segmentation methods for underwater coral images into two groups: patch classification based on random point annotation and semantic segmentation based on pixel-level labels. By delineating differences in input data and network output, the segmentation methods are further refined into three distinct types: patch-based classification, superpixel-based segmentation and software-labeled segmentation. Notably, superpixel-based segmentation and software-labeled segmentation are both identified as forms of semantic segmentation based on pixel-level labels, as illustrated in Figure 2.

Figure 2. The process diagram of (a) patch-based classification, (b) superpixel-based segmentation, and (c) software-labeled segmentation.

2.1. Patch classification based on random point annotation

This method involves segmenting underwater coral images into multiple small blocks or “patches” and classifying them based on annotations of random points within these patches. Each patch is assigned a category label based on the features it contains, and these patches can then be further processed to produce a classification outcome for the entire image. The research community refers to patch classification based on random point annotation as “local image segmentation,” where low-resolution patches can be assigned class labels, then color-coded and combined to create classification masks. These methods are generally based on CNN architectures, widely used for various computer vision tasks, including convolution layer, pooling layer, full connection layer and output layer. In the context of underwater coral image segmentation, patch classification based on random point annotation typically involves extracting non-overlapping cropped patches from high-resolution images as inputs to CNN models. Studies have shown that considering various patch sizes during training, rather than relying solely on a fixed scale for input data (Modasshir, Li, and Rekleitis 2018). The final classification target in CNNs can be either an image patch or the entire image, predicting each patch’s class label or assign a class to the entire image based on majority vote among patch labels (Roy et al. 2019). Similarly, images can be subdivided into overlapping patches for classification, subsequently assembling them into the probability heatmap of the entire image, highlighting potential target areas (Lam et al. 2018; Wang et al. 2016).

In the field of coral image segmentation, there have been the application of various CNN architectures, such as Inception V1 and Inception V3, have been applied in the field of coral image segmentation. These architectures are often combined with a set of CNNs (Lu et al. 2015) or a two-stage framework (Yan et al. 2016) to extract features from multiple image patches. Additionally, researchers have explored novel convolution operation, where random patches extracted from the image serve as the convolution kernels to generate feature maps with input data (Xu et al. 2018). Although CNNs are the most common and effective automatic feature extractors in image segmentation tasks, other automatic feature extractors include Stacked Sparse Autoencoder (SSAE) by Shan and Li (2016), Deep Belief Network (DBN) (Hirra et al. 2021), and Long Short-Term Memory (LSTM) (Bunk et al. 2017), all of which can be used for image classification or segmentation. To capture spatial relationships between patches and restore global context, some networks have incorporated attention mechanisms (Wang et al. 2016). In the context of neural networks, the attention mechanism refers to a set of computational processes that enable the model to selectively focus on specific regions or features within the input data. It is worth noting that although the identification of precise boundaries through patch classification based on random point annotation is not the primary goal, the method focuses more on identifying and classifying the main content or objects present in the image (King, Bhandarkar, and Hopkinson 2018). Because it facilitates the rapid processing of large amounts of data, especially when annotation resources are limited, it is suitable for the rapid preliminary determination of metrics such as coral cover.

2.2. Semantic segmentation based on pixel-level labels

Unlike patch-based classification, semantic segmentation focuses on classifying each pixel in the image, aiming to segment the image into multiple regions representing different categories (e.g. various types of corals, underwater terrain, etc.). Each pixel is assigned a category label, thereby generating a detailed understanding of the entire image content. Due to the complexity of coral structures, two different label generation approaches are commonly used for pixel-wise coral segmentation, addressing the annotation challenges. The superpixel-based method extends the ground truth labels from annotated points to estimate the main object boundaries in the image, providing a label for each pixel. This concept has been widely applied in medical image segmentation, achieving a coarse-to-fine division of pathological regions through superpixel segmentation (Albayrak and Bilgin 2019; Daoud et al. 2019; Kopecky et al. 2023; Zhang et al. 2019). Moreover, some post-processing techniques for optimization have been explored (Xiong et al. 2017). Generating pixel-wise labels is relatively complex and typically involves the use of open-source semi-automated annotation tools.

In recent years, semantic segmentation based on pixel-wise annotation has gained popularity in coral image classification, owing to the advancements in Fully Convolutional Neural Networks (FCNNs). FCNN represents a significant advancement by adapting an existing CNN for semantic segmentation tasks. It can process images of any size and replaces fully connected layers with convolutional layers to produce spatial feature maps instead of feature vectors (Long, Shelhamer, and Darrell 2015). Besides FCNN architectures, encoder-decoder architectures have also become popular in pixel-wise coral segmentation (Badrinarayanan, Kendall, and Cipolla 2017; Ronneberger, Fischer, and Brox 2015). This design effectively leverages the spatial information of the image to achieve precise segmentation. Additionally, several methods are proposed to improve the network grasp of global context information, including the use of conditional random field (Chen et al. 2014), dilated convolutions (Mahmood et al. 2017; Yu and Koltun 2015), skip connection (Ronneberger, Fischer, and Brox 2015), attention mechanisms (Hu, Shen, and Sun 2018), multi-scale processing (Chen et al. 2018), and so on. Figure 1 showcases three different approaches to underwater coral image segmentation or classification, clearly illustrating the differences in annotation methods and the significant variation in network prediction results from CNN to FCNN. Specifically, patch-based classification provides a coarse-grained classification of coral types in the image, superpixel-based segmentation offers a more compact and structured representation of the coral reef, and dense-labeled segmentation delivers pixel-level accuracy in identifying coral and other underwater elements. Although semantic segmentation based on pixel-level labels requires precise annotation of each pixel’s category in the image, this method offers a deep understanding of the image content, suitable for applications requiring high-precision analysis of underwater coral reefs. Of course, the choice of method depends on the specific monitoring goals and available resources.

3. Benchmark datasets and challenges in underwater coral image segmentation

3.1. Benchmark datasets

Traditional image segmentation methods rely on predetermined color and texture classifiers to segment underwater coral images, limiting their flexibility and adaptability when dealing with complex environments (King, Bhandarkar, and Hopkinson 2018; Shihavuddin et al. 2013). In contrast, human do not just rely on visual cues like color and texture for object recognition; they also utilize a wealth of contextual knowledge, a process that traditional methods often fail to mimic. As mentioned earlier, compared to traditional approaches, deep learning techniques can effectively model domain knowledge using datasets of labeled pixels. This means that deep networks can capture more complex image features and patterns, more effectively mimicking human visual perception and handling complex image segmentation tasks with increased accuracy and robustness. However, the success of deep learning methods heavily depends on large, diverse datasets of labeled data. These datasets provide the necessary information to train networks to recognize a variety of image features and objects, making the construction and management of these datasets key to efficient image segmentation. Moreover, for a fair evaluation and comparison of different image segmentation algorithms, standardized datasets representing specific domains are needed. This ensures the validity of comparisons while also fostering collaboration and progress within the research community. This section highlights some of the popular and diverse datasets used for coral image segmentation, which vary in size, annotation methods, and the locations and specific environments from which the data was collected, as shown in Table 1.

Table 1. Coral benchmark datasets.

Name	Publication year	Label	Depth	Color balancing	#Class	#Image quantity scale	Location
TasCPC	2011	Sparse	20 to 95 m	Done	13	Thousands	Tasmania
MLC	2012	Sparse	2 to 5 m, 10 m, 17 m	None	9	Thousands	Moorea
EILAT	2013	Sparse	–	Done	8	Thousands	Eilat in Red Sea
RSMAS	2013	Sparse	–	None	14	Hundreds	Different locations
Benthoz15	2015	Sparse	10 to 50 m	Done	148	Thousands	Australia
EFC	2016	Sparse	3 to 15 m	Done	9	Hundreds	Eilat
UCSD Mosaics	2017	Dense	10 m	None	34	Thousands	Palmyra Atoll
SSPQICD	2019	Sparse	10 m	Done	182	Millions	860 locations
ATCRC	2020	Sparse	~3 to 9 m	None	47	Hundreds	Curaçao

Here is a more detailed introduction and the web addresses where these datasets, listed in Table 1, can be accessed.

• TasCPC (Barrett et al. 2011): The Tasmania Coral Point Count (TasCPC) dataset contains of 1,258 benthos images captured by Autonomous Underwater Vehicle (AUV), with each image having 50 random point annotated by experts. The dataset includes 13 types of labels covering biological species (such as types of sponges, corals, algae and others), abiotic elements (such as sand, gravel, rock, shells, etc.), and categories of unknown data, all these labels are indicated in CSV file containing the image meta data. Available at http://marine.acfr.usyd.edu.au/datasets/.

• MLC (Beijbom et al. 2012): The Moorea Labeled Corals (MLC) dataset is a subset of the Moorea Coral Reef Long Term Ecological Research (MCR LTER) project, which has been collecting image data from Moorea Island since 2005. It includes 2,055 images from 2008, 2009, and 2010, collected from three habitats: fringing reef, outer 10 m, and outer 17 m. The dataset features random point annotation (row, column, label) with nine labels, including five coral genera: (1) Acropora, (2) Pavona, (3) Montipora, (4) Pocillopora, and (5) Porites, along with four non-coral labels: (6) Crustose Coralline Algae (CCA), (7) Macroalgae, (8) Sand, and (9) Turf algae. These nine classes account for 96% of the annotations, totaling almost 400,000 points. Available at http://vision.ucsd.edu/data.

• EILAT (Shihavuddin et al. 2013): The EILAT dataset contains 1123 image patches of size 64 × 64, taken with the same camera from coral reefs near Eilat in the Red Sea, and it was named after the location, Eilat. Experts have visually classified these images into eight classes: branches type III, branches type II, branches type I, brain coral, urchin, favid coral, dead coral and sand. Available at https://data.mendeley.com/datasets/86y667257h/2.

• RSMAS (Shihavuddin et al. 2013): The RSMAS dataset consists of 766 image patches of size 256 × 256, collected by divers from the Rosenstiel School of Marine and Atmospheric Sciences at the University of Miami. The images, taken with different cameras at different times and places, and include 14 classes: Acropora cervicornis, Acropora palmata, Colpophyllia natans, Diadema antillarum, Diploria strigosa, Gorgonians, Millepora alcicornis, Montastraea cavernosa, Meandrina meandrites, Montipora spp., Palythoas palythoa, Sponge fungus, Siderastrea siderea and tunicates. Available at https://data.mendeley.com/datasets/86y667257h/2.

• Benthoz15 (Bewley et al. 2015): The Australian benthic dataset contains 407,968 expert-labeled points of 9874 georeferenced images with associated sensor data which captured by AUVs, and up to 50 pixels were randomly annotated in each image. The images were collected from nine sites around Australia over 2008–2013. The dataset features 148 kinds of expert labels, available in a CSV file where each line represents a single labeled point within an image. Available at http://squidle.acfr.usyd.edu.au.

• EFC (Beijbom et al. 2016): The EFC dataset divides the 212 annotated image-pairs into 142 training image-pairs and 70 verification image-pairs. Each image-pair contains a reflectance image and its corresponding fluorescence image, with 200 annotated points in each image. The dataset has nine labels: Acropora, Bare-subst., Faviidae, Millepora, Platygyra, Pocillopora, Other Inv., Other Hard Coral and Unknown. Available at http://10.5061/dryad.t4362.

• UCSD Mosaics (Edwards et al. 2017): UCSD Mosaics is the only publicly available coral dataset that provides dense ground truth annotations. The original dataset contains 16 mosaics with resolution of over 10K × 10K. The version of the dataset provided by Alonso et al. (2019) includes 4193 training images and 729 test images, each with a size of 512 × 512 pixels. The dataset features 34 dense semantic labels in addition to the background class. Available at https://sites.google.com/a/unizar.es/semanticseg/home.

• SSPQICD (González-Rivero et al. 2019): The Seaview Survey Photo-quadrat and Image Classification Dataset encompasses over one million standardized downward-facing “photo-quadrat” images, each covering approximately 1 m² of the seafloor. These images were collected between 2012 and 2018 at 860 transect locations worldwide, including regions like the Caribbean, Bermuda, Indian Ocean, etc. and also feature multi-temporal images for specific sites. The dataset contains a total of six function groups (Algae, Sponge, Hard Coral, Soft Coral, Other Invertebrates, and Others) and 182 subcategories. It includes 11,383 expert-annotated training images with 50 random points, and 1.1 million images. Available at https://espace.library.uq.edu.au/view/UQ:734799.

• ATCRC (Rashid and Chennu 2020): The dataset comprises 147 underwater scenes of coral reefs near Curaçao, surveyed between August 4th and 26th, 2016. Captured through hyperspectral imaging, each 50-m transect contains approximately 2.29 billion pixels, providing spectral data across 400 wavelengths. Annotations of 31 hyperspectral images, including 23 comparative transects, detail 47 labels for habitat descriptors and biodiversity, focusing on sessile biota, substrate types, and other abiotic elements. Available at https://doi.org/10.3390/data5010019.

3.2. Challenges

In this section, we will explore the key challenges encountered by deep learning algorithms when effectively delineating corals in underwater images. Our exploration encompasses a thorough examination of diverse factors that exert a substantial impact on segmentation accuracy, as outlined below:

3.2.1. Image quality

Due to the complex imaging environment and challenging illumination conditions, effects like color shift, scattering, lens distortion, and chromatic aberration can modify the visual attributes of objects (Fayaz, Parah, and Qureshi 2023; Zhou et al. 2024). Coral reefs often display blurred outlines and misclassification at varying distances, causing confusion for both human observers and recognition algorithms (Pavoni et al. 2021). In Figure 3(a), the left image represents a standard, unprocessed underwater image. To address the color defects in underwater images, researchers have conducted extensive studies on restoring their degradation using deep learning techniques (Islam et al. 2020a; Li et al. 2019; Zhou et al. 2023, 2024). These approaches aim to enhance the visual quality and improve the suitability of underwater images for subsequent analysis. Some work explores the use of orthophotos for coral segmentation (Pavoni et al. 2019, 2020). Orthophotos, corrected for geometric distortions, resulting in a map-like representation that fixes the distance from the lens to the object to ensure consistent perspective across all data. However, it is essential to acknowledge that the interpolation process involved in orthophoto generation may lead to local ghosting and blurring. The stitching and mosaicking operations may also result in geometric distortion of the final map. As demonstrated in the right image of Figure 3(a), it shows the potential distortion on corals caused by orthographic image processing. In conclusion, image quality significantly impacts the difficulty of identifying and segmenting coral regions, thus affecting the overall accuracy of image segmentation.

Figure 3. Challenges in coral image segmentation: (a) color defect of underwater image and distortion due to orthophoto generation; (b) complex morphology of corals and background in distinguish ability; (c) variations in abundance among coral classes; (d) mislabeling of dead corals as alive by experts (pink: live Pocillopora, light pink: dead Pocillopora, light blue: Porites).

3.2.2. Complex morphology

Regarding the dense population of coral reefs, it is crucial to account for the complex spatial morphology exhibited by various corals. Coral reef skeletons are hugely variable in morphology, including branching, encrusting, or massive forms. These differences reflect the innate characteristics of different coral species that change with age (Todd 2008). Moreover, significant intraspecific variability exists among samples within the same class. Identifying dead corals presents a unique set of challenges, as their surfaces often display distinctive features, such as wormholes, rust spots, or extensive algae coverage. These visual changes are due to higher rates of bioerosion in dead corals compared to their living counterparts (Glynn and Manzello 2015). Figure 3(b) provides a visual representation of the complex morphology observed in coral reefs, further emphasizing the difficulties associated with the segmentation tasks.

3.2.3. Class imbalance

The distribution of corals is closely related to their habitat. Generally, there are significant variations in the abundance of distinct coral functional groups within the same area, often by factors of ten or even hundreds (Bellwood et al. 2004). The distribution pattern results in images captured having imbalanced frequencies to varying degrees. The presence of class imbalance has a detrimental effect on classifier output, as it tends to pay of foreground class to varying degrees. The presence of class imbalance detrimentally affects the classifier’s output, as it tends to focus more on the major classes while penalizing the minority ones. As depicted in Figure 3(c), notable disparities in coverage among different coral classes. In this area, Porites, marked in light blue, represents an underrepresented class, make it a challenging case for the accurate automatic segmentation due to the scarcity of positive examples.

3.2.4. Human error

Human observers can make recognition errors in recognizing different benthic groups in underwater videos or images (Curtis et al. 2024). Interestingly, hard corals, which play a vital role as builders and creators of coral reef ecoysystems, exhibited the least accurate and most variable classification levels (Ninio et al. 2003). Furthermore, the subjective annotation of training samples by different experts also introduces an additional factor that may impact the network’s reliability. An example illustrating this effect is presented in Figure 3(d), where an expert erroneously labels a dead Pocillopora as alive, potentially leading to misguided network learning. This highlights the substantial influence of human error on the quality of training data and subsequent segmentation results. To address these concerns, establishing robust protocols for expert annotation and ensuring inter-annotator agreement to enhance the overall reliability of the dataset.

4. Data annotation, partitioning, and augmentation

4.1. Data annotation

The process of transforming captured underwater coral images into quantitative data is a resource-intensive task, demanding significant time and labor. In the domain of marine research, there is a growing need for efficient image analysis tools to assist coral class annotation. One such tool is CoralNet Beta, which employs random point annotation, and its standard operating procedures have been comprehensively documented in certain studies (Lamirand et al. 2022). In 2021, CoralNet 1.0 was launched as a new iteration of this tool, boasting an impressive 18.4% reduction in annotation error rates (Chen et al. 2021a). Other annotation tools based on random point annotation have also been succinctly summarized and organized in relevant literature (Ayroza et al. 2015; Gomes-Pereira et al. 2016). For simplicity, we provide a straightforward list in Table 2. However, to fulfill the demands of semantic segmentation, superpixel techniques can extend point annotations into approximate dense semantic labels (Alonso et al. 2017). The selection of sparsity in random sampling for superpixels also exerts a notable influence on segmentation outcomes (Yuval et al. 2021). Subsequent research on this tool has yielded several enhancements in both accuracy and processing speed (Alonso et al. 2019; Pierce et al. 2020). Nevertheless, acquiring precise pixel-wise labels for high-precision tasks, such as fine-grained image segmentation, is still closely linked to human involvement. Various annotation tools for general object segmentation, including Labelbox, are available and are also employed in the semantic segmentation of underwater coral images. Recently, there have been advancements in manual annotation techniques utilizing semi-automatic pixel-wise annotation tools in specific studies (King, Bhandarkar, and Hopkinson 2018, 2019; Pavoni et al. 2020). An exemplary open-source pixel-wise segmentation tool called TagLab (Pavoni et al. 2022), meticulously tailored for underwater coral imagery, has emerged in this context. It uses an optimization of DeepLab V3+ to support the analysis of large orthophotos generated through the photogrammetric pipeline. These advancements in image annotation tools and techniques represent progress in the field of underwater coral image segmentation. They enable researchers to more effectively annotate underwater coral images, thereby expediting the analysis process and enhancing the accuracy of subsequent segmentation models. Notably, AI-assisted annotation tools within software, like those found in CoralNet and TagLab, are increasingly utilized for underwater coral image automatic segmentation and analysis (Kopecky et al. 2023; Williams et al. 2019).

Table 2. Annotation tools used in coral image segmentation.

	Tool	OS	Costs	Code availability	Output file	Description
Point Annotation	CATAMI	Web based	Free	Yes	.csv	-public access or data private -three point sampling methods (random, fixed 5-point, and uniform grid)
	CoralNet	Web based	Free	Yes	.csv	-public access or data private -three point sampling methods (random, stratified random, and uniform grid) -machine learning-based annotator
	CpCe	Win	Free	–	.xls	–four point sampling methods (random, stratified random, uniform grid and equally spaced grid) –code file integrity checker
	Squidle+	Web based	Free	–	–	-public access or data private -three point sampling methods (random, stratified random, and uniform grid) -preloaded label set
Pixel-wise Annotation	Labelme	Win, Linux	Free	Yes	.json	-object-labeled -two annotation types (polygon boundary and bounding boxes)
	Labelbox	Web based	Free for 5k images	Yes	.csv, .json, .xml	-AI-assisted annotation -two annotation types (polygon boundary and bounding boxes)-
	TagLab	Win, Linux	Free	Yes	.jpg, .tiff, .csv,	-AI-assisted annotation -semi-automatic annotation (4-clicks tool/positive-negative clicks tool to label boundary) -automatic refine operation

4.2. Data partitioning

When training a neural network model using data from a dataset, it is essential to consider an optimal strategy. This strategy demands ensuring that the model is provided with an adequate number of training examples during the training process while avoiding overfitting. Simultaneously, one must recognize that insufficient training examples will result in inadequate model training, leading to poor performance during the testing phase. In the case of large datasets, a common practice is hold-out validation. In this approach, the available input data is randomly divided into two parts: a training dataset for model learning and a validation dataset for tuning the network’s hyperparameters. In certain instances, a separate test dataset may also be employed to evaluate the model’s performance and generalization ability. For datasets with a limited number of samples, the prevalent technique is k-fold cross validation. The sample data is partitioned into k folds and k-1 folds for learning and one-fold for validation. The evaluation results are then averaged across k iterations, with k = 5 being a common choice in coral segmentation scenarios (Gómez-Ríos, Tabik, Luengo, Shihavuddin, and Herrera 2019; Gómez-Ríos, Tabik, Luengo, Shihavuddin, Krawczyk, et al. 2019; Xu et al. 2019). Researchers have also explored various techniques to balance the distribution of training and validation datasets. Steffens et al. (2019) created N fixed-size training and validation sets from randomly selected images and applies cosine distances weighted by the overall class-distribution to compare the training/validation distributions. In another study (Mizuno et al. 2019), coral regions are segregated into low transparency datasets and high transparency datasets based on coral coverage for coral prediction evaluations. Moreover, when dividing multiple images as a dataset directly from an entire coral map, a biologically-inspired dataset partition method can be employed (Pavoni et al. 2019). This method assigns weights to three landscape ecology metrics describing the spatial patterns of benthic communities (measuring the standard deviation, quantity and density of specimens respectively) as similarity scores (S). Finally, the three sub-areas with the best S value are selected as three distinct datasets. These approaches emphasize the significance of appropriate data partitioning and provide valuable insights into the various strategies employed in underwater coral image segmentation studies.

4.3. Data augmentation

Data augmentation serves as a crucial solution to address the challenges posed by limited data in deep learning. It improves the robustness and generalization capabilities of deep learning models by expanding the size and improving the quality of training datasets. underwater coral image segmentation, as an application domain, faces constraints in accessing extensive datasets due to the high costs associated with acquiring high quality images of marine communities in underwater environments. To overcome this limitation, various data augmentation algorithms have been developed, including geometric transformations, colorimetric transformations and Generative Adversarial Network (GAN)-based methods, etc.

Geometric transformations such as flipping (horizontal or vertical), cropping, scaling, rotation, and translation, are commonly used to introduce additional variations in the data and improve model performance. In terms of color correction, previous studies have shown that coral image preprocessing in Lab color space has better performance than RGB, HSV or gray color space (Beijbom et al. 2012). A Lab color space is a color-opponent space with dimension L for lightness and a and b for the color-opponent dimensions of redness – greenness and blueness – yellowness, respectively, based on nonlinearly compressed CIE XYZ color space coordinates. Moreover, researchers have discovered that the Cr band in the YCbCr color space effectively mitigates the impact of illumination changes and shadows, making it well-suited for habitat segmentation (Mohamed, Nadaoka, and Nakamura 2022). Other color spaces such as CIELAB and YUV have also been employed in various computer vision tasks and could potentially offer benefits for coral image analysis (Poongodi, Hamdi, and Wang 2022). In addition to color space conversion, color normalization is one of the most commonly employed data augmentation methods to improve the contrast in submarine coral images (Pavoni et al. 2021).

However, before employing these basic data augmentation algorithms, a unique challenge arises in the context of large-scale and small-number high quality underwater coral remote sensing images. This problem mainly involves two aspects: how to efficiently extract as much useful information as possible from a limited number of images, and how to deal with and balance the issue of class distribution imbalance in the dataset. This is particularly relevant for methods using orthomosaic images for image segmentation. The conventional approach involves using a sliding window division to mitigate these challenges. It typically serves as the first step in data extraction, generating a large number of training samples from large-size images, but it may fail to address the problem of class imbalance as it provides equal sampling probability for each target (Shihavuddin et al. 2013). There is also a data partition method based on Poisson disk sampling, which can generate non-overlapping random samples through a Dart Throwing algorithm with a class-dependent radius (Pavoni et al. 2020). These oversampling strategies leverage the fact that certain corals frequently cover smaller areas, and are techniques used to deal with the issue of class imbalance by increasing the number of samples of the minority classes (either by duplicating existing samples or generating new ones) to balance the distribution of classes in the dataset. Hard negative sampling is a strategy of selecting those negative samples that the model incorrectly predicts (i.e. samples that the model finds difficult to classify correctly) for focused training (Kim, Hong, and Byun 2021). This method helps the model better learn to distinguish hard-to-recognize categories or patterns, improving the model’s robustness and accuracy. These two strategies are essentially further processing of samples on the basis of sliding window sampling, as previously mentioned, focusing respectively on addressing the class imbalance and learning difficulties that may arise in datasets generated by sliding window sampling.

To further effectively expand the number of available images, Pavoni et al. (2019) propose an oversampling method based on the actual area coverage of the specimens. A cutting window technique is applied to subdivide each annotated coral into a set of overlapping subparts, the number of which is proportional to the size of the coral. These are added to the dataset to address class imbalance and ensure the representation of both small and large corals during network training. Furthermore, GAN-based data augmentation techniques have proven effective in improving the accuracy of underwater image classification by training a generator network to generate synthetic data with statistical properties similar to the original data (Xu et al. 2017). However, while GANs (Goodfellow et al. 2014) have been utilized for underwater image segmentation, their application in marine biological data, particularly for such organisms with complex morphologies like corals, remains an area that requires further exploration. In the future, it could be beneficial to explore more deep learning-based oversampling strategies for enhancing underwater coral image samples, such as neural style transfer (Tobin et al. 2017; Ye et al. 2023) and meta-learning schemes (Gama et al. 2023; Zoph and Le 2016), which could help generate new samples reflecting underrepresented coral features.

5. Network design and optimization techniques

The intricate structures and growth patterns of corals contribute significantly to the high morphological variation observed within and among populations. This diversity makes it challenging to provide comprehensive descriptions of corals using features from a single type, hierarchy, or scale. Consequently, there is a pressing need to establish generalized characteristics that can effectively describe corals within the context of underwater environments. Given the complexities of coral morphology, it becomes imperative to focus on the design, selection and optimization of network models suitable for underwater coral image segmentation. Moreover, the application of techniques like transfer learning and ensemble methods are also critical. These approaches play a pivotal role in our quest for accurate and robust coral image segmentation, ultimately advancing our understanding of coral ecosystems.

5.1. Convolutional neural network (CNN)

5.1.1. Network design and hierarchical feature extraction

Elawady (2015) first introduces CNN for coral classification task. They adapt a network model based on LeNet-5, with the input layer comprising three basic color channels, along with additional channel for texture and shape descriptors. These descriptors include the Weber local descriptor, phase consistency, and zero component analysis (Song et al. 2016) presents a two-level hierarchical framework that includes two-level feature extractors designed to capture global and local features, respectively. Global feature extractor is trained using the entire image, while local feature extractors focused on different parts of the image, such as textures, shapes, and edges. The fusion of features from both levels is input into a linear SVM for the final prediction. Moreover, the combination of features from different convolutional layers of the network can result in a more powerful image representation (Mahmood et al. 2020) combines the features extracted from the last three convolutional layers of a pre-trained ResNet to yield higher classification accuracy for coral classification. These combined feature vectors are then input into classifiers such as SVM, Softmax, or a shallow CNN for classification.

5.1.2. Multi-scale and multi-model data input

A common technique in coral classification involves using patches of different scales centered on annotated pixels. This approach has been adopted in numerous coral classification studies (Mahmood et al. 2016a, 2018; Modasshir, Li, and Rekleitis 2018). Among these, the multi-patch network MDNet (depicted in Figure 4) proposed by (Modasshir, Li, and Rekleitis 2018) not only learns the texture and morphology of corals by cropping patches of different scales but also introduces dense connections between layers to reduce overfitting. Furthermore, leveraging multi-modal data can provide diverse information sources, thus addressing the limitations of single-modal data and enhancing the accuracy and robustness of segmentation models. Beijbom et al. introduce the application of FluorIS (Beijbom et al. 2016), a low-cost modified consumer camera, to capture wide-band wide-field-of-view fluorescence images. To obtain high-contrast corals as well as information on non-fluorescing substrates, this approach combines fluorescence and reflectance images. Similarly (Xu et al. 2019), utilizes a conditional GAN based image translator trained by paired multi-modal data from the EFC dataset to learn the mapping from reflectance to fluorescence images, generating translated images that provide complementary information. Moreover (Mahmood et al. 2016b), utilizes the approach proposed in (Shihavuddin et al. 2013) which incorporates Gabor filter responses, Gray Level Co-Occurrence Matrix, and Complete Local Binary Pattern as texture descriptors, along with relative angle and hue color histograms as color descriptors to represent hand-crafted features. These features are then combined with the corresponding CNN features obtained from multi-scale patches. The nViewNet, proposed by (King, Bhandarkar, and Hopkinson 2019), introduces feeding images corresponding to each viewpoint into the trained network for classification. Each image casts a vote for the grid face to predict its class. Finally, nViewNet aggregates the votes of all images and outputs the class label with the highest number of votes as the final classification label for that mesh face. The purpose of this approach is to leverage information from multiple viewpoints to improve classification accuracy.

Figure 4. MDNet. (a) The structure of MDNet; (b) the structure of Dense Block.

5.1.3. Transfer learning and network integration

In addition to network design and data preparation, transfer learning has been employed to enhance the performance of coral image classification. Various backbone networks, including VGG16, ResNet50, ResNet101, Efficientnet-B0, Efficientnet-B1 and Efficientnet-B4, are tested for their suitability in the proposed CoralNet classification engine (Chen et al. 2021a). Experimental results show that EffecentNet-B0 achieves the balance of accuracy and speed (Gonzalez-Rivero et al. 2020). utilizes the VGG-D 16 convolutional neural network architecture, pretrained on the large ImageNet dataset, and fine-tuned its parameters with a specific coral training dataset (Jackett et al. 2023). develops a deep learning system for automatic reef-building coral identification, employing the ResNet50 network for transfer learning. The pre-training fine-tuning method demonstrated superior learning efficiency compared to random initialization and fixed pretraining. Network integration methods have also been explored (Marre et al. 2020). extracts the features from global average pooling layers of four ResNet18s and feeds them into a Multilayer Perceptron made of three fully connected layers. In another study (Gómez-Ríos, Tabik, Luengo, Shihavuddin, and Herrera 2019), a two-stage classifier consisting of three ResNets is designed to automatically classify corals from texture and structure images. Multiple CNN models, including DenseNet (Huang et al. 2017), are fine-tuned using different strategies, and their predictions are combined to achieve higher performance compared to a single CNN model (Lumini, Nanni, and Maguolo 2020).

5.2. Fully Convolutional Neural Network (FCNN)

Different from a typical CNN that uses a fully connected layer to output a fixed-length feature vector after the convolutional layer, FCNNs employ a deconvolution layer to upsample the feature map from the last convolutional layer, recovering it to the original size as the input image. Meanwhile, FCNNs offer the advantage of processing images of any size while preserving the original spatial information. In the field of semantic segmentation of underwater coral images, popular neural networks can be categorized into three main types: context-based, deconvolution-based, and feature-enhancement-based methods, which extract semantic features from different perspectives (Hao, Zhou, and Guo 2020). Here, we present examples of commonly used networks for each type in coral segmentation. DilatedNet (Yu and Koltun 2015), as a context-based approach, introduces a context module that systematically aggregates multi-scale context information by employing dilated convolutions with different dilation rates. This is achieved without introducing extra computational overhead. Both UNet and DeepLab v3+ are feature extraction-based networks that combine shallow and deep features to preserve local and global image details. UNet (Ronneberger, Fischer, and Brox 2015) is a symmetric encoder-decoder structure that employs skip connections for feature fusion on channel dimensions. DeepLabv3+ (Chen et al. 2018) takes DeepLabv3 (Chen et al. 2017) as an encoder and adds a simple yet effective decoder module to refine segmentation results. It integrates the Atrous Spatial Pyramid Pooling module, which enables multi-scale analysis and context aggregation. SegNet (Badrinarayanan, Kendall, and Cipolla 2017) has a similar structure to UNet, but it saves the location of pooling in index during the encoding phase and recovers it through unpooling, effectively recovering spatial information. Bayesian SegNet (Kendall, Badrinarayanan, and Cipolla 2015) is an extension of SegNet using Bayesian-based models.

5.2.1. Network design and hierarchical feature extraction

In this section, we present various network improvements related to the semantic segmentation of underwater coral images (Mizuno et al. 2020). constructs a deep neural network based on UNet and integrates a 50% dropout layer into the first three decoder layers to mitigate overfitting and enhance model generalization (Zhang, Grün, and Li 2022) adds a residual refinement module after the UNet network output to improve the prediction boundary of the coral. DeeperLabC, an extension of the DeepLab family, is created to classify coral and non-coral areas in the single-channel images (Song et al. 2021). The backbone of DeeperLabC is derived from a fine-tuned pre-trained ResNet34, and a Class Activation Map (CAM) module is incorporated at the end to visualize feature maps for semantic segmentation (Islam et al. 2020b) introduces SUIM-Net (as shown in Figure 5), a deep residual model following the encoder-decoder architecture, with a core component called Residual Skip Block, which offers optional hierarchical skip connections on its core building block. When skip = 0, skip connection can be fed from an intermediate convolution layer; otherwise from the input for local residual learning.

Figure 5. SUIM-Net. (a) The structure of SUIM-Net; (b) the structure of Residual Skip Block.

5.2.2. Multi-model data input

Furthermore, leveraging multimodal data has emerged as a promising direction to enhance the prediction accuracy in deep learning-based coral semantic segmentation (King, Bhandarkar, and Hopkinson 2019). designs TwinNet, a stereo FCNN architecture utilizing the front end of Dilation8. Left-perspective and right-perspective images are processed independently within the network but share weights. The separated outputs of Dilation8 are fed to different Siamese subnetworks and concatenated on the channel axis before producing the final output. Distance information contained in depth maps is also available (Zhong et al. 2023). They effectively capture shape information in depth images by replacing vanilla convolutions with ShapeConv (Cao et al. 2021). The incorporation of grayscale information has also shown benefits in coral prediction. In (Thomas et al. 2022), they employ two distinct UNet models to handle RGB images and grayscale images separately. By combining outputs of dual-model methodology significantly improves the algorithm’s accuracy by effectively capturing the intricate features present in coral reef images. Even if one model fails, the other model can still continue to work and generate the corresponding masks.

5.2.3. Transfer learning and network integration

In the realm of underwater coral image segmentation based on deep learning, aside from model optimization, techniques such as transfer learning and ensemble networks are also commonly employed. For instance, in their work (Pavoni et al. 2019, 2020), choose the pre-trained CNN framework Bayesian SegNet and DeepLab v3+ for semantic underwater coral image segmentation task. Furthermore, selectively freezing the lower layer weights of neural networks is a popular fine-tuning method that encourages the network to adapt to the specialized features provided by the coral dataset while preserving the general characteristics of the standard dataset (Jackett et al. 2023; Thomas et al. 2022). In terms of ensemble methods (Sui et al. 2022), presents an automated three-stage parallel deep learning pipeline for processing coral data in turbid water conditions. First, color correction is performed using UGAN. Second, coral areas are distinguished from non-coral areas using UNet based on EfficientNet-B6. Lastly, coral species are classified using DenseNet-201. This ensemble framework for coral reef monitoring significantly reduces the time required for data processing, and parallel ensemble methods also demonstrate the stability in their calibration across dataset shifts (Wyatt et al. 2022).

6. Evaluation metrics and loss functions

6.1. Evaluation metrics

To evaluate the validity and effectiveness of segmentation methods, it is important to use standard and widely recognized metrics for a rigorous performance assessment (Garcia-Garcia et al. 2017). Here are some commonly used evaluation metrics for image segmentation that can be employed to quantify the performance of segmentation methods:

6.1.1. Pixel accuracy (PA)

It is the simplest evaluation metric, indicating the ratio of correctly classified pixels to all pixels in the image. However, it simply counts the correct pixels, which may not fully reflect the network’s performance, especially in scenarios involving multi-class segmentation.

6.1.2. Mean pixel accuracy (mPA)

Calculates the ratio of correctly classified pixels for each class to the total number of pixels within that class providing an average accuracy measure across all classes.

6.1.3. Precision, recall and f-score

Precision evaluates the model’s ability to correctly distinguish negative samples, while Recall reflects its performance on positive samples. The F-score, which is the harmonic mean of Precision and Recall, offers a balanced assessment of overall accuracy.

6.1.4. Mean intersection over union (mIoU)

It is widely regarded as the standard metric for segmentation tasks. It computes a ratio between the intersection and union of true labels and predictions for each class, and then averages the IoU values across all classes. In most cases, mIoU yields lower values compared to mPA because the denominator of the IoU expression for a class includes pixels that the network has misclassified.

6.1.5. Boundary IoU

It is proposed by Cheng et al. (2021). This metric focuses on evaluating boundaries errors between prediction results and ground truth. It calculates the set of intersection-over-union of mask pixels within a certain distance from the corresponding ground truth or prediction boundary contours. The inclusion of Boundary IoU allows for a more precise assessment of the quality of coral boundaries in segmentation results. While many studies on coral segmentation traditionally rely on metrics like mPA and mIoU for evaluation, recent works have introduced Boundary IoU as an additional evaluation metric to assess the quality of coral boundary delineation in segmentation results (Zhang, Grün, and Li 2022).

6.2. Loss functions

The loss function plays a pivotal role during the model training stage by quantifying the disparity between the predicted value and the ground truth. These functions can generally be categorized into four types: distribution-based, region-based, boundary-based, and compounded loss (Jadon 2020). In this section, we provide a brief overview of the commonly used loss functions in the context of coral image segmentation.

Distribution-based losses are designed to minimize differences between probability distributions of random variables within a dataset. One of the widely used distribution-based loss functions in coral image segmentation is Cross-Entropy loss. Cross-Entropy loss is favored for its ease of gradient calculation. However, in situations characterized by class imbalance, cross-entropy loss might bias the model toward predicting the majority class while neglecting the minority class. To address this issue, the Weighted Cross-Entropy Loss function is introduced. It assigns greater weight to specific classes, thereby enhancing the model’s learning efficiency for these classes. Another frequently adopted loss function for coral image semantic segmentation is the Focal Loss (Lin et al. 2017). By introducing the focal factor into the cross-entropy loss function, this approach successfully mitigates the class-imbalance between the foreground and background, which effectively reduces the weight of generous simple negative samples during training. Dice Loss (Milletari, Navab, and Ahmadi 2016), a standard region-based loss function, is well-suited for scenarios marked by severe class imbalance. It demonstrates greater robustness to class imbalances compared to cross-entropy loss. Dice loss incorporates both the ground truth and prediction results during its calculation, making it more sensitive to minority class predictions. Tversky Loss (Salehi, Erdogmus, and Gholipour 2017) represents another commonly employed loss function that controls the balance between precision and recall through the adjustment of two additional parameters. Additionally, Boundary Loss (Kervadec et al. 2021), as a boundary-based loss, is introduced to address issues associated with region-based loss in highly imbalanced segmentation tasks. In their exploration of various loss functions for model optimization (Pavoni et al. 2020), found that Focal Tversky Loss is the most efficient cost function for training coral segmentation models, though it may not be the ideal choice for handling datasets with extreme class imbalances. Consequently, the field often resorts to the use of weighted loss functions or hybrid loss functions to effectively address these challenges (Qin et al. 2022; Steffens et al. 2019).

7. Experiments and discussion

In this section, we conduct a comparative analysis of representative fully supervised semantic segmentation methods using a same specific dataset. More specifically, the selected competitors include MultiResUNet (Ibtehaz and Rahman 2020), DeepLab V3+ (Chen et al. 2018), UNet (Ronneberger, Fischer, and Brox 2015), TransUNet (Chen et al. 2021b), PointRend (Kirillov et al. 2020), CCNet (Huang et al. 2019), SegFormer (Xie et al. 2021), Segmenter (Strudel et al. 2021), SwinUNet (Cao et al. 2022), LEDNet (Wang et al. 2019), BiSeNet (Yu et al. 2018), ShelfNet (Zhuang et al. 2019) and STDC (Fan et al. 2021), covering multiple kinds of approaches discussed in Section 5, as well as state-of-the-art methods that have yet to be applied in the domain of coral semantic segmentation. For these methods, we employ the official models provided by the respective authors for our assessment. It is important to note that the existing published datasets differ in their annotation methods, acquisition scenarios, and data collection conditions, making them often tailored for specific research methods. Therefore, our evaluation was conducted on a custom coral dataset, which represents a significant contribution of this paper. This dataset can be utilized for testing and developing various pixel-level semantic segmentation models. The dataset comprises millimeter-resolution images captured around Moorea Island by the Moore Island Digital Ecosystem Avatar project. Specifically, it includes images of five underwater coral reef plots, covering a total seabed area of about 180 square meters. The images used for training and validation were densely labeled semiautomatically with semantic annotations by experts with the support of TagLab, featuring two main coral categories (Elkhorn coral and Pocillopora) and three health states (living coral, dead coral, and bleached coral). All data were collected during the summer months from 2017 to 2019 using an underwater photography system.

Our experiments are executed on a computer running the Ubuntu operating system, equipped with 20 Intel Core i7-12700F CPUs, 64GB RAM, and two GeForce RTX 3060 Ti GPUs. Table 2 presents an objective evaluation of performance indicators, including mPA, mIoU, F1 scores. Beyond predicted accuracy, the execution speed is also significant for semantic segmentation algorithms. Therefore, frames per second (FPS) values are also included in Table 3. For a more intuitive representation of the results, please refer to Figure 6.

Figure 6. Scatter plots depict the evaluation metrics, mIoU and FPS, for various network models trained on a customized coral segmentation dataset. Models exhibiting superior performance are positioned closer to the upper-right corner. The model with the highest mIoU is highlighted along the vertical axis, while the model with the highest FPS is highlighted along the horizontal axis. The figure reveals that BiSeNet emerges as the most efficient and top-performing model, showcasing its proximity to the upper-right corner, signifying optimal trade-offs between mIoU and FPS.

Table 3. The performance of segmentation algorithms on custom dataset.

	Methods	mPA	mIoU	F1	FPS (224×224)
Deep Segmentation	MultiResUNet	64.96	52.98	64.47	63.1
	DeepLab V3+	71.57	58.25	69.59	52.0
	UNet	70.10	60.92	71.22	83.2
	TransUNet	73.57	62.61	72.69	35.9
	PointRend	72.69	59.50	70.04	41.2
	CCNet	70.71	60.37	71.61	22.7
TransFormer	SegFormer	60.07	46.47	59.97	78.5
	Segmenter	68.03	53.28	64.58	66.1
	SwinUNet	72.58	60.47	71.87	50.1
Light Segmentation	LEDNet	62.74	53.32	63.34	52.0
	BiSeNet	72.03	60.39	71.25	104.1
	ShelfNet	69.12	60.96	71.02	46.8
	STDC	68.59	59.80	70.96	67.1

Note: The bold indicates the best result of the objective index in these algorithms.

Examples of images predicted by various methods are available in Figure 7.

Figure 7. Predicted results comparison on the image patches taken from custom dataset. (a) Denotes the input test image, (b) the ground truth mask. (c–o) Exhibit the outcomes predicted by different network models.

Notably, all experiments maintain consistent testing environments, hyperparameters, and other settings. The objective of these comparative experiments is not merely to depict the present landscape of coral semantic segmentation research but also to aid readers in swiftly and comprehensively selecting a model suitable for their specific task demands. In fact, it is not easy to make a completely fair comparison among these competitors due to the complexity of settings. However, we can make the following conclusions from the comparison results on accuracy, inference speed and visual quality: (i) In data-limited tasks like coral semantic segmentation, a balance of accuracy and efficiency can be achieved by choosing lightweight networks. (ii) Overall, predicted images exhibit noticeable visual disparities from their corresponding ground truth counterparts, particularly at edges and low-contrast regions. For instance, these models frequently generate corals with smooth boundaries. (iii) Compared to FCNNs, Transformers offer inferior predictive performance. SegFormer and Segmenter, except for SwinUNet, suffer from overfitting despite using their smallest network structures. This potentially be attributed to their extensive parameter range and their primary design for natural language processing tasks. (iv) Models that incorporate pretrained weights, such as TransUNet, SwinUNet, and BiseNet, etc., demonstrate superior performance. This highlights the critical role of transfer learning in our study.

8. Current challenges and future directions

The extensive research reviewed in Section 5 indicates that the study of segmentation of underwater coral imagery has garnered increasing attention from researchers as the scientific community gains access to a wealth of underwater imagery data. Efforts have been made to apply a wide array of advancements from the fields of general photogrammetric computer vision and machine learning to large underwater datasets, including those containing coral images, leading to innovative applications and ongoing exploration, with significant progress being achieved. However, as highlighted in the literature review and current status discussions earlier, we recognize that the primary issues to be addressed are related to datasets and image segmentation algorithms. Therefore, in this section, we will further discuss the challenges and promising directions of coral semantic image segmentation, focusing on these two key topics.

8.1. Image datasets

The data for model training may be equally as crucial as the model architecture for the study on coral image semantic segmentation, especially for deep learning-based solutions. In recent years, several universities and research institutions have developed underwater coral imagery datasets, as listed in Table 1. These datasets have undoubtedly facilitated progress within the research community in the task of underwater coral image segmentation. However, compared with extensive general-purpose datasets featuring common objects such as buildings, vehicles, and landscapes, the lack of standard underwater coral image datasets for segmentation is still striking. Furthermore, the design of coral datasets is often tailored to specific objectives. For instance, coral images from EILAT dataset comprise low-resolution patches with texture details, yet they fail to reflect the complete coral structure; while UCSD Mosaics dataset presents comprehensive depictions of coral individuals, the absence of underwater imagery processing leads to perplexing color features. Certainly, we have also discovered that the SSPQICD dataset from the University of Queensland contains over a million underwater coral images collected from 860 locations around the world. It boasts more than ten thousand annotated images, most accompanied by geolocation data and subjected to photogrammetric processing, surpassing the quality of most datasets we have reviewed. However, these annotations are limited to sparse labeling, and the actual coverage area of each image is merely one square meter. Additionally, most of the data lacks multi-temporal in-situ observations. The dataset we built in our experiment not only has rigorous underwater geographical references, processed through strict photogrammetry and radiometric calibration, and provides three years of continuous coral observation images and dense annotations at the same location, but its volume is still not large enough, and the types of corals covered are not diverse enough. Considering that data is the foundation for more effective precision segmentation of underwater coral images based on deep learning in the future, we still face challenges related to the difficulty of acquiring larger, more representative, and richer underwater coral image datasets, as well as the cost of annotation. Of course, we also believe that with the deployment of low-cost, more autonomous Unmanned Underwater Vehicles (UUVs) equipped with consumer-grade observation equipment and the emergence of more artificial intelligence-driven annotation tools, in the near future, we may find solutions to these challenges. At the same time, we invite fellow researchers to consider the methodologies and approaches discussed in this review as potential tools in their own research endeavors. Our aim is to contribute to the collective knowledge and encourage the exploration of diverse strategies that have shown promising results in our experience.

8.2. Application-oriented segmentation algorithms

Segmentation of underwater coral images is a crucial tool for the more accurate and fine-grained monitoring of coral health and growth distribution. There is an urgent need for further research and the enhancement of practical application innovations. To begin with, coral images captured in underwater environment are likely to suffer from distinctly different degradations, posing challenges to existing semantic segmentation algorithms. Therefore, it is necessary to develop models that can adapt to ever-changing environments while ensuring robust generalization. In addition, most current coral semantic segmentation methods rely on large models, such as deep networks, consequently demanding extensive computational resources and time for forward inference, as well as substantial parameter storage. However, these resources are often limited in practical applications, with only a handful of international corporations having access to hundreds of thousands of Graphics Processing Units. Non-profit universities and research institutions find it difficult to afford them. Additionally, some application scenarios may not be able to utilize such large-scale computing resources due to constraints on energy and space. For example, designing a lightweight model that can be integrated with UUVs to achieve rapid and accurate image processing without increasing hardware is one of the current major needs for underwater coral ecosystem monitoring. Inspiration for such a lightweight network design can be obtained from Figure 6. Furthermore, the use of different imaging devices makes the already labor-intensive coral image annotation even more complex. Therefore, developing segmentation models that can operate with limited training data or even in an unsupervised manner is a direction that should be explored.

9. Conclusions

In recent years, due to the ongoing global warming, incidents of coral bleaching and death have become increasingly frequent. Consequently, geospatial information scientists and oceanographers are in growing need of advanced image segmentation tools to extract quantitative data from underwater coral remote sensing images. These tools enable more precise and detailed assessments of coral health and recovery capabilities. It is precisely because this field of research is receiving growing attention that it is undergoing unprecedented development, urgently requiring timely reviews to provide the research community with an in-depth and comprehensive understanding. This paper primarily reviews and critically summarizes the supervised image segmentation algorithms and strategies for underwater coral images based on annotated data, including block-based classification algorithms and pixel-level precise segmentation algorithms. In addition, the paper provides a detailed summary of open-source datasets, advanced data augmentation techniques, and comprehensive accuracy assessment metrics for training and evaluating underwater coral image segmentation models. Our analysis indicates that, despite recent significant progress in underwater coral image segmentation research, there are still numerous challenges mentioned in the paper that require further investigation and resolution. These challenges include the availability of generic datasets for model training and testing, as well as the development of robust deep learning neural network models tailored to applications. Hence, we have also created and made open source a multi-temporal underwater coral dataset, which undergoes rigorous photogrammetric processing and offers densely labeled images that correspond one-to-one. This dataset provides a foundation for the research community to evaluate and develop new algorithms. In conclusion, we hope that these open questions can serve as beacons for our future research directions, guiding the way for promising avenues of study. We believe that, through the joint efforts of geospatial information scientists and oceanographers, with the aid of deep learning technology, we can apply successful experiences from terrestrial ecosystems to rapidly changing underwater coral ecological remote sensing monitoring. This will provide powerful advanced technological means for the sustainable prosperity and protection of coral reef ecosystems in the near future. All abbreviations used in this paper can be found in Appendix Table A1.

Acknowledgments

We especially would like to thank Prof. Matthias Troyer for his financial and scientific support provided. Special thanks are extended to the Institute of Geodesy and Photogrammetry and the Institute for Theoretical Physics at ETH Zürich.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work is supported by the U.S. National Science Foundation [Grant No.OCE 2224354] and earlier awards for the Moorea Coral Reef LTER, the Italian Minister of University and Research [Grant No.PNRA18 00263-B2], and the National Natural Science Foundation of China [Grant No.41901407].

Notes on contributors

Ming Li

Ming Li received his PhD degree from Wuhan University. He is also a Research Scientist Assistant at ETH Zürich and an Associate research Professor at Wuhan University. His main research interests include the principles and methods of machine learning, photogrammetric computer vision, robotics, underwater photogrammetry, and remote sensing.

Hanqi Zhang

Hanqi Zhang is a master degree student at Wuhan University. Her main interests lie in machine learning and photogrammetric computer vision, particularly focusing on underwater photogrammetry and semantic image segmentation based on deep learning.

Armin Gruen

Armin Gruen received his PhD degree from the Technical University Munich (TUM) and an Honorary Doctorate from Aristotle University. Since 1984, he has been a Full Professor at ETH Zürich, Switzerland. He is a member of the Swiss National Academy of Natural Sciences. He has been awarded the Brock Gold Medal in recognition of outstanding contribution to photogrammetry in 2008, along with a number of other internationally recognized awards and honors. His research interests include 3D cloud mapping and tracking, sensor modeling and image matching of three-line-scanner imagery, digital photogrammetry and remote sensing for Cultural Heritage and natural hazards, satellite and UAV photogrammetry, underwater photogrammetry for coral monitoring, bathymetric Lidar, etc.

Deren Li

Deren Li received his PhD degree from the University of Stuttgart and was awarded an honorary doctorate from ETH Zürich. Since 1986, he has been a Full Professor at Wuhan University, China. He is a member of both the Chinese Academy of Sciences and the Chinese Academy of Engineering. He has been awarded the Brock Gold Medal in recognition of outstanding contribution to photogrammetry in 2020, along with a number of other internationally recognized awards and honors. His research interests include photogrammetry and remote sensing, global navigation satellite system, geographic information system, and their innovation integrations and applications, etc.

Appendix

Table A1. Nomenclature.

Acronyms	Meaning
AUV	Autonomous Underwater Vehicle
ATCRC	A Trillion Coral Reef Colors
CNN	Convolutional Neural Network
CCA	Crustose Coralline Algae
DBN	Deep Belief Network
EFC	Eilat Fluorescence Coral
FPS	Frames Per Second
FCNN	Fully Convolutional Neural Network
GAN	Generative Adversarial Network
HSV	Hue, Saturation, Value
LSTM	Long Short-Term Memory
MCR LTER	Moorea Coral Reef Long Term Ecological Research
MLC	Moorea Labeled Corals
mPA	Mean Pixel Accuracy
mIoU	Mean Intersection over Union
NOAA	National Oceanic and Atmosphere Administration
PA	Pixel Accuracy
RGB	Red, Green, Blue
SOTA	State-of-the-art
SVM	Support Vector Machine
SSAE	Stacked Sparse Autoencoder
SSPQICD	Seaview Survey Photo-quadrat and Image Classification Dataset
UGAN	Unsupervised Generative Adversarial Network
UCSD	University of California San Diego
UUV	Unmanned Underwater Vehicle
VGG	Visual Geometry Group