Dual generative adversarial networks for merging ocean transparency from satellite observations

ABSTRACT

Satellite ocean transparency data have low spatial coverage due to cloud shading, sun glint, swath width, and temporal revisit. Merging multiple satellite ocean transparency data can improve spatial coverage and create a high-accuracy data set. This study proposed a new satellite ocean transparency merging model based on dual generative adversarial networks (Z_SD-merging GAN), and the products of full-coverage and high-accuracy ocean transparency were produced. The ZSD-merging GAN comprises the guess GAN and the merging GAN. The guess GAN is used to generate the guess of the ocean transparency merged product, while the merging GAN combines the guess and satellite ocean transparency data to produce the merged product. The experiments show that the spatial coverage of the ZSD-merging GAN product is 100%. The root-mean-square error (RMSE) and average relative error (ARE) between the ZSD-merging GAN product and unmerged ocean transparency data from the Visible Infrared Imaging Radiometer Suite (VIIRS) on JPSS1 are 4.31 m and 11%, respectively, which are better than 5.59 m and 17% for historical average, 5.55 m and 19% for guess product, 5.08 m and 17% for Poisson blending product, and 5.12 m and 21% for Kriging interpolation product.

KEYWORDS:

• Ocean transparency
• merging
• generative adversarial network (GAN)
• satellite

1. Introduction

Ocean transparency is the maximum visible depth of the Secchi disk (Z_SD) vertically sinking into the seawater, which is closely related to the physical properties and chemical contents of seawater, suspended solids in the seawater, and ocean dynamic processes (Doron et al. 2007, 2011; Kirby et al. 2021; Zhan et al. 2021). Obtaining ocean transparency with full coverage and high accuracy is significant for studying seawater’s physical and chemical properties, marine ecological monitoring, and fishery production. Currently, in-situ measurement and satellite remote sensing are the two major ways to observe ocean transparency. In-situ measurement uses the Secchi disk to accurately measure transparency on the ship (Lee et al. 2015). However, mapping the spatial distribution of ocean transparency is challenging using in situ measurements, and achieving a large-area continuous observation is logistically challenging. Satellite imagery can be used to observe the transparency of large ocean areas (He et al. 2017). However, cloud shading and sun glint can sometimes affect these remotely sensed data, resulting in partial or complete data contamination. Additionally, the narrower swath width and lower temporal resolution of the satellite imagery can challenge comprehensibly capturing ocean transparency over larger areas. Merging ocean transparency data from multiple satellites can improve spatial coverage and produce a highly accurate unified ocean transparency data set, facilitating the research of seawater’s physical and chemical characteristics, marine ecological monitoring, fishery production, and underwater operation.

The European Space Agency’s GlobColour project (D’Andon et al. 2009) has released the ocean transparency merged data with an error-weighted averaging method. Its weights are the confidence of each sensor compared to the other (Pottier et al. 2006). This method can produce unified ocean transparency products, but the improvement of spatial coverage is relatively limited. For example, the daily spatial coverage of ocean transparency merged products from the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) on OrbView-2, the Medium Resolution Imaging Spectrometer Instrument (MERIS) on ENVISAT and the Moderate Resolution Imaging Spectroradiometer (MODIS) on Aqua is less than 30% (Shi et al. 2015). The European Copernicus Marine Environmental Monitoring System (CMEMS) uses the Kriging method with regional anisotropic covariance models for merging satellite ocean transparency data (Saulquin, Gohin, and Fanton d’Andon 2019). However, its variogram parameters are based on climatological monthly means, making it difficult to accurately describe the daily variation characteristics of ocean transparency. Wang et al. (2016) and Tian (2013) merged satellite ocean transparency data with optimal interpolation to produce complete maps of ocean transparency. Its background error covariance matrix is assumed to be horizontally homogeneous and isotropic, making it unable to accurately describe ocean transparency’s spatial characteristics, and the calculation cost is high. In addition, Bayesian maximum entropy, objective analysis, and other ocean color fusion methods can also be used to merge satellite ocean transparency data to improve data quality, but the daily spatial coverage is still limited.

Deep learning has attracted widespread attention recently for its powerful spatiotemporal feature mining and data representation capabilities (Li, Huang, and Gong 2019; Li et al. 2020). It has been applied in merging satellite and gauge precipitation by combining the convolutional neural networks (CNNs) and the long-short-term memory networks (LSTMs) (Wu et al. 2020), merging active and passive microwave remote sensing data with the CNNs (Malmgren-Hansen et al. 2021), and merging visible and infrared images with a generative adversarial network (GAN) (Hou et al. 2021; Ma et al. 2019), which provides a new technology path for merging satellite ocean transparency. In this study, we aim to propose a spatiotemporal deep fusion model to yield multiple satellite ocean transparency merged products with full coverage and high accuracy. The reconstruction of missing ocean transparency data and the fusion of reconstructed and satellite-observed ocean transparency data are two issues that must be addressed. To produce the full-coverage and high-accuracy ocean transparency merged products, we propose a new satellite ocean transparency merging model based on dual generative adversarial networks (ZSD-merging GAN), which contains the guess GAN and the merging GAN. The guess GAN is used to generate the guess of ocean transparency, which reconstructs missing ocean transparency data. The merging GAN combines the guess and satellite ocean transparency data to produce ocean transparency merged products. This paper is organized as follows: Section 2 describes the study area and data acquisition. The details of the model are given in Section 3. Section 4 evaluates the spatial completeness and accuracy of the ocean transparency merged products. A discussion is given in Section 5, and the whole paper is concluded in Section 6.

2. Study area and data description

2.1. Study area

The study area is located in Offshore China and the Northwest Pacific Ocean, between 2.5° and 45.0°N latitude and 103.5° to 147.5°E longitude. As shown in Figure 1, it has complex coastlines, numerous islands, and significant variations in water depth. Several big rivers enter the sea in this region, such as the Yellow River, Mekong River, Yangtze River, and Pearl River. Moreover, the Kuroshio also passes through it (Shi et al. 2014; Wang et al. 2016). Therefore, ocean transparency in this area has a complex spatial distribution and apparent seasonal variation, which is suitable for developing and validating a new merging model.

Figure 1. Study area.

2.2. Data acquisition

2.2.1. Satellite ocean transparency data

The ocean transparency data from the Sea-viewing Wide Field-of-view Sensor (SeaWiFS) on OrbView-2, the Medium Resolution Imaging Spectrometer Instrument (MERIS) on ENVISAT, the Moderate Resolution Imaging Spectroradiometer (MODIS) on Aqua, the Visible Infrared Imaging Radiometer Suite (VIIRS) on Suomi-NPP and JPSS1, the Ocean and Land Colour Instrument (OLCI) on Sentinel-3A and Sentinel-3B were used to develop and validate a new merging model. They were downloaded from the European Space Agency’s GlobColour project (http://www.globcolour.info/) with a spatial resolution of 4 km and a temporal resolution of 24 h. The swath width, equator crossing time, and time range of data of SeaWIFS, MERIS, MODIS, VIIRS, and OLCI are described in Table 1. The normalized remote sensing reflectances at 490 and 560 nm are used to calculate ocean transparency with the DORON method, which is valid both in coastal and oceanic waters (Doron et al. 2007, 2011). The ocean transparency data typically range between 0 and 60 m. Their coverage is deficient because of the limitations of swath width, temporal revisit, sun glint, and clouds. Figure 2 shows ocean transparency maps from MODIS/Aqua, VIIRS/NPP, VIIRS/JPSS1, OLCI/Sentinel-3A, OLCI/Sentinel-3B, and the five combined sensors on 1 May 2022. The combined ocean transparency maps are produced using the average of data within three standard deviations in each pixel. Their coverage is 18.0% for MODIS/Aqua, 13.7% for VIIRS/NPP, 11.3% for VIIRS/JPSS1, 21.0% for OLCI/Sentinel-3A, 21.3% for OLCI/Sentinel-3B, respectively. The combined coverage for the five sensors is 51.7%. Merging ocean transparency data from the satellites mentioned above can improve spatial coverage and produce high-accuracy unified ocean transparency products.

Figure 2. Ocean transparency maps from (a) MODIS/Aqua, (b) VIIRS/NPP, (c) VIIRS/JPSS1, (d) OLCI/Sentinel-3A, (e) OLCI/Sentinel-3B, and (f) the five combined sensors on 1 May 2022.

Table 1. Swath width, equator crossing time, and time range of data of SeaWIFS, MERIS, MODIS, VIIRS, and OLCI.

Sensor	SeaWIFS	MERIS	MODIS	VIIRS		OLCI
Platform	Orbview-2	ENVISAT	Aqua	Suomi-NPP	JPSS1	Sentinel-3A	Sentinel-3B
Swath width (km)	1500 (global area coverage)	1150	2330	3000	3000	1270	1270
Equator crossing time	12:20 (descending)	10:00 (descending)	13:30 (ascending)	13:30 (ascending)	13:30 (ascending)	10:00 (descending)	10:00 (descending)
Time range of data	09/04/1997 −12/11/2010	04/29/2002 −04/08/2012	07/04/2002-present	01/02/2012 -present	12/13/2017 -present	04/25/2016 -present	05/14/2018 -present

2.2.2. CMEMS ocean transparency merged product

The CMEMS ocean transparency merged products (OCEANCOLOUR_GLO_OPTICS_L4_REP_OBSERVATIONS_009_081) were downloaded from https://marine.copernicus.eu/ and used for pre-training the ZSD-merging GAN, which has 100% spatial coverage with a spatial resolution of 4 km and a temporal resolution of 24 h. CMEMS ocean transparency merged products are created using the Kriging method (Saulquin, Gohin, and Fanton d’Andon 2019) to merge SeaWiFS/OrbView-2, MERIS/ENVISAT, MODIS/Aqua, VIIRS/Suomi-NPP, VIIRS/JPSS1, OLCI/Sentinel-3A, and OLCI/Sentinel-3B ocean transparency data. It should be noted that the variogram parameters of the Kriging method are based on climatological monthly means, and the daily variation characteristics of ocean transparency may be lost. Therefore, CMEMS ocean transparency merged products are only used for pre-training.

3. Methodology

The existing merging models with deep learning, such as the convolutional neural network (CNN) architecture for Sentinel-1 and AMSR2 data fusion (Malmgren-Hansen et al. 2021), the GAN for Infrared and Visible Image Fusion (Hou et al. 2021; Ma et al. 2019), combine different types of data to generate more robust and informative data. It is worth noting that these data have full coverage, and these merging models mainly solve the problem of combining different data types. Unlike the data mentioned above, there is a large amount of missing data for satellite ocean transparency due to cloud shading, sun glint, swath width, and temporal revisit. Consequently, the ocean transparency merging model needs to solve the problem of reconstructing the missing data and merging the reconstructed and satellite-observed ocean transparency. We employ deep learning to extract the spatiotemporal variation features of ocean transparency to solve the problem of reconstructing the missing data. We then use the data of the preceding m-days to estimate the guess of the ocean transparency merged products, which serve as the reconstructed ocean transparency. In order to solve the problem of merging the reconstructed and satellite-observed ocean transparency, deep learning is used to combine the full coverage of the reconstructed data and the high accuracy of satellite data.

According to the above idea, we propose the ZSD-merging GAN that contains the guess GAN and the merging GAN, which were implemented using PyTorch. Figure 3 shows the architecture of ZSD-merging GAN. T represents a given day. T-m represents m days before T day. The guess GAN utilizes ocean transparency merged products on T-1, T-2, … , and T-m days to generate the guess of the ocean transparency merged products on T day. Hence, the guess generator can be defined as follows:

Figure 3. Architecture of Z_SD-merging GAN.

(1) ${\hat{x}}_{\mathit{T}}=G_1\left(x_{\mathit{T}-1},x_{\mathit{T}-2},\cdots ,x_{\mathit{T}-\mathit{m}}\right),$(1)

where $\mathit{\hspace{0.17em}}{x}_{\mathit{T}-1},x_{\mathit{T}-2},\cdots ,x_{\mathit{T}-\mathit{m}}$ are ocean transparency merged products on T-1, T-2, T-m day.${\hat{x}}_{\mathit{T}}$ is the guess of ocean transparency merged product on T day, $G_1$ represents the guess generator.

The merging GAN generates the full-coverage and high-accuracy ocean transparency merged products from the guess and satellite observations. The merging generator can be formally defined as follows:

(2) $x_T=G_2\left({\hat{x}}_{\mathit{T}},{\tilde{x}}_{\mathit{T}}\right),$(2)

where ${\tilde{x}}_{\mathit{T}}$ is satellite ocean transparency data on T day,$G_2$ represents the merging generator.

3.1. Guess GAN

3.1.1. Architecture and training process

The guess GAN, composed of a generator and a discriminator, obtains the ability to generate the guess of ocean transparency by training with long-time ocean transparency merged products. As seen in Figure 3, the guess generator uses the preceding m-days ocean transparency merged product to generate the guess on T day, and the guess discriminator identifies the generated guess from the actual ocean transparency merged product. The training process is as follows. Ocean transparency merged products on the consecutive T~T-m days are selected. The data on T-1~T-m days are input into the guess generator during training, and the T-day guess is output after the mining and analysis of spatiotemporal features. The difference between the guess and the ocean transparency merged product on T day is calculated according to the loss function, and the optimizer will adjust the parameters of the guess generator. At the same time, the guess discriminator identifies the T-day guess produced by the guess generator from the T-day ocean transparency merged product, which will feed back into the guess generator and further improve it. The T-day guess produced by the improved guess generator is more consistent with the T-day ocean transparency merged product than the preceding T-day guess, so the guess discriminator will further be updated to differentiate them. Repeating this process, the guess generator and discriminator confront each other and finally reach Nash equilibrium (Goodfellow et al. 2014).

3.1.2. Guess generator

The guess generator uses an encoder-decoder architecture and consists of five layers. Figure 4 shows the detailed architecture of the guess generator. The encoder module includes the space-to-depth operation, the ConvGRU (Shi et al. 2017), the residual block I, and the residual block II, which reduces the data resolution to accelerate the model processing and extracts the spatiotemporal features from the data on T-1~T-m days. The space-to-depth operation can reduce the resolution by rearranging elements from the spatial dimensions into the channel dimensions. The ConvGRU plays an essential role in the guess generator. Unlike the conventional GRU, the ConvGRU utilizes the convolution operator instead of the fully connected operator, which can better capture the spatiotemporal correlations of ocean transparency. The residual blocks I and II are designed in the guess generator to alleviate the gradient disappearance problem caused by increasing the network depth. As illustrated in Figure 5(a), the residual block I consists of two 3 × 3 convolutional layers and a skip connection, which is employed to downsample the input data and initialize the ConvGRU. Unlike the residual block I, group normalizations are added to the residual block II. Sampling units can be flexibly configured as downsampling, upsampling, or null (as shown in Figure 5(b)). In the encoder module, the residual block II is configured with the downsampling unit to reduce the data resolution and is used to connect the convGRUs of different layers.

Figure 4. Architecture of guess generator.

Figure 5. Architecture of (a) residual block I, (b) residual block II, and (c) output block.

The decoder module includes the residual block II and the output block. The residual block II is configured with a bilinear interpolation unit, which is utilized to restore the data resolution of each layer. Figure 5(c) shows that the output block consists of the group normalization, the Relu activation function, the 1 × 1 convolution, and the Tanh activation function. This structure further restores the data resolution and outputs the guess of the T-day ocean transparency merged product. Note that the residual block II on each layer of the encoder module makes a skip connection with the ConvGRU on the same layer of the decoder module, which effectively integrates shallow features of the encoder module with deep features of the decoder module and enables the guess generator to extract features better.

3.1.3. Guess discriminator

As mentioned above, the guess discriminator aims to identify the generated guess from the ocean transparency merged product. Referring to the PatchGAN discriminator (Isola et al. 2017), we designed the guess discriminator with the 7-layer convolutional neural network (see Figure 6 for details). The LeakyReLU activation function follows the first six convolutional layers, while the Sigmoid activation function follows the last convolutional layer. Unlike the conventional GAN discriminator, the guess discriminator divides the input into N × N size of matrices called patches and then classifies if each N × N patch is real or fake, which can help to generate a high-accuracy guess of the ocean transparency merged product on T day.

Figure 6. Architecture of guess discriminator.

3.1.4. Loss function

The loss function includes the adversarial loss and the content loss. The adversarial loss represents the distance between the distribution of the generated guess and the distribution of the real ocean transparency merged product, which guides the training of the generator and discriminator. The generator aims to fool the discriminator by making the guess as similar to the actual ocean transparency merged product as possible, and the discriminator accounts for updating the classifier to differentiate them (Tian et al. 2020). The adversarial loss is formally defined as follows:

(3) $\underset{\mathrm{G}}{min}\underset{\mathrm{D}}{max}{\mathrm{L}}_{\mathrm{adv}}\left({\mathrm{G}}_1,\mathrm{D}\right)={\mathrm{E}}_{\mathrm{x}\sim {\mathrm{P}}_{\mathrm{data}}\left(\hat{x}\right)}\left[\mathrm{logD}\left(\mathrm{x}\right)\right]+{\mathrm{E}}_{\hat{x}\sim {\mathrm{P}}_{{\mathrm{G}}_1}\left(\hat{x}\right)}\left[\log \left(1-\mathrm{D}\left(\hat{x}\right)\right)\right],$(3)

where $\mathrm{D}$ represents the guess discriminator,$\mathit{\hspace{0.17em}}{\mathrm{P}}_{\mathrm{data}}$ represents the distribution of ocean transparency merged products and ${\mathrm{P}}_{{\mathrm{G}}_1}$ represents the distribution of guess.

The content loss is the Manhattan distance between the guess and the ocean transparency merged product on T day. The study area covers both ocean and land. The gradient of ocean transparency near the sea-land junction is considerable. If ocean and land are not distinguished when calculating the content loss, the guess’ error near the sea-land junction will be enormous. Therefore, the content loss is calculated separately for ocean and land, which is defined as follows:

(4) $L_{\mathit{content}}=L_{\mathit{land}}+\mathit{\mu }{L}_{\mathit{ocean}},$(4)

(5) $L_{\mathit{land}}=\mathit{E}\left[\parallel m_{\mathit{land}}\odot \left(\mathit{x}-\hat{x}\right){\parallel }_1\right],$(5)

(6) $L_{\mathit{ocean}}=\mathit{E}\left[\parallel m_{\mathit{ocean}}\odot \left(\mathit{x}-\hat{x}\right){\parallel }_1\right],$(6)

where $\mathit{\mu }$ is a constant and here taken as 6, H and W are, respectively, the height and width of the guess on T day, $m_{\mathit{ocean}}$ is the mask of ocean, $m_{\mathit{land}}$ is the mask of land, $\cdots { }_1$ represents the Manhattan norm which is the sum of the absolute values of the dimensions of the vector, $\odot$ represents the dot product.

The loss functions of guess generator and discriminator are defined as follows:

(7) $L_{G_1}=E_{\hat{x}\sim P_{G_1}\left(\hat{x}\right)}\left[\log \left(1-\mathit{D}\left(\hat{x}\right)\right)\right]+\mathit{\lambda E}\left[\parallel m_{\mathit{land}}\odot \left(\mathit{x}-\hat{x}\right){\parallel }_1\right]+\mathit{\lambda \mu E}\left[\parallel m_{\mathit{ocean}}\odot \left(\mathit{x}-\hat{x}\right){\parallel }_1\right],$(7)

(8) $L_D={\mathrm{E}}_{\mathit{x}\sim {\mathit{P}}_{\mathit{data}}\left(\mathit{x}\right)}\left[\mathit{logD}\left(\mathit{x}\right)\right]+{\mathrm{E}}_{\hat{x}\sim P_{G_1}\left(\hat{x}\right)}\left[\log \left(1-\mathit{D}\left(\hat{x}\right)\right)\right],$(8)

where $L_{G_1}$ is the loss of guess generator, $\mathit{\lambda }$ is a constant and here taken as 60,$L_D$ is the loss of guess discriminator.

3.2. Merging GAN

3.2.1. Architecture and training process

The merging GAN comprises a merging generator, a global discriminator, and a local discriminator. Figure 3 shows the architecture of the merging GAN. The merging generator is trained to produce the ocean transparency merged product to fool the dual discriminators. The global discriminator identifies the ocean transparency merged product from the Poisson blending product, the combination of the guess and satellite ocean transparency data in the gradient domain, which helps to ensure the global consistency and continuity of ocean transparency merged products. The local discriminator identifies the ocean transparency merged product from the satellite ocean transparency data in the satellite observation area, which helps to ensure the regional authenticity of the products. Due to the randomness of the spatial coverage of the daily satellite ocean transparency data, global authenticity can be identified by training a large number of samples.

The training process is as follows. The satellite ocean transparency products and the T-day guess are input into the merging generator. The ocean transparency merged product on T day is generated after a series of spatial feature mining analyses. Meanwhile, we create the Poisson blending product by combining the guess and satellite ocean transparency data using the Poisson equation(Perez, Gangnet, and Blake 2003). According to the loss function, we, respectively, calculate the difference between the ocean transparency merged product and the Poisson blending product, as well as the difference between the ocean transparency merged product and the satellite ocean transparency data on T day. Then, the optimizer will adjust the parameters of the merging generator. At the same time, the global discriminator identifies the ocean transparency merged product from the Poisson blending product on T day, and the local discriminator identifies the ocean transparency merged product from the satellite ocean transparency data on T day. The connected feedback of the two discriminators is sent to the merging generator and further improves the generator. Repeating this process, the merging generator and the two discriminators confront each other and finally reach Nash equilibrium (Li, Huang and Gong 2019). At this time, the ocean transparency merged product on T day has better global consistency, continuity, and authenticity.

3.2.2. Merging generator

The merging generator composes an encoder and a decoder with a 5-layer network architecture. This architecture is shown in Figure 7 and includes the space-to-depth operation, the output block, and the residual block II. Here, the residual block II is mainly used to connect the shallow and deep networks in the merging generator and solves the problem of gradient disappearance due to the difficulty of gradient flow to the shallow network during backpropagation. The encoder module uses the residual block II configured with the average pooling unit to reduce the data resolution and speed up the model processing. The decoder module uses the residual block II configured with the bilinear interpolation unit to restore the data resolution. A skip connection is adopted between the encoder and decoder at the same layer to integrate shallow features of the encoder module with deep features of the decoder module so that the merging generator can obtain better feature extraction capability and reduce the loss of the decoder module.

Figure 7. Architecture of merging generator.

3.2.3. Global discriminator and local discriminator

As shown in Figure 8, the global discriminator uses the network architecture similar to the guess discriminator to identify the ocean transparency merged product from the Poisson blending product on T day. The local discriminator uses a seven-layer convolutional neural network, and a spatial attention module is added in the second and fifth layers. Unlike the original spatial attention module (Woo et al. 2018), random pooling is used instead of average pooling to enhance the generalization ability, as shown in Figure 9. When the global discriminator is unable to distinguish between the ocean transparency merged product and the Poisson blending product, and the local discriminator is unable to distinguish between the ocean transparency merged product and the satellite ocean transparency data during the process of network training, the ocean transparency merged product can maintain both global consistency and continuity from the Poisson blending product, as well as the authenticity of satellite observation from satellite ocean transparency data. In other words, through training, the ocean transparency merged product can simultaneously maintain global consistency, continuity, and authenticity, making it a more accurate and reliable ocean transparency product.

Figure 8. Local discriminator.

Figure 9. Spatial attention module.

3.2.4. Loss function

The loss function of the merging generator includes the adversarial loss and the content loss. The adversarial loss comprises two components: one is the adversarial loss between the merging generator and the global discriminator, and another is the adversarial loss between the merging generator and the local discriminator, which is formally defined as follows:

(9) $L_{\mathit{adv}}\left(G_2\right)={\mathrm{E}}_{x\sim P_{G_2}\left(\mathit{x}\right)}\left[\log \left(1-D_a\left(\mathit{x}\right)\right)\right]+{\mathrm{E}}_{x\sim P_{G_2}\left(\mathit{x}\right)}\left[\log \left(1-D_s\left(\mathit{x}\right)\right)\right],$(9)

where $L_{\mathit{adv}}\left(G_2\right)$ represents the adversarial loss of merging generator,$G_2$ represents the merging generator,$D_a$ represents the global discriminator,$D_s$ represents the local discriminator, $P_{G_2}$ represents the distribution of ocean transparency merged products and $\mathit{x}$ is the ocean transparency merged product.

The content loss is used to measure the differences between the ocean transparency merged product and the Poisson blending product, as well as the differences between the ocean transparency merged product and the satellite ocean transparency data in the satellite observation area. The $l_2$ distance and gradient are used to calculate the data differences to ensure global consistency and continuity when measuring the differences between the first two. When measuring the differences between the latter two, the $l_1$ distance (Wang et al. 2022) and structure similarity are used to calculate the data differences. The content loss is defined as follows:

(10) $L_{\mathit{content}}=\mathit{E}\left[\parallel \mathit{x}-x^{\prime }{\parallel }_2\right]+\mathit{\alpha }\mathit{E}\left[\parallel \mathrm{\nabla x}-\mathrm{\nabla }{x}^{\prime }{\parallel }_2\right]+\mathit{\beta E}\left[\parallel m_{\mathit{sat}}\odot \left(\mathit{x}-\tilde{x}\right){\parallel }_1\right]+\mathit{\gamma }\frac{\left(2{\mu }_x{\mu }_{\tilde{x}}+c_1\right)\left(2{\sigma }_{x\tilde{x}}+c_2\right)}{\left({\mu }_x^2+{\mu }_{\tilde{x}}^2+c_1\right)\left({\sigma }_x^2+{\sigma }_{\tilde{x}}^2+c_2\right)},$(10)

where $\mathit{x}{ }^{\mathrm{\prime }}$ is the Poisson blending product,$\tilde{x}$ is the satellite ocean transparency data,$\mathit{\alpha },\mathit{\beta },\mathit{\gamma }$ are constants with values of 4, 2, and 80, respectively;$\mathrm{\nabla }$ represents the gradient operator,$\cdots { }_1$ represents the $l_1$ distance; $\cdots { }_2$ represents the $l_2$ distance; $m_{\mathit{sat}}$ is the mask of the satellite observation area; $\frac{\left(2{\mu }_x{\mu }_{\tilde{x}}+c_1\right)\left(2{\sigma }_{x\tilde{x}}+c_2\right)}{\left({\mu }_x^2+{\mu }_{\tilde{x}}^2+c_1\right)\left({\sigma }_x^2+{\sigma }_{\tilde{x}}^2+c_2\right)}$ indicates the structure similarity; ${\mu }_x$ is the mean of $\mathit{x}$,${\mu }_{\tilde{x}}$ is the mean of $\tilde{x}$, ${\sigma }_x^2$ is the variance of $\tilde{x}$, ${\sigma }_{x\tilde{x}}$ is the covariance between $\mathit{x}$ and $\tilde{x}$,$c_1$ and $c_2$ are constants with the values of 0.0004 and 0.0036, respectively.

The loss function of the merging generator can be represented as follows:

(11) $L_{G_2}=L_{\mathit{adv}}\left(G_2\right)+\mathit{\delta }{L}_{\mathit{content}},$(11)

where $\mathit{\delta }$ is a constant which is set to 60.

The loss function for the global discriminator is represented as follows:

(12) $L_{D_a}={\mathrm{E}}_{\mathit{x}{\mathit{ }}^{\mathrm{\prime }}\sim P_{\mathit{data}1}\left(\mathit{x}{ }^{\mathrm{\prime }}\right)}\left[\mathit{log}{D}_a\left(\mathit{x}{ }^{\mathrm{\prime }}\right)\right]+{\mathrm{E}}_{x\sim P_{G_2}\left(\mathit{x}\right)}\left[\log \left(1-D_a\left(\mathit{x}\right)\right)\right],$(12)

where $P_{\mathit{data}1}$ represents the distribution of the Poisson blending product.

The loss function for the local discriminator is represented as follows:

(13) $L_{D_s}={\mathrm{E}}_{\tilde{x}\sim P_{\mathit{data}2}\left(\tilde{x}\right)}\left[\mathit{log}{D}_s\left(\tilde{x}\right)\right]+{\mathrm{E}}_{x\sim P_{G_2}\left(\mathit{x}\right)}\left[\log \left(1-D_s\left(\mathit{x}\right)\right)\right],$(13)

where $P_{\mathit{data}2}$ represents the distribution of the satellite ocean transparency data.

3.3. Experimental design

The deep learning framework used for training was PyTorch on two NVIDIA Tesla V100s GPUs with CUDA parallel framework and cuDNN acceleration library. The number of batch size and epochs are 32 and 1000. The guess GAN and the merging GAN both used the ADAM optimizer. The learning rate for the generative and discriminative models is both 0.0002, and they are alternatively optimized.

The training dataset covers 1 January 2008 to 31 December 2017. The validation dataset covers the period from 1 January 2018 to 31 December 2019, and the testing dataset covers the period from 1 January 2020 to 31 December 2020. Figure 10 shows the training, evaluation, and test data’s probability density curves. The mean and standard deviation of the training set are 37.95 m and 16.01 m, respectively. These numbers are close to the evaluation set (36.64 m and 15.57 m) and the test set (36.50 m and 15.67 m). This indicates that there is no significant difference between the training set, the evaluation set, and the test set. The training set is used to train the ZSD-merging GAN model, which is verified and tested using the evaluation set and test set. The data with a size of 1024 × 1024 on T-1~T-m days are input into the guess GAN during training, and the T-day guess is output. More than 3 days of data are required to capture the time-varying characteristics of ocean transparency. Due to hardware limitations, 5 days of data are selected here, and m is equal to 5. The merging GAN is trained using multi-source satellite ocean transparency data and the T-day guess of ocean transparency from the guess GAN to generate the ocean transparency merged product. In this study, the CMEMS ocean transparency merged products are only used to pre-train the Z_SD-merging GAN model because they may lose some daily variation characteristics, and then the newly generated ocean transparency merged products are used to replace the CMEMS products to fine-tune the Z_SD-merging GAN model.

Figure 10. The probability density curves of the training set, the evaluation set, and the test set.

The experiment evaluates the ocean transparency merged product using metrics such as the root-mean-square error (RMSE) and the average relative error (ARE). The merged data sources include satellite ocean transparency data from SeaWiFS/OrbView-2, MERIS/ENVISAT, MODIS/Aqua, VIIRS/Suomi-NPP, OLCI/Sentinel-3A, and OLCI/Sentinel-3B. As shown in Table 1, VIIRS/JPSS1 ocean transparency data cover the period from 13 December 2017 to the present and are not used for training. Regarding independence, the validation data come from VIIRS/JPSS1, which provides higher-accuracy ocean transparency data.

4. Results and discussion

4.1. Results

A comparison of spatial coverage before and after merging multi-source satellite ocean transparency data on 10 July 2020 is shown in Figure 11. Due to cloud shading, sun glint and its swath width, and temporal revisit, the daily spatial coverage of ocean transparency obtained by combining remote sensing data from the five sensors, including MODIS/Aqua, VIIRS/NPP, VIIRS/JPSS1, OLCI/Sentinel-3A, and OLCI/Sentinel-3B, is about 46% in Figure 11(a). The ocean transparency combined map from the five sensors misses the spatial distribution characteristics in the East China Sea, Kuroshio, and other seas. Figure 11(b) shows that the spatial coverage of the ocean transparency merged product from the ZSD-merging GAN model is 100%, indicating that the ZSD-merging GAN model can reconstruct the missing data. The ZSD-merging GAN product shows the fine spatial distribution characteristics of ocean transparency. The ocean transparency in Offshore China is significantly less than that in the Northwest Pacific, which is consistent with the spatial distribution of Case-1 waters (Lee and Hu 2006). Figure 12 shows the ocean surface current on 10 July 2020 from the GLORYS12V1 Reanalysis. Ocean transparency is closely related to ocean circulation as it is stirred by the ocean current to form surface patterns that follow fronts and eddies. The ocean current from GLORYS12V1 provides a good approximation for qualitatively evaluating the ocean transparency. By comparing Figures 11(b) and 12, it is evident that the spatial distribution of ocean transparency from the ZSD-merging GAN product is consistent with the ocean surface current, such as areas A and B.

Figure 11. (a) Ocean transparency combined map from MODIS/Aqua, VIIRS/NPP, VIIRS/JPSS1, OLCI/Sentinel-3A, and OLCI/Sentinel-3B, (b) Z_SD-merging GAN product on 10 July 2020.

Figure 12. Ocean surface current on 10 July 2020 from the GLORYS12 reanalysis.

Figure 13 shows the RMSE and ARE of ZSD-merging GAN products versus the test set’s VIIRS/JPSS1 ocean transparency data. The RMSE is the square root of the deviation between the ZSD-merging GAN product and VIIRS/JPSS1 ocean transparency data. The ARE is the average value of the relative error, which is generally expressed by the absolute value of the average relative error (Zhan et al. 2021). The RMSE and ARE are defined as follows:

Figure 13. (a) RMSE, (b) ARE, and (c) number of ZSD-merging GAN product versus VIIRS/JPSS1 ocean transparency data from the test set.

(14) $\mathrm{RMSE}={\left(\frac{1}{\mathrm{N}}\sum _{\mathrm{i}-1}^{\mathrm{N}}{\left[{\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{M}}-{\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{J}}\right]}^2\right)}^{0.5}$(14)

(15) $\mathrm{ARE}=\left(\frac{1}{\mathrm{N}}\sum _{\mathrm{i}-1}^{\mathrm{N}}\left|\frac{{\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{M}}-{\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{J}}}{{\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{J}}}\right|\right)\times 100\mathrm{\%}$(15)

where ${\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{M}}$ represents the Z_SD-merging GAN product,${\mathrm{Z}}_{\mathrm{S}\mathrm{D}}^{\mathrm{J}}$ represents VIIRS/JPSS1 ocean transparency data, N is the number of matchup data. Figure 13 shows RMSE, ARE, and number of ZSD-merging GAN products versus VIIRS/JPSS1 ocean transparency data at each grid point with observations in JPSS1 products from January to December 2020. As shown in Figure 13(a,b), the RMSE in the Northwest Pacific is significantly larger than that in Offshore China, and the ARE in the Northwest Pacific is significantly smaller than that in Offshore China, which is because the Northwest Pacific is dominated by Case-1 waters with greater ocean transparency (Lee and Hu 2006). Figure 13(c) shows that the RMSE and ARE are available for the study area except for the Yangtze River Estuary to the Subei Shoal in China, where no matching points exist.

To evaluate the accuracy of ocean transparency merged product from Z_SD-merging GAN, we chose historical average, guess product, Poisson blending product, and Kriging interpolation product as the baselines for comparison. The historical average is the average of daily satellite ocean transparency data from 2008 to 2020. The guess product is generated from the guess GAN. The Poisson blending product combines the guess and satellite ocean transparency data in the gradient domain using the Poisson equation. The Kriging interpolation product is provided by the CMEMS (OCEANCOLOUR_GLO_OPTICS_L4_REP_OBSERVATIONS_009_081).

Figure 14 shows scatterplots of the historical average, guess product, Poisson blending product, Kriging interpolation product, and Z_SD-merging GAN product versus VIIRS/JPSS1 ocean transparency data from the test set. The data distribution of the historical average is the most dispersed, indicating that the error caused by using the historical average to estimate ocean transparency is the largest. The Kriging interpolation product is notably underestimated. The accuracy of the Poisson blending product is superior to the guess product as it incorporates satellite observation. The Z_SD-merging GAN product exhibits the best consistency with VIIRS/JPSS1 satellite ocean transparency, indicating that the Z_SD-merging GAN has the highest precision.

Figure 14. (a) Historical average, (b) Guess product, (c) Poisson blending product, (d) Kriging interpolation product, and (e) Z_SD-merging GAN product versus VIIRS/JPSS1 ocean transparency data from the test set.

Table 2 presents the RMSE and ARE of the historical average, guess product, Poisson blending product, Kriging interpolation product, and Z_SD-merging GAN product versus VIIRS/JPSS1 ocean transparency data. According to the RMSEs in Table 2, it can be observed that the historical average has a maximum RMSE, with values of 4.35, 6.05, and 5.59 m for the ranges of 0–20, 20–40, and all, respectively. The Guess product has a maximum RMSE of 6.34 m in the 40–60 m range. On the other hand, the Kriging interpolation product has a minimum RMSE of 3.97 m in the range of 20–40 m, and the Z_SD-merging GAN product has a minimum RMSE with values of 2.31, 5.04, and 4.31 m for the ranges of 0–20, 40–60, and all. According to the AREs in Table 2, for the 0–20 m range, we can find that the guess product has a maximum ARE of 51%, and the Z_SD-merging GAN product has a minimum ARE of 20%. For the 20–40 and 40–60 m ranges, the Kriging interpolation product has a maximum ARE with values of 19% and 16%, and the Z_SD-merging GAN product has a minimum ARE with values of 10% and 8%. For the 0–60 m ranges, the Kriging interpolation product has a maximum ARE of 21%, while the Z_SD-merging GAN product has a minimum ARE of 11%.

Table 2. RMSE and ARE of historical average, guess product, Poisson blending product, Kriging interpolation product, and ZSD-merging GAN product versus VIIRS/JPSS1 ocean transparency data. The bold font indicates a minimum RMSE or ARE in each row.

Error type	Ocean transparency (m)	Historical average (m)	Guess product (m)	Poisson blending product (m)	Kriging interpolation product (m)	ZSD-merging GAN product (m)
RMSE	0~20	4.35	3.66	2.91	2.88	2.31
	20~40	6.05	5.01	5.00	3.97	4.14
	40~60	5.64	6.34	6.27	5.49	5.04
	All	5.59	5.55	5.08	5.12	4.31
ARE	0~20	39%	51%	39%	34%	20%
	20~40	16%	12%	12%	19%	10%
	40~60	9%	11%	10%	16%	8%
	All	17%	19%	17%	21%	11%

By comparing the experimental results from Figure 14 and Table 2, it can be concluded that the Z_SD-merging GAN product has significantly lower RMSE and ARE than the historical average, guess product, Poisson blending product, and Kriging interpolation product. These results indicate that the proposed Z_SD-merging GAN in this study is superior to the baseline models and that the ZSD-merging GAN product is highly accurate.

4.2. Discussion

4.2.1. Model limitation

The essence of the Z_SD-merging GAN model is to generate the ocean transparency merged product, which preserves information on satellite observations from the day while filling missing data with information from the preceding days. Without any controls, there will be visible differences between satellite observations and filled data. The content loss and the global discriminator of merging GAN aim to control the differences and ensure the consistency and continuity of ocean transparency merged products. However, the differences between satellite observations are not considered in the study. We use the SeaWiFS/OrbView-2, MERIS/ENVISAT, MODIS/Aqua, VIIRS/Suomi-NPP, VIIRS/JPSS1, OLCI/Sentinel-3A, and OLCI/Sentinel-3B ocean transparency data from the GlobColour project, which are computed uniformly according to the Doron model (Doron et al. 2007; Doron et al. 2011), and their differences are slight. However, the Chinese Ocean Color and Temperature Scanner (COCTS) on Haiyang-1 and the Medium Resolution Spectral Imager on Fengyun-3 have varying designs and characteristics and are not calibrated for consistency. There are differences between COCTS/Haiyang-1, MERSI/Fengyun-3, and GlobColour satellite ocean transparency data that cannot be ignored. As a result, the Z_SD-merging GAN model cannot be applied to merge the COCTS/Haiyang-1 and MERSI/Fengyun-3 ocean transparency data before the calibration.

4.2.2. Model applicability

The ZSD-merging GAN model can merge satellite ocean transparency for larger areas. First, we divided the area of interest into many equal-sized subareas. Then, the model parameters of each subarea are trained separately. Figure 15 shows that the study area is divided into four smaller subareas with black solid boxes. In order to remove inconsistencies at the edges, the ZSD-merging GAN model should be applied to the blue/yellow/red/green dotted subareas instead of black solid subareas, and the model parameters of blue/yellow/red/green dotted subareas are trained separately. Finally, we select black solid subareas from blue/yellow/red/green dotted subareas and then combine black solid subareas using the Poisson equation (Perez, Gangnet, and Blake 2003), which can remove inconsistencies at the edges. According to the above method, the ZSD-merging GAN model can merge the global satellite ocean transparency by splicing multiple subareas. In the future work, we will validate it globally and assess each subarea’s model accuracy and transferability.

Figure 15. Sketch map of combining subareas to remove inconsistencies at the edges.

Training the ZSD-merging GAN model is more expensive than the other approaches tested. However, after the ZSD-merging GAN model is completely trained, the execution time to generate a merged product is less than 1 s, which is much faster than the other approaches tested (Alzubaidi et al. 2021). In addition, the architecture of the ZSD-merging GAN model is also applicable to merge other ocean parameters from satellite observations, such as chlorophyll-a concentration and sea surface temperature.

5. Conclusion

The satellite ocean transparency has many missing data due to cloud shading, sun glint, swath width, and temporal revisit, so its spatial coverage is deficient. It is challenging to meet the application requirements of research on seawater’s physical and chemical properties, marine ecological monitoring, marine fishery production, and underwater operation. By merging multi-source satellite ocean transparency products, it is possible to reconstruct the missing data and improve spatial coverage. Traditional ocean transparency merged algorithms have some problems, such as low spatial coverage, weak ability to express details and large computation. In the study, the ZSD-merging GAN is established by the deep learning method and consists of a guess GAN and a merging GAN. The guess GAN utilizes ocean transparency merged products on T-1, T-2, … , and T-5 days to generate the guess of the ocean transparency merged products on T day, which solves the problem of missing data reconstruction for ocean transparency. The proposed GAN model can generate full-coverage and high-accuracy ocean transparency products from multiple satellite observations.

In order to demonstrate the advancement of the Z_SD-merging GAN and the accuracy of the Z_SD-merging GAN product, we use the satellite ocean transparency from VIIRS/JPSS1 in the period from the test set, which was not involved in the fusion, to analyze and compare the accuracy with the baseline models such as historical average, guess product, Poisson blending product, and Kriging interpolation product in Offshore China and Northwest Pacific Ocean. The RMSE and ARE of the proposed Z_SD-merging GAN product are 4.31 m and 11%, which are better than 5.59 m and 17% for the historical average, 5.55 m and 19% for guess product, 5.08 m and 17% for Poisson blending product, and 5.12 m and 21% for Kriging interpolation product. It indicates that the proposed Z_SD-merging GAN model is superior to the baseline models.

A full-coverage and high-accuracy ocean transparency merged product is significant for seawater physical and chemical properties research, marine ecological monitoring, marine fishery production, and underwater operation. Using the full-coverage and high-accuracy ocean transparency merged products, we can analyze the characteristics of oceanic water mass movement and flow system changes, identify seawater pollution and eutrophication, determine the range and quantity of fish activities, and also provide essential data support for underwater operations. With the operational application of satellite ocean transparency products such as COCTS-2/Haiyang-1 and MERSI-2/Fengyun-3, the number of available satellite data sources for merging ocean transparency is gradually increasing. The next step is to develop ZSD-merging GAN to adapt to new satellite data sources and extend it globally.

Acknowledgments

In this study, SeaWiFS/OrbView-2, MERIS/ENVISAT, MODIS/Aqua, VIIRS/Suomi-NPP, VIIRS/JPSS1, OLCI/Sentinel-3A, and OLCI/Sentinel-3B ocean transparency data were downloaded from the European Space Agency’s GlobColour project (http://www.globcolour.info/). The CMEMS ocean transparency merged products were downloaded from the website of CMEMS (https://marine.copernicus.eu/).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available on request from the corresponding author.

Additional information

Funding

This work was supported by the National Key Research and Development Program of China No. [2021YFB3901300].