Recurrent U-Net based dynamic paddy rice mapping in South Korea with enhanced data compatibility to support agricultural decision making

ABSTRACT

The integration of remote sensing and state-of-the-art deep learning models has enabled the generation of highly accurate semantic segmentation maps to serve the agricultural sector, for which continuous land monitoring is required. However, despite their wide presence in the research field, only a few such products are used in on-site decision-making processes. This is due to their incompatibility with existing datasets that are at the core of current operating processes. In this study, paddy rice mapping in South Korea was examined to determine whether it produces qualified products that can complement on-site surveys and simultaneously be compatible with existing domestic datasets. Cases of early predictions for timely rice supply control were examined using a recurrent U-Net architecture with diverse applications: chronological batch training (CBT), time-inversed padding (TIP), and super-resolution (SR). In addition, the paddy area was confirmed using diverse datasets by standardizing its spatial extent in the definition of each data manual and calibrating the levee error, which was considered a major source of incompatibility. The robustness of the recurrent U-Net in early predictions dramatically increased upon CBT and TIP, recording an F1 score of over 0.75 on July 10, when the on-site survey was performed; meanwhile, the best performance score was 0.81 at the end of the growing period. SR enhanced the spatial details of rice mapping near the levee area, which had an estimated width of 60 cm; however, the area was more similar to that in existing datasets when it was calibrated with the predicted probability of the levee ratio rather than SR. The calibration was scalable from the patch to city level, with the paddy area at both levels recording high R² for the farm map and statistics (0.99 for both the farm map and statistics at the city level, and 0.93 and 0.95, respectively, at the patch level). This study shows that remote-sensing-based paddy rice mapping can produce not only accurate but also timely and compatible predictions by integrating deep learning applications. The results show that the predictions are compatible with domestic datasets as much as they are with each other; therefore, remote-sensing approaches are expected to be more actively and practically integrated into agricultural decision-making processes.

KEYWORDS:

• Recurrent U-Net
• paddy rice mapping
• decision-making
• early prediction
• remote sensing
• semantic segmentation

1. Introduction

Owing to advancements in remote-sensing instruments and their integration with state-of-the-art modeling methods, such as deep learning, the accuracy of land cover segmentation in most recent studies has increased to more than 90% (Jo et al. 2020; Sitokonstantinou et al. 2021; Xiao, Xu, and He 2021). However, despite the outstanding performance of such techniques, which have the potential to generate data for decision-making, their practical usage is sometimes restricted due to factors such as untimeliness, insufficient precision, and incompatibility with existing national datasets, which may have subtle but critical differences in values and definitions of the target, and can hence change decisions. In addition, since the accuracy of the classifier is equal or superior to the human level (Hestness, Ardalani, and Diamos 2019), the classification results have reached to an accuracy of man-made labeling; therefore, higher accuracy alone may not necessarily guarantee a more accurate reflection of reality. The value of decision-making data also depends on whether the classification results can be incorporated into operational decision-making processes and harmonized with existing datasets.

Paddy rice mapping in South Korea is a representative case characterized by both great potential and limitations in applying state-of-the-art technologies for decision-making. South Korea is chronically overproducing paddy rice; rice consumption has decreased due to the westernization of diets, but aging farmers are holding on to the cultivation of rice because it is the most mechanized and least labor-intensive crop. The Korean government has started responding to the imbalance of supply and demand by implementing diverse long-term policies to reduce paddy rice production. According to the Grain Management Act, revised in 2020, management plans for overproduction should be established by every October 15 before a full-scale harvest; overproduction is isolated from the market when it is greater than 3% of the expected or estimated production.

The deep-learning models applied to remote-sensing data have significant potential for supporting policy decisions as they enable accurate and efficient nationwide segmentation of paddy rice, allowing for more accurate estimates of rice yields. Since the cultivation area and yield are both crucial factors in determining rice production, and because advanced yield modeling requires comprehensive information on the spatial distribution of paddy fields, mapping these fields is essential for providing evidence to support policy decisions. Moreover, the reliability of reference data must be asserted by their compatibility with relevant datasets, unless the incompatibility can be clearly explained and calibrated. In addition, modeling will only be useful if it is performed before formulating the management plans; the current decision-making dataset is prepared between July and August using an on-the-spot survey (see Section 2.2).

However, paddy rice mapping in the ongoing decision-making process remains supplementary because of mis-qualifications such as area mismatches caused by impreciseness, a lack of agreement in defining the paddy field area, and belated information. Statistics Korea, which provides a national decision-making dataset, has been trying to improve the reliability of paddy area statistics using remote sensing since 2013 (Statistics Korea 2020). However, transforming the existing sampling approach into wall-to-wall mapping may decrease compatibility with past records. Instead, remote sensing is only being integrated into the existing statistical approach as a source of ancillary data for selecting the sampling area and visually interpreting some of the sampling spots.

Likewise, despite the progress made in preparing more accurate paddy rice maps using remote-sensing data, there are limitations to the methods used in previous studies. For example, time-series MODIS images have been exploited for their strength in tracking the phenological changes in rice (Tornos et al. 2015; Clauss, Yan, and Kuenzer 2016; Zhang et al. 2020). However, a coarse resolution of at least 250 m results in mixed pixel values, making the results valuable only in the context of large-scale monitoring, such as in ecological environment modeling.

Recent approaches for paddy rice mapping widely use radar sensors with higher spatial resolution, such as Sentinel-1. These sensors are capable of stable time-series observation that is invariant under different weather conditions, allowing for the preparation of a large volume of remote-sensing data that can then be utilized for deep-learning models to achieve state-of-the-art performance. Wei et al. (2019) applied U-Net to a time-series Sentinel-1 dataset after introducing analysis of variance and the Jeffries – Matusita distance to reduce multi-temporal data redundancy. The deep-learning model was found to be optimal, producing values more similar to the crop land statistics of Jilin Province, China, than those generated by traditional machine-learning approaches. Jo et al. (2020) analyzed time-series Sentinel-1 images using diverse recurrent neural network-based deep-learning applications and increased the accuracy of mapping heterogeneous environments in South Korea.

Both of these studies demonstrated the effectiveness of using deep learning for accurate paddy rice mapping by examining representative neural networks that can exploit spatial and temporal features, respectively. However, when the results of gridded paddy rice mapping and national statistics were compared in these studies, reasons for mismatches, such as over- or underestimation and bias, were not clearly explained; clarity is necessary for narrowing the gap between the on-going processes for paddy area estimation.

Jo et al. (2020) suggested further research on the effects of levees at the boundaries of paddy fields, which can lead to over-estimation when they are grouped with paddy rice during labeling. Xu et al. (2021) also recognized the importance of processing the boundaries for small and fragmented paddy mapping in Thailand, the characteristics of which are similar to those of paddies in the mountainous regions of South Korea. In their study, the probability result of the U-Net classifier with a fully connected conditional random field module was further processed so that the paddy rice mapping could preserve compact shapes and field edges. Wang et al. (2022) delineated paddy boundaries with fine-resolution satellite images processed using U-Net and analyzed time-series Sentinel-1 images to identify rice cultivation in the parcels. Using images with a 0.5 m resolution, they delineated boundaries with sub-meter widths and reduced the near-border errors.

Although these proposed models added more precision to paddy rice mapping, they were only examined for planting to harvesting observations; their applicability to early prediction for timely decision-making was not examined. Moreover, the use of fine-resolution images for boundary delineation can be costly on a national scale, particularly in South Korea, where frequent updates are needed to reflect rapidly changing land use.

A study performed in South Korea aimed at developing a scalable random-forest model for paddy rice mapping with the application of pseudo-labeling reported dynamic classifications throughout multiple cultivation periods (Sitokonstantinou et al. 2021). Although the accuracy of the results was excellent and the use of dynamic predictions to improve the value of paddy rice mapping for agricultural decision-making was novel, the study was limited to sample areas of the country and did not explore model application at the national scale. In addition, the dynamic prediction was performed by training a separate set of parameters for each prediction period, adding more computational cost to the modeling.

Therefore, the current study was aimed at developing a paddy rice detection model for South Korea to produce data that are qualified for reliable decision-making. In addition to pursuing higher accuracy, deep-learning applications based on a recurrent U-net (RU-Net) were employed to 1) dynamically map paddy rice, including the early stage of rice cultivation for agricultural decision-making, 2) produce paddy area data at the national scale that are compatible with the existing national dataset, and 3) calibrate errors originating from paddy boundaries. For this, we used technical applications such as chronological batch training (CBT) and time-inversed padding (TIP) that enhance the ability to extract time series features within a single end-to-end learning process, thus simplifying the procedures compared to training multiple models for each period. We also examined the data generation methods for each national datasets – statistics and GIS dataset – to identify if the paddy area includes the levee; therefore, the national-scale labeling of paddy rice was based on a common definition and an agreed extent of paddy area, contributing to a model that is more compatible with national datasets. Finally, we investigated super-resolution (SR) techniques and post-calibration using the probability of levee ratio (PLR) over cultivation area to employ the spatial information on levees for harmonizing the mapping results and national datasets, which can contribute to decision-making.

2. Materials

2.1. Study area

The study area comprised all of South Korea except Ulleung-gun in the East Sea, where no rice cultivation is done (Figure 1). In South Korea, more than 99.9% of rice is cultivated in inundated paddies. Owing to irrigation availability and suitable environmental circumstances, most paddies are located in the western and southern plain regions, whereas some are sparsely distributed in the mountainous east (Chung et al. 2011). There is a variance of approximately one month in rice phenology according to the rice type, and transplantation onto inundated paddies starts in early to mid-May in the southern region and ends in late to mid-June in the northern region (Rural Development Administration Korea 2017; Kim et al. 2009). The ripening stage is from mid-August to early September, and the harvest is conducted no later than the end of October.

Figure 1. Paddy rice distribution in South Korea.

The paddy rice cultivation area in South Korea has been decreasing for more than 30 years and is expected to decrease by 1.1% per year after 2021, from 732,000 ha. However, consumption has decreased more rapidly (by 2.3% on average) over the past 10 years, accelerating overproduction (Kim and Park 2021). To stabilize the supply and demand balance, the Ministry of Agriculture, Food and Rural Affairs (MAFRA) quarantines an average of 365,000 tons of rice during the harvest season, which is compatible with a yield of 70,000 ha, and releases it through five rounds of public sales during the next year. The quarantined amount is decided based on yearly yield predictions by Statistics Korea or by referring to other national datasets, such as MAFRA, when statistics are not available.

2.2. National datasets on paddy rice

Statistics Korea provides paddy rice area statistics every year by updating previous statistics through on-the-spot surveys in the sample areas. This statistical approach employs the agricultural area and surveys of cultivation areas. Among 870,000 sample areas with a size of approximately 2 ha each, the agricultural area survey encompasses 32,000 areas to determine whether they are used for agricultural purposes. The previous statistics are then updated with the proportion of the changed area determined from the survey:

(1) $A_i=A_{i-1}\left(1+\frac{\mathrm{\Delta }{A}_i^s}{A_i^s}\right),\hspace{0.17em}$(1)

where A_i indicates the agricultural area of year $i$, and $A^s$is the investigated agricultural area in the sample areas.

The cultivation area survey includes 22,000 areas among the selected agricultural sample areas for crop type and investigates actual cultivation area without margin or fallow. The cultivation area is calculated as the proportion of cultivation area to the agricultural area added to a fully surveyed area where little agricultural land exists:

(2) $C_i=A_{i-1}\left(\frac{C_i^s}{A_i^s}\right)+C_i^f,\hspace{0.17em}$(2)

where C_i indicates the cultivation area of year i and C^f is the full survey cultivation area. The surveys on paddy rice are conducted in July, and city-level statistics are announced at the end of August after a review. Although statistics are a major reference for the current decision-making process, this study used the data as an auxiliary labeling source as they do not include explicit spatial information but are designated to administration boundaries (see Figure 1).

Farm map is a national agricultural dataset produced by MAFRA; it was used to label paddy rice in this study (Figure 2). Farm map records the agricultural area and crop type in a vector format; this record was visually interpreted with high-resolution satellite images and aerial photos by referring to other national GIS data. The visual interpretation was carried out by dividing the country on a regional scale; it took 2–3 years for the entire country to be assigned. The acquired version of the farm map comprised the statuses of 2017, 2018, and 2019. The boundary of the farm map is very accurate because the demarcation was performed by experts using very high-resolution images (Lee et al. 2021). However, the boundary indicates the field area rather than the exact cultivation area, as it does not separate margins less than 3 m in width.

Figure 2. Agricultural land cover labeling in the farm map.

2.3. Remote-sensing data

In this study, time-series Sentinel-1 images were used to detect paddy rice as they are beneficial for monitoring seasonal changes in inundated paddy fields. Since it uses an active sensor with C-band microwaves that pass through clouds, Sentinel-1 enables stable time-series monitoring in all weather conditions. Moreover, Sentinel-1 is sensitive for capturing ground surface texture and effectively identifies the phenological features of paddy rice that are clearly distinguishable from those of other crops because paddies are flooded during the planting season. Considering the phenological stages of paddy rice, images of eight time steps were composited throughout the growing season, from May 10 to October 20, at intervals of 20 days (Figure 3).

Figure 3. Time-series sentinel-1 images composited based on rice phenology.

Minimum values were composited in the planting season to capture generalized features of low backscattering in flooded paddies invariant to transplanting time. Likewise, the maximum values were composited in the booting season to capture generalized features of high backscattering from the rapidly growing grains, and the mean values were composited in the other seasons. Sentinel-1 images were collected from 2017 to 2019 for each area to coincide with the farm map. Images with a 10 m resolution were divided into patches of 256 × 256 pixels, and 12,942 learning samples were constructed by filtering only patches containing rice paddies in the farm map.

3. Methodology

Figure 4 illustrates the workflow of this study, which was aimed at developing a timely paddy rice detection model that is compatible in terms of the definition of the paddy boundary and its inner area with domestic datasets. Since the definition of the paddy area varies even among domestic datasets, the spatial extent of the farm map was calibrated according to the definition of cultivation area statistics to create accurate labels. Calibrated farm map (C-farm map) differentiated paddy boundaries more distinctly by labeling both cultivation and levee areas.

Figure 4. Research flow for developing paddy rice detection model.

A C-farm map was used to train paddy rice detection models, which were designed with an RU-net architecture for timely detection during the growing season (Figure 6). Additionally, various applications were examined to improve robustness over time and enhance spatial details. The model was trained on a time series of Sentinel-1 data with a 10 m spatial resolution, and the C-farm map was rasterized to match this resolution, except for examining SR methods, which required mapping with a 5 m resolution. The training and validation areas were randomly selected from the learning samples in a 3:2 ratio, and the trained model was tested from 2017 to 2021 for the entire study area.

The generated paddy maps were evaluated in three steps to assess their effectiveness for both timely and compatible prediction at the pixel, patch, and city levels (see the numbers of validation and calibration in Figure 4). Firstly, the semantic segmentation of paddy rice was evaluated using the F1 score by comparing the segmentation with the C-farm map. Secondly, the paddy rice area segmentation for each city were aggregated and compared to the cultivation area statistics. Based on the comparison, the paddy rice area was further calibrated to minimize the mismatch in scales that was greater than the pixel level caused by errors near the levee boundary. The PLR was used to calibrate the paddy area, and its scalability was examined at both the city and patch levels. Finally, the calibration result, farm map, and cultivation area statistics were compared to each other to assess whether the paddy rice area determined using remote-sensing data processed with deep learning was compatible with domestic datasets; the baseline of compatibility was set to the correlation between the two domestic datasets.

3.1. Compatibility between national datasets

To enhance the compatibility of the model with the national datasets, the compatibility between farm map and cultivation area statistics was first examined. As the outline of farm map indicates the paddy boundary, the data-derived paddy area is different from that obtained using cultivation area statistics, which specifically records the paddy rice area without including levees. The C-farm map was created to improve compatibility with the national datasets in terms of both numerical extent and definition. This was achieved using the same methods used for generating statistics to estimate cultivation area from agricultural area by excluding the length of levees from parcel boundaries (see Section 2.2); in cases where levees are shared by multiple parcels, only half the length was excluded (Figure 5). Therefore, a certain length of levee was found to minimize the root mean squared error (RMSE) between the cultivation areas determined using the C-farm map and cultivation area statistics. The inner boundary of the calibrated area was labeled as a cultivation area, and the reduced area was labeled as a levee.

Figure 5. Farm map calibration by introducing levee area.

3.2. RU-Net applications

For a model to produce timely paddy rice segmentation in the middle of the growing season, its deep-learning architecture should be effective in extracting both spatial and temporal contexts from images. The proposed approach is to integrate U-Net with recurrent neural networks (RNNs) that are specifically used for extracting spatial and temporal features, respectively (Yang et al. 2019; Wang et al. 2019).

In this study, an RU-Net was designed to follow a standard U-Net architecture, while the encoder comprising a series of recurrent modules (Figure 6). Each recurrent module consisted of convolutional layers, a dropout, and spatial maximum pooling. Each time series shared the convolutional layers, and the calculation of the previous time step was added to the next time step so that the phenological contexts could be passed on for the subsequent calculation. Considering both the preservation of temporal features and computational efficiency, the max-pooling layers at the skip connections were applied to the time axis so that the adjacent time steps at the same phenological stage were pooled, and half the size of the features was passed on to the decoder.

Figure 6. Architecture of proposed recurrent U-Net.

For the RU-Net to produce timely predictions without fully obtained time steps and to enhance high-frequency spatial details, the following methods were proposed for the training process and architecture.

1) CBT: The prediction with the entire time step and the early prediction with confined time steps share a common feature extraction process; however, the signal intensity through the neural network can be greatly altered by the activation process. Therefore, if the input of the aforementioned time series is provided in a random order during the training phase, the loss function is hardly optimized and over-fitted to the last-seen training instances, which is associated with the problem of catastrophic forgetting that newly acquired information disrupts previously learned information (French 1999). The problem could be mitigated by sharing knowledge among the tasks and iteratively training each task in turn (Choi and Park 2018). Therefore, we manipulated the training order, as described in Figure 7, so that the model could sequentially learn from the data of each time step. Specifically, for each epoch, we started to provide the cases of very early prediction as input (e.g. only 1 time step), followed by those of additional time steps (e.g. the full growing season in the end). By training the model with chronologically ordered batches, the parameters were gradually updated with an additional time step after every new batch type, which was more likely to preserve the knowledge gained from the values of each new time step. RU-Net without and with CBT was denoted as RU-Net A and RU-Net B, respectively.

Figure 7. Concept of chronological batch training.

1. TIP: The proposed method involves zero-padding the unobtained time step with an inverse time sequence. Once the input is in a time order without inversion, the padded time steps are affected by the last valid time step because of the sequential process of the RNNs (Yoon and Yu 2020). Since paddy rice dynamically changes during its phenological stage, the features indicated by the last valid time step and the subsequent padded time steps are greatly affected by the prediction time. Upon inverting the time order, the effect of padding becomes independent of the last valid time step, and the valid time steps maintain their location regardless of the prediction time. Therefore, the padded time step no longer duplicates the spatial features of the last valid time step and only affects the activation process as a bias (Dwarampudi and Reddy 2019), and the properly trained bias can compensate for the lack of signal at missing time steps. RU-Net with CBT and TIP was denoted as RU-Net C.

2. SR: An SR approach with an additional up-sampling layer was examined to validate whether enhancing the precision could minimize the near-paddy boundary error and improve compatibility with the existing datasets. Up-sampling was expected to extract clearer high-frequency details from the multiple feature layers, in comparison with interpolation of the single-layer output, which results in blurred or jagged images (Zhao et al. 2018). RU-Net with CBT, TIP, and SR was denoted as RU-Net D.

3.3. Validation & calibration

The F1 scores for all the proposed RU-Nets from A to D and for eight time steps from May to October were calculated to validate the performance at the pixel level. The effect of each application on enhancing timely prediction was validated by comparing the F1 scores for multiple time steps. In general, each pixel is classified based on the argument of maximum among the class probabilities: probability of cultivation area (P_c), levee (P_l), and others (P_o). However, in this study, an option for integrating the probability of levee with that of cultivation area was examined because, if the pixel size is much larger than the levee width, the pixels labeled as levees inevitably include a large portion of the cultivation area, even if the center of the pixels are located in the levee areas.

In addition, to make the best use of each probability for paddy area estimation, the difference between the predicted area and that obtained from the C-farm map was investigated for the existence of a levee by calculating the PLR using the following equation:

(3) $\mathrm{P}\mathrm{L}\mathrm{R}=\frac{P_l\hspace{0.17em}}{P_c}.$(3)

A scalable area-calibration method to differentiate paddy rice areas from the gridded results with an empirical equation was proposed by estimating the correlation between the PLR and the result error at both the patch and city levels. Finally, the correlation between the modeling results and the two national datasets was compared to validate whether the results were more or less compatible with the two national datasets than they were with each other.

4. Results

4.1. Spatial calibration of farm map

Table 1 compares the statistics of paddy rice cultivation areas with those obtained using the C-farm map with diverse versions of calibration. Without calibration, the paddy area on the farm map was found to be approximately 5% greater than that reported in the statistics. By removing a certain width of the levee from the farm map to produce cultivation areas of C-farm map, the paddy area became comparable to that reported in the cultivation area statistics, with a difference of less than 1% when the width of the removed levee was set to 60 cm. When the area was compared at city-level aggregation, the RMSE decreased as the levee width increased up to 60 cm; however, the RMSE increased when the levee width exceeded this value. Based on the results, the most compatible calibration of the farm map was set to a levee width of 60 cm, which also corresponds to the width mentioned in the domestic report (National Academy of Agricultural Sciences 2014). Therefore, the C-farm map with a levee width of 60 cm was used for labeling in this study.

Table 1. Results of farm map calibration according to levee width.

Dataset (Levee width, cm)	Area (ha)	Relative Area (%)	RMSE (City Level)
Statistics	746,247	100	0
Farm Map	783,477	104.99	759.70
C-Farm Map (50)	753,834	101.02	679.50
C-Farm Map (60)	743,842	99.68	667.21
C-Farm Map (70)	742,117	99.45	668.72
C-Farm Map (90)	730,480	97.89	670.98

4.2. Pixel-level validation

Figure 8 shows the results of validating the paddy rice detection models at each time step. Throughout each period, the maximum value of P_l did not exceed 0.2, implying that none of the pixels was classified as a levee. Therefore, the F1 score was calculated using the binary classification of the cultivation and other areas by comparing the model prediction and C-farm map. In every case of separation and integration of P_c and P_l, the integrated value had minor but continuous superiority over the compared prediction; this is shown as a representative case of RU-Net C. Owing to the superior performance at pixel-level validation, the evaluation of each model was based on the prediction indicating that the combined value of P_c and P_l was greater than P_o.

Figure 8. Pixel level-validation through prediction time.

The results showed little difference when all the input time steps were provided, recording an F1 score of more than 0.77 on October 20. The F1 scores at the end of the prediction period were 0.78, 0.79, 0.81, and 0.81 for RU-Nets A, B, C and D, respectively. When the proposed training methods were not applied, the performance of RU-Net A decreased dramatically as the prediction time became earlier and it recorded an almost zero F1 score before August 20. For RU-Nets B – D, which were trained with CBT, the F1 score of early predictions failed to reach that of October but remained over 0.71 after June 20 and over 0.75 after July 10. Without TIP, the results of RU-Net B were inferior to those of RU-Nets C and D at most time steps. RU-Net D exhibited the most stable and monotonously increasing performance over time.

A comparison of the results of RU-Nets A and B indicated that CBT increased the robustness of early prediction by overcoming catastrophic forgetting. In accordance with the designed purpose of chronologically ordered training, the knowledge of activating signals using only confined time steps was successfully transferred to the next training with more time steps in the harmonious development of the feature extraction process. However, the F1 scores on August 20 and September 10 were lower than that on July 31 despite the inclusion of additional time-series information. This fluctuation in performance implies that additional information does not always contribute to accurate mapping, especially without TIP, when the last valid time step affects the subsequent zero paddings under the recurrent architecture. A confusion matrix for the pixel-based validation is provided in Appendix A (Table A1).

To further investigate segmentation accuracy near boundaries for each RU-Net application, Boundary Intersection-over-Union (BIOU) was calculated using the following equation:

(4) $\mathrm{B}\mathrm{I}\mathrm{O}\mathrm{U}\left(L,\hspace{0.17em}P\right)=\frac{\left|\left(L_d\cap L\right)\cap \left(P_d\cap P\right)\right|}{\left|\left(L_d\cap L\right)\cup \left(P_d\cap P\right)\right|},$(4)

where $L$ and $P$ are the label and prediction, respectively, and $L_d$ and $P_d$ are the corresponding areas within d pixel distance from the label and prediction boundaries. BIOU is measured to evaluate segmentation that focuses on boundary quality by estimating boundary alignment between prediction and labeling masks (Cheng et al. 2021). Table 2 shows the BIOU between paddy rice labeling and each RU-Net application at a 5 m resolution with full period dataset. Unlike that of RU-Net D, which already had a resolution of 5 m, the predictions of RU-Nets A – C were bilinearly resampled to 5 m. Since the overall accuracy was validated with F1 scores, BIOU was explicitly calculated for performance near the boundary by setting short d to 10 m. The boundary delineation accuracy continuously improved in the order of RU-Net A<B<C<D, with the greatest BIOU of 19.31% acquired using RU-Net D.

Table 2. Boundary intersection-over-union (BIOU) measured for recurrent U-Net (RU-Net) applications.

	RU-Net A	RU-Net B	RU-Net C	RU-Net D
			Number of Pixels
$\left|\left(L_d\cap L\right)\cap \left(P_d\cap P\right)\right|$	10,366,477	11,502,935	11,464,274	13,155,338
$\left|\left(L_d\cap L\right)\cup \left(P_d\cap P\right)\right|$	65,050,502	67,700,172	65,421,331	68,120,738
BIOU	15.93%	16.99%	17.52%	19.31%

4.3. Scalable area calibration

Since the levee width was lower than the data precision of 10 m, a small proportion of the levee area was designated as a levee in grid-format labeling, and, therefore, none of the pixels were classified as levees in the predictions. However, labeling both the levee and paddy rice areas enabled the model to differentiate their distinct features. Therefore, adding two probabilities of levee and cultivation area for pixel-by-pixel mapping should be recognized as different from combining the two classes in the labeling process, as it provides additional information on land surface composition that enables calibration based on PLR.

Figure 9 shows the correlation between PLR and the ratio of prediction error to cultivation area aggregated at the city and patch levels; cities with more than 100 ha of paddy rice and patches of more than 50 ha were selected to remove outliers. The prediction error and PLR were negatively correlated in RU-Net C at both the city and patch levels (Figure 9(a,b)), whereas no correlations were found in RU-Net D (Figure 9(c,d)). Although the city-level samples deviated from the trendline in the early predictions shown in Figure 9a, the overall trend was similar to that of the prediction made on October 20, with a slope of ˗8.9102 and a bias of 0.3317. Despite the low R² value in the patch-level comparison shown in Figure 9b, the slope and bias of the trendline were almost similar to those in the city-level comparison, at ˗8.7297 and 0.3406, respectively. The similarity in trendlines across scales suggests that this calibration of paddy rice areas is scalable, as it corrects systematic errors according to the following equation:

Figure 9. Correlation between PLR and error/cultivation area ratio:(a) RU-Net C at city level, (b) RU-Net C at patch level for time steps after June, (c) RU-Net D at city level and (d) patch level on October 20.

(5) $\mathrm{C}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{b}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{A}\mathrm{r}\mathrm{e}\mathrm{a}=\frac{\mathrm{A}\mathrm{r}\mathrm{e}\mathrm{a}}{1+f_{Calibration}\left(PLR\right)},\hspace{0.17em}$(5)

where the calibration function is the empirical correlation between PLR and the error over the cultivation area. In this study, the slope and bias of the calibration function for RU-Net C were determined by averaging the trendlines at two different scales, resulting in values of ˗8.8199 and 0.3362, respectively.

4.4. Compatibility with national datasets

Figure 10 compares the predicted paddy rice areas with those in existing national datasets. As the correlation between the existing national datasets (statistics and farm map) recorded an R² value of 0.9788, it could be used as a baseline for assessing the predictions as decision-making data. The R² values of the correlations between RU-Net C and the farm map and statistics predicted on October 20 were 0.9933 and 0.9796, respectively, recording a value higher than that of the correlation between the two national datasets. When the prediction was made on July 10 (the approximate date on which the statistics are announced, or slightly earlier), the R² values for the farm map and statistics were 0.9794 and 0.9675, respectively; both full-time and early predictions exhibited greater correlation with the farm map. The R² values of the correlations between RU-Net D and the farm map and statistics were 0.9931 and 0.9809, respectively, on October 20, and 0.9801 and 0.9752, respectively, on July 10. Although the correlation between the predictions and statistics was less than the baseline at the early prediction stage, it exceeded the baseline in most cases. When RU-Net C was calibrated with PLR, the overall R² of the farm map and statistics increased to 0.9937 and 0.9848, respectively, at the end prediction, and 0.9854 and 0.9777, respectively, at the early prediction. Moreover, the calibration also enhanced the correlation between the predictions and the farm map at the patch level from an R² of 0.932 and 0.8161 to 0.945 and 0.8525 for the end and early predictions, respectively. Before the calibration, the distribution of patch samples was skewed such that a greater number of underestimated patches were recorded with less paddy area; in contrast, a dominant overestimation was recorded as the paddy area increased. However, the skewness decreased after calibration.

Figure 10. Correlation between national datasets and predictions: (a) Farm map to statistics, (b) Farm map to RU-Net C at city level, (c) Statistics to RU-Net C at city level, (d) Farm map to RU-Net D at city level, (e) Statistics to RU-Net D at city level, (f) Farm map to calibrated RU-Net C at city level, (g) Statistics to calibrated RU-Net C at city level, (h) Farm map to RU-Net C at patch level, and (i) Statistics to calibrated RU-Net C at patch level where paddy rice exists.

5. Discussion

5.1. Dynamic prediction for timely paddy rice mapping

For the timely paddy rice mapping, the satellite images taken from May to July were chosen to include key features for detecting paddy rice, as the lack of inputs in that period greatly deteriorated the performance compared to that obtained in the absence of other time steps. These results coincided with those of previous research showing that the interpretation of the planting season from May to June is predominant in the modeling procedure (Dong and Xiao 2016; Xiao et al. 2002). In July, the dramatic change in the backscattering value at the tillering stage was found to be effective in differentiating the inundated paddy from water.

When both CBT and TIP were applied to the model, the robustness of early prediction improved, as indicated by high performance on July 10, when key features were sufficiently acquired. The performance was nearly the same as that with full observation in October. Considering that the current statistical survey on paddy rice area was conducted from July 1 to 20, validated in the subsequent three days, and reported at the end of August, the results with CBT and TIP showed that the remote-sensing-based paddy rice mapping could investigate the paddy area within a time period similar to or shorter than that taken by the statistical approach and at much less cost.

The use of a single model for dynamic predictions of paddy rice, enabled by the application of CBT and TIP in this study, offers several advantages over the approach used in a previous study of training multiple models for each prediction period (Sitokonstantinou et al. 2021). The single model simplifies the machine-learning process and enables the machine to learn how to interpret time-series data with varying lengths for crop mapping. This knowledge can then be applied to other crop-mapping tasks that use time-series data. Future studies should examine the applicability of trained RU-Net to diverse regions, times, and crop types through transfer learning.

5.2. Minimizing errors originating from paddy boundaries

This study was aimed at minimizing mismatches in paddy rice areas originating from paddy boundaries by applying SR (RU-Net D) and calibrating with PLR (calibrated RU-Net C). As illustrated in Figure 11, owing to the contrasting P_c and P_l, paddy boundaries can be identified even when the gap between parcels is less than the spatial resolution of Sentinel-1 images (10 m). Upon the application of SR, the resolution of RU-Net D became higher than that of RU-Net C. Even when RU-Net C was resampled to 5 m to match the resolution of RU-Net D, the latter produced clearer high-frequency details, as the resolution was enhanced by exploiting multiple layers and preserving more spatial information; this finding supports the results presented in Table 2. The stable and monotonic increase in the F1 score over time and the absence of systematic error induced by PLR also suggest that RU-Net D provided a clearer representation of the paddy boundary (Figures 8 and 9(c,d).

Figure 11. Comparison of spatial details in predictions of RU-Nets C and D.

Unlike that obtained using RU-Net D, the paddy area indicated by RU-Net C had a systematic error, which led to an underestimation of the paddy area as the PLR increased. The PLR equation, which comprises the ratio between P_l and P_c, represents the level of parcel fragmentation since PLR decreases at clustered parcels that share the levee area and increases at fragmented parcels that have their own levee. Therefore, the systematic error in RU-Net C implies that some parts of the paddy rice are not detected without SR when parcels are fragmented. Leaving aside the similar absolute errors in RU-Nets C and D, in contrast, RU-Net D exhibited an advantage in the unbiased interpretation of paddy areas in both fragmented and clustered paddies.

Nevertheless, the errors in RU-Net C were partially explainable with PLR in both large- and small-scale (city and patch level) sampling. The correlation between the errors and PLR was maintained across scales, allowing for scalable calibration at multiple levels with the expected errors from PLR. Hence, calibrated RU-Net C provided more accurate estimates of paddy areas at the national and regional levels, which may be useful for agricultural decision-making. Although PLR-based calibration enables accurate estimation of paddy areas, it does not improve precision or high-frequency details for binary classification, which is important for delineating paddy boundaries. Therefore, the results should be chosen based on the research purpose (area estimation or boundary delineation), and further research is needed to determine how much high-frequency detail can be obtained by incorporating more time-series data into the SR model.

5.3. Applicability to agricultural decision-making

To further investigate the beneficial effect of remote-sensing-based paddy rice mapping on agricultural decision-making, five years of paddy rice statistics and predictions were analyzed to determine whether they are interrelated with the rice wholesale price, which was acquired from the Korea Agricultural Marketing Information Service (https://www.kamis.or.kr). As the rice price is mainly affected by the amount of harvested paddy rice until the next harvest, the monthly prices were averaged for the harvest cycle from October to the next year’s September (Table 3). Rice yield was calculated from the yearly paddy areas estimated using each data source by multiplying the yield/ha statistics for each city (Table 4). Thereafter, the inversed relative yield (IR) of paddy rice was calculated by dividing the five-year average yield by the yield of each year, as expressed in the following equation:

Table 3. Monthly rice prices according to harvest cycle

	Harvest year					Next year														Avg. Price
Year	Oct		Nov		Dec	Jan		Feb	Mar		Apr	May		Jun	Jul	Aug		Sep		(20 kg/won)
2017		35,941	37,287	38,032			38,806	40,888	42,418	43,695		43,555	43,713		44,738		45,030		45,560	41,638
2018		47,727	48,799	48,680			48,596	48,580	48,598	48,475		48,380	48,243		48,180		48,181		47,967	48,367
2019		45,727	46,252	46,245			46,350	46,305	46,332	46,220		46,194	47,414		47,698		48,387		49,986	46,926
2020		52,468	54,644	55,229			55,547	56,319	56,950	57,422		57,569	57,762		57,937		57,796		56,063	56,309
2021		54,401	52,461	51,709			51,165	51,016	50,757	49,105		47,761	49,251		49,627		50,003		50,380	50,636

*Rice prices from Jun. to Sep. 2022 are expected values according to the price trend in the past five years

Table 4. Paddy rice area and expected rice yield based on different datasets.

Year (y)	2017	2018	2019	2020	2021
			Paddy rice area (ha)
Cultivation Area Statistics	754,339	737,408	729,585	726,180	732,070
RU-Net C	788,783	758,310	765,671	716,045	715,328
RU-Net C (Calibrated)	836,768	774,604	797,198	734,893	735,535
			Expected rice yield (Y, ton)
Cultivation Area Statistics	3,971,609	3,867,415	3,743,868	3,505,984	3,880,312
RU-Net C	4,147,045	3,980,404	3,931,803	3,457,057	3,790,726
RU-Net C (Calibrated)	4,396,556	4,063,705	4,092,714	3,547,009	3,895,447

(6) $I{R}_y\hspace{0.17em}\left(Inversed\hspace{0.17em}Relative\hspace{0.17em}Yield\right)=\frac{1}{5}{\sum }_{i=2017}^{2021}{Y}_i/Y_y,$(6)

where $Y_y$ indicates the rice yield in year y. Considering that excessive supply lowers the price, the time-series pattern of IR was presumed to coincide with that of the price.

As illustrated in Figure 12, the overall increase in rice price matched the increase in IR caused by the decrease in paddy rice area. The short-term pattern indicated that the IR based on remote-sensing-based predictions, especially that of calibrated RU-Net C, surpassed that indicated by the statistics in explaining the 2019 price reduction. Although the paddy area is one of the diverse factors that determine market price and the predictions are substantially overestimated compared to the statistics, the interrelated patterns show the usefulness of deep-learning-based remote-sensing analysis for agricultural decision-making and the need for steady development of its utility.

Figure 12. Time-series pattern of rice price and inversed relative yield.

6. Conclusion

In this study, we developed a paddy rice detection model that provides timely and compatible predictions to maximize their value as decision-making data. The proposed RU-Net model was trained using time-series Sentinel-1 images composited through rice phenology and labeling of paddy rice and levee areas. Owing to the application of CBT and TIP, RU-Nets C and D produced robust early predictions with an F1 score of more than 0.75 in a similar or smaller time period than that taken by the statistics, which were produced in July, and showed minor differences in quality in terms of paddy rice mapping at the end of the growing season, which recorded F1 scores of 0.807 and 0.806, respectively. An additional SR layer at the end of the architecture in RU-Net D enhanced the output resolution from 10 to 5 m, preserving high-frequency details such as paddy rice boundaries and levees. The clear representation of edges was found to decrease the bias of paddy areas according to the level of fragmentation, but it could not reduce the absolute error. By exploiting the systemic error expected due to PLR in RU-Net C, the paddy area could be calibrated to reduce the bias, making it more compatible with existing domestic datasets such as statistics and the farm map. The calibrated result correlated with the farm map and statistics, with R² values of 0.9854 and 0.9777, respectively, on July 10 and 0.9937 and 0.9848, respectively, on October 20. This could be assessed as a compatible and reliable decision-making dataset considering that the R² between the two existing national datasets – cultivation area statistics and farm map – was 0.9788.

The results show that deep-learning-based paddy rice mapping provides timely and compatible decision-making data comparable to the existing national datasets with cost- and labor-efficient progress. Moreover, the scalable calibration method, which effectively uses PLR, compensates for the discrete interpretation of a unit pixel, allowing representation of sub-pixel paddy areas, and produces paddy rice areas compatible with the existing national dataset even with its wall-to-wall mapping. However, it should be noted that datasets from different sources should be carefully compared to avoid misunderstanding agricultural activities and market prospects, as a levee width of 60 cm may result in a 5% difference in paddy area when aggregated to the national scale, resulting in skewed predictions. Therefore, it is recommended to use the spatially enhanced paddy rice map of RU-Net D to identify paddy rice distribution and the calibrated paddy rice area of RU-Net C to identify cultivation areas. As remote-sensing-based paddy rice mapping with the proposed scalable calibration method provides a sub-city-level paddy area, future research should integrate these results with on-the-spot surveys for downscaling statistics. In addition, although five years of paddy area predictions explain the rice price pattern, the substantial gap between the paddy area predictions and statistics should be studied from the perspective of a temporal imbalance in the backscattering intensity, since this study mainly focused on the skewness in spatial extent.

Author contribution

H.W.J designed this study, analyzed and interpreted the data, and wrote the manuscript. E.P, V.S., A.K., and W.K.L contributed to part of the methodology and review. J.K, and S.L., conducted the data investigation. All authors provided comments and approved the final manuscript. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

This work was supported by the International Research and Development Program of the National Research Foundation of Korea (NRF), funded by the Ministry of Science and ICT [2021K1A3A1A78097879], and partially supported by the European Commission under contract H2020-CALLISTO [101004152].

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

Paddy rice prediction maps from 2017 to 2021 and the labeled time-series Sentinel-1 patch dataset are available in a publicly accessible repository. If these datasets are used, please cite this publication. The “Paddy Rice Maps South Korea (2017 ~ 2021)” data are available at https://github.com/Agri-Hub/Callisto-Dataset-Collection.

Additional information

Funding

This work was supported by the International Research and Development Program of the National Research Foundation of Korea (NRF), funded by the Ministry of Science and ICT [2021K1A3A1A78097879], and partially supported by the European Commission under contract H2020-CALLISTO [101004152].

Appendix A

Table A1. Confusion matrix for paddy rice mapping.

		May 31	Jun 20	Jul 10	Jul 31	Aug 20	Sep 10	Sep 30	Oct 20
RU-Net 1	T-P	0	0	0	0	19	1823012	16496983	22224000
	F-N	30319352	30319352	30319352	30319352	30319333	28496340	13822369	8095352
	F-P	0	0	0	0	48	121135	2215511	4772441
	T-N	308960520	308960520	308960520	308960520	308960472	308839385	306745009	304188079
	F1	0	0	0	0	0	0.113	0.673	0.775
RU-Net 2	T-P	17295195	21857530	22107629	23791811	23179026	21656761	23105661	24001439
	F-N	13024157	8461822	8211723	6527541	7140326	8662591	7213691	6317913
	F-P	12092125	9260231	5911903	7147682	6253859	4484139	5442282	6446273
	T-N	296868395	299700289	303048617	301812838	302706661	304476381	303518238	302514247
	F1	0.579	0.712	0.758	0.777	0.776	0.767	0.785	0.790
RU-Net 3 (Pc)	T-P	12942663	21287828	22937157	23371645	23021308	23177821	23978962	23977913
	F-N	17376689	9031524	7382195	6947707	7298044	7141531	6340390	6341439
	F-P	6137297	7618023	7726879	5938278	5310070	4448764	4851484	5189637
	T-N	302823223	301342497	301233641	303022242	303650450	304511756	304109036	303770883
	F1	0.524	0.719	0.752	0.784	0.785	0.800	0.811	0.806
RU-Net 3 (Pc +Pl)	T-P	13217399	21558491	23183557	23616712	23271791	23425516	24208036	24204059
	F-N	17101953	8760861	7135795	6702640	7047561	6893836	6111316	6115293
	F-P	6426533	7946556	8030778	6223685	5569023	4693719	5114991	5448197
	T-N	302533987	301013964	300929742	302736835	303391497	304266801	303845529	303512323
	F1	0.529	0.721	0.754	0.785	0.787	0.802	0.812	0.807
RU-Net 4	T-P	62517071	83982416	91283550	92048913	95548729	96846895	98295881	98188905
	F-N	57499006	36033661	28732527	27967164	24467348	23169182	21720196	21827172
	F-P	31903542	24593697	24490869	21173266	24803501	25025191	26106965	25296363
	T-N	1205199869	1212509714	1212612542	1215930145	1212299910	1212078220	1210996446	1211807048
	F1	0.583	0.735	0.774	0.789	0.795	0.801	0.804	0.806