A multimodal fusion framework to diagnose cotton leaf curl virus using machine vision techniques

Abstract

Cotton diseases are disastrous for quality and sustainable production of the yield. Cotton leaf curl virus (CLCuV) is one of the most damaging diseases for cotton crops. Symptoms-based CLCuV identification is tedious, time consuming, error prone and needs exceptional expertise. Sensor-based machine vision approaches have great potential to detect the CLCuV at early stages. This research study proposes a machine vision-based multimodal fusion framework to diagnose various CLCuV severity levels. Our designed model is based on three contrasting datasets: digital photographic, multispectral and fused datasets. A digital camera was used to acquire the digital photographic dataset, the multispectral dataset was obtained by a multispectral radiometer-5 (MSR5), and the two datasets were fused to formulate the third one. From the digital photographic dataset, 269 texture features were extracted and optimized to the most discriminant 30 texture features, the multispectral dataset consisted of 5 spectral features, and the fused dataset was formed by combining the two. The 30 most discriminant features from the digital photographic dataset were selected by incorporating fisher co-efficient, probability of error plus average correlation and mutual information (MI). To diagnose CLCuV, four machine-learning classifiers, namely simple logistics (SL), multilayer perceptron (MLP), sequential minimal optimization (SMO), and random forest (RF), were deployed separately on each dataset. The maximum CLCuV diagnosing accuracies attained from digital photographic, multispectral, and fused datasets were 81.263%, 91.177%, and 96.313%, respectively.

Keywords:

• MSR5
• CLCuV
• Machine-learning classifiers
• MLP
• SL

REVIEWING EDITOR:

• Manuel Tejada, Universidad de Sevilla, Spain

SUBJECTS:

• Artificial Intelligence
• Computation
• Databases
• Testing

1. Introduction

During the last four decades, machine vision approaches have been applied sizably in many areas, such as automatic inspection, process control, assessing yield production, disease detection, etc. (Radhakrishnan, 2012). Disease detection in plants is a crucial process. Detection of plant leaf diseases based on visual assessment is challenging and requires considerable expertise and domain knowledge but is still error-prone. It can be successfully achieved by machine vision (Pantazi et al., 2019). Disease detection in crops at an early stage increases advantages and reduces losses. Machine learning and deep learning data analysis methods accurately detect crop diseases (Shahi et al., 2023). Bacterial, fungal, and viral infections of insects instigate plant diseases. Viral diseases affect the plant’s natural growth, infections, or mechanical injury to leaves or plants (Kerkech et al. 2018). Nevertheless, cotton plants are also affected by diseases in seeding, cotton bolls, cotyledons, and mature cotton leaves (Hassan et al., 2016; Prajapati et al., 2016). CLCuV is a damaging disease and a serious threat to cotton crops. Plants infected by CLCuV disease exhibit various symptoms due to their severity levels. The symptoms of CLCuV appear in photosynthate blockage, causing tertiary veins to be yellowish and thicken. Later, leaf curling and blockage of secondary veins reduce the leaf surface area for photosynthesis and block the passage of photosynthesis. Consequently, the infected plant seems to be dwarf. Leaves are curled downward or upward; growth of the leaves is also stunted due to inter-nodal distance reduction in severe conditions, and an outgrowth called enation appears on the lower side of a curled leaf (Saeed et al., 2018).

The evolutionary story of CLCuV disease in cotton crops is fascinating. The disease was first recorded in 1912 in Nigeria on Gossypium peruvianum, in 1924 in Sudan, in 1926 in Tanzania and in 1959 in the Philippines. CLCuV was reported in Pakistan for the first time in 1967 in District Multan of Punjab. Similar symptoms were reported on tobacco and other plants much earlier than 1967 in Pakistan. CLCuV was not measured as a serious disease until 1987, but in the early 1990s, it was assumed to be an epidemic proportion in Punjab. A huge loss of 1.3 million bales of cotton over 24.28 hectares was recorded in Punjab. Sindh remained a free province till 1997. CLCuV was first observed in Sindh in 1997 and then varied from area to area, year-wise. Therefore, it is very hard to estimate the loss caused by CLCuV overall in Pakistan (Farooq et al., 2014).

Cotton is considered a cash and fiber crop in Pakistan and is called ‘White Gold’ or ‘king of fibers’. Pakistan is in 4th position in the world for the production of cotton. Approximately, 55% of Pakistan’s foreign exchange earnings are due to cotton. Two provinces, Punjab and Sindh, are growing nearly 65% of Pakistan’s cotton. More than 15% of the cultivated area is devoted to cotton, and almost 26% of farmers in Pakistan grow a cotton crop. 4.5% of the agriculture gross domestic product (GDP) is added by cotton, which is 0.8 percent of the GDP. The country’s agro-industrial sector provides 17 percent employment opportunities (Rana et al., 2020). Cotton production in Pakistan has decreased during recent decades due to several factors, including the CLCuV, insect pests and various abiotic strains. CLCuV is a highly destructive plant disease that significantly reduces Pakistan’s ability to produce cotton (Akhtar et al., 2004).

One of the major diseases that cause economic and production loss in Pakistan and India is cotton leaf curl disease (CLCuD). CLCuD belongs to the Geminiviridae and is caused by a monopartite plant virus. Disease detection in plants requires expert opinions, as little variation in symptoms can cause economic and production loss. Based on the inappropriate diagnosis of diseases, farmers must input pesticides, resulting in an incredible amount of economic and production consequences. Plant disease diagnosis at the manual level can be error-prone and costly. The computer is used for automatic disease detection in the current information age (Zubair et al., 2017). Symptom-based detection of plant leaf diseases remains a headache due to its complexity. Many experienced plant pathologists and agronomists need help accurately diagnose a specific disease. Detection of plant diseases based on visual assessment is prone to error. Therefore, a farmer has to bear a large economic and production loss (Ferentinos, 2018). In the case of large-scale cultivation, it is very hard for farmers and experts to visit the field and observe each infected plant physically. Proper treatment can be made timely for accurate detection and diagnosis of plant diseases at the initial stage by deciding the right pesticide. Ultimately, economic and production loss can be avoided (Ilic et al., 2018).

The human eye cannot see within a visual spectral range of 400–700 nm, while remote sensing (RS) devices can measure radiations in numerous parts of the Light spectrum ranging from 10 to 255 nm (Nilsson, 1995). RS measures electromagnetic radiations that are emitted or reflected from an object. Numerous non-invasive procedures, like RS, imaging-based and spectroscopy-based, provide precise and reliable support to detect and monitor diseases in the plant in a large-scale and real-time environment. In addition, digital imaging sensors and hyperspectral or multispectral sensors are also used to detect plant diseases. Multispectral sensor normally works on an electromagnetic spectrum range of 400–2500 nm. These wavebands can easily capture a minor change in plant pathology by its diseases at their early stages (Zhang et al., 2020). Data Fusion combines data and information from multiple sources to produce more accurate results than separately using data sources (Ouhami et al., 2021). Data fusion can be classified into three major categories. 1) Observation level fusion; 2) Feature level fusion; 3) Decision level fusion. Observation-level fusion is used to combine raw data from two sources. Feature level fusion is used to integrate extracted features of different sources, while decision level is used to incorporate classification results achieved by each source alone to reach a common final decision (Schmitt & Zhu, 2016).

Researchers used deep learning and support vector machine (SVM) technologies to recognize patterns and detect four types of rice leaf diseases by using healthy and infected images. Features of infected rice leaf images were extracted using a convolutional neural network (CNN). After feature extraction, the SVM model was used to classify and predict the infected rice leaf. Pattern recognition accuracy was improved to 96.8% when both deep learning and SVM were combined (Jiang et al., 2020). Machine learning algorithms and hyperspectral imaging (HSI) techniques were used to detect tomato spotted wilt virus (TSWV) infection in tobacco at its early stages. A spectral range from 400 to 1000 nm with 28 bands was used. Four machine learning techniques, RF, SVM, classification and regression tree (CART) and boosted regression tree (BRT), and three wavelength selection methods (BRT, successive projection algorithm (SPA) and genetic algorithm (GA)) were used. BRT algorithm performs best with the SPA wavelength feature selection technique at 10-fold cross-validation and achieves 85.2% accuracy. Results also showed that Healthy and infected tobacco were differentiated by near-infrared (NIR) spectral region (Gu et al., 2019).

An approach was demonstrated to identify leaf diseases from different crop species. Various leaf images of species were used. One class classifier was deployed to classify powdery mildew, downy mildew, black rot and healthy images, while local binary patterns (LBPs) were used for feature extraction. The algorithm achieved a 95% success rate when tested on 46 plants while trained on vine leaves only (Pantazi et al., 2019). CNN was developed to detect and diagnose plant diseases using healthy and diseased plant images with deep learning models. 87,848 images of 25 different plants with 58 distinct classes from the open database were used for training. Training several architectures with plant disease combinations achieved a high success rate. VGG-CNN achieved 99.53% accuracy. Due to the 99.53% success rate with very deep neural network VGG, the suggested model could be used as a warning and advisory tool to identify diseases (Ferentinos, 2018).

Dielectric spectroscopy was suggested to detect basal stem rot (BSR) in oil palm leaves using electric properties like dielectric constant, impedance, capacitance and dissipation factor. Four types of leaf samples (healthy, mild, moderate and severely infected) were collected, and a solid test fixture was used to classify oil palm leaves by BSR. RF, support vector machine-feature selection (SVM-FS) and GA were used to analyze the dataset with significant electrical properties. Significant frequencies 100 kHz–30 MHz with 300 spectral intervals were also selected. Artificial neural network (ANN) and SVM were used to determine the classification accuracy. The SVM-FS model showed 88.64% accuracy, while the ANN classifier produced results with 80% accuracy (Khaled et al., 2018).

CLCuV can be predicted and managed through bio-product and resistant germplasm. Every year, there is significantly reduced cotton yield due to CLCuV. Multiple regression analysis evaluated 15 varieties to predict disease incidence abiotic environmental variables like humidity, minimum air temperature, maximum air temperature and rainfall. Two bio-products were used against the whitefly population to control the disease. No one variety was found out of the fifteen highly resistant against CLCuV (Saeed et al., 2018). A deep learning approach was developed for detecting vine diseases using unmanned aerial vehicle (UAV) images with vegetation and colorimetric indices. Early detection of symptoms prevents serious disease attacks in grape leaves, and disease management can be addressed better. Visible domain was used in UAV images. Color information and CNN were used to detect disease symptoms in vineyards. Different vegetation indices, color spaces, and combinations of both were used to compare the performance of CNN. The best results were achieved with 95.8% accuracy when CNN was combined with YUV color space, ExG, ExR and ExGR vegetation indices (Kerkech et al., 2018).

A deep convolutional neural network (DCNN) model was proposed to diagnose cucumber diseases using leaf symptoms like Powdery mildew, downy mildew, target leaf spots and anthracnose. The segmentation was performed on cucumber leaf images. The data augmentation method increased the dataset to minimize the chance of data overfitting. Fourteen thousand two hundred eight symptoms-based images were used in the dataset. AlexNet, SVM and RF classifiers were used to compare results obtained by DCNN. DCNN has proven to be the best tool for recognizing cucumber diseases, with an accuracy of 93.4% (Ma et al., 2018). A technique was developed for detecting citrus diseases based on symptoms by textural descriptors and color histograms. A color difference algorithm ΔE was used to classify the healthy and infected areas. For feature reduction, principal component analysis (PCA) was used, and state-of-the-art classifiers like K-Nearest Neighbor (KNN), SVM and Bagged Tree Ensemble (BTE) classifier were used. BTE outperformed with 99.9% accuracy results (Ali et al., 2017).

Bacteria Blight (BB) and CLCuV were detected among healthy and unhealthy leaves by RESNET50, a transfer learning algorithm, and KNN, a machine learning algorithm. RESNT50 obtained 95% accuracy, while KNN outperformed with 98% accuracy (Kotian et al., 2022). A framework was implemented to diagnose BB, Ball Rot (BR) and CLCuV using SVM, KNN, Naïve Bayes (NB) and Inception V4 based on CNN. Data was collected from Sindh, Pakistan. Inception V4-based CNN achieved an accuracy of 98.26% (Anwar et al., 2022). A deep learning model, MobileNet, Snapshot Ensemble (SE) and Vision Transformers (VT), was deployed to diagnose real-time CLCuV, BB and Fusarium Wilt (FW), three common diseases of cotton leaf. The suggested framework was designed to detect early stage diseases to overcome yield loss. It was observed that VT is less efficient, with an accuracy of 95.29%, MobileNet achieved 96.74% and the SE technique got 97.21% accuracy (Gudela et al., 2023).

A machine vision spectral statistical framework was suggested to classify different types of land cover data, such as Sutlej River land, fertile cultivated land, green pasture, desert range land and bare land. Multispectral radiometer-5 (MSR5), a handheld device with 5 spectral bands (red, green and blue [RGB], shortwave infrared and NIR), was used to acquire land cover data, while a digital camera was used to acquire texture data. Digital images were transformed into 229 texture features. Three statistical feature selection methods, MI, Fisher and (PA + ACC), were used to obtain the most discriminant features. Multispectral data was verified by linear discriminant analysis (LDA), while non-linear discriminant analysis (NDA) was applied to selected texture data. An ANN: n-class was used to classify multispectral and texture data. A cross-validation of 80:20 was implemented. 91.32% and 96.40% accuracy for texture and multispectral data were achieved, respectively (Qadri et al., 2016). Data fusion is applied to texture and multispectral data. MLP, NB, j48 and RF machine learning classifiers were used to achieve the best results. An accuracy of 99.60% is obtained using an MLP classifier on a fused dataset (Qadri et al., 2017).

Machine learning techniques were implemented to diagnose cotton crop diseases like BB, CLCuV, Whitefly infection (WI) and Jassid at early stages. Three rounds were used with different training and testing capacities. A TensorFlow classifier got 86.3%, 88.6% and 91% accuracy in the first, second and third rounds, respectively (Pechuho et al., 2020). A DJANGO framework was implemented, using VGG16 and VGG19, to detect multiple plant leaf diseases like Bacterial spot (BS), CLCuV, BB and Tomato mosaic virus (TMV). VGG16 got an accuracy of 86.32%, while VGG19 got an average of 93.61% accuracy (Gelli et al., 2020). A Rice-Fusion framework was proposed to diagnose rice diseases. A camera and agrometeorological sensor collected the dataset. The dataset contains 3200 healthy rice images. In the Rice-Fusion framework, numerical features were extracted from the agrometeorological dataset, and visual features were extracted from the image dataset. An accuracy of 95.31% was achieved by fusing numerical data with visual features as opposed to the unimodal framework, which had 91.25% and 82.03% accuracy based on MLP and CNN Architecture (Patil & Kumar, 2022).

Data fusion strategies have great potential to detect rice diseases like leaf blight, rice blast and rice sheath blight by using three spectroscopic techniques, including NIR HSI spectra, mid-infrared spectroscopy (MIR) and Laser-induced breakdown spectroscopy (LIBS). Rice diseases were classified by fusing different types of spectral features. Features were extracted by PCA and autoencoder algorithms. Raw data fusion, feature fusion and decision-level fusion were used. MLP and CNN models were used to identify rice diseases. CNN achieved an accuracy of 93%. HSI-based models performed better outcomes than MIR and LIBS (Feng et al., 2020).

It has been concluded that most authors identified leaf diseases using machine learning classifiers such as SVM to discriminate healthy and infected leaves with notable accuracy. Others applied multiple classifiers, such as RF, KNN, NB and MLP, to classify the healthy and infected leaves, using CLCuV, BB, FW and BR diseases as the infected leaves dataset. Some researchers also experimented with deep learning models like AlexNet, VGG, RESNET50 and Inception V4, classifying infected and healthy ones and showing suitable results. The survey also revealed that almost all the research was to discriminate between healthy and infected leaves. If some research studies used multiple diseases to represent the infected leaves, they only collect them as a single dataset of unhealthy leaves. It is also to be noted that few research studies used feature optimization, such as LDA, PCA and RDA. Finally, the survey showed that data fusion needed to be improved. The summary of the related work is given in Table 1.

Table 1. Summary table of related work.

References	Features and methodology	Classification type	Remarks
Pantazi et al. (2019)	Disease detection from various crop species, LBP was used for feature extraction, one class classifier	Black rot, powdery and downy mildew classification	One class classifier achieve 95% accuracy
Kerkech et al. (2018)	Vine leaf disease detection through UAV images, YUV color space, color information and CNN model	Healthy vs. un-healthy classification	CNN attain 95% accuracy
Ferentinos (2018)	Various plants disease detection, texture features, AlexNet and VGG model	Healthy vs. diseased plant classification	VGG Model gives 99.53% accuracy.
Jiang et al. 2020)	Rice leaf disease identification, texture features, deep learning and SVM model	Healthy vs. infected leaf classification	SVM gave maximum 96.8% accuracy
Gu et al. (2019)	TSW Virus disease detection, Spectral features, RS, SVM, CART and BRT model	Healthy vs. infected leaf classification	BRT obtained best accuracy of 85.2%
Khaled et al. (2018)	BSR disease detection, spectral features, SVM and ANN model	Healthy vs. diseased plant classification	SVM showed 88.64% accuracy
Ma et al. (2018)	Disease detection using leaf symptoms, Texture features, SVM, RF, AlexNet and DCNN model	Target leaf spots, anthracnose, downy and powdery mildew	DCNN achieved best accuracy of 93.4%
Ali et al. (2017)	Symptoms-based disease detection from citrus leaf, textual descriptors and color information, SVM, KNN and BTE model	Healthy vs. infected leaf classification	BTE outperformed with 99.9% accuracy
Kotian et al. (2022)	BB and CLCuV detection, Texture Feature, RESNET50 and KNN model	healthy vs. non healthy leaf classification	KNN showed better classification of 98%
Anwar et al. (2022)	BB, BR and CLCuV disease detection, Texture features, XVM, KNN, NB, and Inception V4 model	Healthy vs. diseased leaf classification	Inception V4 obtained 98.6% accuracy
Gudela et al. (2023)	CLCuV, BB and FW symptoms-based disease detection, texture features, MobileNet, VT and SE model	Healthy vs. infected leaf classification	SE got 97.21% accuracy
Qadri et al. (2016)	Land cover classification, Texture and Spectral dataset, RDA, LDA, PCA and ANN model	Various land cover classification	ANN gave accuracy of 96.40% for spectral dataset
Qadri et al. (2017)	Land cover classification, fused features, MLP, NB and RF model	Various land cover classification	MLP outperformed with 99.60% accuracy
Pechuho et al. (2020)	BB, CLCuV, WI and Jassid disease detection, Texture features, TensorFlow	Different cotton leaf diseases, healthy vs. infected detection	TensorFlow attain maximum of 91% accuracy
Gelli et al. (2020)	BS, CLCuV, BB and TMV, leaf disease detection, texture features, VGG16 and VGG19 classifiers on DJANGO framework	Healthy vs. infected disease detection	VGG19 gave 93.61% accuracy.
Patil and Kumar (2022)	Rice disease detection, fused features, MLP and CNN model	Healthy vs. diseased classification	CNN achieved an accuracy of 95.31%
Feng et al. (2020)	Disease detection from rice leaf, spectral data, MLP and CNN model	Healthy vs. infected leaf classification	CNN got 93% accuracy

This research work is instanced and based on three research contributions. First, Smruti, with colleague researchers (Kotian et al., 2022), extracted texture features from a digital photographic dataset to identify CLCuV. Second, Qing, with his fellow researchers (Gu et al., 2019), used the multispectral dataset to identify the curl virus, and finally, Rutuja, with colleague authors (Patil & Kumar, 2022) applied data fusion approach to identify the leaf disease. It is examined that most researchers performed bi-class classification, i.e. just discriminating the infected leaves from healthy ones. Rare work was found on CLCuV, though few efforts were noticed to diagnose CLCuV and a healthy one. No research work was conducted to discriminate the severity levels of CLCuV using multiple and fused datasets and hybrid feature selection in the selected regions. These indicated parameters clearly argue the novelty of this research study.

It is analyzed that cotton production is dominant for the economy of any cotton-yield country, but it is essential for developing countries like Pakistan. Cotton crops may be damaged substantially by many harmful diseases. Late disease identification is disastrous, and we need efficient and timely disease detection to set aside the loss. Human eye observation of diseases needs to be reassessed, whereas machine vision approaches have great potential to diagnose diseases accurately and efficiently. This leads to many practical benefits, like minimizing the loss of yield production and conquering economic damage. CLCuV is a destructive disease for cotton yields, and early and accurate detection of CLCuV is inevitable. This research study proposes a machine-based solution to the problem by developing a multimodal fusion framework to discriminate various CLCuV severity levels by incorporating three different CLCuV datasets, which include digital images, multispectral and fused datasets. The designed framework comprised data acquisition, image preprocessing, feature extraction, feature selection, classification and evaluation.

2. Material and methods

This research study designs an automated system for early detection of the CLCuV using image processing and RS technologies. Digital photographic data was captured for texture features, while radiometric/RS data was acquired for spectral features. The digital Conon camera model IXUS 185, which has 20.00 megapixels, was used for digital photographic data. A CROPSCAN device named MSR5 was used to capture radiometric data. Data was collected from Bahawalpur (29° 23′ 44′′ (North) latitude and 71° 41′ 1′′ (East) longitude) and Multan (30° 15′ 75′′ (North) latitude and 71° 52′ 49′′ (East) longitude) Division of Punjab, Pakistan as Punjab is the province producing more than 70% cotton of the country.

2.1. Proposed methodology

Four levels were used to accomplish this study, out of which one is healthy, and three are maximum, medium and minimum affected by CLCuV. No specific experimental setup was required for this study. A digital photographic dataset was collected from a cotton field using a digital camera. The multispectral dataset was acquired from the same cotton fields used to collect digital photographic datasets. The novel multimodal fusion framework to diagnose the CLCuV is shown in Figure 1.

Figure 1. Proposed multimodal fusion framework to diagnose CLCuV.

Image processing software Irfan’s view, CVIP Tools version 5.7e, built at Southern Illinois University Edwardsville, United States, MaZda version 4.6, created at the Institute of Electronics, Technical University of Lodz, Poland, and data mining tool WEKA version 3.8.3, developed at the University of Waikato, New Zealand, were used on Intel^® Core i3 2.4 GHz processor, 8 GB RAM with 64-bit Microsoft^® Windows version 8.1 Pro operating system to implement the proposed framework (Tariq et al., 2020) (Szczypiński et al. 2009) (Aher and Lobo 2011). The steps used to implement the proposed framework are discussed below.

2.2. Photographic data acquisition

Digital photographic images of different cotton varieties were captured by a digital Canon IXUS 185 model camera with 20.00 Megapixel with four severity levels (healthy, maximum effected, medium effected and minimum effected) of CLCuV, which are shown in Figure 2. A total of 600 digital photographic images with dimensions 5152 × 3864 and 24-bit depth of joint photographic expert group (jpg) format are captured for each severity level.

Figure 2. Healthy, maximum effected, medium effected and minimum effected leaf images.

All images were acquired in an open environment at 39 °C–44 °C in August and September 2020 from 12:00 PM to 3:00 PM in GMT + 5 Time Zone. There was no shadow at the image-capturing time. For each severity level, 100 digital photographic images were selected from 600 digital images by consulting with domain experts. Luxmeter Extech LT300 is used to measure sunlight intensity levels during data acquisition. Sunlight intensity with time is shown in Table 2.

Table 2. Time and sunlight intensity information.

Sr. no	Severity level	Time	Sunshine intensity
1	Healthy	1:00 PM	34,300
2	Maximum effected	1:50 PM	34,800
3	Medium effected	12:30 PM	34,600
4	Minimum effected	2:20 PM	34,900

2.3. Remote sensing data acquisition

RS data of cotton plants with four severity levels, healthy, maximum effected, medium effected, and minimum effected, was gathered by MSR5 with serial no. 566 is made by CROPSCAN, Inc. USA. MSR5 has five bands, three visible (blue, green and red). Two are invisible (NIR and shortwave infrared) bands, as shown in Table 3. Five hundred ninety-four MSR scans for each severity level were captured at a distance of 5 feet in August and September 2020 at 39 °C–44 °C from 12:00 PM to 3:00 PM in GMT + 5 time zone. Each MSR scan covers a half area 2.5 feet of its height.

Table 3. Multispectral feature table.

MSR types	Blue	Green	Red	Near infrared	Shortwave infrared
MSR5 (Generic)	450–520 nm	520–600 nm	630–690 nm	760–930 nm	1550–1750 nm
MSR5 (S. No. 566)	485 nm	560 nm	660 nm	830 nm	1650 nm

2.4. Image preprocessing

Six hundred colored digital photographic images for each category were acquired from twelve different cotton fields, each with an area of 43,560 square feet. The 100 best images were selected by consulting with experts and plant pathologists to obtain better and more accurate results. These selected images contain irrelevant areas, so captured images were cropped to various sizes from 3215 × 2776 to 5152 × 3826 pixels to get prominent leaves only. These cropped images were then resized to 64 × 64, 128 × 128, 256 × 256 and 512 × 512 pixels for each severity level and stored in bitmap (.bmp) image format. This study uses a digital photographic dataset of 400 × 4 = 1600 images to diagnose different severity levels of CLCuV. Each pixel requires 8 × 3 = 24 bits of memory to process each image in the RGB color model. So, each color image is transformed into grayscale to occupy only 8 bits (Yan et al., 2016). Grayscale images were enhanced at contrast level, and salient features like shadow and sharpness were restored. MaZda version 4.6, an image texture analysis tool, was used to obtain texture features. It is an efficient and reliable tool for image segmentation, texture feature computation, feature selection and feature extraction (Szczypiński et al., 2009).

The region of interest (ROI) of three-pixel dimensions for each image size was used to obtain better results. Digital photographic images were captured from 3 to 4 feet in height, depending upon the cotton plant size so that the whole cotton plant could be observed. Acquired images may contain many irrelevant areas, so large and equal pixel dimensions ROI cannot be used for all images. Therefore, various ROI sizes were used on an image dataset. ROI sizes 8 × 8, 10 × 10, and 12 × 12 were used for 64 × 64 image size. ROI sizes 12 × 12, 14 × 14, and 16 × 16 were used for 128 × 128 image size. ROI sizes 16 × 16, 18 × 18, and 20 × 20 were used for 256 × 256 size image. ROI sizes 20 × 20, 22 × 22, and 24 × 24 were used for 512 × 512 image size. The digital photographic dataset contains 1600 × 3 = 4800 images in this study. ROIs of 512 × 512 image size for various severity levels are shown in Figure 3.

Figure 3. ROIs of healthy, maximum effected, medium effected and minimum effected leaf.

2.4.1. Feature extraction

In feature extraction, raw data is altered into numerical values while original dataset information is preserved. Extracted features have the potential to produce better results than processing raw data using machine learning classifiers (Kumar & Bhatia, 2014). Five non-overlapping sub-images or ROIs from each image were drawn to obtain texture features. The digital photographic dataset used a total of 4800 × 5 = 24,000 ROIs or sub-images/segments. For each ROI, 9 first-order histogram features, 11 second-order co-occurrence features with 5 inter-pixel distances in four directions (0°, 45°, 90° and 135°) (11 × 5 × 4 = 220), 5 second-order run-length features in four directions (0°, 45°, 90° and 135°) (5 × 4 = 20) and 20 wavelet features, a total of 9 + 220 + 20 + 20 = 269 features were extracted. The total feature vector space is 24,000 × 269 = 6,456,000, while 538,000 features were used for the same ROI size in all severity levels. Three-pixel dimensions for each image size were used to obtain better results, and the dataset was also increased.

2.4.1.1. First-order histogram features

The first-order histogram or statistical feature calculates the intensity of individual pixels. Histogram features are popular due to their simplicity and compact performance. Data analysis based on histogram features provides the best solution for image processing problems to recognize and classify objects (Blachnik & Laaksonen, 2008). In this study, nine histogram features were calculated for each sub-image or ROI. Probability and various features of first-order histogram are described by Equations (1)–(5). (1) $${\mathrm{ }P}_{\left(\alpha \right)}=\frac{S_{\left(\alpha \right)}}{T}$$(1)

where $S_{\left(\alpha \right)}$ is a measure of pixels in gray level image and $T$ is number of pixels in an image or sub image (ROI). First-order histogram features like mean, variance, skewness, kurtosis, etc., are explained below.

Mean is average value of light and dark brightness in an image or sub-image (ROI) (2) $$\mathrm{\text{Mean}}=\sum _x\sum _y\frac{Z_{\left(c,d\right)}}{M}$$(2)

Variance is used to measure pixel value to the mean pixel value. (3) $$\text{Variance}=\frac{1}{S_{p\hspace{0.17em}}}{{\displaystyle \sum }}_{t=0}^{S_{p\hspace{0.17em}}}{\left(G_r-\hspace{0.17em}\overline{G}\right)}^2$$(3)

Skewness is the degree of asymmetry about the mean value. (4) $$\mathit{\text{Skewness}}=\frac{\frac{1}{S_{p\hspace{0.17em}}}{{\displaystyle \sum }}_{t=0}^{S_{p\hspace{0.17em}}}{\left(G_r-\hspace{0.17em}\overline{G}\right)}^3\hspace{0.17em}}{{\left(\frac{1}{S_{p\hspace{0.17em}}}{{\displaystyle \sum }}_{t=0}^{S_{p\hspace{0.17em}}}{\left(G_r-\hspace{0.17em}\overline{G}\right)}^2\right)}^3}$$(4)

Kurtosis is the measure of peakedness of distribution in an image or sub-image (ROI) (5) $$\mathit{\text{Kurtosis}}=\frac{\frac{1}{S_{p\hspace{0.17em}}}{{\displaystyle \sum }}_{t=0}^{S_{p\hspace{0.17em}}}{\left(G_r-\hspace{0.17em}\overline{G}\right)}^4\hspace{0.17em}}{{\left(\frac{1}{S_{p\hspace{0.17em}}}{{\displaystyle \sum }}_{t=0}^{S_{p\hspace{0.17em}}}{\left(G_r-\hspace{0.17em}\overline{G}\right)}^2\right)}^2}$$(5)

2.4.1.2. Second-order texture features

Second-order texture features calculate the co-occurrence and spatial inter-dependency of two neighboring pixels. Gray-level co-occurrence matrix (GLCM) and run length matrix (RLM) methods commonly calculate second-order texture features. Second-order texture features give more accuracy in the classification process than first-order histogram features (Feng et al., 2015). Eleven GLCM features up to the 5-pixel distance, and five RLM features in all four directions (0°, 45°, 90° and 135°) were calculated for each ROI.

GLCM features like contrast, correlation, entropy, entropy difference, sum entropy, etc., are described below and showed by Equations (6)–(15).

Contrast is the measure of variation in local intensity of pixels. (6) $$\mathit{\text{Contrast}}={\sum }_{x=1}^{Z_a}{\sum }_{y=1}^{Z_a}{\left(x-y\right)}^{2}S\left(x,y\right)$$(6)

Correlations are used to determine similarity among pixels. (7) $$\mathit{\text{Correlat}}=\frac{1}{S_{x\hspace{0.17em}}{S}_y}{\displaystyle \sum }_x{\displaystyle \sum }_y\left(L\hspace{0.17em}-\hspace{0.17em}{\omega }_x\right)\left(M-\hspace{0.17em}{\omega }_y\right)\left(x,y\right)$$(7)

Entropy is the degree of randomness and contained the average information of an image or sub-image (ROI). (8) $$\mathit{\text{Entrop}}=-\sum _x\sum _y\left(x,y\right){\mathrm{\text{log}}}_2\left(x,y\right)$$(8)

Difference entropy is used to compute neighborhood intensity value difference variability. (9) $$\mathit{\text{Diff}}\_\mathit{\text{Entropy}}={\sum }_{x=0}^{Z_a-1}{S}_{a-b}\left(x\right){\mathrm{\text{log}}}_2\left(S_{a-b}\right(x)+\in$$(9)

Sum entropy is used to measure sum of intensity value difference of neighbor pixels. (10) $$\mathit{\text{Sum}}\_\mathit{\text{Entropy}}={\sum }_{x=2}^{2Z_c}{S}_{a+b}\left(x\right){\mathrm{\text{log}}}_2\left(S_{a+b}\right(x)+\in$$(10)

RLM features are used to compute the length or range of run. Gonzales described length or range as a grayscale section or the same color level in a particular direction for a specified θ. RLM features like run length non-uniformity (RLNU), run length non-uniformity normalized (RLNN), long run emphasis (LRE), short run emphasis (SRE) and run percentile (RP)/fraction, etc., are described below.

RLNU measures similarity index among run length of an image, high homogeneity factor contains low RLNU value. (11) $$\mathit{\text{RLNU}}=\frac{{{\displaystyle \sum }}_{x=1}^{Z_a}{\left[{{\displaystyle \sum }}_{y=1}^{Z_b}\in \left(x,y\hspace{0.17em}\sigma \right)\right]}^2}{Z_b\left(\sigma \right)}\hspace{0.17em}$$(11)

RLNU is normalized to RLNN with substantial enhancement. RLNN also measures the similarity index among run length of an image, high homogeneity factor contains high RLNN value. (12) $$\mathit{\text{RLNN}}=\frac{{{\displaystyle \sum }}_{y=1}^{Z_a}{\left[{{\displaystyle \sum }}_{x=1}^{Z_b}\in \left(x,y\hspace{0.17em}\sigma \right)\right]}^2}{Z_b{\left(\sigma \right)}^2}\hspace{0.17em}$$(12)

LRE computes the distribution of long run length. High value of LRE indicates the coarser texture. (13) $$\mathit{\text{LRE}}=\frac{{{\displaystyle \sum }}_{y=1}^{Z_a}{\left[{{\displaystyle \sum }}_{x=0}^{Z_b}\in \left(x,y\hspace{0.17em}\sigma \right)\right]}^2}{Z_b\left(\sigma \right)}$$(13)

SRE computes the distribution of short run length. High value of SRE indicates finer texture. (14) $$\mathit{\text{SRE}}=\frac{{\sum }_{x=0}^{Z_a}{\sum }_{y=1}^{Z_b}\frac{\in (x,y\mathrm{ }|\sigma )}{x^2}}{Z_b\left(\sigma \right)}$$(14)

RP or fraction computes the coarseness of texture. (15) $$R\mathrm{ }P=\frac{Z_z}{Z_b}$$(15)

2.4.1.3. Wavelet features

A signal is decomposed into a spatial domain with finite energy called wavelet transform. The wavelet feature is widely used in texture classification to measure energy distribution (Huang & Aviyente, 2008). Many researchers use wavelet features to compress and denoise an image, pattern recognition and image analysis (Barbhuiya & Hemachandran, 2013). For each ROI, 20 wavelet features were calculated. In discrete wavelet transform 2 M × 2 M matrix is converted into same dimension matrix. Energy can be measured by Equation (16)(16) $$\mathit{\text{Energy}}={\mathrm{\phi }}^{-1}\sum _{a,bϵ\mathit{\text{ROI}}}{x}_{a,b}^2$$(16) . (16) $$\mathit{\text{Energy}}={\mathrm{\phi }}^{-1}\sum _{a,bϵ\mathit{\text{ROI}}}{x}_{a,b}^2$$(16)

2.4.2. Feature selection

A dataset with hundreds of thousands of features increases time and complexity to compute results. In the machine learning process, too many features cause a dimension disaster. The curse of dimensionality produces a bad impact on classification results. So, high-dimensional data, which may contain many irrelevant and redundant features that overfit in machine learning methods, is reduced to low-dimensional data to obtain efficient and prominent features using feature selection to overcome such a problem (Lutu & Engelbrecht, 2010). Feature selection is considered a critical process in machine learning applications. Feature selection methods are classified into filter, embedded and wrapped methods. Filter feature selection methods select features with a high correlation score to other outcome features. No dependency exists between selected features and machine learning classifiers. Filter feature selection methods are low-cost computational methods. Embedded feature selection methods select features during the execution of machine learning classifiers. Such methods are highly dependent on classifiers and have too much computational cost than other methods. Wrapper feature selection methods select features that are dependent on specific classifiers. A dependency exists between selected features and machine learning classifiers. The computational cost of such methods is much higher than filter feature selection methods (Jovic et al., 2015).

The digital photographic dataset has 269 × 400 × 5 = 538,000 extracted features for the same ROI size in all severity levels, which is a large-scale dataset and requires high computational power, time, and memory. All 269 extracted features of each ROI are not equally significant. The most significant features are extracted by fisher co-efficient (F), probability of error plus average correlation coefficient (POE + ACC = PA) and mutual information coefficient (MI) filter feature selection techniques to obtain more accurate results. MaZda combines three approaches (F + PA + MI) and gives the thirty most significant features for each ROI, ten features by each method, reducing the dataset to 30 × 400 × 5 = 60,000 features. Most significant thirty features selected by (F + PA + MI) are shown in Table 4.

Table 4. Feature selection (F + PA + MI) for ROIs (512 × 512 image size).

F		PA		MI
1	S(2,2)DifEntrp	11	WavEnLH_s-4	21	S(4,0)DifEntrp
2	S(4,4)DifEntrp	12	S(5,5)Correlat	22	S(3,0)DifEntrp
3	S(5,5)DifEntrp	13	WavEnHL_s-4	23	S(0,5)DifEntrp
4	S(3,3)DifEntrp	14	Vertl_LngREmph	24	S(5,−5)DifEntrp
5	Perc.99%	15	S(4,−4)Correlat	25	S(0–4)DifEntrp
6	S(2,2)InvDfMom	16	Kurtosis	26	S(4,−4)DifEntrp
7	S(0,3)DifEntrp	17	S(1,0)Correlat	27	S(0,1)SumEntrp
8	S(5,0)DifEntrp	18	WavEnHL_s-3	28	S(2,−2)DifEntrp
9	S(1,-1)SumEntrp	19	Skewness	29	S(3,3)InvDfMom
10	S(3,-3)DifEntrp	20	Mean	30	S(2,0)DifEntrp

Mathematically, F, PA and MI formulation and description are given below.

Fisher coefficient (F) calculates the ratio between inter-class and intra-class variation, is given by Equation (17)(17) $$\text{Fisher}=\frac{M^2}{N^2}=\frac{\left(\frac{1}{1‐\hspace{0.17em}{{\displaystyle \sum }}_{x=1}^X{O}_x^2}\right){{\displaystyle \sum }}_{x=1}^X{{\displaystyle \sum }}_{y=1}^X{O}_x\hspace{0.17em}{O}_y\hspace{0.17em}{\left(P_x‐\hspace{0.17em}{P}_y\right)}^2\hspace{0.17em}}{{{\displaystyle \sum }}_{x=1}^X{O}_x\hspace{0.17em}{N}_x^2}$$(17) . (17) $$\text{Fisher}=\frac{M^2}{N^2}=\frac{\left(\frac{1}{1‐\hspace{0.17em}{{\displaystyle \sum }}_{x=1}^X{O}_x^2}\right){{\displaystyle \sum }}_{x=1}^X{{\displaystyle \sum }}_{y=1}^X{O}_x\hspace{0.17em}{O}_y\hspace{0.17em}{\left(P_x‐\hspace{0.17em}{P}_y\right)}^2\hspace{0.17em}}{{{\displaystyle \sum }}_{x=1}^X{O}_x\hspace{0.17em}{N}_x^2}$$(17)

Probability of error plus average correlation coefficient (POE + ACC = PA) analyze ratio between misclassified samples and total samples in dataset, is described by Equations (18)–(21). (18) $$\mathit{\text{POE}}\left(X_m\right)=\frac{\mathit{\text{Mis}}\_\mathit{\text{classified}}\_\mathit{\text{samples}}}{\mathit{\text{Total samples}}}$$(18) (19) $$X_j=X_m:{\mathrm{\text{min}}}_m\mathrm{ }\left[\mathrm{ }\mathit{\text{POE}}\mathrm{ }\left(X_m\right)+\left|\mathit{\text{Corrlt}}\mathrm{ }\left(X_i,X_j\right)\right|\right]$$(19) (20) $$X_n=X_m:{\mathrm{\text{min}}}_m\mathrm{ }\left[\mathrm{ }\mathit{\text{POE}}\mathrm{ }\left(X_m\right)+\frac{1}{T-1}\left|\mathit{\text{Corrlt}}\mathrm{ }\left(X_i,X_m\right)\right|\right]$$(20) (21) $$S=X_l:{\mathrm{\text{min}}}_l\mathrm{ }\left[\mathrm{ }{U}_i\mathrm{ }\mathit{\text{POE}}\mathrm{ }\left(X_l\right)+U_j\mathrm{ }\mathit{\text{ACC}}\mathrm{ }\left(X_l\right)\right]$$(21)

MI compute dependency between two variables, explained in Equation (22)(22) $$\mathit{\text{MI}}\mathrm{ }\left(S_i,J\right)=\sum _{J=1}^{C_a}\sum _{n=1}^{D_b}P\mathrm{ }\left(S_i^J,X_n\right){\mathrm{\text{log}}}_2\left[\frac{P(S_i^J,X_n)}{P\mathrm{ }\left(S_i^J\right)P\left(X_n\right)}\right]\mathrm{ }$$(22) . (22) $$\mathit{\text{MI}}\mathrm{ }\left(S_i,J\right)=\sum _{J=1}^{C_a}\sum _{n=1}^{D_b}P\mathrm{ }\left(S_i^J,X_n\right){\mathrm{\text{log}}}_2\left[\frac{P(S_i^J,X_n)}{P\mathrm{ }\left(S_i^J\right)P\left(X_n\right)}\right]\mathrm{ }$$(22)

2.4.3. Classification

A supervised machine learning process in which input data is characterized into classes based on one or more variables is called classification (Lever et al., 2016). Waikato environment for knowledge analysis (WEKA) is used for data analysis and predictive modeling, a machine learning application developed by the University of Waikato, New Zealand. It accepts attribute relation file format (ARFF) and provides many classifiers (Aher & Lobo, 2011). Many ML classifiers were used to detect CLCuV, but SL, MLP, SMO and RF outperformed for digital photographic, spectral, and fused datasets. SL is the most common and fundamental ML classifier, which is easy to implement (Ray, 2019). MLP is a powerful multi-layered neural network, an effective classifier for complex and large datasets (Feng et al., 2020). RF is another widely used ML classifier in classification problems, which computes the final result by combining the results from multiple sub-decision trees. It is a flexible and easy-to-implement classifier (Ma et al., 2018). SMO is one of the compelling ML classifiers designed particularly for sparse datasets (Wang et al., 2018). It is based on the SVM model but also differs concerning the requirement of quadratic programming. The hyperparameters of deployed ML classifiers are shown in Table 5.

Table 5. Hyperparameters of ML classifiers.

ML classifier	Hyperparameters
SL	heuristicStop	maxBootingIterations	weightTrimBeta
MLP	hiddenLayers	learningRate	momentum
SMO	penalityFactor c	toleranceParameter	epsilon
RF	bagSizePercent	numIterations	numExecutionSlots

2.4.4. Performance evaluation

Various evaluation parameters are used to identify the performance of proposed model on given dataset. These performance evaluation parameters are explained by Equations (23)–(29) below.

Kappa statistics (KS) is a measure of agreement between two or multiple classes. (23) $$\mathit{\text{KS}}=\frac{S_{o\mathrm{ }}-S_{e\mathrm{ }}}{1-S_{e\mathrm{ }}}\mathrm{ }$$(23)

Mean absolute error (MAE), is an absolute difference, which is a positive value, between actual and predicted value measured by a model. (24) $$\mathit{\text{MAE}}=\frac{1}{S}\hspace{0.17em}\sum \hspace{0.17em}\left|\hspace{0.17em}G-\hspace{0.17em}\overline{G}\right|$$(24)

Root mean squared error (RMSE) measures the average difference between actual and predicted value and calculates the square root of that average. (25) $$\mathit{\text{RMSE}}=\sqrt{\frac{{{\displaystyle \sum }}_{t=1}^s{\left(y_t-\hspace{0.17em}\overline{y_t}\right)}^2}{S}}$$(25)

True positive (TP) rate or sensitivity, is a measure of perceived positive prediction with total positive prediction. (26) $$\mathit{\text{TP rate}}=\frac{\mathit{\text{TP}}}{\mathit{\text{TP}}+\mathit{\text{FN}}}$$(26)

False positive (FP) rate, is a ratio of negative predictions that are wrongly categorized as positive predictions. (27) $$\mathit{\text{FP rate}}=\frac{\mathit{\text{FP}}}{\mathit{\text{FP}}+\mathit{\text{TN}}}$$(27)

F-measure/F-score measures the accuracy of classification model on a given dataset. (28) $$F-\mathit{\text{Score}}=\frac{2\times \mathrm{\text{precision}}\times \mathrm{\text{recall}}}{\mathrm{\text{precision}}+\mathrm{\text{recall}}}$$(28)

Mathew’s correlation coefficient (MCC) is used to measure correlation co-efficient between predicted and actual value. (29) $$\mathit{\text{MCC}}=\frac{\mathit{\text{TN}}\times \mathit{\text{TP}}-\mathit{\text{FN}}\times \mathit{\text{FP}}}{\sqrt{\left(\mathit{\text{TP}}+\mathit{\text{FP}}\right)}(\mathit{\text{TP}}+\mathit{\text{FN}})(\mathit{\text{TN}}+\mathit{\text{FP}})(\mathit{\text{TN}}+\mathit{\text{FN}})}\mathrm{ }$$(29)

Receiver-operating characteristics (ROC), are used to measure the performance of each class in classification model.

Precision recall curve (PRC), is used to summarize the trade-off between TP rate and FP rate.

3. Results and discussion

Four supervised machine learning classifiers named SL, MLP, SMO and RF were used to detect various severity levels (healthy, maximum effected, medium effected and minimum effected) of CLCuV at digital photographic, spectral, and fused dataset with k-fold cross validation where k is set to 10. All employed classifiers compute the average result of all turns by splitting each dataset into ten folds. In each turn, one is used for testing, and the remaining nine are used for training. All deployed ML classifiers give unsatisfactory accuracy results for less than 512 × 512 images with various ROI sizes. As the image size increased to 512 × 512 with ROI size of 24 × 24, more promising results were achieved. Surprisingly, Results accuracy decreased as the image size increased to 1024 × 1024 with different ROI sizes. Performance evaluation parameters like KS, MAE, RMSE, TP rate, FP rate, F-measure/F-Score, MCC, ROC and PRC were calculated to achieve accuracy. All accuracy tables and confusion matrices (CM) were computed based on averages of 10-fold results.

In the first step, ML classifiers SL, MLP, SMO and RF were deployed on a digital photographic dataset. ML classifiers were implemented on 64 × 64, 128 × 128, and 256 × 256 image sizes with various ROIs feature datasets, but promising results were not found and have result accuracy less than 80%. The same ML classifiers were deployed on 512 × 512 image size with 24 × 24 ROI size feature dataset. It was observed that SL outperformed with an accuracy of 81.263%, while classification results accuracy for MLP, SMO and RF are 80.051%, 80.707% and 79.647%, respectively, which are shown in Table 6.

Table 6. ML classifiers detailed accuracy on 512 × 512 image size for digital photographic dataset.

Classifier	KS	MAE	RMAE	TP rate	FP rate	F-Score	MCC	ROC	PRC	Accuracy (%)
SL	0.750	0.143	0.264	0.813	0.062	0.813	0.750	0.953	0.885	81.263
MLP	0.734	0.108	0.288	0.801	0.066	0.800	0.734	0.942	0.863	80.051
SMO	0.743	0.271	0.343	0.807	0.064	0.807	0.743	0.917	0.738	80.707
RF	0.729	0.174	0.277	0.796	0.068	0.796	0.729	0.948	0.870	79.647

Figure 4 shows the comparative result accuracy of SL, MLP, SMO, and RF classifiers to diagnose CLCuV on a digital photographic dataset for 512 × 512 image size. Table 7 illustrates the CM prediction result summary of 512 × 512 image size with 24 × 24 ROI size for healthy, maximum effected, medium effected and minimum effected severity levels of CLCuV.

Figure 4. Accuracy results for CLCuV detection on digital photographic dataset.

Table 7. CM showing result of SL classifier for 512 × 512 image size of digital photographic dataset.

Class	Healthy	Maximum effected	Medium effected	Minimum effected	Total
Healthy	409	9	59	18	495
Maximum effected	2	411	42	40	495
Medium effected	68	20	383	24	495
Minimum effected	17	60	12	406	495

The same ML classifiers were deployed on a multispectral dataset in the next step. MLP classifier has outperformed on a multispectral dataset with 91.177% accuracy, while the classification accuracy results for SL, SMO, and RF are 88.202%, 82.023% and 88.483%, respectively, as shown in Table 8. Comparative result accuracy of SL, MLP, SMO, and RF classifiers for CLCuV is shown in Figure 5. The prediction result summary is shown in CM Table 9 for the MLP classifier for all severity levels of CLCuV.

Figure 5. Accuracy results for CLCuV detection on multispectral dataset.

Table 8. ML classifiers detailed accuracy on multispectral dataset.

Classifier	KS	MAE	RMAE	TP rate	FP rate	F-Score	MCC	ROC	PRC	Accuracy (%)
SL	0.843	0.112	0.219	0.882	0.037	0.880	0.846	0.968	0.915	88.202
MLP	0.882	0.087	0.201	0.912	0.027	0.911	0.884	0.974	0.934	91.177
SMO	0.760	0.271	0.344	0.820	0.059	0.809	0.765	0.905	0.751	82.023
RF	0.846	0.111	0.210	0.885	0.037	0.885	0.848	0.981	0.951	88.483

Table 9. CM showing results of multispectral dataset using MLP.

Class	Healthy	Maximum effected	Medium effected	Minimum effected	Total
Healthy	512	6	8	68	594
Maximum effected	6	582	1	5	594
Medium effected	1	3	588	2	594
Minimum effected	85	1	4	504	594

Finally, data fusion was performed by combining multispectral and digital photographic datasets to obtain more powerful results. Fusion-based dataset have 35 (5 multispectral and 30 digital photographic) features. SL, MLP, SMO, and RF ML classifiers also classified the fused dataset. SL ML Classifier was the best classifier with 96.313% result accuracy. MLP, SMO, and RF classifiers performed well with accuracy results of 96.061%, 93.586%, and 90.354%, respectively, which is better accuracy than individual datasets, as shown in Table 10. ML classifier’s detailed accuracy results for the fused dataset are shown in Figure 6, while the comparative analysis of all ML classifiers is presented in Figure 7. Table 11 shows the CM prediction analysis summary for a fused dataset. A detailed comparison between current state-of-the-art technologies and the proposed framework is shown in Table 12.

Figure 6. Accuracy results for CLCuV detection on fused dataset.

Figure 7. Comparative results graph for spectral, digital, and fused dataset.

Table 10. ML classifiers detailed accuracy for fused dataset.

Classifier	KS	MAE	RMAE	TP rate	FP rate	F-Score	MCC	ROC	PRC	Accuracy (%)
SL	0.951	0.037	0.126	0.963	0.012	0.963	0.951	0.995	0.988	96.313
MLP	0.948	0.023	0.129	0.961	0.013	0.961	0.948	0.996	0.989	96.061
SMO	0.915	0.257	0.321	0.936	0.021	0.936	0.915	0.973	0.906	93.586
RF	0.871	0.111	0.200	0.904	0.032	0.903	0.871	0.987	0.967	90.354

Table 11. CM showing results of SL classifier for fused dataset.

Class	Healthy	Maximum effected	Medium effected	Minimum effected	Total
Healthy	465	5	4	21	495
Maximum effected	3	482	3	7	495
Medium effected	2	1	492	0	495
Minimum effected	15	8	4	468	495

Table 12. Comparison between state-of-the-art current technologies and proposed multimodal fusion framework.

References	Features	Classifiers	Accuracy (%)
Pantazi et al. (2019)	Texture features	One class classifier	95
Kerkech et al. (2018)	Texture features	CNN	95.8
Gu et al. (2019)	Spectral features	RF, SVM, CART and BRT	85.2
Khaled et al. (2018)	Spectral features	SVM-FS	88.64
Ma et al. (2018)	Texture features	SV, RF andAlexNet	93.4
Pechuho et al. (2020)	Texture features	TensorFlow	91
Gelli et al. (2020)	Texture features	VGG16 and VGG19	93.61
Patil and Kumar (2022)	Texture + numerical features	MLP and CNN	91.25
Feng et al. (2020)	Spectral features	LR, SVM and CNN	93
Proposed methodology	Texture + spectral features	Simple logistic	96.313

It is perceived that most of the research studies applied machine learning models, such as SVM, RF, CART, and BRT to distinguish healthy and CLCuV-infected leaves, but their classification accuracies remained less than 90%. In some other research studies, deep learning techniques like CNN, VGG, TensorFlow, and MLP were used to identify CLCuV with healthy leaves, but the result accuracy of most of these models was below 95%. In comparison with other methodologies, discrimination of CLCuV severity levels, multiple and fused datasets and hybrid feature selection are the parameters that make this research study exclusive. Additionally, our designed framework achieved a high accuracy of 96.313%.

4. Conclusion

This research study has designed a novel multimodal fusion framework to diagnose different severity levels of CLCuV. For the research experiment, three distinct datasets have been conceived. These are the digital photographic dataset, multispectral dataset and their fused composition. After preprocessing, the feature extraction process extracts relevant features from the designated datasets. The hybrid feature optimization (F + PA + MI) gave the thirty most optimized features from the digital photographic dataset, whereas the five spectral features were extracted from the multispectral dataset. The fused dataset has been assembled by integrating the features extracted from the digital photographic and multispectral datasets. Each contrived dataset is input to the four designated machine vision classifiers, SL, MLP, SMO, and RF, to diagnose the CLCuV. SL is the best classifier for the digital photographic dataset, with a diagnosing accuracy of 81.263%. Next, the MLP classifier gained a considerably good accuracy of 91.177% on the multispectral dataset. Ultimately, the SL classifier again outperformed the fused dataset and achieved 96.313% diagnosing accuracy. Even though digital photographic and multispectral datasets have also given good results, it has been observed that the CLCuV diagnosing accuracy results were significantly improved for the fused dataset.

Future work

The proposed framework can also be used in decision-making, defect detection, yield assessment, water quality analysis, verification process, water stress in crops, seed quality classification and bioinformatics in the future.

Disclosure statement

The authors declare no conflitcs of interest.

Additional information

Notes on contributors

Nazir Ahmad

Mr. Nazir Ahmad is Assistant Professor and Ph.D research scholar in the Department of Information Technology, The Islamia University of Bahawalpur. He is an esteemed teacher, having good knowledge in pattern recognition, data mining, machine learning and artificial intelligence. His research focuses on machine learning application in agriculture and health care. Mr. Nazir Ahmad has worked as a co-author in many publications, and supervised almost 10 MPhil students. He also has supervised more than 100 undergraduate students. He is recognized for his commitment to education. He fosters student’s growth in technology innovations.