A multimodal fusion framework to diagnose cotton leaf curl virus using machine vision techniques

Figures & data

Table 1. Summary table of related work.

References	Features and methodology	Classification type	Remarks
Pantazi et al. (2019)	Disease detection from various crop species, LBP was used for feature extraction, one class classifier	Black rot, powdery and downy mildew classification	One class classifier achieve 95% accuracy
Kerkech et al. (2018)	Vine leaf disease detection through UAV images, YUV color space, color information and CNN model	Healthy vs. un-healthy classification	CNN attain 95% accuracy
Ferentinos (2018)	Various plants disease detection, texture features, AlexNet and VGG model	Healthy vs. diseased plant classification	VGG Model gives 99.53% accuracy.
Jiang et al. 2020)	Rice leaf disease identification, texture features, deep learning and SVM model	Healthy vs. infected leaf classification	SVM gave maximum 96.8% accuracy
Gu et al. (2019)	TSW Virus disease detection, Spectral features, RS, SVM, CART and BRT model	Healthy vs. infected leaf classification	BRT obtained best accuracy of 85.2%
Khaled et al. (2018)	BSR disease detection, spectral features, SVM and ANN model	Healthy vs. diseased plant classification	SVM showed 88.64% accuracy
Ma et al. (2018)	Disease detection using leaf symptoms, Texture features, SVM, RF, AlexNet and DCNN model	Target leaf spots, anthracnose, downy and powdery mildew	DCNN achieved best accuracy of 93.4%
Ali et al. (2017)	Symptoms-based disease detection from citrus leaf, textual descriptors and color information, SVM, KNN and BTE model	Healthy vs. infected leaf classification	BTE outperformed with 99.9% accuracy
Kotian et al. (2022)	BB and CLCuV detection, Texture Feature, RESNET50 and KNN model	healthy vs. non healthy leaf classification	KNN showed better classification of 98%
Anwar et al. (2022)	BB, BR and CLCuV disease detection, Texture features, XVM, KNN, NB, and Inception V4 model	Healthy vs. diseased leaf classification	Inception V4 obtained 98.6% accuracy
Gudela et al. (2023)	CLCuV, BB and FW symptoms-based disease detection, texture features, MobileNet, VT and SE model	Healthy vs. infected leaf classification	SE got 97.21% accuracy
Qadri et al. (2016)	Land cover classification, Texture and Spectral dataset, RDA, LDA, PCA and ANN model	Various land cover classification	ANN gave accuracy of 96.40% for spectral dataset
Qadri et al. (2017)	Land cover classification, fused features, MLP, NB and RF model	Various land cover classification	MLP outperformed with 99.60% accuracy
Pechuho et al. (2020)	BB, CLCuV, WI and Jassid disease detection, Texture features, TensorFlow	Different cotton leaf diseases, healthy vs. infected detection	TensorFlow attain maximum of 91% accuracy
Gelli et al. (2020)	BS, CLCuV, BB and TMV, leaf disease detection, texture features, VGG16 and VGG19 classifiers on DJANGO framework	Healthy vs. infected disease detection	VGG19 gave 93.61% accuracy.
Patil and Kumar (2022)	Rice disease detection, fused features, MLP and CNN model	Healthy vs. diseased classification	CNN achieved an accuracy of 95.31%
Feng et al. (2020)	Disease detection from rice leaf, spectral data, MLP and CNN model	Healthy vs. infected leaf classification	CNN got 93% accuracy

Figure 1. Proposed multimodal fusion framework to diagnose CLCuV.

Figure 2. Healthy, maximum effected, medium effected and minimum effected leaf images.

Table 2. Time and sunlight intensity information.

Sr. no	Severity level	Time	Sunshine intensity
1	Healthy	1:00 PM	34,300
2	Maximum effected	1:50 PM	34,800
3	Medium effected	12:30 PM	34,600
4	Minimum effected	2:20 PM	34,900

Table 3. Multispectral feature table.

MSR types	Blue	Green	Red	Near infrared	Shortwave infrared
MSR5 (Generic)	450–520 nm	520–600 nm	630–690 nm	760–930 nm	1550–1750 nm
MSR5 (S. No. 566)	485 nm	560 nm	660 nm	830 nm	1650 nm

Figure 3. ROIs of healthy, maximum effected, medium effected and minimum effected leaf.

Table 4. Feature selection (F + PA + MI) for ROIs (512 × 512 image size).

F		PA		MI
1	S(2,2)DifEntrp	11	WavEnLH_s-4	21	S(4,0)DifEntrp
2	S(4,4)DifEntrp	12	S(5,5)Correlat	22	S(3,0)DifEntrp
3	S(5,5)DifEntrp	13	WavEnHL_s-4	23	S(0,5)DifEntrp
4	S(3,3)DifEntrp	14	Vertl_LngREmph	24	S(5,−5)DifEntrp
5	Perc.99%	15	S(4,−4)Correlat	25	S(0–4)DifEntrp
6	S(2,2)InvDfMom	16	Kurtosis	26	S(4,−4)DifEntrp
7	S(0,3)DifEntrp	17	S(1,0)Correlat	27	S(0,1)SumEntrp
8	S(5,0)DifEntrp	18	WavEnHL_s-3	28	S(2,−2)DifEntrp
9	S(1,-1)SumEntrp	19	Skewness	29	S(3,3)InvDfMom
10	S(3,-3)DifEntrp	20	Mean	30	S(2,0)DifEntrp

Table 5. Hyperparameters of ML classifiers.

ML classifier	Hyperparameters
SL	heuristicStop	maxBootingIterations	weightTrimBeta
MLP	hiddenLayers	learningRate	momentum
SMO	penalityFactor c	toleranceParameter	epsilon
RF	bagSizePercent	numIterations	numExecutionSlots

Table 6. ML classifiers detailed accuracy on 512 × 512 image size for digital photographic dataset.

Classifier	KS	MAE	RMAE	TP rate	FP rate	F-Score	MCC	ROC	PRC	Accuracy (%)
SL	0.750	0.143	0.264	0.813	0.062	0.813	0.750	0.953	0.885	81.263
MLP	0.734	0.108	0.288	0.801	0.066	0.800	0.734	0.942	0.863	80.051
SMO	0.743	0.271	0.343	0.807	0.064	0.807	0.743	0.917	0.738	80.707
RF	0.729	0.174	0.277	0.796	0.068	0.796	0.729	0.948	0.870	79.647

Figure 4. Accuracy results for CLCuV detection on digital photographic dataset.

Table 7. CM showing result of SL classifier for 512 × 512 image size of digital photographic dataset.

Class	Healthy	Maximum effected	Medium effected	Minimum effected	Total
Healthy	409	9	59	18	495
Maximum effected	2	411	42	40	495
Medium effected	68	20	383	24	495
Minimum effected	17	60	12	406	495

Figure 5. Accuracy results for CLCuV detection on multispectral dataset.

Table 8. ML classifiers detailed accuracy on multispectral dataset.

Classifier	KS	MAE	RMAE	TP rate	FP rate	F-Score	MCC	ROC	PRC	Accuracy (%)
SL	0.843	0.112	0.219	0.882	0.037	0.880	0.846	0.968	0.915	88.202
MLP	0.882	0.087	0.201	0.912	0.027	0.911	0.884	0.974	0.934	91.177
SMO	0.760	0.271	0.344	0.820	0.059	0.809	0.765	0.905	0.751	82.023
RF	0.846	0.111	0.210	0.885	0.037	0.885	0.848	0.981	0.951	88.483

Table 9. CM showing results of multispectral dataset using MLP.

Class	Healthy	Maximum effected	Medium effected	Minimum effected	Total
Healthy	512	6	8	68	594
Maximum effected	6	582	1	5	594
Medium effected	1	3	588	2	594
Minimum effected	85	1	4	504	594

Figure 6. Accuracy results for CLCuV detection on fused dataset.

Figure 7. Comparative results graph for spectral, digital, and fused dataset.

Table 10. ML classifiers detailed accuracy for fused dataset.

Classifier	KS	MAE	RMAE	TP rate	FP rate	F-Score	MCC	ROC	PRC	Accuracy (%)
SL	0.951	0.037	0.126	0.963	0.012	0.963	0.951	0.995	0.988	96.313
MLP	0.948	0.023	0.129	0.961	0.013	0.961	0.948	0.996	0.989	96.061
SMO	0.915	0.257	0.321	0.936	0.021	0.936	0.915	0.973	0.906	93.586
RF	0.871	0.111	0.200	0.904	0.032	0.903	0.871	0.987	0.967	90.354

Table 11. CM showing results of SL classifier for fused dataset.

Class	Healthy	Maximum effected	Medium effected	Minimum effected	Total
Healthy	465	5	4	21	495
Maximum effected	3	482	3	7	495
Medium effected	2	1	492	0	495
Minimum effected	15	8	4	468	495

Table 12. Comparison between state-of-the-art current technologies and proposed multimodal fusion framework.

References	Features	Classifiers	Accuracy (%)
Pantazi et al. (2019)	Texture features	One class classifier	95
Kerkech et al. (2018)	Texture features	CNN	95.8
Gu et al. (2019)	Spectral features	RF, SVM, CART and BRT	85.2
Khaled et al. (2018)	Spectral features	SVM-FS	88.64
Ma et al. (2018)	Texture features	SV, RF andAlexNet	93.4
Pechuho et al. (2020)	Texture features	TensorFlow	91
Gelli et al. (2020)	Texture features	VGG16 and VGG19	93.61
Patil and Kumar (2022)	Texture + numerical features	MLP and CNN	91.25
Feng et al. (2020)	Spectral features	LR, SVM and CNN	93
Proposed methodology	Texture + spectral features	Simple logistic	96.313

Pantazi, X. E., Moshou, D., & Tamouridou, A. A. (2019). Automated leaf disease detection in different crop species through image features analysis and One Class Classifiers. Computers and Electronics in Agriculture, 156, 96–104. https://doi.org/10.1016/j.compag.2018.11.005

 Web of Science ®Google Scholar

Kerkech, M., Hafiane, A., & Canals, R. (2018). Deep learning approach with colorimetric spaces and vegetation indices for vine diseases detection in UAV images. Computers and Electronics in Agriculture, 155, 237–243. https://doi.org/10.1016/j.compag.2018.10.006

 Web of Science ®Google Scholar

Ferentinos, K. P. (2018). Deep learning models for plant disease detection and diagnosis. Computers and Electronics in Agriculture, 145, 311–318. https://doi.org/10.1016/j.compag.2018.01.009

 Web of Science ®Google Scholar

Jiang, F., Lu, Y., Chen, Y., Cai, D., & Li, G. (2020). Image recognition of four rice leaf diseases based on deep learning and support vector machine. Computers and Electronics in Agriculture, 179, 105824. https://doi.org/10.1016/j.compag.2020.105824

 Web of Science ®Google Scholar

Gu, Q., Sheng, L., Zhang, T., Lu, Y., Zhang, Z., Zheng, K., Hu, H., & Zhou, H. (2019). Early detection of tomato spotted wilt virus infection in tobacco using the hyperspectral imaging technique and machine learning algorithms. Computers and Electronics in Agriculture, 167, 105066. https://doi.org/10.1016/j.compag.2019.105066

 Web of Science ®Google Scholar

Khaled, A. Y., Abd Aziz, S., Bejo, S. K., Nawi, N. M., & Abu Seman, I. (2018). Spectral features selection and classification of oil palm leaves infected by Basal stem rot (BSR) disease using dielectric spectroscopy. Computers and Electronics in Agriculture, 144, 297–309. https://doi.org/10.1016/j.compag.2017.11.012

 Web of Science ®Google Scholar

Ma, J., Du, K., Zhen, F., Zhang, L., Gong, Z., & Sun, Z. (2018). A recognition method for cucumber diseases using leaf symptom images base on deep convolutional nerual network. Computers and Electronics in Agriculture, 154, 18–24. https://doi.org/10.1016/j.compag.2018.08.048

 Web of Science ®Google Scholar

Ali, H., Lali, M. I., Nawaz, M. Z., Sharif, M., & Saleem, B. A. (2017). Symptom based automated detection of citrus diseases using color histogram and textural descriptors. Computers and Electronics in Agriculture, 138, 92–104. https://doi.org/10.1016/j.compag.2017.04.008

 Web of Science ®Google Scholar

Kotian, S., Ettan, P., Kharche, S., Saravanan, K., & Kumar, K. A. (2022). Cotton leaf disease detection using machine learning. 2nd International Conference on Advancement in Electronics & Communication Engineering, Ghaziabad, India. https://doi.org/10.2139/ssrn.4159108

 Google Scholar

Anwar, S., Kolachi, A. R., Baloch, S. K., & Soomro, S. R. (2022). Bacterial blight and cotton leaf curl virus detection using inception V4 based CNN model for cotton crops . IEEE 5th International Conference on Image Processing Applications and Systems (IPAS). Genova, Italy. https://doi.org/10.1109/ipas55744.2022.10052835

 Google Scholar

Gudela, Y. S., Jangam, E., & Mogili, S. (2023). Cotton crop disease detection using mobilenet snapshot ensemble technique and vision transformers. 8th International Conference on Communication and Electronics Systems (ICCES). Coimbatore, India. https://doi.org/10.1109/icces57224.2023.10192615

 Google Scholar

Qadri, S., Khan, D. M., Ahmad, F., Qadri, S. F., Babar, M. E., Shahid, M., Ul-Rehman, M., Razzaq, A., Shah Muhammad, S., Fahad, M., Ahmad, S., Pervez, M. T., Naveed, N., Aslam, N., Jamil, M., Rehmani, E. A., Ahmad, N., & Akhtar Khan, N. (2016). A comparative study of land cover classification by using multispectral and texture data. BioMed Research International, 2016, 8797438–12. https://doi.org/10.1155/2016/8797438

 PubMed Web of Science ®Google Scholar

Qadri, S., Khan, D. M., Qadri, S. F., Razzaq, A., Ahmad, N., Jamil, M., Nawaz Shah, A., Shah Muhammad, S., Saleem, K., & Awan, S. A. (2017). Multisource data fusion framework for land use/land cover classification using machine vision. Journal of Sensors, 2017, 1–8. https://doi.org/10.1155/2017/3515418

 Web of Science ®Google Scholar

Pechuho, N., Khan, Q., & Kalwar, S. (2020). Cotton crop disease detection using machine learning via tensor flow. Pakistan Journal of Engineering and Technology, 3(2), 126–130. https://doi.org/10.51846/vol3iss2pp126-130

 Google Scholar

Gelli, J. S., Madduri, L. A., Tanveer, R., Bhanu, U., & Kishore, G. K. (2020). Plant disease detection using VGG and DJANGO. International Research Journal of Engineering and Technology, (IRJET ), 7(8), 4369–4373. https://irjet.net/archives/V7/i8/IRJET-V7I8753.pdf

 Google Scholar

Patil, R. R., & Kumar, S. (2022). Rice-fusion: A multimodality data fusion framework for rice disease diagnosis. IEEE Access, 10, 5207–5222. https://doi.org/10.1109/ACCESS.2022.3140815

 Web of Science ®Google Scholar

Feng, L., Wu, B., Zhu, S., Wang, J., Su, Z., Liu, F., He, Y., & Zhang, C. (2020). Investigation on data fusion of multisource spectral data for rice leaf diseases identification using machine learning Methods. Frontiers in Plant Science, 11, 577063. https://doi.org/10.3389/fpls.2020.577063

 PubMed Web of Science ®Google Scholar