Deep neural network-based emotion recognition using facial landmark features and particle swarm optimization

Figures & data

Table 1. Prior work on Facial emotion recognition.

	Dataset
Year	Name	Samples	Features	Classification	Merits	Demerits
2014 [12]	MUG & CK+	458	70 Landmark Points	SVM	An individual set of images for training and testing	Pre-processing is not done
2015 [13]	Indian & JAFFE	2500	Sub Task	P-SIFT	Addressed all approaches	Less accuracy compared to other approaches
2018 [14]	MUG & CK + & RAFD	3276	Face	CNN	Addressed CNN cons	Not efficient for real-time applications
2019 [15]	JAFFE	213	Entire face	CNN	Increased system performance	Less no of samples
2019 [16]	CK+, MUG, OULU, RAFD	N/A	Facial Features	HI-Net	Visual saliency	NIL
2020 [17]	CK + & JAFFE and YALE FACE	N/A	Face	CNN	3 Different dataset	Low accuracy
2021 [18]	CK+	148	Face	CNN	NIL	Less sample size
2022 [19]	CK+, JAFFE, RAFD	239	Facial Landmark	MLP + DNN	Increased Accuracy	NIL
2023 [20]	CIFAR	72	Entire Face	DCNN	NIL	Sample size can be increased

Figure 1. MUG database basic expressions with one sample image.

Figure 2. GEMEP Corpus with four sample frames.

Figure 3. Proposed approach for emotion recognition.

Figure 4. Face Localization and Landmark Detection.

Figure 5. Manually annotated 66 dimensions Geometrical Face Points.

Table 2. Facial Landmark coordinates for feature extraction.

Facial Geometric Features
Distance				Angle
#ID	Points pair	#ID	Points pair	#ID	$\mathit{\theta }$
D1	18, 22	D25	34, 52	A1	18, 20, 22
D2	19, 21	D26	49, 55	A2	23, 25, 27
D3	18, 20	D27	52, 58	A3	38, 37, 42
D4	20, 22	D28	49, 52	A4	39, 40, 41
D5	23, 27	D29	52, 55	A5	44, 43, 48
D6	24, 26	D30	49, 58	A6	45, 46, 47
D7	23, 25	D31	58, 55	A7	32, 31, 36
D8	25, 27	D32	37, 49	A8	33, 31, 35
D9	20, 25	D33	46, 55	A9	40, 28, 43
D10	22, 23	D34	40, 28	A10	32, 30, 36
D11	22, 40	D35	43, 28	A11	51, 63, 53
D12	23, 43	D36	31, 28	A12	59, 67, 57
D13	37, 40	D37	32, 34	A13	50, 49, 60
D14	38, 42	D38	36, 34	A14	62, 61, 68
D15	39, 41	D39	62, 68	A15	54, 55, 56
D16	43, 46	D40	63, 67	A16	64, 65, 66
D17	44, 48	D41	64, 66	A17	33, 52, 35
D18	45, 47	D42	50, 60	A18	8, 58, 10
D19	40, 43	D43	54, 56	A19	22, 28, 23
D20	40, 32	D44	53, 57
D21	43, 36	D45	51, 59
D22	32, 36	D46	40, 58
D23	31, 32	D47	43, 58
D24	31, 36

Figure 6. Flowchart  for optimizing the DNN Hyperparameters using PSO.

Table

Algorithm: Novel Particle Swarm Optimization algorithm for optimal parameter selection
Input: MUG and GEMEP Dataset, 66-dimensional feature vector, label and Initial parameter range
Output: Hyper-parameters: Hl, Hln, On, Af, Ne, Lr, Bs, Do, Of, Oc#Define the accuracy of the DNN as a fitness function $\mathrm{Acc}=\left[\frac{\mathrm{tp}+\mathrm{tn}}{\mathrm{tp}+\mathrm{fp}+\mathrm{tn}+\mathrm{fn}}\right]$
#Define initial parameter values as random positions within the bound defined in the input range
1: Initialize the parameters, search space (X), and velocity (Vi).
2: Initialize the best positions Pbest for each particle.
3: while continuing until convergence is achieved up to the max iterations reached, then do
4:  for each particle do
5:   update the velocity resting in the population
6:   revise the particle position
7:    determine the fitness (Acc) of each individual using DNN
8:   if the current fitness is better than Pbest then
9:     modify the Pbest and fitness
10:   end if
11:      for each surrounding, do
12:       modify the gbest and fitness
13:     end for
14:  end for
15: end while
16: #Use the gbest position as Optimal Hyperparameters.

Figure 7. Feature Distribution of emotions in the MUG dataset.

Figure 8. Feature Distribution of emotions in the GEMEP dataset.

Figure 9. Deep Neural Networks Architecture.

Table 3. Hyperparameter setting for training the DNN using the PSO Algorithm.

Hyper-parameters	Range of values searched for Optimization	The optimal value obtained for MUG Dataset	The optimal value obtained from the GEMEP Dataset
Number of hidden layers (Hl)	[3–10]	3	3
Number of units per hidden layer (Hln)	[128–512]	128	256
Optimizer (On)	[Adam, Adamax, SGD]	Adamax	Adamax
Activation functions (Af)	[ReLu, Leaky ReLu, Sigmoid, Tanh]	ReLu	ReLu
Epochs number (Ne)	[50–200]	100	100
Learning Rate (Lr)	[0.0001–0.01]	0.001	0.001
Batch Size (Bs)	[4–128]	16	16
Drop out (Do)	[10–50%] in hidden layers	30%	10%
Output function (Of)	[Softmax, Sigmoid]	Sigmoid	Sigmoid

Figure 10. Confusion Matrix for the MUG Dataset with 66 features.

Table 4. Performance measure of seven basic emotions.

Emotions	Accuracy	Precision	Recall	F1-score
angry	99.05	99.05	99.05	99.74
disgust	99.38	99.07	99.22	99.79
fear	98.77	97.56	98.16	99.49
happy	99.46	98.4	98.92	99.66
sadness	97.53	99.28	98.40	99.62
neutral	98.62	99.44	99.03	99.70
surprise	98.40	98.66	98.53	99.53

Figure 11. GEMEP Dataset Confusion matrix outcomes.

Table 5. Performance measure of micro-coded emotions.

Emotions	Accuracy	Precision	Recall	F1-score
admiration	95.49	97.69	96.58	99.68
amusement	95.75	96.21	95.98	99.39
angry	96.89	97.91	97.40	99.64
anxiety	97.96	100	98.97	99.89
contempt	98.23	100	99.11	99.93
despair	89.6	97.84	93.54	99.11
disgust	98	100	98.99	99.93
interest	100	100	100	100
irritation	98.6	95.93	97.25	99.57
joy	99.44	96.72	98.06	99.75
panic fear	100	90.37	94.94	99.53
pleasure	97.93	100	98.95	99.79
pride	100	97.16	98.56	99.86
relief	98.31	94.57	96.4	99.53
sad	98.9	100	99.45	99.93
surprise	97.65	96.51	97.08	99.82
tenderness	92.93	91.09	92	99.43

Figure 12. (a) and (b) Model Performance for the MUG Dataset, (c) and (d) Model Performance for the GEMEP Dataset.

Figure 13. Emotion Recognition from real-time Dataset.

Table 6. Cutting-edge results achieved in the emotional datasets.

Reference	Method	Dataset	Accuracy in (%)
Proposed Work	Geometric Features + DNN	GEMEP	90.3
[6]	Dense optical flow + CNN		96.6
Proposed work	Geometric features using
	PSO + DNN		97.7
[16]	HiNet using visual saliency	MUG	87.8
[28]	Dense SIFT + SVM		91.8
[12]	Distance Manifolds using SVM		92.7
Proposed work	Geometric Features + DNN		93.1
	Geometric Features +
	PSO + DNN		98.7

Aifanti N, Delopoulos A. Linear subspaces for facial expression recognition. Signal Process Image Commun. 2014;29(1):177–188. doi:10.1016/j.image.2013.10.004

 Web of Science ®Google Scholar

Chickerur S, Reddy T, Shabalina O. Parallel scale invariant feature transform based approach for facial expression recognition. Commun. Comput. Inf. Sci. 2015;535:621–636.

 Google Scholar

Fathallah A, Abdi L, Douik A. Facial expression recognition via deep learning. In: Proc. IEEE/ACS Int. Conf. Comput. Syst. Appl. AICCSA, vol. 2017; Oct 2018. p. 745–750. doi:10.1109/AICCSA.2017.124

 Google Scholar

Taghi Zadeh MM, Imani M, Majidi B. Fast facial emotion recognition using convolutional neural networks and gabor filters. In: 2019 IEEE 5th Conf. Knowl. Based Eng. Innov. KBEI 2019; 2019. p. 577–581. doi:10.1109/KBEI.2019.8734943

 Google Scholar

Verma M, Vipparthi SK, Singh G. Hinet: hybrid inherited feature learning network for facial expression recognition. IEEE Lett. Comput. Soc. 2019;2(4):36–39. doi:10.1109/LOCS.2019.2927959

 Google Scholar

Ravi R, Yadhukrishna SV, Prithviraj R. A face expression recognition using CNN LBP. In: Proc. 4th Int. Conf. Comput. Methodol. Commun. ICCMC 2020; 2020. p. 684–689. doi:10.1109/ICCMC48092.2020.ICCMC-000127

 Google Scholar

Chowdary MK, Nguyen TN, Hemanth DJ. Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Comput Appl 2023;35(32):23311–23328. doi:10.1007/s00521-021-06012-8

 Web of Science ®Google Scholar

Siam AI, Soliman NF, Algarni AD, et al. Deploying machine learning techniques for human emotion detection. Comput Intell Neurosci. 2022;2022:8032673. doi:10.1155/2022/8032673

 PubMedGoogle Scholar

Talaat FM, Ali ZH, Mostafa RR, et al. Real-time facial emotion recognition model based on kernel autoencoder and convolutional neural network for autism children. Soft Comput 2024; 1–14. doi:10.1007/s00500-023-09477-y

 Web of Science ®Google Scholar

Sekar V, Jawaharlalnehru A. Semantic-based visual emotion recognition in videos-a transfer learning approach. Int J Electr Comput Eng. 2022;12(4):3674–3683. doi:10.11591/ijece.v12i4.pp3674-3683

 Google Scholar

Vaijayanthi S, Arunnehru J. Dense SIFT-based facial expression recognition using machine learning techniques. In: Proceedings of the 6th International Conference on Advance Computing and Intelligent Engineering; 2023. p. 301–310.

 Google Scholar