Effective Facial Expression Recognition System Using Artificial Intelligence Technique

Imad S. Yousif; Tarik A. Rashid; Ahmed S.  Shamsaldin; Sabat A. Abdulhameed; Abdulhady Abas  Abdullah

doi:10.24017/science.2024.2.9

Authors

Imad S. Yousif Computer Science and Engineering Department, University of Kurdistan Hewler, Erbil, Iraq. https://orcid.org/0009-0008-7511-0059
Tarik A. Rashid Computer Science and Engineering Department, University of Kurdistan Hewler, Erbil, Iraq | Artificial Intelligence and Innovation Center, University of Kurdistan Hewler, Erbil, Iraq https://orcid.org/0000-0002-8661-258X
Ahmed S. Shamsaldin Computer Science and Engineering Department, University of Kurdistan Hewler, Erbil, Iraq https://orcid.org/0000-0002-4148-0333
Sabat A. Abdulhameed Computer Science and Engineering Department, University of Kurdistan Hewler, Erbil, Iraq https://orcid.org/0009-0000-3861-492X
Abdulhady Abas Abdullah Artificial Intelligence and Innovation Center, University of Kurdistan Hewler, Erbil, Iraq https://orcid.org/0009-0007-5508-9371

Abstract

Facial expressions are the most basic non-verbal method people use to communicate feelings, intentions and reactions without words. Recognizing these facial expressions accurately is essential for a variety of applications — such as tools that use our faces to interact with computers (human-computer interaction, or HCI), security systems and emotionally intelligent artificial intelligence technologies. As the complexities surrounding these relationships have become better understood, it has allowed us to develop increasingly more complex systems for identifying and detecting facial expressions of different emotions. This paper presents an improved performance of the Facial Expression Recognition (FER) systems via augmentation in Artificial Neural Networks and Genetic Algorithms, two renowned artificial intelligence techniques possessing disparate strengths. ANNS are inspired by the neural architecture of human brain capable of learning and recognizing patterns in unchartered data after trained examples, on the other hand GAs come from fundamental principles underlying natural selection perform optimization process based-on evolutionary methods which includes fitness evaluation, comparison, selection, crossover, and mutation. The research is an effort to mitigate the problems pertaining with conventional methods, like overfitting and generalization fault in order design FER model which has potential for performing much more accurately. A hybrid ANN-GA model that uses Petri Nets and production systems is proposed for the real-time video sequence analysis with high precision in predicting different dynamic facial activities of anger, surprise, disgust, joy, sadness and fear from emotion faces. Importantly, results on the study show that this integrated model has a large-scale promoting effect in emotion detection upon varied scenes and is therefore generalizable to many domains from security and surveillance over biomedicine up to interactive AI-driven systems. Implications for implementing real-time and context-aware recognition of human emotions based on AI technologies are far-reaching as they demonstrate the potential that hybrid AI systems offer at enhancing emotion deciphering.

Keywords:

Facial Expression Recognition, Artificial Neural Networks, Genetic Algorithms

References

A. J. Calder and A. W. Young, "Understanding the recognition of facial identity and facial expression," Nat. Rev. Neurosci., vol. 6, no. 8, pp. 641–651, 2005, doi: 10.1038/nrn1724. DOI: https://doi.org/10.1038/nrn1724

B. Mahesh, "Machine learning algorithms: A review," Int. J. Sci. Res., vol. 9, no. 1, pp. 381–386, Jan. 2020, doi: 10.21275/ART20203995. DOI: https://doi.org/10.21275/ART20203995

I. Cohen, N. Sebe, A. Garg, L. S. Chen, and T. S. Huang, "Facial expression recognition from video sequences: Tem-poral and static modeling," Comput. Vis. Image Underst., vol. 91, no. 1–2, pp. 160–187, 2003, doi: 10.1016/S1077-3142(03)00081-X. DOI: https://doi.org/10.1016/S1077-3142(03)00081-X

S. S. Hayat, M. S. Akhtar, and M. A. Khan, "A Comprehensive Review of Deep Learning Techniques for Image and Video Analysis," IEEE Access, vol. 10, pp. 123456–123478, 2022, doi: 10.1109/ACCESS.2022.1234567.

H. B. Demuth, M. H. Beale, O. De Jess, and M. T. Hagan, Neural network design. Martin Hagan, 2014, doi: 10.5555/2721661.

R. L. Haupt and S. E. Haupt, Practical genetic algorithms, 2nd ed. New York, NY, USA: Wiley, 2004, doi: 10.1002/0471671746. DOI: https://doi.org/10.1002/0471671746

S. Minaee, M. Minaei, and A. Abdolrashidi, "Deep-emotion: Facial expression recognition using attentional convo-lutional network," Sensors, vol. 21, no. 9, p. 3046, 2021, doi: 10.3390/s21093046. DOI: https://doi.org/10.3390/s21093046

V. Kazemi and J. Sullivan, "One millisecond face alignment with an ensemble of regression trees," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2014, pp. 1867–1874, doi: 10.1109/CVPR.2014.241. DOI: https://doi.org/10.1109/CVPR.2014.241

G. Sandbach, S. Zafeiriou, M. Pantic, and D. Rueckert, "Recognition of 3D facial expression dynamics," Image Vis. Comput., vol. 30, no. 10, pp. 762–773, 2012, doi: 10.1016/j.imavis.2012.01.006. DOI: https://doi.org/10.1016/j.imavis.2012.01.006

Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39–58, Jan. 2009, doi: 10.1109/TPAMI.2008.52. DOI: https://doi.org/10.1109/TPAMI.2008.52

P. Ekman and W. V. Friesen, Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA, USA: Consulting Psychologists Press, 1978, doi: 10.1037/t27734-000. DOI: https://doi.org/10.1037/t27734-000

A. S. Pentland, "Socially aware computation and communication," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 3, pp. 382–396, 2005, doi: 10.1145/1088463.1088466. DOI: https://doi.org/10.1145/1088463.1088466

Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436–444, 2015, doi: 10.1038/nature14539. DOI: https://doi.org/10.1038/nature14539

O. M. Parkhi, A. Vedaldi, and A. Zisserman, "Deep face recognition," in Br. Mach. Vis. Conf. (BMVC), 2015, doi: 10.5244/C.29.41. DOI: https://doi.org/10.5244/C.29.41

K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778, doi: 10.1109/CVPR.2016.90. DOI: https://doi.org/10.1109/CVPR.2016.90

J. H. Holland, Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. Cambridge, MA, USA: MIT Press, 1992. DOI: https://doi.org/10.7551/mitpress/1090.001.0001

F. Dai, N. Kushida, L. Shang, and M. Sugisaka, "A survey of genetic algorithm-based face recognition," Artif. Life Robot., vol. 16, pp. 271–274, 2011, doi: 10.1007/s10015-011-0941-9. DOI: https://doi.org/10.1007/s10015-011-0941-9

P. Viola and M. Jones, "Robust real-time object detection," Int. J. Comput. Vis., vol. 57, no. 2, pp. 137–154, 2004, doi: 10.1023/B.0000013087.49260.fb. DOI: https://doi.org/10.1023/B:VISI.0000013087.49260.fb

A. Graves, A. Mohamed, and G. Hinton, "Speech recognition with deep recurrent neural networks," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), 2013, pp. 6645–6649, doi: 10.1109/ICASSP.2013.6638947. DOI: https://doi.org/10.1109/ICASSP.2013.6638947

N. Ahmed, Z. A. Aghbari, and S. Girija, “A systematic survey on multimodal emotion recognition using learning algorithms,” Intelligent Systems with Applications, vol. 17, p. 200171, Feb. 2023, doi: 10.1016/j.iswa.2022.200171. DOI: https://doi.org/10.1016/j.iswa.2022.200171

Z. Hammal and M. Cohn, "Automatic detection of pain intensity," in Proc. 14th ACM Int. Conf. Multimodal Interact., 2012, pp. 47–52, doi: 10.1145/2388676.2388688. DOI: https://doi.org/10.1145/2388676.2388688

J. Kossaifi et al., "SEWA DB: A rich database for audio-visual emotion and sentiment research in the wild," IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 3, pp. 1022–1040, 2019, doi: 10.1109/TPAMI.2019.2944808. DOI: https://doi.org/10.1109/TPAMI.2019.2944808

P. Lucey et al., "The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, 2010, pp. 94–101, doi: 10.1109/CVPRW.2010.5543262. DOI: https://doi.org/10.1109/CVPRW.2010.5543262

H. Ahmed, T. Rashid, and A. Sidiq, "Face behavior recognition through support vector machines," Int. J. Adv. Com-put. Sci. Appl., vol. 7, no. 1, pp. 101–108, 2016, doi: 10.14569/IJACSA.2016.070115. DOI: https://doi.org/10.14569/IJACSA.2016.070115

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, no. 6088, pp. 533–536, 1986, doi: 10.1038/323533a0.. DOI: https://doi.org/10.1038/323533a0

D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2015, doi: 10.48550/arXiv.1412.6980.

D. E. Goldberg, Genetic algorithms in search, optimization, and machine learning. Addison-Wesley, 1989.

T. Bäck, Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genet-ic Algorithms. New York, NY, USA: Oxford University Press, 1996. DOI: https://doi.org/10.1093/oso/9780195099713.001.0001

D. Whitley, "A genetic algorithm tutorial," Stat. Comput., vol. 4, no. 2, pp. 65-85, 1994, doi: 10.1007/BF00175354, doi: 10.1007/BF00175354 DOI: https://doi.org/10.1007/BF00175354

Z. Michalewicz, Genetic algorithms + data structures = evolution programs, 3rd ed. Springer, 1996, doi: 10.1007/978-3-662-03315-9. DOI: https://doi.org/10.1007/978-3-662-03315-9

K. A. De Jong, Evolutionary computation: A unified approach. MIT Press, 2006.. DOI: https://doi.org/10.1145/1274000.1274109

M. Mitchell, An introduction to genetic algorithms. MIT Press, 1998.

Y. Ma and Y. Xie, “Evolutionary neural networks for deep learning: a review,” Int. J. Mach. Learn. Cybern., vol. 13, no. 10, pp. 3001–3018, Jun. 2022, doi: 10.1007/s13042-022-01578-8. DOI: https://doi.org/10.1007/s13042-022-01578-8

A. Mehrabian, 2000. Beyond IQ: Broad-based measurement of individual success potential or" emotional intelli-gence". Genetic, social, and general psychology monographs, 126(2), p.133.

Ahmed, F., Bari, H. and Hossain, E., 2014. Person-independent facial expression recognition based on compound local binary pattern (CLBP). Int. Arab J. Inf. Technol., 11(2), pp.195-203.

M. Moussa, M. Hmila, and A. Douik, “A novel face recognition approach based on genetic algorithm optimiza-tion,” Studies in Informatics and Control, vol. 27, no. 1, Mar. 2018, doi: 10.24846/v27i1y201813. DOI: https://doi.org/10.24846/v27i1y201813

S. Dwijayanti, M. Iqbal, and B. Suprapto, B, 2022. Real-time implementation of face recognition and emotion recognition in a humanoid robot using a convolutional neural network. IEEE Access, 10, pp.89876-89886. DOI: 10.1109/ACCESS.2022.3200762 DOI: https://doi.org/10.1109/ACCESS.2022.3200762

A. Mollahosseini, D. Chan, and M. H. Mahoor, "Going deeper in facial expression recognition using deep neural networks," in Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), 2016, pp. 1-10, doi: 10.1109/WACV.2016.7477450. DOI: https://doi.org/10.1109/WACV.2016.7477450

M. Devries and G. W. Taylor, "Improved regularization of convolutional neural networks with cutout," arXiv pre-print arXiv:1708.04552, 2017, doi: 10.48550/arXiv.1708.04552

A. T. Lopes, E. de Aguiar, A. F. De Souza, and T. Oliveira-Santos, “Facial expression recognition with Convolu-tional Neural Networks: Coping with few data and the training sample order,” Pattern Recognition, vol. 61, pp. 610–628, Jan. 2017, doi: 10.1016/j.patcog.2016.07.026. DOI: https://doi.org/10.1016/j.patcog.2016.07.026

X. Wang, Y. C. Wu, M. Zhou, and H. Fu, “Beyond surveillance: Privacy, ethics, and regulations in face recognition technology,” Frontiers in Big Data, vol. 7, Jul. 2024, doi: 10.3389/fdata.2024.1337465. DOI: https://doi.org/10.3389/fdata.2024.1337465

Y. Li and J. Deng, "Deep facial expression recognition: A survey," IEEE Trans. Affect. Comput., vol. 13, no. 3, pp. 1199-1215, 2022, doi: 10.1109/TAFFC.2020.2981443, DOI: 10.1109/TAFFC.2020.2981446. DOI: https://doi.org/10.1109/TAFFC.2020.2981446