Multi-class emotion classification through visual content analysis using deep learning techniques

Florin Octavian  Robu; Simona Moldovanu; Ioana Diana Moldovanu

doi:10.52846/stccj.2025.5.1.65

Authors

Florin Octavian Robu Faculty of Automation, Computers, Electrical Engineering and Electronics Dunarea de Jos University of Galati
Simona Moldovanu Dunarea de Jos University
Ioana Diana Moldovanu Department of Psychology Danubius International University Galati, Romania

DOI:

https://doi.org/10.52846/stccj.2025.5.1.65

Keywords:

Deep Learning, Emotion classification, Convolutional Neural Networks CNNs, Face Expression Recognition Dataset (FERD)

Abstract

Facial emotion recognition plays an important role for human-computer interaction, including therapeutic applications, security and behavioral analysis of a person.

Despite several strategies for facial emotion recognition hybrid deep learning models, particularly Convolutional Neural Network (CNN) and Machine Learning (ML), brought significant potential robustness in automatic feature extraction capabilities and computational efficiency. In this study, we attain the highest single-network classification accuracy on the Face Expression Recognition Dataset (FERD) with the anger, disgust, fear, happiness, neutrality, sadness, and surprise facial expression. We utilize a custom-built CNN and Pycaret AutoML, carefully optimize its hyperparameters, and explore diverse optimization techniques and pre-processing methods. To the best model attains an accuracy of 67.26% on FERD with Ensemble Stacking Classifier (ESC) on proposed CNN architecture.

References

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.

Ohéix, J. (n.d.). Face Expression Recognition Dataset. Kaggle. Retrieved September 01, 2025, from https://www.kaggle.com/datasets/jonathanoheix/face-expression-recognition-dataset/data.

Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321-357.

Woo, S., Park, J., Lee, J.-Y., & Kweon, I. S. (2018). CBAM: Convolutional Block Attention Module. In Proceedings of the European Conference on Computer Vision (ECCV), 3-19.

Teja, R. (2023). Emotion Classification on Face Expression Dataset. Kaggle. Retrieved September 01, 2025, from https://www.kaggle.com/code/ravitejakemidi/emotion-classification-on-face-expression-dataset.

Kharwal, A. (2023). Facial Emotion Recognition with Kaggle Dataset. Kaggle. Retrieved September 01, 2025, from https://www.kaggle.com/code/amanjharwal/facial-emotion-recognition-with-kaggle-dataset.

Li, J.; Jin, K.; Zhou, D.; Kubota, N.; Ju, Z. Attention mechanism-based CNN for facial expression recognition. Neurocomputing 2020, 411, 340–350.

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., ... & Zheng, X. (2016). TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems.

Chollet, F. (2015). Keras: Deep Learning library for Theano and TensorFlow. https://keras.io.

Bradski, G. (2000). The OpenCV Library. Dr. Dobb's Journal of Software Tools, 120; 122-125.

King, D. E. (2009). Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research, 10, 1755-1758.

Dlib. (n.d.). shape_predictor_68_face_landmarks.dat. Retrieved September 01, 2025, from http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2.

Li, S., Deng, W., & Du, J. (2017). Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Goodfellow, I., et al. (2013). Challenges in Representation Learning: A Report on Three Machine Learning Contests.