Manara - Qatar Research Repository
10.1007_s11042-023-15008-6.pdf (883.96 kB)

Analysis of facial emotion expression in eating occasions using deep learning

Download (883.96 kB)
journal contribution
submitted on 2024-01-03, 07:38 and posted on 2024-01-03, 07:41 authored by Elif Yildirim, Fatma Patlar Akbulut, Cagatay Catal

Eating is experienced as an emotional social activity in any culture. There are factors that influence the emotions felt during food consumption. The emotion felt while eating has a significant impact on our lives and affects different health conditions such as obesity. In addition, investigating the emotion during food consumption is considered a multidisciplinary problem ranging from neuroscience to anatomy. In this study, we focus on evaluating the emotional experience of different participants during eating activities and aim to analyze them automatically using deep learning models. We propose a facial expression-based prediction model to eliminate user bias in questionnaire-based assessment systems and to minimize false entries to the system. We measured the neural, behavioral, and physical manifestations of emotions with a mobile app and recognize emotional experiences from facial expressions. In this research, we used three different situations to test whether there could be any factor other than the food that could affect a person’s mood. We asked users to watch videos, listen to music or do nothing while eating. This way we found out that not only food but also external factors play a role in emotional change. We employed three Convolutional Neural Network (CNN) architectures, fine-tuned VGG16, and Deepface to recognize emotional responses during eating. The experimental results demonstrated that the fine-tuned VGG16 provides remarkable results with an overall accuracy of 77.68% for recognizing the four emotions. This system is an alternative to today’s survey-based restaurant and food evaluation systems.

Other Information

Published in: Multimedia Tools and Applications
See article on publisher's website:


Open Access funding provided by the Qatar National Library.



  • English


Springer Nature

Publication Year

  • 2023

License statement

This Item is licensed under the Creative Commons Attribution 4.0 International License.

Institution affiliated with

  • Qatar University

Usage metrics

    Qatar University



    Ref. manager