Moritz Moß
Moritz Moß
Advisors
Daniel Krauß (M. Sc.), Dr. Heike Leutheuser, Prof. Dr. Matthias Berking, Prof. Dr. Björn Eskofier
Duration
06 / 2023 – 10 / 2023
Abstract
Charles Darwin already examined the relation between facial expressions and emotions [1]. In the late 20th century, Paul Ekman studied the cross-cultural perception of emotions and the measurement of facial expressions [2–5]. Various facial coding systems have been invented to describe human feelings, e.g. Monadic Phases Coding System [6], Maximally Discriminative Facial Movement Coding System [7] or Facial Animation Parameters [8]. One of the most used is the Facial Action Coding System (FACS), which was originally invented by P. Ekman in 1978 [9] and updated in 2002 [10]. FACS consists of 12 action units (AU) in the upper and 18 in the lower facial area, as well as 14 miscellaneous actions. The system also rates the AUs with different intensities, mostly using a five-point scale (in the letters A-E). In addition, the facial expression can also be divided into temporal segments (onset, apex, offset). Besides frontalview analysis, also profil-view analysis is possible [11]. Although there are many automatic detection algorithms, this work focus on an open source digital biomarker (openDBM) application [12].
Automatic detection of facial action units found numerous applications over the last few decades [13–18]. What all examples have in common, however, is that the facial expressions were calculated from awake people.
Sigmund Freud already pointed out the relation between sleep and mood [19]. A specific correlation between emotions and sleep architecture has already been shown, for example after watching a stressful movie [20] or after a stressful day [21]. Lack of sleep, on the other hand, leads to stronger negative emotions during the day [22, 23]. However, studies that extract the facial expressions of sleeping people and examine the connection with the mood the next day were not examined so far.
The goal of this bachelor’s thesis is therefore to implement a pipeline that covers the extraction of facial action units and the computation of facial emotions. For this purpose, a dataset of 25 nights was recorded containing full-night videos from three infrared cameras with respect to different spatial perspectives. Every night was preceded by questionnaires assessing the subjective mood of the participant during three different time points in the day. Analyzing this dataset, the correlation between facial expressions during the night and the mood of the day afterwards, based on patient surveys, will be studied.
References
[1] C. Darwin, P. Prodger, and P. Ekman, The expression of the emotions in man and animals: Essay on the history of the illustration / by Philip Prodger, 3rd ed. Oxford, New York: Oxford University Press, op. 1998.
[2] P. Ekman, “BODY POSITION, FACIAL EXPRESSION, AND VERBAL BEHAVIOR DURING INTERVIEWS,” Journal of abnormal psychology, vol. 68, no. 3, pp. 295–301, 1964, doi: 10.1037/h0040225.
[3] P. Ekman, “Differential communication of affect by head and body cues,” Journal of Personality and Social Psychology, vol. 2, no. 5, pp. 726–735, 1965, doi: 10.1037/h0022736.
[4] P. Ekman, “Facial expression and emotion,” The American psychologist, vol. 48, no. 4, pp. 384–392, 1993, doi: 10.1037/0003-066x.48.4.384.
[5] P. Ekman and W. V. Friesen, “Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion,” J. Pers. Soc. Psychol., vol. 53, no. 4, pp. 712–717, 1987.
[6] E. Tronick, H. Als, and T. B. Brazelton, “Monadic phases: A structural descriptive analysis of infant-mother face to face interaction,” Wayne State University Press, vol. 26, no. 1, 1980.
[7] M. Lewis, S. M. Alessandri, and M. W. Sullivan, “Violation of expectancy, loss of control, and anger expressions in young infants,” Developmental Psychology, vol. 26, no. 5, pp. 745–751, 1990, doi: 10.1037/0012-1649.26.5.745.
[8] X. Wei, Z. Zhu, L. Yin, and Q. Ji, “A Real Time Face Tracking And Animation System,” in 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA, 2004, p. 71.
[9] P. Ekman and W. V. Friesen, “Facial Action Coding System: A technique for the measurement of facial movement.,” Environmental Psychology & Nonverbal Behavior, 1978, doi: 10.1037/t27734-000.
[10] P. Ekman, W. V. Friesen, and J. C. Hager, Facial action coding system, 2002.
[11] M. Pantic and I. Patras, “Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences,” IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society, vol. 36, no. 2, pp. 433–449, 2006, doi: 10.1109/tsmcb.2005.859075.
[12] C. Floricel et al., Opening Access to Visual Exploration of Audiovisual Digital Biomarkers: an OpenDBM Analytics Tool, 2022.
[13] Q. Ji, P. Lan, and C. Looney, “A Probabilistic Framework for Modeling and Real-Time Monitoring Human Fatigue,” IEEE Trans. Syst., Man, Cybern. A, vol. 36, no. 5, pp. 862–875, 2006, doi: 10.1109/TSMCA.2005.855922.
[14] P. Lucey et al., “Automatically Detecting Pain in Video Through Facial Action Units,” IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics : a publication of the IEEE Systems, Man, and Cybernetics Society, vol. 41, no. 3, pp. 664–674, 2011, doi: 10.1109/TSMCB.2010.2082525.
[15] K. Wendin, B. H. Allesen-Holm, and W. L. Bredie, “Do facial reactions add new dimensions to measuring sensory responses to basic tastes?,” Food Quality and Preference, vol. 22, no. 4, pp. 346–354, 2011, doi: 10.1016/j.foodqual.2011.01.002.
[16] D. DeVault et al., “SimSensei Kiosk: A virtual human interviewer for healthcare decision support,” in Proceedings of the 2014 International Conference on Autonomous Agents and Multiagent Systems, 2014, pp. 1061–1068.
[17] P. M. Blom et al., “Towards Personalised Gaming via Facial Expression Recognition,” Proceedings of the Tenth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 10, no. 1, pp. 30–36, 2014.
[18] M. Gavrilescu and N. Vizireanu, “Predicting Depression, Anxiety, and Stress Levels from Videos Using the Facial Action Coding System,” Sensors (Basel, Switzerland), vol. 19, no. 17, 2019, doi: 10.3390/s19173693.
[19] S. Freud, THE INTERPRETATION OF DREAMS. Vienna, 1931. [Online]. Available: https://www.degruyter.com/document/doi/10.7312/kurz91842-004/html
[20] L. M. Talamini, L. F. Bringmann, M. de Boer, and W. F. Hofman, “Sleeping Worries Away or Worrying Away Sleep? Physiological Evidence on Sleep-Emotion Interactions,” PloS one, vol. 8, no. 5, e62480, 2013, doi: 10.1371/journal.pone.0062480.
[21] J. M. Blaxton, C. S. Bergeman, B. R. Whitehead, M. E. Braun, and J. D. Payne, “Relationships Among Nightly Sleep Quality, Daily Stress, and Daily Affect,” J Gerontol B Psychol Sci Soc Sci, vol. 72, no. 3, pp. 363–372, 2017, doi: 10.1093/geronb/gbv060.
[22] J. P. R. Scott, L. R. McNaughton, and R. C. J. Polman, “Effects of sleep deprivation and exercise on cognitive, motor performance and mood,” Physiology & behavior, vol. 87, no. 2, pp. 396–408, 2006, doi: 10.1016/j.physbeh.2005.11.009.
[23] D. Tempesta et al., “Lack of sleep affects the evaluation of emotional stimuli,” Brain research bulletin, vol. 82, 1-2, pp. 104–108, 2010, doi: 10.1016/j.brainresbull.2010.01.014.