Background
We recorded 13 EEG hyperscanning experiments in which groups of three acquainted people held 12 conversational trials each, spanning free speech, consensus finding, and conflict. Alongside 96-channel EEG at 2.5 kHz and multi-microphone audio, every session was filmed. The video is stored as .mkv and is already synchronised to the EEG through a photodiode signal, so frame-accurate alignment to the neural data is a solved problem you can build on rather than a risk you have to absorb.
So far the video has been used for exactly one thing: extracting lip motion to help verify who was speaking. This is a considerable waste. The face is the richest continuously available signal of a person’s affective state during conversation, and we have three faces recorded simultaneously alongside their three brains.
Our published work from this dataset shows that occipital alpha power and delta-band inter-brain synchrony both predict how engaged participants report feeling. What we do not know is whether these neural markers have a visible affective signature, and whether the emotional dynamics of a conversation explain variance that the self-report ratings miss.
*This project is being done in collaboration with the Neurotech team.
Objective
Extract facial features from the video recordings, classify them into affective states, and relate those states to the EEG metrics already computed in our pipeline and to the behavioural ratings.
Requirements
- Solid Python; experience with computer vision or deep learning frameworks is an advantage.
- Comfort with statistics beyond a t-test, or willingness to learn mixed models.
Resources
A recorded multimodal dataset, an established EEG analysis pipeline with metrics already computed, GPU compute.
Supervisors
Please send me an email (amirreza.asemanrafat@fau.de) with your resume and transcripts to apply for the topic. We will then get in contact with you if we are interested.
Please send me an email (amirreza.asemanrafat@fau.de) with your resume and transcripts to apply for the topic. We will then get in contact with you if we are interested.