Scientific publications
Read about the research that supports the FaceReader Ecosystem
Over the past 20+ years, our facial coding platform and its embedded technologies have been the subject as well as the preferred instrument for numerous accredited scientific studies. Below we present a comprehensive overview of the literature that has emerged from these studies, highlighting and validating the cutting-edge technology of FaceReader Online.
2015
226 citations
A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system
Harley, Bouchet, Hussain, Azevedo, Calvo
This study evaluates the synchronization of three emotional measurement methods—automatic facial expression recognition, self-report, and electrodermal activity—and their agreement regarding learners’ emotions. Data were collected from 67 undergraduates at a North American university who learned about a complex science topic while interacting with MetaTutor, a multi-agent computerized learning environment. Videos of learners’ facial expressions, captured with a webcam, were analyzed using automatic facial recognition software . Learners’ physiological arousal was recorded using Affectiva’s Q-Sensor 2.0 electrodermal activity measurement bracelet. Learners self-reported their experience of 19 different emotional states on five different occasions during the learning session, which were used as markers to synchronize data from FaceReader and Q-Sensor. The study found a high agreement between the facial and self-report data , but low levels of agreement between them and the Q-Sensor data, suggesting that a tightly coupled relationship does not always exist between emotional response components.
2015
30 citations
Association between facial expression and PTSD symptoms among young children exposed to the Great East Japan Earthquake: a pilot study
Fujiwara
“Emotional numbing” is a symptom of post-traumatic stress disorder characterized by a loss of interest in usually enjoyable activities, feeling detached from others, and an inability to express a full range of emotions. Emotional numbing is usually assessed through self-report, and is particularly difficult to ascertain among young children. We conducted a pilot study to explore the use of facial expression ratings in response to a comedy video clip to assess emotional reactivity among preschool children directly exposed to the Great East Japan Earthquake. This study included 23 child participants. Child PTSD symptoms were measured using a modified version of the Parent’s Report of the Child’s Reaction to Stress scale. Children were filmed while watching a 2-min video compilation of natural scenes followed by a 2-min video clip from a television comedy . Children’s facial expressions were processed the using Noldus FaceReader software, which implements the Facial Action Coding System . We investigated the association between PTSD symptom scores and facial emotion reactivity using linear regression analysis. Children with higher PTSD symptom scores showed a significantly greater proportion of neutral facial expressions, controlling for sex, age, and baseline facial expression . This pilot study suggests that facial emotion reactivity, measured using facial expression recognition software, has the potential to index emotional numbing in young children. This pilot study adds to the emerging literature on using experimental psychopathology methods to characterize children’s reactions to disasters.
2015
16 citations
Deceit and facial expression in children: the enabling role of the “poker face” child and the dependent personality of the detector.
Gadea, Alino, Espert, Salvador
This study examines the interplay between children’s deceptive behaviors and their facial expressions, focusing on the “poker face”—a neutral expression that conceals emotions. It also explores how the personality traits of the observer, particularly dependency, influence the detection of deceit. The research suggests that children who can maintain a poker face are more successful in deceiving others. Additionally, observers with dependent personalities may be less adept at identifying deceit, potentially due to their reliance on others and desire for approval. These findings highlight the complex dynamics between a child’s ability to mask emotions and the observer’s personality in the context of deception.
2015
216 citations
Deep learning based FACS Action Unit occurrence and intensity estimation
A. Gudi, H. E. Tasli, T. M. den Uyl and A. Maroulis
Ground truth annotation of the occurrence and intensity of FACS Action Unit activation requires great amount of attention. The efforts towards achieving a common platform for AU evaluation have been addressed in the FG 2015 Facial Expression Recognition and Analysis challenge . Participants are invited to estimate AU occurrence and intensity on a common benchmark dataset. Conventional approaches towards achieving automated methods are to train multiclass classifiers or to use regression models. In this paper, we propose a novel application of a deep convolutional neural network to recognize AUs as part of FERA 2015 challenge. The 7 layer network is composed of 3 convolutional layers and a max-pooling layer. The final fully connected layers provide the classification output. For the selected tasks of the challenge, we have trained two different networks for the two different datasets, where one focuses on the AU occurrences and the other on both occurrences and intensities of the AUs. The occurrence and intensity of AU activation are estimated using specific neuron activations of the output layer. This way, we are able to create a single network architecture that could simultaneously be trained to produce binary and continuous classification output.
2014
117 citations
Facial expressions and autonomous nervous system responses elicited by tasting different juices
Danner, Haindl, Joechl, Duerrschmid
The study aimed to understand reactions elicited by tasting different juices by examining self-reported liking, autonomic nervous system responses, and both implicit and explicit facial expressions. Eighty-one participants tasted banana, grapefruit, mixed vegetable, orange, and sauerkraut juices. Measurements included skin conductance level , skin temperature , heart rate , pulse volume amplitude , and facial expressions. Results showed significant differences in SCL and PVA responses, as well as in the intensity of several facial expressions across the juices. A moderate correlation was found between these physiological responses and self-reported liking, allowing differentiation between liked, disliked, and neutral samples. Notably, in the implicit approach, participants displayed minimal positive emotions for liked samples, whereas in the explicit approach, they exhibited strong positive emotions. Negative emotions were more pronounced for disliked samples in both approaches. The findings suggest that self-reported liking, ANS responses, and facial expressions provide distinct information about taste experiences.
The article was accepted on 4 June 2014.
2014
209 citations
Predicting advertising effectiveness by facial expressions in response to amusing persuasive stimuli
P. Lewinski, M. L. Fransen, E. S. H. Tan
We present a psychophysiological study of facial expressions of happiness produced by advertisements using the FaceReader system for automatic analysis of facial expressions of basic emotions . FaceReader scores were associated with self-reports of the advertisement’s effectiveness. Building on work describing the role of emotions in marketing research, we examined the relationship between the patterns of the FEBE and the perceived amusement of the advertisements, attitude toward the advertisement and attitude toward the brand . Differences were observed between FEH scores in response to high-, medium-, and low-amusing video advertisements . Positive correlations were found between FEH and AAD and FEH and AB in high- and medium- but not in low-AVAs. As hypothesized, other basic emotions did not predict advertisement amusement or advertisements’ effectiveness. FaceReader enabled a detailed analysis of more than 120,000 frames of video-recordings contributing to an identification of global patterns of facial reactions to amusing persuasive stimuli. For amusing commercials, context-specific FEH features were found to be the major indicators of advertisement effectiveness. The study used video-recordings of participants in their natural environments obtained through a crowd-sourcing platform. The naturalistic design of the study strengthened its ecological validity and demonstrated the robustness of the software algorithms even under austere conditions. Our findings provide first evidence for the applicability of FaceReader methodology in the basic consumer science research.