D. Dupré, E. G. Krumhuber, D. Küster, G. McKeown
In the wake of rapid advances in automatic affect analysis, commercial software tools for facial recognition have attracted considerable attention in the past few years. While several options now exist to analyze dynamic video data, less is known about the relative performance of these classifiers, in particular when expressions are spontaneous rather than posed. The present work, we tested eight out-of-the-box machine classifiers (Affectiva’s Affdex, CrowdEmotion’s FaceVideo, Emotient’s Facet, Microsoft’s Cognitive Services, MorphCast’s EmotionalTracking, Neurodatalab’s EmotionRecognition, VicarVision’s FaceAnalysis), compared their performance to human observers. For this, a total of 938 videos were sampled from two large databases conveyed basic six emotions (happiness, sadness, anger, fear, surprise, disgust) either posed (BU-4DFE) or spontaneous (UT-Dallas) form. Results revealed that human observers held an advantage over classifiers. Among the eight tested, there was variance in accuracy ranging 49% – 62%. Subsequent analyses per type of expression revealed that the best performing classifier approximated those of human observers, suggesting high agreement with posed expressions. However, spontaneous expressions were consistently lower and reflected less affective behavior. Overall, happiness was the most successful emotion across databases, whereas confusion rates suggested system-specific biases favoring classification of certain emotions over others. The findings indicate shortcomings in existing systems measuring emotions, and highlight the need for more datasets that can act as benchmark for training and testing systems.
