Culture and ambience: Investigating the role of social environments on classification and generation of facial and verbal expressions

Social environments play a critical function in how humans express themselves in non-verbal communication with other humans and robots. For instance, culture may change how one expresses emotions on their face, while ambiance may change how they express themselves vocally. Little research has been done in affective computing to take into account such social environments when building affective systems. The current thesis aims to investigate these two sources of influence. In study 1, a support vector machine was leveraged to classify three negative emotions across three different cultures using a curated dataset compiled from YouTube videos. In addition, a one-way ANOVA was used to analyze the differences that exist between each culture in terms of the level of activation of underlying social signals. In study 2, we investigate modifications to a robot's speech, using various speech generation tools, to maximize acceptability in various social and acoustic contexts, starting with a use case for service robots in varying restaurants. An original dataset was collected over Zoom with participants conversing in scripted and unscripted tasks given seven different ambient sounds and images. Voice conversion, and altered Text-to-Speech that matched ambient specific data, were implemented for speech generation tasks. A subjective perception study showed that humans favour generated speech that matches the ambient environment, ultimately preferring more human-like voices. This work provides three important contributions to culture-specific emotion expression recognition, as well as ambient appropriate generated voices: (1) understand how different cultures express themselves through their facial expressions, (2) understand how humans adapt their voices to different ambiences, and (3) taking data-driven approaches to perform classification and generation tasks using context-sensitive machine learning methods and novel data collection protocols.