Between bits and emotions , affective computing is a technological frontier.

Between bits and emotions , affective computing is a technological frontier.

Feec research group develops resources that expand the possibility of communication between people and machines

One of the most intriguing characters in cinema is a machine endowed with artificial intelligence (AI). Created by Stanley Kubrick and Arthur C. Clarke as the antagonist in the film 2001: A Space Odyssey (1968), the HAL 9000 computer commands the Discovery One spacecraft on its way to Jupiter, and converses with the crew members in a more natural way than we do today with devices like Alexa, Amazon's virtual assistant. At a certain point, a mistake by HAL causes the two astronauts on board to question its operational capabilities and plan its replacement. The computer senses the distrust of its mission colleagues and, cornered, begins to sabotage them.

What makes this computer such a memorable character is its ability to drive the plot forward by using its ability to understand the astronauts' emotions and make decisions based on that. In the film, HAL loses the contest and ends up shutting down. In the real world, however, AI systems have been gaining more and more ground.

According to a 2024 survey by McKinsey consulting, conducted with 1.363 representatives from different business sectors, 72% of companies claim to already adopt AI to perform tasks and 65% plan to invest in generative AI systems, which create new content based on patterns identified in training data, as is the case with ChatGPT, launched in 2022 by OpenAI.

In the search for AIs that increasingly resemble the human capacity to make decisions, scientists and developers are dedicated to creating computational resources capable of identifying, expressing and simulating emotions. “Historically, AI has developed based on logical models. For a long time, the role of emotions in decision-making and in building social engagements, an aspect of fundamental importance for learning new skills, was disregarded. Affective computing is our last frontier,” says Paula Dornhofer, a professor at the School of Electrical and Computer Engineering (Feec) at Unicamp and coordinator of one of the research lines at the Hub of Artificial Intelligence and Cognitive Architectures (H.IAAC).

One of the team's work fronts is the development of systems capable of recognizing patterns of personality and human emotions expressed in speech, facial expressions and gestures. Using machine learning, the technology can reproduce these emotions in different modalities.

The aim is for such technologies to make AI systems more autonomous and effective in their communication – not to the point of taking control of a spaceship, as in the case of HAL 9000, but in a way that revolutionizes computing and the interaction between humans and machines, just as 2001: A Space Odyssey revolutionized science fiction.

Training emotions

AI systems work by using algorithms to identify patterns in databases, which serve as an initial repertoire. The training of these systems aims to enable them to perform a specific task, such as identifying colors or shapes in images, or reproducing a type of sound. As the process progresses, the systems begin to perform these same tasks autonomously, with new data, not previously presented.

In the case of generative AIs, the systems go further, creating texts, images, music, and other content, also based on the data framework provided. The more sophisticated the task to be performed or the content to be created, and the greater the autonomy desired for the system, the larger the repertoire of data offered and the more extensive the training to be performed with the algorithms must be.

In the case of affective computing, the new feature is the recognition and expression of emotions, which are now one of the tasks of AI. Therefore, it is necessary to provide algorithms with databases that allow this work. If a system needs to recognize facial expressions related to emotions, the training data must inform this, just as in the case of recognizing a harsher or gentler tone of speech, or a pattern of gestures that indicate a more extroverted or introspective personality.

The challenge is to deal with the complexity involved in human communication. “Emotions and expressiveness have multiple layers, and human beings combine all of these,” Dornhofer reflects. According to the professor, unlike other AI systems in which it is possible to isolate the skills for performing tasks in different applications, communication processes combine several elements – voice, facial expression, gestures – that complement each other and also inform each other. Thus, researchers seek to build systems capable of synthesizing human expression in the most complete way possible. “We want to develop systems that have coherent nonverbal communication.”

Speech, expressions and gestures

The Unicamp Journal learned about three of the research projects developed by H.IAAC focused on emulating aspects of human personality and expressiveness. The first is dedicated to creating virtual characters that autonomously and naturally express gestures related to extroverted, introverted, and neutral personalities, based on voice sounds. “We envisioned creating a virtual assistant or a game character with whom it would be possible to interact more realistically,” comments Rodolfo Tonoli, a doctoral student and member of the group.

The first step was to build a database of body movements combined with speech. To do this, the researchers recorded professional actors performing a script that included aspects of the three personality patterns. In order to capture the range of body movements, sensors were attached to the hands, elbows, heads and other parts of the actors' bodies.

“We guided the actors according to a protocol, so that the algorithm would not be biased,” he explains. The images were processed and converted into training data, so that the AI ​​could begin to generate synthetic movements in a kind of virtual model from recorded audio,” said Tonoli.

The researcher says that, at first, the system performed random movements. However, as more data was input, the AI ​​began to learn the dynamics of human movement and its correlation with voice, delivering more energetic gestures in emphatic speech or a restrained posture in more neutral speech. To confirm the results, volunteers watched the synthetic gestures and indicated which type of personality they identified. According to the researcher, the impression corresponded to what was expressed by the system. “The great success is that we can achieve the same perception results in synthetic and real movements,” he said.

Another project focuses on expressive speech synthesis, that is, reproducing human speech not only with intonations that express emotions or mood, but also autonomously, to adapt to the speaker's mood, making communication more efficient. The researchers give the example of a customer service system that informs about flight delays at airports, with the ability to recognize possible frustration from the passenger's tone of voice. "An intelligent system needs to make the decision to communicate, but knowing that the message may be unwanted and identifying the emotion of the person listening to it," explains Dornhofer.

Here, the algorithms are trained in the same way, by identifying speech patterns – tone, rhythm and emphasis – expressed in previously provided data. One advantage of the research, carried out in partnership with the Telecommunications Research and Development Center (CPQD), is that it contributes to increasing diversity in AI systems by enabling the reproduction of accents from different regions of the country.

“There are few databases of Brazilian Portuguese speech. Our research helps fill this gap,” says Leonardo Boulitreau, a master’s student in the group, who emphasizes the care taken to provide the system with typical everyday speech, so that the algorithm is not restricted to archetypes.

The third research study is dedicated to the production of synthetic facial expressions from audio in Portuguese. In this case, two algorithms with complementary action are used: the first processes recorded images of an actress to transform into data the relationship between speech sounds and corresponding facial movements, called visemes.

In a second step, another algorithm converts new speech audio into synthetic faces that reproduce facial movements and expressions corresponding to the emotion used in the speech. The technique is similar to that used in so-called deep fakes , where speech is realistically attributed to someone. This study, however, aims to expand the possibilities of communication between users and machines, as well as to disseminate the necessary caution regarding scams and disinformation campaigns. "With these technologies, our benchmark for judging whether something is real or not needs to be raised," assesses Brayan Bernardo, a master's student and researcher in the group.