Between bits and emotions, affective computing is technological frontier

Between bits and emotions, affective computing is technological frontier

Feec research group develops resources that expand the possibility of communication between people and machines

One of the most intriguing characters in cinema is a machine with artificial intelligence (AI). Created by Stanley Kubrick and Arthur C. Clarke as the antagonist in the film 2001: Uma Odisseia no Espaço (1968), the HAL 9000 computer is responsible for commanding the Discovery One spacecraft towards Jupiter, as well as communicating with crew members in a more natural way than we do today with devices like Alexa, Amazon's virtual assistant. At a certain point, an error by HAL causes the two astronauts on board to question his operational capabilities and plan to replace him. The computer senses the distrust of his fellow missionaries and, cornered, begins to sabotage them.

What makes this computer such a memorable character is its ability to drive the plot forward by using its ability to understand the astronauts' emotions and make decisions based on that. In the film, HAL loses the contest and ends up shutting down. In the real world, however, AI systems have been gaining more and more ground.

According to a 2024 survey by McKinsey consulting, conducted with 1.363 representatives from different business sectors, 72% of companies claim to already adopt AI to perform tasks and 65% plan to invest in generative AI systems, which create new content based on patterns identified in training data, as is the case with ChatGPT, launched in 2022 by OpenAI.

In the search for AIs that increasingly resemble the human capacity to make decisions, scientists and developers are dedicated to creating computational resources capable of identifying, expressing and simulating emotions. “Historically, AI has developed based on logical models. For a long time, the role of emotions in decision-making and in building social engagements, an aspect of fundamental importance for learning new skills, was disregarded. Affective computing is our last frontier,” says Paula Dornhofer, a professor at the School of Electrical and Computer Engineering (Feec) at Unicamp and coordinator of one of the research lines at the Hub of Artificial Intelligence and Cognitive Architectures (H.IAAC).

One of the team's work fronts is the development of systems capable of recognizing patterns of personality and human emotions expressed in speech, facial expressions and gestures. Using machine learning, the technology can reproduce these emotions in different modalities.

The aim is for technologies of this type to make AI systems more autonomous and effective in their communication – not to the point of taking control of a spaceship, as in the case of HAL 9000, but in a way that revolutionizes computing and the interaction between humans and machines, as 2001: Uma Odisseia no Espaço revolutionized science fiction.

Training emotions

AI systems work by using algorithms to identify patterns in databases, which serve as an initial repertoire. The training of these systems aims to enable them to perform a specific task, such as identifying colors or shapes in images, or reproducing a type of sound. As the process progresses, the systems begin to perform these same tasks autonomously, with new data, not previously presented.

In the case of generative AIs, the systems go further, creating texts, images, music, and other content, also based on the data framework provided. The more sophisticated the task to be performed or the content to be created, and the greater the autonomy desired for the system, the larger the repertoire of data offered and the more extensive the training to be performed with the algorithms must be.

In the case of affective computing, the new feature is the recognition and expression of emotions, which are now one of the tasks of AI. Therefore, it is necessary to provide algorithms with databases that allow this work. If a system needs to recognize facial expressions related to emotions, the training data must inform this, just as in the case of recognizing a harsher or gentler tone of speech, or a pattern of gestures that indicate a more extroverted or introspective personality.

The challenge is to deal with the complexity involved in human communication. “Emotions and expressiveness have multiple layers, and human beings combine all of these,” Dornhofer reflects. According to the professor, unlike other AI systems in which it is possible to isolate the skills for performing tasks in different applications, communication processes combine several elements – voice, facial expression, gestures – that complement each other and also inform each other. Thus, researchers seek to build systems capable of synthesizing human expression in the most complete way possible. “We want to develop systems that have coherent nonverbal communication.”

Speech, expressions and gestures

O Journal of Unicamp learned about three of the research projects developed by H.IAAC aimed at emulating aspects of human personality and expressiveness. The first is dedicated to creating virtual characters that express gestures related to extroverted, introverted and neutral personalities, in an autonomous and natural way, based on the sound of the voice. “We thought about creating a virtual assistant or a game character with whom it would be possible to interact in a more realistic way”, comments Rodolfo Tonoli, a doctoral student who is part of the group.

The first step was to build a database of body movements combined with speech. To do this, the researchers recorded professional actors performing a script that included aspects of the three personality patterns. In order to capture the range of body movements, sensors were attached to the hands, elbows, heads and other parts of the actors' bodies.

“We guided the actors according to a protocol, so that the algorithm would not be biased,” he explains. The images were processed and converted into training data, so that the AI ​​could begin to generate synthetic movements in a kind of virtual model from recorded audio,” said Tonoli.

The researcher says that, at first, the system performed random movements. However, as more data was input, the AI ​​began to learn the dynamics of human movement and its correlation with voice, delivering more energetic gestures in emphatic speech or a restrained posture in more neutral speech. To confirm the results, volunteers watched the synthetic gestures and indicated which type of personality they identified. According to the researcher, the impression corresponded to what was expressed by the system. “The great success is that we can achieve the same perception results in synthetic and real movements,” he said.

Another project focuses on expressive speech synthesis, that is, reproducing human speech not only with intonations that express emotions or mood, but also autonomously, to adapt to the speaker's mood, making communication more efficient. The researchers give the example of a customer service system that informs about flight delays at airports, with the ability to recognize possible frustration from the passenger's tone of voice. "An intelligent system needs to make the decision to communicate, but knowing that the message may be unwanted and identifying the emotion of the person listening to it," explains Dornhofer.

Here, the algorithms are trained in the same way, by identifying speech patterns – tone, rhythm and emphasis – expressed in previously provided data. One advantage of the research, carried out in partnership with the Telecommunications Research and Development Center (CPQD), is that it contributes to increasing diversity in AI systems by enabling the reproduction of accents from different regions of the country.

“There are few databases of Brazilian Portuguese speech. Our research helps fill this gap,” says Leonardo Boulitreau, a master’s student in the group, who emphasizes the care taken to provide the system with typical everyday speech, so that the algorithm is not restricted to archetypes.

The third research study is dedicated to the production of synthetic facial expressions from audio in Portuguese. In this case, two algorithms with complementary action are used: the first processes recorded images of an actress to transform into data the relationship between speech sounds and corresponding facial movements, called visemes.

In a second step, another algorithm converts new speech audios into synthetic faces that reproduce the facial movements and expressions corresponding to the emotion used in the speech. The technique is similar to that used in so-called deep fakes, in which someone is attributed speech in a realistic way. The study, on the other hand, aims to expand the possibilities of communication between users and machines, in addition to spreading the necessary caution with scams and disinformation actions. “With these technologies, our yardstick for judging whether something is real or not needs to be increased,” says Brayan Bernardo, a master and researcher in the group.