AI identifies personality through facial analysis.

Technology dissected 15-second videos to extract information from the faces of the people analyzed

Scientists from Unicamp have developed an artificial intelligence (AI) model to identify personality traits by analyzing facial images in videos lasting approximately 15 seconds. The research, conducted by systems analyst Alan Gonçalves as a dissertation for a master's degree in technology, under the guidance of professor Marco Antonio Garcia de Carvalho, from the Faculty of Technology (FT), integrates computer vision techniques, artificial intelligence and concepts from psychology.

The study is based on the OCEAN model, also known as the Big Five, a psychological conceptual framework that structures human personality into five dimensions: openness to experience , conscientiousness – a trait of diligent and responsible people –, extraversion , agreeableness , and neuroticism – a characteristic of individuals prone to negative feelings such as anxiety, depression, envy, and fear. To achieve the results, the researcher employed Transformers neural networks, the computational architecture that underpins systems like ChatGPT.

Gonçalves reports that the first attempt to identify personality traits in videos, using image processing methods through convolutional networks – a type of neural network widely used in facial recognition tasks – did not produce the expected results. “The performance was practically equivalent to flipping a coin,” explains the researcher, highlighting that the model was unable, for example, to establish distinctions between extroverted and introverted people.

The dissertation advisor believes that the difficulties encountered at this stage are likely related to the volume of information: "We assume that the amount of information obtained via deep learning was very large and the technology was unable to identify a pattern." Faced with this challenge, the two decided to reformulate the research methodology, with the support of researchers from the University of Calgary (Canada).

The solution consisted of abandoning image processing as a whole, focusing on the so-called “fiducial points” – coordinates that demarcate parts of the face, such as the corners of the eyes, the line of the eyebrows and the contour of the mouth. These points, extracted frame by frame, were transformed into numerical sequences that fed the model: the Transformers architecture. This technology uses an input sequence to obtain an output sequence, learning the context and tracking the relationships between the components of the sequence.

“Each 15-second video provides us with approximately 400 frames , and each frame contains information that is processed from beginning to end,” the researcher explains. These images then undergo processing using the Transformers model.

This second approach yielded more promising results. At the end of the study, the model was able to successfully classify traits such as extroversion and openness to experience, although it performed poorly on other aspects, such as neuroticism.

Source and infrastructure

The research used a sample from the First Impressions V2 dataset , an open database containing approximately 10 15-second videos previously analyzed and classified by humans who had access to both the images and audio of the recordings. This material served as the basis for both the training and testing stages of the model, allowing for analysis of the accuracy of the responses provided by the algorithm.

Using the MediaPipe Face Mesh programming library (a set of pre-written codes for reuse in software projects), 468 facial points were extracted and later organized into smaller groups of 68, 96 and 136 points. These points underwent the binary classification process of the Transformers architecture, answering questions such as: “Is the person extroverted?” The model assigned each individual a value of 0 or 1 – no and yes – for each personality trait analyzed.

Processing this volume of data required computational resources made available through a partnership with the Renato Archer Information Technology Center, located in Campinas.

Challenges, limitations and potentials

According to Carvalho, one of the challenges faced during his master's degree was obtaining data for the research. There is a scarcity of publicly available databases in this area. Although other databases exist, they are private, which would involve financial costs for their use. Despite this limitation, the advisor believes that using First Impressions offers a benefit. "The advantage of this dataset is the possibility of comparing our method with a broad set of existing literature, providing more foundation for our research," Carvalho notes.

In his dissertation, Gonçalves explains that his research is experimental and exploratory in nature, and is a first step towards developing more in-depth studies. “I cannot say that 15 seconds are enough for an analysis. This was the sample available to work with,” explains the researcher.

It is worth noting that the model developed in the master's degree only analyzes visual data, unlike the people who labeled the original videos based on the audio and subtitles available. According to the advisor, the possibility of incorporating these elements in future research has not been ruled out, with the aim of investigating whether the accuracy of the model can be improved.

One of the challenges in advancing technology is creating datasets that are appropriate for each context. In the dataset used in the study, for example, the videos showed people speaking English and not Brazilian Portuguese, which can introduce biases in the results.

“The dataset we have available was labeled based on more information than that used in our analysis,” explains Carvalho. This means that, for an effective commercial application, it would be necessary to develop a more robust and diverse dataset.

According to the researchers, obtaining such sets is a task that demands financial resources and time. “It’s not just about capturing the person’s image. I’m not a psychologist. It would be necessary to hire professionals in the field of psychology to perform this analysis. Because without this information, it becomes impossible to train the model,” explains Gonçalves.

For the systems analyst, the research has the potential to be applied in recruitment and selection processes, helping with hiring according to personality traits identified by the system. Carvalho, in turn, sees possibilities for application in the educational field.

The developed model operates within a controlled environment, according to the advisor. For the technology to achieve reliable status , more data, more research, and more human support are needed.