The current four-year evaluation of postgraduate studies (2021-2024) will provide information on the social impacts of all programs in the country. This four-year evaluation joins the previous one, which has already begun to capture this information from the programs.
How it became a consensus to attribute more than 90% of Brazilian scientific production to postgraduate studies[1], it is worth examining what lies ahead in terms of the impacts of Brazilian research.
The 50 evaluation forms, one for each area of knowledge created by the Coordination for the Improvement of Higher Education Personnel (Capes) – and by researchers working on Capes committees –, contain three sets of indicators that the programs must meet, including their numbers and texts so that they can be evaluated by the respective committees.
These three sets of indicators, with equal weights in the final grades, are: program, training and impact on society. The first two have to do with indicators directly related to the functions that Capes itself has historically attributed to postgraduate studies: scientific production, training of people and coherence and consistency of programs.
What interests us here is the third group – that of impacts on society.
Examining the 50 assessment sheets for the knowledge areas of the four-year period that will be assessed now in 2025, we see that this third group of indicators is divided into three sub-items:
3.1 – Impact and innovative nature of intellectual production based on the nature of the program;
3.2 – Economic, social and cultural impact of the program;
3.3 – Internationalization, insertion (local, regional, national) and visibility of the program.
Of these three sub-items defined to measure impacts on society, only the second (item 3.2) actually asks for indicators of social and economic impacts and it is precisely this one that has the lowest weights in the evaluation forms – they are on average just below 30% of this group, which would represent around 10% of the total evaluation indicators for this four-year period.
Items 3.1 and 3.3 rarely have indicators of social and economic impact. The first (3.1) is related to the scientific impact of publications and the third (3.3) to the international, regional or national scope of the research.
Even in item 3.2, some areas detail indicators that, in any manual or methodological guide on science, technology and innovation (ST&I), could not be considered to have an economic and social impact, such as the “holding of conferences, advanced schools and workshops national and regional”. Okay. I also consider this to be an important indicator, but it already has a guaranteed place in other items on the evaluation sheets.
It is interesting to note that the areas understand socioeconomic impact in very particular ways and distribute the weights of these indicators in a particular way. Areas such as philosophy, history, and religious sciences and theology, for example, give greater weights to item 3.2 than those of astronomy, physics, and mathematics.
All areas of knowledge, without exception, could develop more and better indicators of socioeconomic impact. There are examples around the world. This is what has been happening in several countries for some years now.
Um article published in this year 2025 in the magazine Research Evaluation analyzes how three countries are evaluating their research systematically and using indicators of social and economic impact, sometimes also environmental and cultural, and other dimensions. They are: the United Kingdom, with the Research Excellence Framework (REF of 2014 and 2021); Australia, with the Engagement and Impact Assessment (EI of 2018); and Hong Kong, with the Research Assessment Exercise (RAE, of 2020). These indicators are analyzed to understand what is being discovered as the impacts of scientific and technological research in those countries.
It's quite interesting. Let's see.
The three countries have similar assessment models, whether or not they are conditional performance with funding. The study carried out by the authors of the above article analyzed 7.275 impact studies from the three evaluation systems of the respective countries.
The research question they proposed was: “How is the appreciation of research expressed in the three countries that have similar impact assessment models?”
Using an artificial intelligence (AI) tool, the authors categorized four socioeconomic impact themes: a) in policies, b) in the media, c) in applications, and d) in economic impacts. Table 1 below shows the frequency with which these themes appear in the assessments.
Table 1 – Frequency of impact themes in the countries studied
As can be seen, the spectrum is broad and reveals how research in these countries has been addressing impacts beyond the academic.
When analyzing the relative frequency of these impact indicators (or topics), the result shows prevalence of the theme Application (applications). Table 2 details the indicators for each assessment topic.
Table 2 – Impact indicators of the four impact themes
As can be seen in Table 2, Application is by far the topic that researchers and their institutions most often address the impacts of their research. Perhaps because it is a generic topic that encompasses a wide variety of outcomes. As can be seen in Table 3, Application It is by far the topic that researchers and their institutions most address the impacts of their research.
Table 3 – Distribution of impacts by sub-themes of the theme Application
Informing procedures, practices and protocols; informing guidelines and strategies; and changing attitudes and behaviors are the most addressed indicators. Next comes counting on services or products in use by society. The least frequent item is the use of research results to finalize/end procedures, practices or protocols.
If we combine indicators that overlap, we can say that the most frequent effects of the research results reported in the three cases have been “creating evidence and information that is used by society”.
Although less frequently detected in the study, the other themes, impacts on policies, economics and the media, as seen in Table 2, also stand out and are used with different intensities in the countries, revealing, according to the study, the cultural characteristics and preferences of the countries.
Practices have shown that it is perfectly possible to measure impacts beyond academic ones with relatively convincing indicators validated from a scientific point of view and the context in which they occur.
Changing the course of an assessment as structured, rigorous and consolidated as Capes takes time. For obvious reasons, but mainly because it involves learning and accepting new things and abandoning crystallized things, especially those that have created hierarchies in the world of science.
Almost everyone agrees that it is worth knowing the impacts of research on society, especially since there is no shortage of impacts. But when it comes to doing it, few follow through. There is a question of power in this story, which acts as a brake on change.
The last two four-year evaluations, especially the current one (2021-2024), will allow us to understand whether and what is being changed in the Capes evaluation model. To do so, we will need an evaluation of the evaluation.
I'm sure that's not the intention, but care must be taken so that some indicators on the evaluation sheets are not interpreted as "impactwashing".
This text does not necessarily reflect the opinion of Unicamp.
[1] This number, mentioned in the article McManus et al. (2021), would need to be revisited. It is known that postgraduate programs report as their own all possible publications by faculty, researchers, and even students. As a hypothesis, yet to be proven, it is possible that the numbers appear inflated.
