Anyone who follows the reports from the Federal Court of Accounts (TCU) or the results of basic education assessments can hardly say that the Brazilian state does not conduct assessments. The TCU produces careful audits, points out design and execution flaws, identifies waste, management problems, and risks that could often be corrected before causing greater damage. In education, we have accumulated decades of assessments, with indicators and information that allow us to know, with reasonable precision, how much students learn, where the greatest difficulties lie, how these difficulties are distributed among regions, school systems, and social groups, and how they evolve over time.
It's no small feat, especially for a developing country. But those who follow public policies know that, despite knowing more and more, we continue to live with many of the same problems. Educational assessments follow one another and again reveal known deficiencies; audits identify flaws that reappear later, in other programs or even in the same ones; studies are discussed, recommendations are made, and, after some time, a new assessment finds a good part of the problems still unresolved.
It would be unfair and false to conclude that evaluations are useless or that many policies have not improved. However, it is difficult to deny that technically sound information, often produced by the state itself, struggles to translate into changes in the policies being evaluated. Why does this happen?
In our two previous conversations, we discussed some limitations of the evaluation process that help to understand this difficulty. In this last one, I want to examine more closely whether the State itself has the necessary conditions to use the results of evaluations in the review and correction of policies.
The State has learned to evaluate…
In Brazil, evaluating public policies has ceased to be an occasional activity, carried out by researchers external to the government or imposed by international organizations, and has become part of the routine of various state organizations. Progress has been uneven, like almost everything in a federative country marked by profound and far-reaching differences.
In recent decades, the Brazilian Federal Court of Accounts (TCU) has been the most visible example. Alongside the traditional oversight of the legality and use of public resources, the TCU has expanded the scope of audits that examine the functioning of programs, their results, risks, and management problems. The Institute for Applied Economic Research (Ipea), in turn, has accumulated extensive experience in analyzing public policies and brings together personnel with knowledge of the programs and the capacity to work with increasingly broad and sophisticated databases. The National Institute for Educational Studies and Research Anísio Teixeira (Inep) and the Federal Agency for Support and Evaluation of Graduate Education (Capes) have consolidated particularly important experiences in data production and educational evaluation, creating information databases that today allow for much more precise monitoring of performance and inequalities at different levels of the education system. The National School of Public Administration (Enap) has been contributing to the training of civil servants and to the dissemination of knowledge and monitoring and evaluation tools within the public administration.
These examples are far from exhaustive. Ministries and federal agencies have created teams and units focused on monitoring programs; universities have significantly increased their participation in evaluations; states and municipalities have developed their own experiences, some quite advanced. In certain areas, such as education, evaluation has ceased to be an extraordinary event and has become an integral part of the very functioning of policy.
The material conditions for evaluation have also improved significantly. Today, the State has administrative databases, individualized records, historical series, and information processing capacity that simply did not exist a few decades ago. The development of analysis methods and programs has further expanded these possibilities. For many policies, we now have the technical conditions to produce reasonably reliable information about what is happening.
There are many gaps to fill, but the accumulated capabilities are sufficient to conclude that a lack of knowledge or technical competence does not explain the still limited use of assessments. The explanation must be sought elsewhere.
… but learning wasn't enough
Public policies do not exist outside of politics. It is through politics that support is formed, priorities are defined, resources are approved, interests are coordinated, and programs are sustained over time. And, after the evaluation shows the need to change something, it is again through politics that this change will have to be made. Or not, as Caetano would say.
A good evaluation can identify problems, show that certain objectives have not been achieved, and recommend changes, but, as we saw in the first discussion, evidence does not decide for itself. The evaluator rarely has the authority to implement the changes they recommend. This task falls to those responsible for public policy, who need to deal with budgets, administrative rules, commitments made, beneficiary demands, pressure from organized groups, and government priorities. Recognizing the diagnosis does not, therefore, mean that there are conditions to act on it.
Furthermore, by the time the evaluation arrives, this public policy already has a history. There are administrative structures, teams, contracts, committed resources, and groups that benefit from it directly or indirectly. Programs are announced, presented as responses to relevant problems, and associated with governments and leaders. Reviewing, reducing, or ending a public policy often means going against vested interests, and at that point, partisan electoral politics usually outweigh technical arguments.
The very organization of the Brazilian state contributes to increasing the difficulties. An assessment can be produced by the TCU (Federal Court of Accounts), Ipea (Institute for Applied Economic Research), a university, or a specialized team from a ministry, while the measures taken depend on other sectors and, not infrequently, on more than one level of government. A department may recognize the problem but lack the budget to address it; the manager may depend on legal changes that depend on politicians; the federal government may identify a deficiency whose correction lies in the hands of states or municipalities. In complex policies, everyone may agree on the diagnosis and yet little may happen, because responsibilities, resources, and instruments for action are distributed among various organizations.
There is also a tension between evaluation, control, and management that cannot be ignored. A significant part of the advancement of evaluation in Brazil occurred alongside the strengthening of control mechanisms. However, when evaluation is perceived primarily as an instrument to point out flaws and hold managers accountable, the incentive to justify decisions, demonstrate compliance, and avoid risks grows. Public administrators need to be accountable for their actions, but they also need the space to recognize that a policy is not working as intended and to modify it before the problem becomes larger. It is difficult to demand innovation, experimentation, and corrective action in an environment where any deviation can be treated simply as a failure.
Changing a public policy involves resources, opportunities, benefits, and often power. The interests involved are not always illegitimate and have very unequal capacity to make themselves heard. Beneficiaries of an existing policy know what they stand to lose and tend to react; the gains from a change are often uncertain, dispersed, and future-oriented. Evaluation can clarify the consequences of choices and make some problems harder to ignore, but it does not resolve conflicts nor does it remove politicians' responsibility for the decision.
Therefore, expanding the technical capacity for evaluation has not guaranteed the use of its results. It is necessary to create conditions for these results to be incorporated into planning, budgeting, and management, and to influence decisions while it is still possible to correct course. Otherwise, we will continue to produce good evaluations, gaining a better understanding of the problems, and being surprised when a new round of studies finds the same problems in the same place.
This text does not necessarily reflect the opinion of Unicamp.
Cover photo:

