Seven Important Barriers to Effective Policy Evaluation

ìAs increasing pressures are brought to bear on the public sector to perform its role more effectively and efficiently, evaluation will probably become an even greater source of conflict. Negative evaluations of a programís effectiveness and efficiency now will be more likely to lead to the programís termination than in more affluent times. The content of an evaluation, the values that are contained in it, and event he organization performing the evaluation will all affect the final assessment. Evaluation research is now a major industry involving numerous consulting firms (ëbeltway banditsí), universities, and organizations within government itself. These evaluative organizations will have their own perspectives on what is right and wrong in policy and will bring those values with them when they perform an analysis (Peters, 2007, p. 175).î

As such, Peters identifies seven important barriers to effective policy evaluation that can impede the seemingly simple but operationally difficult process of determining what actually occurred as a result of government action and more specifically determining the level of performance of a public policy of program. It is equally important for both policy evaluators and the consumers of policy evaluation to understand the basic nature of these difficulties. Specifically, anyone engaged in policy evaluation must appreciate the degree of bias and unintentional, albeit sometimes intentional, confusion that can occur as a result of the difference between what is being measured during a policy evaluation and what policy-makers thought the policy should achieve. All too often evaluators, administrators, and the public focus on measurement statistics that are easy to obtain but have no real relationship between what has been accomplished compared to what the original intent of the public policy was. The following text block identifies the seven barriers as enumerated by Peters and it is followed by a brief description of each.

1. Goal Specification and Goal Change:

As outlined in Lessons 3 and 4, problem identification and policy formulation are extremely difficult for a variety of reasons. Not the least of which include political bias, incomplete information (bounded rationality), misunderstanding of the nature of the problem, individual value bias (social construct theory), plurality of competing interests, and inadequate or inappropriate understanding of potential policy alternatives. As a result of these significant limitations, adopted policy is frequently comprised of vague language, ambiguous, contradictory, impractical, or impossible policy goals. Consequently, the administrative agency tasked with policy implementation may need to fill in the gaps, which may or may not coincide with the intentions of policy-makers. Finally, lawmakers may not provide sufficient if any resources to adequately support a policy initiative. This requires bureaucrats to make programmatic decisions, which will unequivocally impact the efficiency and effectiveness of any policy or program.

2. Measurement:

Once policy goals have theoretically been identified, communicated, and programs implemented then some type of measurement instrument must be developed to ascertain the extent to which the goals have been achieved. However, most public problems such as national defense, education, poverty, health care, crime, urban and highway planning, and environmental policy are comprised of policy goals that are extremely difficult to measure directly. Consequently, many surrogate metrics are used to circumscribe the level of effectiveness of programs. Frequently these metrics evaluate outputs such as arrest or conviction rates for example because it is nearly impossible to determine if the ìrightî criminal was arrested and convicted, which of course is the goal. If we use the welfare reform example once again we can see that the mere reduction of the number of people receiving welfare benefits because of new program limitations does not mean that poverty has been eliminated. It simply means that less people are eligible to receive benefits and less people are receiving them. Those individuals who are no longer eligible to receive benefits may very well still be living in poverty and may even been worse off then before because they no longer have government assistance.

Many other factors can also significantly impair the adequate and appropriate measurement of program and policy goals. For example, the time span that many policies require for their full impact to be felt can be a major problem when elected officials or other policy actors are looking for quick answers to support their own policy positions. The awareness of the time span problem often results in the formulation of public policy that will generate quick and measurable results in the short-term but do not necessarily address the real problem over the long haul. Other measurement problems include the inability to adequately isolate contributing variables of a major public issue such as health care. For example, in poverty stricken areas poor health may be the result of many factors such as poor nutrition, inadequate housing, minimal education, or poor sanitation in addition to a lack of access to quality health care. Therefore, public programs that simply provide some minimal level of access to medical care may have little impact on the overall health of a community because of these other present and contributing factors. However, measurement statistics will frequently focus on the increased availability to health care providers or the number of patient contacts, etc. This data may be valid statistics but it alone does not evaluate the level of success of eliminating the real problem of poor health in a particular community. The important point to remember is that voluminous measurement and statistical data does not in and of itself prove anything if the measurement instruments are evaluating indicators that are not directly related to the problem at hand.

3. Targets:

The target population whose behavior is the object of policy action is in many cases as difficult to identify and evaluate as is identifying the problem and formulating policy in the first place. ìPrograms that have significant effects on the population as a whole may not have the desired effects on the more specific target population. For example, the Medicare program was intended, in part, to benefit less-affluent older people, although all the elderly are eligible for it. However, although the health of the elderly population in general has improvedÖthe health of the neediest elderly has not improved commensurately. And as the program has been implemented, substantial coinsurance has been required, along with substantial deductibles if the insured enters a hospital, so that it is difficult for the neediest elderly citizens to participate (Peters, 2007, p. 170).î In many cases, the need to achieve political feasibility during the policy formulation and legitimation stages of the policy process results in the approval of expanded eligibility for benefits beyond the target population as indicated in the Medicare example just cited. Therefore the remedial intent of the initial policy is diffused amongst a larger group of recipients diluting the ultimate impact of the policy, and making its evaluation more complicated.