Evaluation Questions and Methods

As we learned in Lesson 1, program evaluation is a kind of applied research. Therefore, many of the principles of conducting research that you learned in basic research methods courses (e.g., PADM 503 in our program) should be applied in it. One of the big issues in any kind of research is how to ask your research questions. In the case of program evaluation, they are evaluation questions. I would like to stress a few important points here based on the information in our text.

What makes a good evaluation question? To ensure that your evaluation questions are reasonable and appropriate, ask sponsors and stakeholders for their views and inputs. Remember, however, our discussions on the influences and involvement of stakeholders in evaluation research: their inputs may be very useful (because they know the situation and problems better than an outside evaluator), but they may also want to influence the research process and its findings to support their positions and promote their influences.

In addition to asking sponsors and stakeholders, the evaluator has to use his or her own expertise, experience, and judgment in determining what a good evaluation question is. One key component to evaluation questions is to determine if evaluation questions are answerable. They are answerable if

• they are stated clearly and specifically (no vague words/concepts are used);
• relevant criteria can be applied in collecting evidence to answer them (you should remember this concept as the operational definition from your basic research methods course, such as P ADM 503); and
• expertise and information sources are available to answer the questions.

As the authors of our textbook discuss in detail, different evaluation questions should be asked at the different stages of program evaluation: needs assessment, assessment of program theory, assessment of program process, outcome and impact assessment, and efficiency assessment. The following is not a substitute for their excellent discussions and examples, but just a brief summary of a few important points.

Needs Assessment Questions

The overall question in needs assessment is this: Is there a need for a program? To answer this question, an evaluator first answers the following more specific questions:

• What is the population of individuals, groups (e.g., households, neighborhoods, organizations), or units of objects (e.g., houses, factories) that the evaluator has been asked to study? For example, how many individuals have been infected with the HIV virus? Or, how many houses need repair?
• What are the specific needs and what should be addressed? What do these poor children need, for example: food, education? How much service should be delivered to whom in a future program? How many additional teachers would be needed for how long, for example?
• How can the specific needs be measured? This may sound like an easy question to answer, but it may actually be very difficult, time consuming, and even contentious. As an example, consider this: How should we determine how many poor children are in the community we want to study?

Easy, right? You can simply apply the “poverty line” statistic that is regularly calculated by the U.S. Census Bureau. (For more information, see the Census Bureau website, http://www.census.gov/hhes/www/poverty.) Those children whose families are under the line are poor and those above it are not. This is a common method used in many poverty studies. However, it is not free of problems. Does the U.S. Census Bureau’s statistic truly measure poverty? It is based on a series of assumptions that are not necessarily agreed upon by everyone. Just keep this problem in mind for now; I will come back to it with an example in the next lesson.

Program Theory Questions

As I mentioned earlier in this lesson, a program theory is the conceptual and organizational structure of a program; an image of how a program is supposed to work. This image should be depicted and then scrutinized to make sure that it can truly guide the research process in the following stages of program evaluation. In two weeks, we will discuss the details of these issues. The following is a preview.

The overall question in assessing a program theory is this: Can this program be evaluated? More specifically, first, what is the accurate image of the program as it was/is conceived by the program sponsors and administrators? In other words, how is the program supposed to work? This may sound easy, but in real-life evaluations it takes quite a while and the cooperation of sponsors and possibly other stakeholders to figure it out. An image of the program should be developed, including

• program goals;
• target population; and
• the organization that will implement the program.

The second more specific question in assessing a program’s theory is this: Is this program evaluable? Once you pictured the goals, population, and the organization that will implement the program, you should determine whether this information is sufficient to evaluate the program. If you are asking “Why should it not be sufficient?” please be patient with me until we discuss the details of the issues in program theory.

Process Evaluation Questions

In a process evaluation study, the overall question is this: Has the program been implemented how it is supposed to be implemented, as it was depicted in the program theory? Why is this an important question?

First, it is an accountability issue. If the program administrators and staff take the money allocated for program implementation and did not do what they were supposed to do, they should be held legally and ethically responsible for their actions. It is important to find out what exactly they did with the money and resources they used.

Second, process evaluation is an important stage before an impact assessment, which is the probably the most important stage in program evaluation. The problem is this: If the program was not implemented properly, then it is not meaningful to conduct an impact assessment. In an impact assessment study you will try to answer the question: Did the program reach its goals? You cannot answer that question without first making sure that the program was implemented as intended in the first place.

Process evaluation is analogous to assessing the skills of a marksman (how close the shots were to the bull’s-eye, for example). If the gun the shooter used was not calibrated properly (not adjusted for precise aiming), then it does not make sense to ask if the marksman was successful in hitting the target. He or she had a faulty gun.

The following questions are typically asked in process evaluation:

• Was there enough money for implementation?
• Were there enough personnel and expertise to implement the program?
• Were the program facilities adequate?
• Was the organization that was in charge of the implementation working properly?

Outcome and Impact Evaluation Questions

Outcome(s) and impact assessments are the most important stages in program evaluation. (They may be considered two separate stages or two sub-stages in one stage.) At this stage, we want to find out if the program worked. Did it “work” in the sense that the goals set by the program sponsors in the beginning were reached? Or, to what extent have the goals been reached?

There is a difference between outcome(s) evaluation (assessment) and impact evaluation (assessment). Outcome evaluation is easier and comes first.

The overall question for an outcome evaluation is: Did we reach our goal? For example, has the percentage of children without health insurance in the community been reduced by 10% in the last five years, as was intended in the beginning of the program?

Impact evaluation is more difficult and can be conducted either after an outcome evaluation is completed, or in conjunction with it. The overall question for impact evaluation is: Did we reach our goal because of our program? For example, did the percentage of children without health insurance in the community drop by 10% in the last five years because of our program? Or were there some other factors that did it, or contributed to the change along with our program? Then what percentage of the drop was caused by our program?

The phrase “because of our program” is the key to understand the nature of impact evaluation. The term suggests that we are looking for a causal relationship between our program and the measured outcomes. You may remember from your introductory research methods course that establishing causality between variables is a major challenge in the social sciences. There are methods you can use (we will discuss them in a few weeks), but you still may not be 100% confident that you have found a causal relationship. We will discuss the delicate issues in establishing causality later in this course.

Efficiency Assessment Questions

If you can reasonably demonstrate that the program you have studied caused the outcomes (i.e., the program reached its goals), then the next, and final, stage of program evaluation is to assess the efficiency of the program. The overall question at this stage is this: Was the cost of the program worth reaching its goal(s)? Of course, it is ultimately the judgment of the sponsors, policy makers, or grant givers that will determine whether the success of the program was worth its cost. There are methods of making the cost calculations, mainly cost-benefit analysis and cost-effectiveness analysis. We will discuss these concepts toward the end of the course (in Lesson 11).