The Evaluation Plan

In our textbook, there is an elaborate discussion on what features of an evaluation situation should be taken into account when making an evaluation plan. I am not going to replicate that discussion here. I will simply highlight some of the important points in that discussion.

There are three important areas an evaluation plan should address:

1. What are the purposes of the evaluation?
2. What are the program structure and circumstances? (Click here to view)
3. What are the resources available to conduct the evaluation? (Click here to view)

Purpose of an Evaluation

The purpose of an evaluation is different from the purpose (goal) of the program it will evaluate. The goal of a program may be to reduce crime in the neighborhood, reduce teenage pregnancy rate in the state, or increase the housing values in the city. A program with one of these goals may be evaluated for different purposes. In general evaluations may be conducted for one of the following three purposes:

1. Program improvement (formative evaluation)
2. Accountability (summative evaluation)
3. Knowledge generation (academic research)
4. Hidden agendas (deception, delaying an action, etc.)

There are very good discussions on each in our textbook. I want to emphasize briefly that our main foci in this course are the first and second purposes, particularly the second one. You will learn how to conduct summative evaluations with the purpose of writing up reports for your sponsors about the successes and failures of their programs. Please note that in many real-life cases, summative and formative evaluations are mixed up. The results of a summative evaluation study may be used to make improvements in programs. Particularly, the results of a process evaluation study (discussed shortly) can be used formatively to improve the implementation of a program.

Evaluation studies can also be used for knowledge generation (third purpose above). However, remember the discussion on applied research versus basic research in Lesson 1. Program evaluation is mainly considered a form of applied research, because its purposes are not knowledge generation, but to solve particular problems. However, as program evaluation developed as a discipline, its borders with basic research disciplines (sociology, psychology, economics, etc.) became blurred. This blurring is evidenced in the fact that program evaluation specialists began publishing their findings in academic journals. They also created their own discipline-specific academic journals. As you will see later in this course, we will use some articles from one of these journals: Evaluation Review.

Using evaluation study results for hidden agendas, of course, is not ethical or legitimate. The authors of our textbook mention it because it happens unfortunately. Be aware of it!

Program Structure and Circumstances

An evaluation plan should include information about the structure and circumstances of a program. The authors of our textbook specifically stress the following three structural or circumstantial elements of a plan:

1. Stages in program development
2. Administrative and political context of a program
3. Conceptual and organizational structure of the program (program theory)

The stages in program development and the corresponding program evaluation functions are summarized in Exhibit 2-e in your textbook, which I copied here. This is an instructive exhibit because it provides an overview of what we will spend most of time for during this course. So I recommend that you study the exhibit and related discussion in the book carefully.

Stages of Program Development Exhibit 2-e

Figure 2.2. Image created base on Exhibit 2-E. Evaluation: a systematic approach by Rossi, P.  H., Lipsey, M. W., Freeman, H. E. Reproduced with permission of Sage Publications Inc. Books in the format Post in a course management system via Copyright Clearance Center.

 

You will remember from Lesson 1 that typically there are multiple stakeholders in program evaluation studies. They are sources of information for your study and they will try to influence your study legitimately or not so legitimately. Rossi and his colleagues call these attempts at influencing circumstantial issues in program evaluation the “administrative and political contexts of the program.” These contexts should be factored into evaluation plans carefully. With a good plan, the evaluator can take advantage of the good information and support some of the stakeholders can provide and prevent unduly influences.

The conceptual and organizational structure of the program, otherwise known as the program theory, is a crucial element of an evaluation plan. As you will see shortly in this lesson and in more detail in Chapter 5 of our textbook, a program theory is the image of how a program is supposed to work. This image is important to discern because only if you have an accurate and clear image, you can determine whether the program worked or not.

Availability of Resources for Evaluation

Obviously no evaluation plan is complete without an estimate of the resources available to the evaluator. The availability or lack of these resources may make or break the evaluation study. It is not necessary to repeat the discussion on this issue in our textbook here. Briefly, an evaluation plan should include

• personnel;
• materials;
• equipment;
• facilities;
• specialized expertise;
• support from program management;
• access to records; and
• time.