Stages of Program Evaluation

We already touched on the stages of program development (Table 2.2). Now we can be more specific. There are different forms, or stages, of program evaluation. In Rossi and his colleagues’ classic formulation, five forms, or stages, of program evaluation are identified:

1. Needs assessment (determining the need for a program) (Chapter 4)
2. Program theory (figuring out the design of a program) (Chapter 5)
3. Process evaluation, monitoring (studying if the program was implemented properly, as designed) (Chapter 6)
4. Outcome and impact assessments (determining whether the desired outcomes have been reached and whether they have been reached because of the implementation of the program) (Chapters 7–10)
5. Efficiency assessment (cost-effectiveness and cost-benefit analyses to determine whether reaching the desired outcome was worth the price paid) (Chapter 11)

If you want to conduct a comprehensive evaluation of a program, you will complete all of these stages in a more-or-less sequential order. That is why they may be called “stages.” However, as Rossi and his colleagues recognize, evaluations usually are not done in a sequence. In many cases, the evaluator may be able to conduct a study in only one of the five stages. You will see in the coming weeks that conducting an evaluation study is not an easy task and it will take large amounts of resources and serious commitments by many parties (e.g., program administrators, policy makers, evaluators, and many other stakeholders) for long periods of time to conduct a comprehensive evaluation. That is why, realistically, as an evaluator you can only do one of the five at a time.

Our discussions of these five stages will take most of our time during this semester. We will spend one week on each stage, but our discussions of Stage #4 (outcome and impact assessments) will extend to multiple weeks. You have probably noticed that in our textbook, four chapters are allocated to the discussion on outcome and impact assessments and for good reason. In it a very important question is asked: Did the program reach its goal(s) because of our program? As you will see in the coming weeks, this is not an easy question answer. Sophisticated methods must be applied to answer it.

Consider this example: The goal of a volunteer patrol program in a neighborhood is to reduce crime. How would you know if the program worked? To answer the question, you would have to decide on what you mean by crime (what kinds of crime: thefts, assaults, murders), a time frame, and how much reduction in crime is acceptable. You may observe that the car thefts in the neighborhood are down during the program’s period of the implementation, but sexual assaults are up in the same period. Did the program work? If the program takes one year to yield successful results, and you take measurements at the end of the sixth month and find that it has not reduced the crime rate, you may be wrong because you have not given the program enough time to generate results. Also, you should decide how much reduction in crime would be counted as a success. If it drops 10% in one year, is that enough? Or, should it be 40%? To complicate the matters even more, even if you observe that the crime rate (let’s say the rate of thefts) has declined in the time-frame you decided and at the amount you specified, what if that was because of the general economic conditions, not because of the program? Perhaps, as the economy got better, people found jobs to sustain themselves and their families better and stopped stealing. How would you sort out the specific impact of the program while economic and social conditions are obviously were also affecting what happened in your neighborhood?

There are methods of disentangling the effects (impacts) of a program from other factors like economic and social conditions. You will learn some of the basic methods in this course. It will take more than a week to clarify what these methods are and how they work. This is why we will spend more time on outcome and impact assessments.

I recommend that you keep the points I made in this section in mind when you read the chapters in the coming weeks. I also would like you to keep this in mind: How Rossi and his colleagues describe program evaluation in their book is not the only way to practice or theorize program evaluation. There are different schools of thought (or “models”) in program evaluation. In this course, we will follow Rossi and his colleagues’ school of thought because it is the most well-established model. I will call it the “Rossi model” or the “orthodox model.”

To understand why it may be called the orthodox model, I recommend, you read about the history of program evaluation (for example, Alkin’s book, Evaluation Roots [Thousand Oaks, CA: Sage, 2004]). In brief, Rossi was one of the founders of the discipline called program evaluation. The book we are using in this course is one of the oldest ones in this field. The current edition of the book is the product of many decades of evolution and refinement. It is considered a classic. (Unfortunately, Rossi passed away in 2006, and the future of this book is uncertain.)

The Rossi model is not the only one, but it is the most mainstream model. I will not discuss all the other models in this course. However, I do not want to bias you against them. The reason I prefer using the Rossi model is because it represents the mainstream, and his book is probably one of the most comprehensive ones available. After learning this model, even if you do not like it and want to learn about the others, this course will provide you with the basic understanding of the orthodox (mainstream) model. If you wish to criticize this orthodoxy, you will know what you are criticizing.