Value and Focus of OD Evaluation

Evaluation is a critical skill set for the OD practitioner. Evaluation of OD and change efforts is important and may

• lead to improvements in change efforts before the change is implemented;
• help ensure that the change effort is directed appropriately;
• assess how well participants are achieving or have achieved pre-established objectives;
• determine how much participants enjoyed the change effort;
• demonstrate the value of change efforts to managers and other stakeholders or decision-makers based on what each group values; and
• help judge the economic effect of training efforts on the organization.

"A great starting point for talking about program evaluation is to get a better understanding of the concepts of efficiency, effectiveness and impact:" (Wikimedia Resource Center, 2022, para. 2).

For the purpose of understanding these terms, put them in context with this example (Brookins, n.d.)

"A small business interested in improving the way projects are managed by employees might use organizational development to map out a plan for creating, or finding, a project management system. With a project management system in place, employees may be able to work smarter, handling tasks in sequential order, identifying which tasks are dependent upon others and outlining which resources are necessary to take projects from start to completion. As a result, customers may receive a better-quality product or service in less time, than if the company hadn't instituted a project management system" (para. 2)

All three concepts should be measured later on, and once you and your classmates have established how you define and use these terms, you will share a common language for everything else that follows (Wikimedia Resource Center, 2022).

Program Efficiency

Program efficiency relates to an analysis of the costs (money, people, time, materials, etc.) that are expended as part of a program in comparison to either their benefits of their effectiveness (Boulmetis and Dutwin, 2011, p. 5). What does this mean in this context? Consider two project management (PM) workshops executed by A and B. Both staged their workshops as one-day events and both had 30 participants attending their events. A asked five trainers to train the participants, whereas B needed 10 to do the same task.

Now, assume that both PM workshops have the same outcome. Every single participant's level of knowledge increased significantly; by the end of the day, every participant was able to understand and apply the principles of project management. Determining the efficiency in those two cases is easy: A's PM workshop was twice as efficient as B's workshop. A only needed half the number of people to achieve the same result as B, whatever the reason for this might have been. Maybe the PM workshop that A organized had a better agenda that enabled a smaller number of trainers to cover the same amount of content. Rather than getting further into the details here, move on to the next term instead.

Program Effectiveness

Program effectiveness relates to the level by which the activities of a program produce the desired effect. Consider a C who receives money for creating online project management training materials. Those training materials introduce employees to the basic concepts of project management. As the materials are freely available online and many people can access them, C has achieved a high level of efficiency. Why's that? C can reach a much larger audience with her online training than A and B, so the PM materials prove to be more cost-efficient (let's just assume that C's investment was not bigger than that of B). But are those online materials that C created also as effective as the trainings that A and B executed?

As it turned out (this is also a hypothetical example; online training can be as efficient as in-person trainings), after measuring the results, C finds out that her online course was not as effective as she'd hoped. Only 10% of the participants who took the online course learned and understood how to apply the principles of project management. Thus C's program was not as effective as A's and B's programs.

So when you examine the effectiveness of your program, you are asking whether the activities did what they were supposed to do. Therefore, a program's effectiveness “is measured in terms of substantive changes in knowledge, attitudes, or skills on the part of the program's clients” (Boulmetis and Dutwin, 2011, p. 6).

Program Impact

Program impact is the extent to which long-term and sustained changes occur in a target population (Boulmetis and Dutwin 2011, p. 7). This is really what doing OD is all about. Consider that one of our OD objectives is to help people improve performance, and align with and contribute to the strategic goals of the organization. Thus, the impact of our programs can be measured by looking into how many people actually improve their performance. To explain this further, go back to the previous example. 6 months after executing their programs, A, B, and C decide to measure the effect of their project management programs. They looked at the participants who went through their workshops (A, B) or used their online materials (C) and counted the number of people whose performance has improved. As it turns out (this is still hypothetical), out of the 60 people who attended A's and B's workshops, 20 saw improvements in performance. C had a different outcome: More than 1,000 people took her online course, yet only 10% of those saw improvements in performance. That's 100 who saw improvements in performance for C's program, whereas A's and B's effort resulted in 20. Although C's program was less effective, it had a bigger long-term impact.

Additional Evaluation Measures

In addition to efficiency, effectiveness, effect, and depending upon the focus of the OD effort, evaluation measures could also focus on

• quality;
• legal compliance;
• the change effort’s contribution to business results;
• improvements in customer satisfaction; and
• comparisons of change objectives to results.

---

References