Obtaining Input in Determining Evaluation Questions

In our textbook, there is generic advice you can use in formulating the evaluation questions in all five stages. Here are some highlights from the book:

• Ask program sponsors first what would be appropriate and relevant questions to ask.
• Also ask stakeholders about appropriate and revelant questions. The sponsors of a program are usually obvious and well-known, but it may take some effort to identify the stakeholders. The authors of our book recommend identifying who the stakeholders are by using
  • the snowball sampling approach;
  • telephone and personal interviews; and
  • focus groups.

The authors also provide a generic list of possible stakeholders. I recommend that you consult this list in your future studies. In addition to the policy makers and obvious program sponsors, the authors list the following:

• Target participants of a program (wait until next week for a discussion of “targets”)
• Program managers
• Program staff
• Program competitors (Yes, there may be other programs competing against the one you are evaluating. Obtaining the sponsors, managers, and/or staff of them may give you a better perspective in designing your evaluation questions.)
• Contextual stakeholders (This includes organizations, groups, or individuals who are in the area the program is implemented and who may be affected by its implementation, although this may or may not be the goal of the program sponsors or policymakers.)
• Evaluation and research community (Remember that evaluation researchers are typically academics and they review and evaluate others’ studies for accuracy, integrity, and appropriateness. You can seek their advice or look into previous similar studies for guidance in designing your questions.)

It is important to identify the stakeholders and ask for their input, but remember the downside of doing this as well. They may try to influence your study in order to promote their own interests. Also remember that multiple stakeholders will probably give multiple and possibly conflicting forms of advice as to which questions are relevant and appropriate. Also, it is highly likely that stakeholders may have not have been involved in any program evaluation studies before and therefore may not know how to formulate questions.

Another possibility is that even sponsors or stakeholders may not know much about their own programs, or they may have a vague understanding of the programs! Are you surprised? Don’t be. Many programs start within political processes (fighting, debating, interpreting differently what the goals are, etc.) and even policy makers, sponsors, or program managers may not have a clear understanding of their goals and how programs are supposed to work. In the end, it is the evaluator’s responsibility to use his or her expertise and experience in guiding stakeholders to formulate evaluation questions. Our discussions on program theory in a couple of weeks will give you a better sense of what this means.

The authors of our textbook recommend that evaluators discuss the following with stakeholders to guide their formulation of evaluation questions:

• Why is the evaluation needed?
  • What are the program goals and objectives (not the goals of the program evaluation)? (See the guidelines for specifying objectives in the book.)
• What are the most important questions for the evaluation to answer?