ADTED531:

During Unit 4 we will continue our deep dive into the instructional design of courses for distance education, but at the same time we will start to shift our focus to the group projects. In the first part of the unit we will examine the notion of deep learning or critical thinking and why we may not currently see evidence of such in our online courses. You will note as you read the article by Schreck, some of the concepts presented in the article have been incorporated into this course, in particular, the structuring of the two-week units that allows for a more in-depth exploration of the topics. As you read the article, reflect on whether you think this approach works across subjects and course levels, or if it is only appropriate for upper-division and graduate-level courses. In your posting and reflections you should discuss in your teams if there are other approaches you have read about for increasing engagement and dialogue in discussions groups, and how these approaches may impact your learning objectives.

Also, in the first week of the unit we'd like each group to pick one of the following scenarios. These scenarios will introduce the nature of budgets for online programs and courses in both the higher education and corporate environments. We have set up separate discussion forums for each team for this exercise, where you can discuss the cases and the questions presented and post a summative one-page response to the questions.

Following your posting and discussions on the Schreck article, your discussion of the budget scenarios, and the submission of your draft proposal, we’d like to turn our attention to the notion of assessment and evaluation. In the second half of the unit we will discuss the role of evaluation in the design and development of distance learning courses and distance education systems: issues regarding what should be evaluated, when evaluation should occur, how it should be undertaken and by whom, and four different frames of reference for selecting evaluation criteria.

For our reading in this unit we will use Chapter 10, "Developing an Evaluation Strategy" from Delivering Digitally, available in eReserves. As you read the chapter and reflect on the points from Chapter 10 that follow, it is important to keep in mind the difference between assessment and evaluation.

For instance, while developing your unit/lesson for your group project, you will be determining what assessment strategy to use in order to assess learning and grades at the micro level. In contrast, evaluation looks at the success of the course or program in meeting its objectives and not necessarily the student learning outcome objectives. This difference may seem subtle, but it is very important. You may want to think of the difference this way: Assessment is what you do at the end of the unit or lesson when you want to assess if the students have mastered the content presented, and how well they have mastered the content. Evaluation is a formative or summative review completed halfway through a course, or at the end of a course or project, to determine if the content was appropriate to support the learning activities and if the content and learning activities are meeting the students' needs so they can meet their desired outcomes. Evaluation lets us take more of a macro view and allows us to make adjustments before the next offering of a course or program.

So as you read the following highlights from Chapter 10, reflect on the differences between assessment and evaluation.

Chapter 10 highlights:

• "Evaluation...cannot only assess and place value upon the outcomes of projects...but...can help to shape them"
   
• "...the use of knowledge media per se are not readily evaluated. All that can be evaluated are particularly teaching and learning activities..."
   
• "The design of learning materials may be based on behaviourist, information processing, cognitive, humanistic or constructivist theories.
  
  Seldom is this last important principle expressed. There seems to be a common unexamined assumption that distance education is necessarily based on only one, usually information processing or cognitive, theory. A few years ago it was presumed to be based on behaviourist theory. Incidentally, when one stresses the importance of having learning objectives as a tool for course designers, people may assume that individual is a behaviorist, but this may not be the case. Often the best solution is to borrow techniques from different schools of practice. The fact is that you can design a course to facilitate any learning theory and many combinations of theories.
  
  Note: If you are not familiar with these theories, return and visit the website recommended in Unit 3 (Natural Learning Principles - Critical for Education), and if possible follow up with reading of your own. It is impossible to be a teacher or designer if you do not understand learning theory.
   
• "Formative evaluation is undertaken during a project in order to monitor progress and modify developments in the project as it proceeds...Summative evaluation is suitable for reflecting upon the success of a project...can inform decisions concerning similar projects in the future."
   
• "The evaluation of needs...should occur prior to the design of a...project. Design evaluation needs to take place prior to the commencement of development of materials..."
   
• "...we can aim to make the means of gathering evaluation data valid and reliable."
   
• Monitoring of progress, regular testing, and assessment are especially important in distance education, where carefully designed feedback replaces the informal feedback common in conventional teaching.
   
• "There are advantages in having an independent evaluator undertake some forms of evaluation while having those involved in a project conduct other forms of evaluation."
  
  (It is important to make sure the independent evaluators really understand what it is that the project is trying to do and to evaluate accordingly, not to evaluate what they think the project should be doing. This is one of several good reasons why evaluators should be associated with the design of the course or project from the beginning. On this point, their involvement means that not only do they understand the aims and objectives, but they also are more likely to establish trust with the designers and others involved in delivery of the course, and that makes a big difference in getting the instructional team to accept the results of the evaluation.)
   
• Evaluation "instruments most commonly used are: expert reviews, surveys, observations."
   
• We have already questioned the advisability of handing "responsibility for selecting the criteria for evaluation to the expert." We won't say this should never be done, but typically, evaluations that are useful are those where the criteria are mutually determined.
   
• "Videotaping use of the product or service can assist in making a detailed analysis of user behaviours."
  
  This technique can be particularly useful during the time of what the authors call "design evaluation." With subject experts it is often difficult to persuade them to simplify their content and their language to a level that students can manage, so a video tape of one or more focus groups taken during a pilot test of a course, when students explain what they can and cannot understand, is a powerful tool for the designers.
   
• "Open-ended forms of evaluation such as loosely structured interviews with users...allow for unexpected responses and provide for depth of reporting. Closed formats like interviewing against a checklist...make data analysis simpler and more reliable."
  
  The text suggests we use the more closed data gathering formats early in the evaluation process and follow up with more open methods. We do not disagree with that, but would add that at the time of course design the more open-ended approach is useful, and during formative or summative evaluation (while you may certainly use both as they suggest) the closed--i.e. larger number of respondents--approach is likely to be more useful.
   
• "To take a quasi-experimental approach requires a theoretical expectation of outcomes..."
  
  We will take a moment here to comment on the often-discussed question of the difference between evaluation and research. We can discuss this more, but for the moment consider that while the above statement of the authors is perfectly correct, technically ALL research--to be defined as research--must have a theoretical foundation. This is the principal difference between evaluation and research. Evaluation has as its sole purpose the improvement of a particular program, course, or system, while research answers a question demonstrated from theory to be of general interest. While the theoretical foundation may not always suggest hypotheses--as is required in experimental or quasi-experimental research--it does provide the rationale for the questions being answered with the data to be gathered.
   
• "Criteria for evaluating innovations involving knowledge media..."
  
  The authors describe four different sources of the criteria by which an evaluation may be undertaken. The first, what they call "goals- or objectives-based criteria," is particularly compelling.  One may not accept their view of the limitations of this approach. A good objectives-based evaluation may, contrary to what they say, reveal unexpected outcomes and can allow for shifts in objectives and need not mean cost issues are ignored. Alternative approaches ask: Has each objective has been met and to what extent, and if not, why not? What components of the program contributed in what ways to those objectives being met? How cost-effectively did those components work and could the objectives be met more cost-effectively? The objectives themselves and the criteria for their evaluation should be evaluated and changed as needed. Without going into more detail or without critiquing each of the other approaches, we hope we have said enough to generate interest in a strict objectives-based approach to evaluation.
   
• "Frames of reference for selecting evaluation criteria..."
  
  This is an interesting and useful elaboration of the different approaches described earlier in the chapter and also the idea that the different approaches can be combined. The reference to Laurillard illustrates some of the concerns with what was said earlier. There should be no problem with evaluating, for example, if the teaching approach is "discursive, allowing teachers and students to access each other's conceptions"...if that is an objective of the program. If the evaluator thinks it is important, that illustrates the view that she should be a member of the design team to ensure that it becomes a focus for evaluation by being incorporated into the objectives of the course. She cannot come along after the course has been designed and delivered and post hoc evaluate against her own criterion.

We have commented at some length on this chapter, so let us refrain from further comment and ask you to read the chapter for yourself. Perhaps you can come back then and look again at the comments above. It is one of the book's stronger chapters and hopefully you will reflect carefully on the issues raised in it. Certainly evaluation is critically important to the success of any course or program, and while there is much lip service paid to this, there is far less good evaluation undertaken than there ought to be.