Author: Peter Ling

Royal Melbourne Institute of Technolgy University, Melbourne

Keywords: Resource based learning, evaluation, innovation, information technology, projects, higher education, computer science, artificial intelligence

Article style and source: Refereed article. This article was first published as a 'work-in-progress' on ultiBASE in 1996. While the article was being formally reviewed, other readers were asked voluntarily to offer comments and opinion on this early version. The comments follow the article below. The finished article is thus the result of both formal peer review and less formal critique received from readers of the World Wide Web.

Contents

• Introduction
• Some issues
• Evaluation and cause
• A case study
• Conclusions
• References
• Your comments to the author

Introduction

Are the anticipated benefits realised and are the benefits worth the effort? This paper considers issues in attempting to evaluate innovations in resource based learning (RBL). Evaluation of RBL is important not only in assessing the value of innovations against their own objectives but to disclose associated costs and benefits. In addition, formative evaluation is likely to have a beneficial effect on project outcomes (Hayden and Speedy, 1995).

Some issues

With regard to the latter, the boundaries of the types of innovation to be evaluated need to be defined. Use of educational technologies can be more or less innovative or innovative in some contexts while established in others. In trying to evaluate the impact of innovation using RBL do we include new uses of old technology; do we include low tech innovations, such as new print materials, as well as high tech? CAUT (Committee for the Advancement of University Teaching in Australia) requires projects under its National Teaching Development Grant scheme to be innovative, its definition of innovation is adopted here. The test of innovative is whether the initiative is likely to result in an improvement in the student learning: that is the initiative must be novel and constructive in the situation in which it is applied; it may be something well developed in another context (CAUT, 1996).

The other challenge is to define RBL. The term resource based learning does not imply any particular form of learning. To put it epistemologically it may involve knowing that, knowing why, knowing how to and being able to. Likewise the learning acquired through RBL may be evidenced through processes requiring recall, understanding or competent or creative behaviours. Neither does RBL in itself imply a particular learning process informed by a particular learning theory. It could involve teacher driven construction, based on behaviourist theory or on information processing precepts. On the other hand it may be premised on individualisation of learning consistent with humanistic or with cognitive structural approaches. It may involve design catering for graded or alternative conceptualisations. The student may be largely passive or active and learning may take place through discovery or through direct instruction. While RBL does not in its design require a particular understanding of learning, evaluation of an RBL initiative, other than in terms of its own goals, will.

The resource based component of the term RBL also requires clarification. RBL implies the use of something more than human resources of speech and action in the generation of learning. That something might be a multimedia package, the internet, print-based open learning materials, a computer aided learning program or a computer managed learning system. RBL cannot be defined in absolute terms. One cannot entirely separate the human element from the rest. There must always be at least the components of intent and design for, while learning per se does not require intent, RBL may be taken to refer to contrived learning. The term RBL cannot be limited to learning which occurs in the absence of a teacher when presence and absence have been made - by available technologies - relative rather than absolute terms. On the continuum of resource based learning, a computer based training package operated by a student alone in an office, without other interactions such as telephone, lies in the RBL camp. On the other hand a lecture, even one illustrated with transparencies, would not. If the transparencies or images were instead displayed on video, covered the subject matter and were automated in presentation, even if it included intermittent live comment by the teacher, it could again be classified as RBL. RBL, like IT based teaching or flexible learning delivery, is not then a discretely definable process, which means that it is not susceptible to an embracive evaluation. All that can be evaluated are teaching and learning processes which involve RBL.

If learning can be more or less resource based what does that do for any comparative element in the evaluation of RBL? The subject of evaluation can only be the particular RBL project. However, the criteria for evaluation may be, but need not necessarily be, applied across projects. There are two broad possibilities for comparison: comparison with a non-RBL approach to the same student cohort, operating in the same context and having the same learning goals (implying the same subject matter as well as the same desired outcomes); or comparison with an external expectation generated from either an understanding of potentialities derived from experience and extrapolation or generated from an ideal derived from learning theory.

There are then alternative approaches to establishing criteria by which to evaluate RBL. A frequently employed approach is to use the goals or objectives of the project if they can be identified. This approach is suitable if the objectives of the particular innovation have been accepted as being worthwhile, unanticipated outcomes are seen as irrelevant and costs are seen as given and acceptable. This is, at least prima facie, the situation which applies to some specially funded projects, such as projects funded by a university or by an outside agency which calls for submissions for projects meeting certain criteria and provides support to a predetermined level. Even in this situation, however, to evaluate a project against its own objectives is limiting. It does not address unanticipated outcomes which may be as educationally or practically important as those intended. It does not allow for shifts in objectives. To focus on project objectives can lead to costs being ignored or at least taken as given, being those specified in a project submission. In fact there are likely to be direct and indirect costs which were not identified in project submissions. To ignore extraneous benefits, costs and other effects limits understandings which could inform future actions.

An alternative to this approach is to compare the outcomes of an RBL initiative with an existing (RBL or non-RBL) approach to the learning task; that is to compare learning under the innovation to a prior or concurrent approach which does not employ the RBL innovation (eg. van der Molen and Predebon, 1995; Tilidetzke, 1992).

There are two sets of issues in attempting this type of comparison. The first is how it could be operationalised. What needs to be kept constant for comparative purposes? If the innovation had very wide application it could be trialed in a variety of circumstances and an inferential approach taken to its outcomes in comparison to traditional approaches, provided a consistent measure of outcomes was used. Wide application is not generally available which suggests the alternative approach of trying to keep as many factors constant as possible. This is likely to prove difficult. Cohorts of students will not be identical. Innovations are complex, changing many elements at once. The learning environment is likely to change in multiple ways. The objectives of the learning task may well have changed as innovations are sometimes sparked by new understandings of learning processes and of the type of learning which is valuable.

The second set of issues in comparing old approaches with RBL approaches is whether there is a commonality which should be compared. It would be unusual for educational initiatives using RBL to simply change the form of teaching without impact on other aspects of the educational transaction. Initiatives are likely to arise from reconceptualisation of learning in the subject area as much from the availability of new teaching technologies. Technology can alter the nature of learning transactions to a point where one is comparing unlike processes and unlike outcomes. The nature of teaching and the role of the teacher may change from instructing expert to guide, facilitator, mentor, fellow learner, resource manager, etc. The role of the learner may change from passive recipient to client, explorer, problem solver, creative manipulator, cognitive apprentice, evaluator, etc. What constitutes the subject area, the definition of its boundaries, relationships between disciplines, who owns the knowledge, who can add to it, who can challenge it, all become open. Who sets the learning agenda and the evaluation criteria are also open. It would not then be appropriate to measure the success of RBL approaches against traditional approaches by using traditional student assessment tools such as a standardised test based on a set text. One could expect the objectives, processes and learning experiences to differ. Take for example an RBL innovation which replaces expository teaching derived from a notion of transfer of information with discovery based learning which values development of discovery techniques as much or more than the information acquired in the process. To apply a test of information acquisition alone as a means of comparison between approaches would be inappropriate.

An alternative approach to comparing RBL innovations to a previous or concurrent approach which does not employ the same learning base is to use benefits and costs anticipated from the literature as a basis for evaluation. The result is to evaluate RBL innovations against benefits which could, on the basis of a conventional wisdom derived from current pertinent publication, be expected to flow from them and to likewise evaluate them on the basis of costs which could be expected to be incurred (though the literature offers less indication of anticipated costs; Montgomery, 1988 and Meyers, 1979 are exceptions). The rationale for this approach is that it can take into account a wider range of benefits and costs than those identified by the designers of a particular project. It takes on board broader experience in the field. It also gives a basis for a broader comparative evaluation than that available when individual project objectives are used to evaluate RBL innovations. Criteria can be developed on this basis in the expectation that RBL innovations will provide new educational experiences, offer greater options for student selection of learning activities, extend information resources, extend opportunities for exchanges between students and between students and teachers, provide better opportunities for monitoring individual student progress or provide wider access to learning. On the cost side, costs relating to hardware requirements and staff time required for development could be anticipated along with some training and facilities costs.

On the other hand you could employ ideal criteria, that is criteria developed from a particular educational, psychological and/or social theory. This approach to evaluation criteria requires a considered and preferably explicit theoretical position on the part of the evaluator. What is required is not just a theoretical position on evaluation but a theoretical position on the realm to be evaluated; for example a position on learning in higher education where learning outcomes are evaluated, a theoretical position on management where organisational costs and benefits are an issue, or a position on equity where access to education is an issue. If one expects learning to be meaningful for the learner and rich in quality it implies criteria for evaluation of an RBL initiative which could lead to estimation of its worth different from an evaluation derived from comparison between approaches on the basis of use of an assessment instrument designed to measure outcomes of a traditional approach to the subject.

Laurillard (1993) and Jones et al. (1994) adopt ideal expectations of RBL initiatives which implicitly or explicitly derive from an intersection of learning theory and an understanding of the potential of various forms of educational technology. These are used to set up criteria against which a particular RBL initiative could be evaluated. Laurillard (1993) defines desirable capacities of educational media based on a principled teaching strategy informed by phenomenography. Media should facilitate teaching approaches which are:

• discursive - allowing teachers and students to access each other's conceptions, allowing them to agree to goals and allowing students to receive feedback on their actions;
• adaptive - responding to the relationships between teacher and student conceptions;
• interactive - providing meaningful intrinsic feedback to student actions to achieve a task goal; and
• reflective - allowing students to link feedback on their actions to the topic goal.

A model for evaluation based on the alternative frames of reference outlined above is tabulated below:

Evaluation and cause

A case study

Teaching difficulties were compounded by the fact that the AI subject could be undertaken by students with a range of academic backgrounds in computer science.

The staff proposed an alternative approach to give students a framework for exploring the subject which would allow them to locate their learning within the subject as a whole, to relate the subject to possibilities for application in the real world and to have greater facility to move in the direction and at the pace preferred by the individual student. The two basic premises of the approach taken were to generate learning from the exploration of problems; and to allow, within some specified expectations, for individual or team decisions about the areas to be explored and the pace and depth of the exploration. Features of the approach adopted were the generation of learning from a focal problem; the location of the learning in a conceptual map of topics; and the use of computer based learning located on the internet to present problems, to provide learning resources and to track progress.

A focal AI problem - the operation of a fully automated taxi system - was used with a view to prompting students to envisage the types of problems faced in developing AI systems, that is to obtain their own overview of the subject. The idea was to allow students to then define and explore sub problems. Texts, references and course materials placed on the internet provided some learning resources, though students might also use their peers, experts and tutors as resources. The computer based learning program comprised two related elements: a course web of learning resources constructed by topic but accessible though hypertext and a problem web which assisted students - as they subdivided the focal problem into sub problems - to locate helpful learning resources housed in the course web.

Evaluation was used at each stage to observe student behaviour with elements of the program and to get feedback on attitudes, difficulties and preferences of students. For example, prior to the construction of any element of the computer program students were broken into small groups (of three) and asked to define sub problems which they would need to address if confronted with the focal AI problem - the operation of a fully automated taxi system. Three levels of tutor intervention - tutor directed, tutor assisted and tutorless - were employed with different groups as they worked with the problem. The groups varied in ethnic composition, gender, and computer science backgrounds as well as in terms of tutor intervention so that likely causes of differences between them were hard to isolate. Outcomes, however, in terms of the capacity to subdivide the focal problem outcomes were encouraging.

Plans for summative evaluation focused on a number of possibilities including evaluation against the project objectives and evaluation against the previous approach. It was intended that the evaluation should take an open format allowing for unexpected outcomes.

A number of observations can be made about the evaluation plans. Firstly, as with many projects, there was an undertaking to evaluate but no detailed plan for evaluation at the time the project submission was drafted. To some extent an evaluation plan has to emerge as projects move through stages from concept to submission to design through trialing, etc. Secondly the form of the project objectives did not allow them readily to translate into evaluation measures. Thirdly costs and disadvantages were not originally envisaged as a component of the evaluation. Fourthly having a control or base group to compare with was not envisaged as it was not feasible to take the old and new approaches simultaneously and for the reason which follows simple comparison with current or past cohorts was inappropriate. Fifthly the staff initiative in establishing the project indicated a change in thinking about both how students might best learn the subject and what it was important to learn. In particular it placed more importance upon learning process and less upon learning outcomes in terms of information acquisition. It placed more importance upon appreciation of the big picture and less on detailed manipulations. It placed more importance on application to world-of-work problems and less on set exercises or on games type problems which can be a feature of AI studies. With these shifts it becomes inappropriate to use a single tool such as the traditional end of semester examination to test student learning under both the former approach and the innovative one.

All three of the frames of reference tabulated above were employed in the evaluation of the project, being driven by the imperatives to improve learning; to confirm project objective were met; and to satisfy stakeholders that the new approach was at least as successful as the previous one measured in traditional terms.

The project objectives which were used to provide one set of evaluation criteria were to:

• Improve staff expertise in problem based learning
• Develop a set of contextualised problem based learning materials in AI
• Provide a method of building up course materials supporting contextualisation
• Improve learning outcomes

For design evaluation purposes and for formative evaluation purposes criteria derived from learning theory were applied:

• Provide for interaction: student/student and student/teacher
• Provide for teacher guidance of participation
• Allow student interaction with content to create concepts
• Allow experimentation with concepts
• Allow collaborative learning
• Provide authentic challenging problems related to pertinent concepts
• Provide multiple resources and be expandable
• Allow for multiple entry conditions and multiple attainments
• Ensure assessment is integrated with learning and is authentic assessment of behaviours and artefacts

Conclusions

References

Hayden, M. and Speedy, G. 1995, Evaluation of the 1993 National Teaching Development Grants. Canberra: CAUT.

Jones, B.J., Valdez, G., Nowakowski, J. & Rasmussen, C. 1994, Designing learning and technology for educational reform : Report. Oak Brook, Illinios : North Central Regional Educational Laboratory.

Laurillard, D. 1993, Rethinking university teaching: a framework for the effective use of educational technology. London : Routledge.

Meyers, N.T. 1979, 'Gold or dust? information technology in higher education, Australian Society of Educational Technology', Yearbook, v4, pp45-54.

Montgomery, A.Y. 1988, 'Information technology in higher education: new frontiers', ASCILITE '88: Computers in learning in tertiary education, Proceedings of the sixth annual ASCILITE conference.

Pence, J.L. 1992, 'Transforming campus culture through resource based learning', New directions for higher education, v22 (2): pp113-122.

Tilidetzke, 1992, 'A comparison of CAI and traditional instruction in a college algebra course', Journal of computers in mathematics and science teaching, v11, (1): pp.53-62.

van der Molen, D.J.L. & Predebon, D.A. 1995, 'The design and application of computer programs for teaching of engineering theory and design', ASCILITE '95: Learning with technology, Proceedings of the thirteenth annual ASCILITE conference.

The author wishes to acknowledge the work of the RMIT Computer Science AI Project Team:

Dr Lawerence Cavedon, Dept. of Computer Science
Dr Vic Cieselski, Dept. of Computer Science
Mr Daryl D'Souza, Dept. of Computer Science
Dr James Harland, Dept. of Computer Science
Mr Steve Hages, Dept. of Computer Science
Mrs Shelia Howell, Dept. of Computer Science
Mr Wayne Jenkins, Dept of Computer Science
Dr Lin Padgham, Dept. of Computer Science

About the author

Copyright © Peter Ling, 1996. For uses other than personal research or study, as permitted under the Copyright Laws of your country, permission must be negotiated with the author. Any further publication permitted by the author must include full acknowledgement of first publication in ultiBASE (http://ultibase.rmit.edu.au). Please contact the Editor of ultiBASE for assistance with acknowledgement of subsequent publication.

Your comments to the author. Please mention that you are commenting on the Ling article.

David Kennedy (d.kennedy@meu.unimelb.edu.au)
The University of Melbourne
Centre for the Study of Higher Education
Multimedia Education Unit
6 August 1996

Peter Ling's paper is a welcome contribution to the ongoing discussion of evaluation. It comes at a time of heightened awareness of the need to demonstrate value and effectiveness - particularly appropriate given the high cost of development of technology or resource-based learning packages and a continuing propensity to pursue technology for its own sake.

I find Peter's introduction of the frame of reference concept particularly useful and practical because it helps the project team to determine the central questions which guide the evaluative task.

The central evaluation questions point to the approach used to gather relevant data and even the specific questions which should be asked of students, observers and others during evaluation phases. The frame of reference is a perspective which the project team can explicitly adopt to help it decide whether particular interview methods and questions get at the heart of what they need and want to know.

These critical decision relationships are illustrated by the case study though more elaboration and explanation would help make this clearer.

We must acknowledge that the adoption of an educationally sound frame of reference does not necessarily lead to an educationally sound and productive evaluation. The project objectives may, for example, be poorly conceptualised: an objective 'to improve student learning outcomes' has no real value unless the desired learning outcomes are clearly articulated in an educationally sound manner (typically a heirarchy of key ideas which underlie and represent the knowledge, skills and attitudes of a specific aspect of the discipline and profession). The intended learning outcomes should also reflect the context and the key concerns which gave rise to the project in the first place - often related to a concept which, past experience and evidence suggests, poses particular difficulties for the learner. So, general, ambiguous or misdirected project objectives and learning outcomes provide no solid basis for evaluation.

Clearly similar comments apply to the other frames of reference. For example, there are rigorous, current learning theories - theories which have been empirically tested in a wide range of settings and remain to be falsified. Yet there are equally idiosyncratic, superseded or empirically untested (quite apart from empirically discredited) learning theories. Choosing a learning theory frame of reference needs obviously to be followed by careful choice of a learning theory.

An important corollary to note here is that a frame of reference, when adopted in the considered manner discussed above, becomes a frame of reference for project design and development, not just project evaluation. Quite possibly, this concept is the key to integrate evaluation wholly in design and development. Scholars of evaluation have long regretted the separation and isolation of evaluation; yet they have arguably not offered a conceptualisation of design and development which effectively and seamlessly incorporates evaluation. It is perhaps more appropriate to view evaluation and design as different perspectives to apply dynamically and continually, in the manner of switching from one to the other, throughout the design and development phases. However this idea is beyond the scope of Peter Ling's article.

I find the discussion of the third frame of reference, which relates to the 'benefits and costs anticipated from the literature', somewhat unsatisfactory. Literature does not in itself offer a frame of reference. However in any specific case an author's analysis and conclusions can adopt or suggest a frame of reference. This issue may be resolved if Peter offered a more explicit elaboration of this concept. It could also help address my suspicion that different learning objectives (refer to the case study) encompass different frames of reference.

Upon reflection, I offer the following point of view on frames of reference:

First, project teams should always carefully articulate desired learning outcomes, other teaching development objectives and yet other relevant criteria usually related to the context of the learning and teaching. These should guide the design, development and evaluation. (I am comfortable, by the way, with these changing during design and development if they become clearer and more rigorous.)

Second, design of evaluation (approach, method and specific question) needs to draw on relevant and peer-acknowledged theory. I have made this point already in relation to learning theory which is clearly central to the general objective 'to improve learning outcomes'. Equally if efficiency is an objective, relevant theory would include theories of costing and analysis of costs and benefits.

Seen in this light, the notion of a 'frame of reference' becomes more focussed, more related to domains of knowledge and more integrative of design, development and evaluation. Most important, it celebrates the skills of creativity and analysis which need to brought to bear when we decide what we want to achieve.

John Milton
Curriculum and Academic Development Unit
Educational Program Improvement Group
RMIT
29th September 1996