[Home]
 
[Current Issue] [About Us] [Subscribe] [Search] [Events] [Resources]
 

Evaluating teaching initiatives which employ Resource Based Learning

Author: Peter Ling

Royal Melbourne Institute of Technology University

Keywords: Resource based learning, evaluation, innovation, information technology, projects, higher education, computer science, artificial intelligence, CAUT

Article style and source: Refereed article. This article was first published as a 'work-in-progress' on ultiBASE in 1996. The finished article is the result of both formal peer review and comments received from readers on the World Wide Web.

Contents

Introduction

Improving the quality of higher education necessitates change. New educational technologies and new uses of existing ones provide opportunities for innovation. Effort is being put into projects which use educational technologies to supplement or replace face to face classroom tuition, sometimes also facilitating distance learning. Innovations based on learning resources which replace or complement face to face tuition are designed to improve learning and/or to provide an operational benefit such as improved access to education.

Are the anticipated benefits realized and are the benefits worth the effort? This paper considers issues in attempting to evaluate innovations in resource based learning (RBL). It discusses alternative approaches to evaluation of RBL initiatives and proposes four frames of reference for evaluating RBL. Evaluation of RBL is important not only in assessing the value of innovations against their own objectives but to disclose associated costs and benefits. In addition, formative evaluation is likely to have a beneficial effect on project outcomes (Hayden and Speedy, 1995). The limitations of evaluation in addressing cause and effect are noted. The paper includes a case study in the evaluation of an RBL innovation. The conclusion to the paper incorporates some recommendations.

Some issues

Evaluating innovations employing RBL raises a number of issues and challenges, some of which derive from focusing on innovations and some of which relate to the definition of RBL and consequences for its evaluation.

Issues arising from a focus on innovations

  • The boundaries of the types of innovation to be evaluated need to be defined. Use of educational technologies can be more or less innovative or innovative in some contexts while established in others.
  • In trying to evaluate the impact of innovation using RBL do we include new uses of old technology; do we include 'low tech' innovations, such as new print materials, as well as 'high tech'?
  • CAUT (Committee for the Advancement of University Teaching in Australia) required projects under its National Teaching Development Grant scheme to be innovative, its definition of innovation is adopted here. The test of innovative is whether the initiative is likely to result in an improvement in the student learning, that is the initiative must be novel and constructive in the situation in which it is applied; it may be something well developed in another context (CAUT, 1996).

Defining RBL

  • The term resource based learning does not imply any particular form of learning. To put it epistemologically it may involve knowing that, knowing why, knowing how to and being able to. Likewise the learning acquired through RBL may be evidenced through processes requiring recall, understanding or competent or creative behaviours.
  • Neither does RBL in itself imply a particular learning process informed by a particular learning theory. It could involve teacher driven construction, based on behaviourist theory or on information processing precepts. On the other hand it may be premised on individualization of learning consistent with humanistic or with cognitive structural approaches. It may involve design catering for graded or alternative conceptualizations. The student may be largely passive or active and learning may take place through discovery or through direct instruction.
While RBL does not in its design require a particular understanding of learning, evaluation of an RBL initiative, other than in terms of its own goals, will.
  • The resource based component of the term RBL also requires clarification. RBL implies the use of something more than human resources of speech and action in the generation of learning. That something might be a multimedia package, the Internet, print-based open learning materials, a computer aided learning program or a computer managed learning system.
  • RBL cannot be defined in absolute terms. One cannot entirely separate the human element from the rest. There must always at least be the components of intent and design for, while learning per se does not require intent, RBL may be taken to refer to contrived learning.
  • The term RBL cannot be limited to learning which occurs in the absence of a teacher when presence and absence have been made - by available technologies - relative rather than absolute terms.
On the continuum of resource based learning, a computer based training package operated by a student alone in an office, without other interactions such as telephone, lies in the RBL camp. On the other hand a lecture, even one illustrated with transparencies, would not. If the transparency images were instead displayed on video, covered the subject matter, and were automated in presentation, even if it included intermittent live comment by the teacher, it could again be classified as RBL. RBL then, like IT based teaching or flexible learning delivery, is not a discretely definable process, which means that it is not susceptible to an embracive evaluation. All that can be evaluated is teaching and learning processes which involve RBL.

Some consequences for evaluation

If learning can be more or less resource based what does that do for any comparative element in the evaluation of RBL?
  • The subject of evaluation can only be the particular RBL project. The criteria for evaluation, however, may be, but need not necessarily be, applied across projects.
  • There are two broad possibilities for comparison:
    • comparison with a non-RBL approach to the same student cohort, operating in the same context and having the same learning goals (implying the same subject matter as well as the same desired outcomes); or
    • comparison with an external expectation generated from either an understanding of potentialities derived from experience and extrapolation or generated from an ideal derived from learning theory.

Approaches to establishing evaluation criteria

There are then alternative approaches to establishing criteria by which to evaluate RBL. Four are differentiated here:
  • The use of objectives of the innovation
  • Comparison with a previous approach
  • Effectiveness against the known potential of the technology
  • Evaluation against an ideal or theoretical position.
A combination of approaches is a possibility.

1. Objectives based criteria

A frequently employed approach is to use the goals or objectives of the project if they can be identified.
  • This approach is suitable if the objectives of the particular innovation have been accepted as being worthwhile, unanticipated outcomes are seen as irrelevant and costs are seen as given and acceptable. This is, at least prima facie, the situation which applies to some specially funded projects, such as projects funded by a university or by an outside agency which calls for submissions for projects meeting certain criteria and provides support to a predetermined level.
  • Even in this situation, however, to evaluate a project against its own objectives is limiting. It does not address unanticipated outcomes which may be as educationally or practically important as those intended. It does not allow for shifts in objectives. To focus on project objectives can lead to costs being ignored or at least taken as given, being those specified in a project submission. In fact there are likely to be direct and indirect costs which were not identified in project submissions. To ignore extraneous benefits, costs and other effects limits understandings which could inform future actions.

2. Comparative criteria

An alternative to this approach is to compare the outcomes of an RBL initiative with an existing (RBL or non-RBL) approach to the learning task; that is to compare learning under the innovation to a prior or concurrent approach which does not employ the RBL innovation (e.g. van der Molen & Predebon, 1995; Tilidetzke, 1992). There are two sets of issues in attempting this type of comparison:
  1. The first is how it could be operationalised. What needs to be kept constant for comparative purposes?
    • If the innovation had very wide application it could be trialed in a variety of circumstances and an inferential approach taken to its outcomes in comparison to traditional approaches, provided a consistent measure of outcomes was used.
    • Wide application is not generally available which suggests the alternative approach of trying to keep as many factors constant as possible. This is likely to prove difficult. Cohorts of students will not be identical. Innovations are complex, changing many elements at once. The learning environment is likely to change in multiple ways. The objectives of the learning task may well have changed as innovations are sometimes sparked by new understandings of learning processes and of the type of learning which is valuable.
  2. The second set of issues in comparing old approaches with RBL approaches is whether there is a commonality which should be compared. It would be unusual for educational initiatives using RBL to simply change the form of teaching without impact on other aspects of the educational transaction.
    • Initiatives are likely to arise from reconceptualization of learning in the subject area as much from the availability of new teaching technologies.
    • Technology can alter the nature of learning transactions to a point where one is comparing unlike processes and unlike outcomes.
    • The nature of teaching and the role of the teacher may change from instructing expert to guide, facilitator, mentor, fellow learner, resource manager, etc.
    • The role of the learner may change from passive recipient to client, explorer, problem solver, creative manipulator, cognitive apprentice, evaluator, etc.
    • What constitutes the subject area, the definition of its boundaries, relationships between disciplines, who owns the knowledge, who can add to it, who can challenge it, all become open.
    • Who sets the learning agenda and the evaluation criteria are also open.
It would not then be appropriate to measure the success of RBL approaches against traditional approaches by using traditional student assessment tools such as a standardized test based on a set text. One could expect the objectives, processes and learning experiences to differ. Take for example an RBL innovation which replaces expository teaching derived from a notion of transfer of information with discovery based learning which values development of discovery techniques as much or more than the information acquired in the process. To apply a test of information acquisition alone as a means of comparison between approaches would be inappropriate.

3. Criteria based on expectations of technology

An alternative approach to comparing RBL innovations to a previous or concurrent approach which does not employ the same learning base is to use benefits and costs anticipated from the literature as a basis for evaluation. The result is to evaluate RBL innovations against benefits which could, on the basis of a conventional wisdom derived from current pertinent publication, be expected to flow from them and to likewise evaluate them on the basis of costs which could be expected to be incurred (though the literature offers less indication of anticipated costs, Montgomery, 1988 and Meyers, 1979 are exceptions).

The rationale for this approach is that it can take into account a wider range of benefits and costs than those identified by the designers of a particular project. It takes on board broader experience in the field. It also gives a basis for a broader comparative evaluation than that available when individual project objectives are used to evaluate RBL innovations. Expectations which arise from conventional wisdom include the expectation that RBL innovations will:

  • provide new educational experiences
  • offer greater options for student selection of learning activities
  • extend information resources
  • extend opportunities for exchanges between students and between students and teachers,
  • provide better opportunities for monitoring individual student progress
  • provide wider access to learning.
On the cost side, costs relating to hardware requirements and staff time required for development could be anticipated along with some training and facilities costs.

4. Ideal criteria

On the other hand ideal criteria could be employed. That is criteria could be developed from a particular educational, psychological and/or social theory. This approach to evaluation criteria requires a (preferably explicit) theoretical position on the part of the evaluator. What is required is not just a theoretical position on evaluation but a theoretical position on the realm to be evaluated; for example a position on learning in higher education where learning outcomes are evaluated, a theoretical position on management where organisational costs and benefits are an issue, or a position on equity where access to education is an issue. If one expected learning to be meaningful for the learner and rich in quality it implies criteria for evaluation of an RBL initiative which could lead to different estimation of its worth from an evaluation derived from criteria based on information acquisition.

Combining approaches

Laurillard (1993) and Jones et al. (1994) adopt ideal expectations of RBL initiatives which implicitly or explicitly derive from an intersection of learning theory and an understanding of the potential of various forms of educational technology. These are used to set up criteria against which a particular RBL initiative could be evaluated. Laurillard (1993) defines desirable capacities of educational media based on a principled teaching strategy informed by phenomenography. Media should facilitate teaching approaches which are:
  • discursive - allowing teachers and students to access each other's conceptions, allowing them to agree goals and allowing students to receive feedback on their actions;
  • adaptive - responding to the relationships between teacher and student conceptions;
  • interactive - providing meaningful intrinsic feedback to student actions to achieve a task goal; and
  • reflective - allowing students to link feedback on their actions to the topic goal.
Educational media can then be evaluated against criteria which emphasize the capacity to describe and redescribe conceptions, which allow adaptations of task goals. which provide for feedback and which allow subsequent adaptation of action.

Jones, et al., drawing on a wide range of writing on learning, establish a 'technology effectiveness framework' setting up quadrants about a learning axis ranging from passive learning (undesirable) to engaged and sustained (desirable) and a technology performance axis ranging from low to high. High technology is indicated by connectivity to resources, inter-connectivity to other participants (teachers or students), inter-opposability between systems (e.g. ability to transfer data between systems), distributed resources rather than one source of knowledge (for example extending from stand alone computers to LANs, WANs and the WWW), the capacity to provide complex problems and complex links, functionality (access to sophisticated peripherals), and user friendliness. In the lowest rated quadrant falls computer aided learning programs based on drill, while approaches which are networked, allow conferencing between participants, provide access to rich resources, and include challenging tasks are at the other end of the scale.

Engaged and sustained learning Passive learning
High technology e.g. challenging tasks with connectivity
Low technology e.g. CAL based on drill

The framework is based on a conception of engaged learning as meaningful to the learner, collaborative, challenging, multidisciplinary and oriented around authentic tasks involving important real-world issues. The student is self-regulated and responsible, having optional routes and strategies. It would be inappropriate to evaluate the value of innovations based on these principles by employing a test of traditional teaching designed on transfer of information concepts of teaching and learning. Insofar as evaluation includes assessment of student achievement, assessment working from this understanding of learning should be interwoven with the learning task, assessing the knowledge constructed by students, observing the processes they adopt and the artifacts they produce.

Frames of reference

Link to the whole model for evaluation based on the alternative frames of reference outlined above. This is a separate file to facilitate printout and download time.

Evaluation and cause

Evaluation by whatever of these approaches constitutes a measure of success and may constitute a measure of comparative success, but it leaves open the question of causal factors. Beyond the evaluation of RBL innovations one might like to know what contextual and integral factors are associated with successful RBL projects and what factors with failure and be able to make pronouncements on necessary steps to ensure success with RBL (as in Pence, 1992). This raises issues of causation in complex educational contexts which again requires a theoretical position and, depending on the position adopted, evaluation circumstances which may not be attainable. When this is combined with the problems in defining the entity to be evaluated, attaining generalizable conclusions is problematic.

A case study

The issues outlined above were confronted in attempting to evaluate an RBL initiative in computer science at RMIT. Staff in the Computer Science Department observed that students undertaking a study of artificial intelligence (AI) using lecture and text book inputs and supported by tutorials had a number of difficulties. They had difficulty in gaining an overview of the subject or even of particular topics such as search or logic. They focused on the detail. Students failed to see the application of the topics to real life situations. The problems they dealt with in practical exercises seemed apart from the world of work for which their course was intended to prepare them.

Teaching difficulties were compounded by the fact that the AI subject could be undertaken by students with a range of academic backgrounds in computer science.

The staff proposed an alternative approach to give students an semi-structured environment for exploring the subject which would allow them to locate their learning within the subject as a whole, to relate the subject to possibilities for application in the real world and to have greater facility to move in the direction and at the pace preferred by the individual student. The two basic premises of the approach taken were to generate learning from the exploration of problems; and to allow, within some specified expectations, for individual or team decisions about the areas to be explored and the pace and depth of the exploration. Features of the approach adopted were the generation of learning from a focal problem; the location of the learning in a conceptual map of topics; and the use of computer based learning using elements of the Internet to present problems, to provide learning resources and to track progress.

A focal AI problem - the operation of a fully automated taxi system - was used with a view to prompting students to envisage the types of problems faced in developing AI systems, that is to obtain their own overview of the subject. The idea was to allow students to then define and explore sub problems. Texts, references and course materials placed on an internet site provided some learning resources, though students might also use their peers, experts and tutors as resources. The computer based learning program comprised two related elements: a 'course web' of learning resources constructed by topic but navigable though hypertext and 'a problem web' which assisted students - as they subdivided the focal problem into sub-problems - to locate helpful learning resources housed in the course web.

Evaluation was used at each stage to observe student behaviour with elements of the program and to get feedback on attitudes, difficulties and preferences of students. For example, prior to the construction of any element of the computer program students were broken into small groups (of three) and asked to define sub-problems which they would need to address if confronted with the focal AI problem - the operation of a fully automated taxi system. Three levels of tutor intervention - tutor directed, tutor assisted and tutorless - were employed with different groups as they worked with the problem. The groups varied in ethnic composition, gender, and computer science backgrounds as well as in terms of tutor intervention so that likely causes of differences between them were hard to isolate. Outcomes, however, in terms of the capacity to subdivide the focal problem outcomes were encouraging.

In formative and summative evaluation of the project all four of the frames of reference tabulated above were employed, being driven by the imperatives:

  1. to confirm project objectives were met
  2. to satisfy stakeholders that the new approach was at least as successful as the previous one measured in traditional terms
  3. to make the most of the technology
  4. to improve learning.
Formative evaluation was conducted with an advisory group of people involved in computer science and computer aided learning. In terms of the frames of reference it focused on:
1. the technical ability of the project to satisfy project objectives, and,
3. drawing on the experience of those on the committee the adequacy of the approach against the educational potential of the technology.

The formative evaluation included a video recorded trial by members of the advisory committee, interested staff and some students. The trial was followed by a debriefing session which led to minor modification to the computer based aspects of the project.

The summative evaluation focused on a number of possibilities including:

1. evaluation against the project objectives
2. evaluation against the previous approach
4. evaluation against some items drawn from learning theory:
  • Provided for interaction student/student and student/teacher
  • Provided for teacher guidance of participation
  • Allowed student interaction with content to create concepts
  • Allowed experimentation with concepts
  • Allowed collaborative learning
  • Provided authentic challenging problems related to pertinent concepts
  • Provided multiple and expandable resources
  • Allowed for multiple entry conditions and multiple attainments
  • Ensured assessment was integrated with learning and was authentic assessment of behaviours and artifacts
The summative evaluation included a semi-structured interview allowing for unexpected observations by participants and unexpected outcomes. Trying to isolate the resource based learning component from other elements of the trial of the project proved difficult in the summative evaluation. The project was trialed through a regular AI subject but it was conducted in an intensive summer school format and some responses related to this format rather than to the project. The project involved problem based learning which was a novelty to the students as well as substantially group focused activity. Students had worked in groups previously but it had not been the major format for their learning activities.

This array of novel features made separating the evaluation of the web based element difficult. The semi-structured interview format, however, allowed the matter to be probed. Students were either as satisfied with the summer school format, problem based learning and group work as with previous subjects they had experienced in the department or they were (greater than half) more satisfied. Reactions to the computer based aspect were mixed. The problem web received a favourable response from most students in setting the problem, assisting in locating in the schema of the subject sub-problems which groups devised, in tracking progress and in directing students to resources. The course web, which is a developing resource, was received less enthusiastically. Most found the resources limited and navigation tedious. The experience was enough to persuade one student to visit the library for resources, which was apparently a new experience and which turned out to be a rewarding one. (Semi-structured interviews do allow the evaluator to elicit unintended outcomes.)

It was agreed by the team that the summative evaluation should at least in part attempt to compare the innovative approach with the established approach and that assessment of students would contribute to that evaluation. Due to concerns with security of individual student assessment and having regard for external expectations it was decided that assessment should include an examination on the traditional subject matter. As a corollary it was decided to incorporate some element of the problem solving approach taken in the project in comparing student performance between the two approaches. This inclusion led to complications in structuring the examination. In the event the performance of individual students equalled or improved on their passed results.

A number of observations can be made about the evaluation:

  1. As with many projects, there was an undertaking to evaluate but no detailed plan for evaluation at the time the project submission was drafted.
  2. The form of the project objectives did not allow them to readily translate into evaluation measures. The project objectives which were used to provide one set of evaluation criteria were to:
    • Improve staff expertise in problem based learning
    • Develop a set of contextualized problem based learning materials in AI
    • Provide a (computer based) method of building up course materials supporting Contextualization
    • Improve learning outcomes
  3. Costs and disadvantages were not originally envisaged as a component of the evaluation.
  4. Having a control or base group to compare with was not envisaged as it was not feasible to take the old and new approaches simultaneously and for the reason which follows simple comparison with current or past cohorts was inappropriate.
  5. The staff initiative in establishing the project indicated a change in thinking about both how students might best learn the subject and what it was important to learn. In particular:
    • It placed more importance upon learning process and less upon learning outcomes in terms of information acquisition.
    • It placed more importance upon appreciation of the big picture and less on detailed manipulations.
    • It placed more importance on application to world-of-work problems and less on set exercises or on games type problems which can be a feature of AI studies.
These shifts make it inappropriate to use a single tool such as the traditional end of semester examination to test student learning under both the former approach and the innovative one.

Conclusions

Evaluation is a matter of attributing value to. It requires a frame of reference. RBL innovations often occur in a context of multiple obligations requiring more than one form of evaluation and implying more than one frame of reference. In the case study the innovation was funded on a project basis with an obligation to report against the objectives specified in the project submission. It replaced a traditional approach in which staff, students and agencies in the field had an interest. The stakeholders had to be satisfied that attainments which they valued in the old course were still being delivered. At the same time the incentive for the innovation was some new understandings among staff of effective teaching/learning transactions in their discipline and success needed also to be reckoned in these terms. The case also suggested that more detailed evaluation planning prior to commencement of RBL projects would allow a more systematic approach than is available when the evaluation plan is devised after the project has commenced. To some extent, however, an evaluation plan has to emerge as projects move through stages from concept to submission, submission to design, and design through to trialing. Innovative projects are dynamic and there needs to be room for responsiveness in forms of evaluation.

References

CAUT (Committee for the Advancement of University Teaching in Australia), 1995. 'National Teaching Development Grants -1997', Guidelines, CAUT, Canberra.

Hayden, M. and Speedy, G. 1995. Evaluation of the 1993 National Teaching Development Grants, CAUT.

Jones, B.J., Valdez, G., Nowakowski, J. & Rasmussen, C. 1994. 'Designing learning and technology for educational reform', Report, North Central Regional Educational Laboratory, Oak Brook, Illinios.

Laurillard, D. 1993. Rethinking university teaching: a framework for the effective use of educational technology, Routledge, London.

Meyers, N.T. 1979. 'Gold or dust? information technology in higher education, Australian Society of Educational Technology', Yearbook, v4, pp.45-54.

Montgomery, A.Y. 1988. 'Information technology in higher education: new frontiers', ASCILITE '88: Computers in learning in tertiary education, Proceedings of the sixth annual ASCILITE conference.

Pence, J.L. 1992. 'Transforming campus culture through resource based learning', New directions for higher education, v22, n2, pp113-122.

Tilidetzke, 1992. 'A comparison of CAI and traditional instruction in a college algebra course', Journal of computers in mathematics and science teaching, v11, n1, pp.53-62.

van der Molen, D.J.L. & Predebon, D.A. 1995. 'The design and application of computer programs for teaching of engineering theory and design', ASCILITE '95: Learning with technology, Proceedings of the thirteenth annual ASCILITE conference.

The author wishes to acknowledge the work of the RMIT Computer Science AI Project Team:

Dr Lawerence Cavedon, Dept. of Computer Science
Dr Vic Cieselski, Dept. of Computer Science
Mr Daryl D'Souza, Dept. of Computer Science
Dr James Harland, Dept. of Computer Science
Mr Steve Hages, Dept. of Computer Science
Mrs Shelia Howell, Dept. of Computer Science
Mr Wayne Jenkins, Dept of Computer Science
Dr Lin Padgham, Dept. of Computer Science

This article was first published as a 'work-in-progress' on ultiBASE in 1996. The finished article is the result of both formal peer review and comments received from readers on the World Wide Web. The original submission has been retained to show the development of this work.

About the author

Associate Professor Peter Ling
Flexible Learning Environment Unit
Educational Program Improvement Group (EPI)
Royal Melbourne Institute of Technology
GPO Box 2476V
Melbourne 3001
Email: pling@rmit.edu.au

Copyright © Peter Ling, 1997. For uses other than personal research or study, as permitted under the Copyright Laws of your country, permission must be negotiated with the author. Any further publication permitted by the author must include full acknowledgement of first publication in ultiBASE (http://ultibase.rmit.edu.au). Please contact the Editor of ultiBASE for assistance with acknowledgement of subsequent publication.
[up]
Send feedback to manager@ultibase.rmit.edu.au
Copyright © 2001 Faculty of Education Language and Community Services
Document URL: http://ultibase.rmit.edu.au/Articles/dec97/lingp2.htm
Last Updated: 17-July-1997 by Marita Mueller		 [RMIT University]
 
current II subscribe II about II search II events II resources