Using Concept Maps to Evaluate Teaching and Learning

Authors: Brad Temple and Helen Marshall

Royal Melbourne Institute of Technology University

Keywords: concept maps, teaching and learning, evaluation, careers, careers education

Article style and source: Original ultiBASE STAR Report.

Contents

• Situation
• Target
• Activities
• Results
  

Situation

RMIT's Graduate Diploma in Careers Education caters for people who are, or who would like to be, careers practitioners (i.e., careers counsellors or careers advisers). The course examines the complex relationship between school, work and society. It looks at trends in the labour market, the organisation of work, and the implications of these for school-to-work transitions, and more broadly, for the working lives of men and women. It encourages critical reflection on the intended and unintended outcomes of education, labour market and career development programs, and asks on what values they are based and whether they achieve their aims.

In particular, the Graduate Diploma in Careers Education examines three major themes which the course teaching team believes are instrumental in influencing client outcomes. They are 'Self', 'Client' and 'Context'. 'Self' refers to the values, attitudes, beliefs, skills and experience that the careers practitioner brings to the careers setting. 'Client' refers to the values, attitudes, beliefs, skills and experience, as well as the needs and wants that the client brings to the same setting, while 'Context' refers to the broader economic, social and political milieu in which the careers interaction occurs.

Like many other professional education programs, the primary purpose of the Graduate Diploma in Careers Education is to encourage students to reflect critically on the validity of the knowledge, skills and outcomes being produced and reproduced in their professional practice. However our experience suggests that traditional learning and assessment techniques fail to take account of the diverse range of conceptions of careers work that our students being to class.

Target

The situation described above makes evaluation of how well the course is achieving its aims difficult. In the course of collating the many pieces of evaluation evidence the teaching team realised that there was little which directly addressed the following questions:

• What concepts of professional practice do students bring into the course?
• Do they leave with concepts altered in any way?
• If their concepts alter during the course, do they shift towards concepts which the teaching team thinks are appropriate for careers practitioners?

Concept maps provide one solution to this dilemma by providing a way of monitoring changes in individual student's understandings of their own professional practice. The maps serve two purposes:

• the process of constructing a concept map encourages the students to modify their current understandings by integrating new, relevant concepts, by constructing new connections or by re-arranging existing connections; and
• the resulting map is an explicit representation of the students' understandings of their cognitive structures which they can use to communicate those understandings to others.

Activities

A concept map is a two dimensional diagram that is used to illustrate ideas and their interrelationship. Novak (1980) suggests that concept maps are analogous to road maps. They not only identify major ideas (i.e., concepts), but they 'also illustrate the relationship among concepts in much the same way that links among a city are illustrated by highways and other roads' (1980, p.III-1). However, unlike a road map, the linkages and the ideas in a concept map are frequently different for different people.

Concept maps are currently being used in a number of different ways in tertiary education. They are being used to develop conceptual frameworks for research projects, to develop curriculum plans, to help students to develop new cognitive structures, and to evaluate students' understanding of course material. In RMIT's Graduate Diploma in Careers Education concept maps are being used both as a teaching tool and as an evaluative instrument.

During their very first class, students are asked to visually represent their view of 'Professional Careers Practice' using a concept map. Their maps provide a starting point for discussions in subjects throughout the course. They also provide a guide to how students view their practice before they are exposed to the course curriculum.

A small refinement on the traditional concept map drawing technique which has been useful in this instance is to give students labels on which to write the elements in their maps. These can be shuffled around the page until the students are sure that they have a good representation of their ideas, and then they are stuck down.

Students draw new maps at the end of the first stage of the course (i.e., after twelve months), and again at the end of the second stage of the course (i.e., after two years). The three maps enable the course teaching team to see the changes that have occurred in studentsŐ views of their professional practice, as well as evaluate how well the course is achieving its aims.

Students' concept maps are analysed in terms of the number of concepts, the number of links between concepts, as well as the presence of concepts relating to the major themes of the course (e.g., 'Self', 'Client', 'Context').

Results

The table below shows the results of the preliminary analysis of the first concept maps (i.e., Sample Stage 1) produced by the 21 students of the 1995 cohort. The number of concepts per map ranged from 9 to 42. The mean number of concepts per map was 16.9 and the standard deviation was 7.64. The number of links per map ranged from 12 to 54. The mean number of links per map was 22.1 and the standard deviation was 9.79. The high degree of variability in the number of concepts and number of links per map is not surprising given the diversity of skills, experience and epistemological perpsectives of the course participants.

      Concepts	Links
Mean	16.9	22.1
SD	7.64	9.79

The table below shows the results of the preliminary thematic (i.e., course themes) analysis of the same concept maps. The mean number of concepts per map relating to 'Self' was 7.2 and the standard deviation was 4.81. The mean number of concepts per map relating to 'Client' was 6.1 and the standard deviation was 3.86. The mean number of concepts per map relating to 'Context' was 3.9 and the standard deviation was 2.46.

	Self	Client	Context
Mean	7.2	6.1	3.9
SD	4.81	3.86	2.46

The relatively small number of concepts relating to 'Context' (i.e., the broader economic, social and political milieu) that were included in the students' concepts maps seems to support the partly sociological orientation of the course. It fits with the course teaching team's belief that careers practitioners frequently operate in ignorance of, or on untested assumptions about, the labour market, the organisation of work, and the implications of these for the working lives of men and women, and that critical reflection on these issues should be an important part of the professional training of careers practitioners.

We are looking forward to comparing these results with those from the second and third concept maps that the students will complete. While we cannot ultimately be sure that it is the course rather than changing work circumstances which has affected students' concepts of practice, we hypothesise that if the course is doings its job there will be an increase in the number of 'context' concepts overall and that individual students will show an increased awareness of the social context (NB., because maps are named it is possible to track how individuals change).

Reference

Novak, J. D. (1980) Handbook for Learning How to Learn Program New York, Cornell University Press.

About the authors