Author: Dr Peter de Vries

Oxford Brookes University

Keywords: Social Science, Bachelor of Arts, dissertations, student assessment, marking criteria, social judgmeents, value judgments, conflict of interest.

Article style and source: Peer reviewed. Original ultiBASE publication..

Contents

• Abstract
• 1. The Nature of the Problem
• 2. Conceptual Underpinning
  • 2.1 Assessment Criteria as Norms/ Rules of Behaviour/ Interaction
  • 2.2 Values
  • 2.3 Social Judgements
• 3. Aim
• 4. Research Question
  • 4.1 Analysis
  • 4.2 The researcher’s Own Value Perspective
  • 4.3 The Sample
• 5. Results
  • 5.1 Nature of Values Addressed in Judgements on Dissertations
  • 5.2 The Orientation of Assessors’ Judgements
  • 5.3 Judgements Related to Structural Devices Used in Building up an Argument
• 6. Discussion
• References

Abstract

1. The Nature of the Problem

The starting premise for the study is that, as assessment is concerned with issues of worth, the types of judgements which assessors make in assessing students’ written work in the social sciences will depend to a great extent on the values they, as assessors, hold. Moreover, the explanations and evidence which assessors will find most acceptable and convincing in students’ work will depend, in part, on what they, the assessors, value most highly in expository writing: this, in turn, will be derived from their views of life, the world, and the subject matter the students are addressing, irrespective of whether or not these are contained in published criteria for marking.

The study is important as it deals with a perennial problem of the validity of assessment decisions and thus of academic standards, and has an impact ultimately on how assessors are socialised into their role as judges of what is good, and what is not so good in academic writing. The context of the study was the assessment of final year dissertations in the Faculty of Social Sciences and Law in a university. Students were required to undertake a research-based dissertation as part of their final year studies for the honours degree. They were expected to work independently on their dissertations under the supervision of a member of the academic staff. The dissertation was required to be a piece of expository writing based on a primary investigation of the students’ own choice, covering at least two thirds of the academic year. For assessment purposes, most academic departments drew up and published criteria to guide the assessors in their judgements. Responsibility for producing assessment criteria rested with dissertation module leaders, who mainly proceeded on the basis of wider discussions of criteria within departments, though some took the views of their subject associations (e.g. British Psychological Association) into consideration, as well as well as making some comparisons with documents concerned with assessment criteria produced by other departments. The context was considered an appropriate vehicle for exploring judgements as students could embark on research on almost any area of study in the social sciences, or in the social sciences in conjunction with another discipline. top

2. Conceptual Underpinning

Three concepts will be explored to come to grips with the study: norms (criteria), values, and judgements.

2.1 Assessment Criteria as Norms/ Rules of Behaviour/ Interaction

Goffman (1963) and Hare (1976:19-59) view rules (they sometimes refer to them as norms, but, the practice in student assessment is to refer to them as criteria) as the regulations which govern all social contacts; for, without regulations, interactions would be confusing, chaotic and arbitrary. These criteria have certain characteristics: they have been viewed as

• the ideas in the minds of members of the group about what should and should not be done.
• the proper ways of acting in a group that have been accepted as legitimate by the members of the group.
• formal written statements of how members of the group should act, though some groups may decide not to codify their expected actions, and as such they will comprise tacit understandings by a group on how they should act.
• rules which are intended to be taken and applied literally.

What is pertinent about this scenario is that, although assessors may have come to agreements beforehand on what criteria to apply to a given situation, there will inevitably be discrepancies among individuals on how the criteria are applied to the same phenomena in the same context. top

2.2 Values

Our views of social reality are influenced by our values; consequently, values also influence our judgements of worth as we have our own views of the relative importance of things compared with what they are worth to other people. The issues we consider to be most important in a topic, and how we think they should be expressed in expository essays, are likely to colour the way in which we think and act, and colour the judgements we make when assessing students’ work. These values could be played out in at least two ways: in the topic the student chooses to write on; and in the way in which the student writes - the style of presenting the argument, the use of language and use of the conventions of writing. First, the arguments we would find most pleasing are dependent, in part, on the things which we regard as most significant in a topic, even though these things may not be incorporated in marking criteria, and if incorporated may not receive the due attention we think they merit. Second, we may attach different priorities to different aspects of academic writing: some of us may place great emphasis on structure of the argument in an essay, or on rigid conformity to writing conventions, such as spelling, punctuation and grammar than, say, to a rich, flowing style replete with ‘errors’. Prosser and Webb (1994) in an Australian study, found that assessors, in general, awarded higher marks for essays which revealed a strong structure than for essays with similar content and weaker structure. Clearly, writing conventions play an important part in many assessors’ judgements of worth. Moreover, in his ‘Academic Tribes and Territories’ Becher (1989) showed that there were different cultures operating within academic departments and within disciplines; consequently, one could expect these cultures to influence judgements made between assessors across these contexts. These differences in value perspective could mean that marks awarded for the same essay could differ between assessors, sometimes by a relatively large margin; this could be expected to occur as we, as persons, have been socialised in different ways and have different levels of knowledge of our subjects, as well as different level of experience of assessing. top

2.3 Social Judgements

Research shows that people make judgements on the basis of anchors or reference points (Sherif and Hovland, 1961). The authors view anchors as internal constructs, based on past experience. These reference points are always present and influence the way a person responds to messages - it is their ‘ego involvement’ which is seen to make a great deal of difference to how they respond to messages. Consequently, any adequate explanation of assessors’ judgements needs to take into account not only how they handle normative expectations, but also how they activate the schemata of their Habitus – their socialisation which occurs through reading, mixing with peers, being trained, consulting (Bourdieu, 1988) - that are relevant to their assessment of students’ scripts. These are the extra-educational contexts which could influence assessors to apply norms in their own idiosyncratic way, especially where there is a systematic or permanent incongruence between normative expectations at the group level and the assessors’ need dispositions on the personality level. Bourdieu has postulated that Habitus and their constancy over time is a more reliable guide to correctness of practice than formal rules and explicit norms. top

3. Aim

4. Research Question

The question which this study seeks to answer is what values the assessors brought to bear on the assessment process. top

4.1 Analysis

Hermeneutics researchers would classify an assessor's marksheet as a text. They would start from the premise that the text itself speaks to them; that it has meanings of its own, divorced from the author’s original intention in writing it; the author’s intent does not prescribe what the text can subsequently be taken to mean (Ricoeur, 1981). The challenge is thus to ascertain the meanings of the text and meanings embedded in the text. We interpret things naturally as part of our every day existence; we always understand experience from the perspective of suppositions or assumptions. Our traditions give us a way of understanding things and we cannot divorce ourselves from that tradition (Heidegger, 1962). History is not separated from the present. We are simultaneously part of the past, in the present and anticipating the future; consequently, we understand an artefact because of what we have learned from history. The meanings we get from a text are the result of our own present-day meanings and those embedded in the language of the text. Our interpretation will be influenced by our own background and experience.

In textual analysis, we use a hermeneutical cycle which involves a movement from experience-near concepts to experience distance concepts (from concepts that have meaning to the members of the culture to concepts that have meaning to outsiders). It becomes a matter of translating between the two so that observers outside the culture can have an understanding of the insiders’ values in a situation. To gain an understanding of an artefact, in this case the assessors’ marksheets, one needs to examine the thing carefully, make inferences and then create an explanation which would make such an experience understandable. In the case in point, when the value perspectives had been identified, they needed to be categorised to place them within the perspective of the reader. top

4.2 The Researcher’s Own Value Perspective

Part of the irony of doing a hermeneutic analysis of other people’s analyses of textual materials is that in so doing, I am giving my readers an opportunity to glean from my interpretation of the material what value perspectives I am bringing to the study. Nevertheless in the interests of openness, it behoves me to be more explicit about my viewpoints. Firstly, I have not adopted a value-neutral position to the study; on the contrary, I started out from the belief that the deterministic cause-and-effect precept in scientific enquiry to higher education issues stultifies depth analysis as it leads to the myth that nothing can be established except through the scientific method of enquiry. In adopting this stance, I reject the use of the ideology of instrumentalism, technical rationality or scientism in higher education. In support of this position, I wish to quote Schön (1983) who asserts that the development of a scientific knowledge base depends on fixed, unambiguous ends, where professional practice is an instrumental activity and properties of specialism, boundedness, scientific application, standardisation of problems, and the uniform application of scientific principles are appropriate. Whereas, by contrast, higher education can be exemplified as a field of activity which suffers from shifting ambiguous ends, based on different value systems, and from unstable institutional contexts and is therefore unable to develop a systematic, scientific knowledge base (Glazer, quoted by Schön, 1983). The reader should thus be aware that, in adopting this value perspective to the subject I was investigating, I was wittingly seeking evidence to substantiate my assumptions about the nature of scientific management of higher education. And, I am also conscious that other persons with different value perspectives from mine could have interpreted the comments on the marksheets in a different way and written a paper with a different orientation to this one. top

4.3 The Sample

Altogether 89 marksheets were gathered for the exercise, distributed as follows: anthropology (7); geography (30); psychology (30); sociology (18); and law, politics and social work (12). There were seven social science departments in the school. However, as there were too few marksheets in law, social work, politics to make any valid inferences, it was decided to concentrate the analysis on the four departments which had the most data for analysis. Thirty-seven marksheets were analysed from four social science departments. Those in geography, psychology and sociology were randomly selected from those available for analysis using random number tables, whereas the total population of marksheets available in anthropology was analysed. top

5. Results

It is necessary to postulate that the assessors adopted a conscious rationality in making their comments on the marksheets otherwise any analysis against the criteria would be meaningless. The marksheets are replete with rich comments that assessors actually made on the dissertations they read, and, as such, provide some indication of the nature of the judgements they were making. One could view the comments on the marksheets as the assessors’ attempts to articulate what judgements they were making; otherwise, there does not appear to be any rationale for their making a comment at all. top

The results of the analysis are reported in terms of the questions posed above:

5.1 Nature of Values Addressed in Judgements on Dissertations

The following are some of the values addressed by the assessors which may have been, but were not necessarily, related to the set criteria.

These different ways in which judgements about dissertations are made, shows how difficult it is to set explicit fixed all-embracing criteria for assessing worth. top

5.2 The Orientation of Assessors’ Judgements

A further analysis shows other dimensions to assessors’ judgements; here the comments have been related to the different orientations that assessors had to the assessment process which in a way are related to their wider value systems: for example, some of the assessors would have been looking for dissertations to be addressing empirical issues; some would have emphasised student centredness; some would have liked to have seen more theory.

top

5.3 Judgements Related to Structural Devices Used in Building up an Argument

The analysis of the marksheets informs one that these are not the only criteria which were used in assessing the value of dissertations; there were other non-stated criteria, concerned with building an argument in expository writing mentioned below followed by a selection of comments made by the assessors:

top

6. Discussion

The analysis reveals that the task of assessment is a multi-faceted, complicated process, involving consideration of many different aspects of learning and different types of judgements. It is a process which, as academic commentators have acknowledged, requires knowledge of the subject; knowledge of what counts as good and not so good in students’ writing among peers in the discipline, as well as experience of assessments of different kinds (Finch, 1996; Trow, 1994; Becher, 1996). From the analysis, one can see that value judgements of different types were being made in assessing the dissertation, some of which were criterion-based, some norm-based, some complying with the requirements of the professional bodies; some related to how the students could improve on their current performance; some based on accepted tenets of writing held by the general body of scholars; and some based on personal preferences.

The act of assessment is principally concerned with establishing truth - what counts as acceptable evidence that the student has reached a certain standard - it is concerned not only with what should be measured but also how it should be measured. But this truth is not absolute and is mediated by other considerations. The comments reveal a conflict of interest in the minds of assessors between completing values: the academic versus the professional body; academic standards versus student-centredness; institutional versus sectoral academic standards. By the nature of the different roles which assessors are required to play, such tensions will always be prevalent and are likely to affect judgements of worth.

One is left with a question as to what should be done about assessors’ value judgements intruding on the assessment process. The answer derived from the work of Becher (1996), Finch (1996) and Trow (1994) appears to lie in the socialisation of assessors, giving them greater exposure to assessments, reviews, external examining, audits and academic judgements of different kinds where they would meet and discuss issues with their peers and place their values under the microscope, as well as training sessions within departments where examples of student work are discussed. top

References

Becher T (1989) Academic Tribes and Territories: Intellectual Enquiry and the Cultures of Disciplines, Open University Press, Milton Keynes.

Becher T (1996) The Hunting of the Gilt-edged Degree in J. Brennan, deVries P and R. Williams (eds.) 'Standards and Quality in Higher Education', Jessica Kingsley, 1996.

BOURDIEU (1988) ‘The Logic of Practice’ Cambridge, Polity Press, p. 54

FINCH J (1996) Power, Legitimacy and Academic Standards in J. Brennan, deVries P and R. Williams (eds.) 'Standards and Quality in Higher Education', Jessica Kingsley Publishers, 1996.

GOFFMAN E(1963) Behaviour in Public Places Free Press, Glencoe Illinois

HARE AP (1976) Handbook of Small Group Research 2nd ed. Free Press, New York

HEIDEGGER M (1962) Being and Times, trans. J Macquarrie and E Robinson, Harper and Row, New York

Prosser M and webb C (1994) Relating the Process of Undergraduate Essay Writing to the Finished Product, ‘Studies in Higher Education’, Vol. 19 (2), pp. 125-138.

Sherif M and Hovland C (1961) ‘Social Judgement’ New Haven Conn. Yale University Press

Ricoeur P (1981) ‘Hermeneutics and the Human Sciences’, Cambridge University Press, Cambridge

Schön D. (1983) ‘The Reflective Practitioner: How Professionals Think in Action’. London, basic Books

TROW M (1994) ‘Academic Reviews and the Culture of Excellence’ Universitietskanslern, Stockholm. top

About the author

Dr Peter deVries is a researcher at the Oxford Centre for Staff Learning and Development at Oxford Brookes University after having previously worked at the Economic and Social Research Council in Swindon and the Open University. He has published in the field of higher education management and collaborative provision, the latest publication being 41 Collaborative Audits and a Code published by the Council of Validating Universities.

Dr. Peter deVries
Oxford Centre for Staff and Learning Development
Oxford Brookes University
Headington
Oxford
OX3 OBP

E-mail: pde-vries@brookes.ac.uk