Universiti Sains Malaysia

Keywords: University Science Malaysia, Centre for Languages, TESOL, ESL,competency, literacy, applied linguistic, postgraduate students, English proficiency test, foreign students.

Article style and source: Peer Reviewed. Original ultiBASE publication. Paper presented at the "Sixth International Literacy and Education Research Network Conference on Learning", Bayview Beach Resort, Penang, Malaysia 27 - 30 September 1999.

Contents

• Abstract
• Introduction
• Profile of students
• English Language Requirement
• Use of English among Foreign Postgraduate Students in USM
• Proficiency
• Assessment
• Conclusion
• References

Abstract

The content of a proficiency test is based on a specification of what candidates have to be able to do in the language in order to be considered proficient (Hughes, 1989; Brown, 1996). This paper aims to describe and report the proficiency assessment for foreign postgraduate students registered with the Institute of Postgraduate Studies (IPS), USM. Since 1991, the number of foreign postgraduate students have been on the rise and currently there are about 272 students in the 1999/2000 academic session. Most of these students are from countries such as the Middle East, Africa and Indonesia. The Centre for Languages, USM has had the task of assessing the English language proficiency of these foreign postgraduate students. However, there has been growing dissatisfaction with the tests used. Data were collected by means of a questionnaire and structured interviews. The different perspectives of the students, staff and administrators will be discussed and some suggestions in overcoming the problems will be put forward.

Introduction

This paper aims to describe and report the problems associated with the proficiency assessment for foreign postgraduate students registered with the Institute of Postgraduate Studies (IPS) in University Science Malaysia (USM).

The growing presence of foreign students in Malaysia is an indication of a trend toward internationalisation of local education. Increasing numbers of foreign students have been coming to various educational institutions in Malaysia to take up courses at all levels. This is partly due to the government’s initiative to promote Malaysia as a centre for excellence in education in the ASEAN region. The largest number of foreign postgraduate students is found in the International Islamic University (IIU) in the state of Selangor. The Ministry of Education has also been instrumental in encouraging local colleges and institutions of higher learning to market their academic courses in other countries. Since the 1980’s, USM has been receiving a steady number of foreign postgraduate students. For the academic session 1999/2000, 279 foreign postgraduate students (from as many as 35 countries around the world) registered in USM. These students have been attracted by USM’s fee structure (which is more attractive than many English-speaking countries) and its excellent facilities for research. Many postgraduate students are also involved in well-funded research programmes as graduate research assistants attached to specific laboratories, playing an important role in deepening and extending research work of USM staff. Graduate work is administered by the Institute of Postgraduate Studies (IPS) in USM and is closely monitored by the Board of Postgraduate Studies. The School or Centre where the candidate is attached undertakes the academic functions and all postgraduates are registered with the IPS.

The IPS was officially established in November 1991 as a main strategy to increase the number of postgraduates in all schools and centres at USM. In its role to promote and diversify postgraduate work, the institute organises the development and expansion of multidisciplinary fields of study. USM currently offers three types of postgraduate programmes: by research, by mixed mode (coursework and research) and by coursework. top

Profile of students

Table 1 presents a profile of the foreign postgraduate students in USM (in 1999) by country of origin and gender. About three quarters of the students are from the Asian region, coming from countries such as Indonesia (119 students), Thailand (19 students), Bangladesh (17 students), India (15 students), Pakistan (14 students) and Myanmar (4 students). The proportion of males (76.6%) is significantly higher than that of females (24.4%). The Middle East (31 students) and Africa (25 students) are the other two regions that make up the foreign postgraduate students population in USM.

Table 2 shows the distribution of the foreign postgraduate students (in 1999) according to the schools and programmes of study. A large number of students (46 students) are enrolled in the School of Management with 29 of them doing their Ph.D. and 17 doing their Masters. The other popular disciplines among the foreign postgraduate students are Humanities (27 students), Social Sciences (24 students), Pharmaceutical Sciences and Computer Sciences (19 students each), Industrial Technology (15 students), Medical Sciences (14 students) and Housing, Building and Planning (12 students). top

English Language Requirement

All foreign postgraduate students are required to have a minimum standard of proficiency in English (which will be decided on an individual basis by the dean of the IPS). Where applicable, the Centre for Languages in USM administers a diagnostic/proficiency test before the start of every semester. Based on their results, students may have to attend remedial English classes offered by the Centre for Languages for its undergraduate students. The proficiency test evaluates the students’ ability to read, write, listen and speak in English. It is essentially an integrative test aiming to profile the students’ English language ability. The test is divided into two parts: Paper 1 (Test of Reading and Writing) and Paper II (Test of Listening and Speaking). In the two and a half-hours’ reading and writing component, candidates read a passage of about 800-1000 words and answer multiple-choice and short answer comprehension questions. They also have two writing tasks and an error analysis exercise. In the listening component, students answer comprehension questions after listening to an excerpt of local news and watching a short documentary on videotape. In the speaking component, the students are interviewed individually for a 10-minute duration by two raters.

For the past three years (1996-1999), the author has been making observations on the issues and practices involved in conducting a valid and reliable testing instrument for the foreign postgraduate students registered in USM. In that time, the researcher has interviewed about 50 foreign postgraduate students and asked them about the proficiency test. It appears that a large number of the students (65.2%) are dissatisfied with the writing and reading component of the test and some of their views are stated below:

1. The content of the reading passage is biased toward disciplines such as the Biological Sciences, Computer Sciences and Economics.
2. It is difficult to understand the abstract nature of the reading selection, as it is not within my area of specialisation
3. The time allocation for Paper I (2 1/2 hours) is too lengthy.
4. The writing tasks are directly linked to the reading passage. So, if one has little knowledge about the content area, one cannot write a good essay on it.

However, most of the students (68.3%) were pleased with the listening and speaking component of their test. These proficiency tests were developed by individual staff members at the Centre for Languages, USM or else they have taken the form of structured comprehension and writing exercises from various ESL methodology books. In most instances, different staff members worked alone on the proficiency test each semester and designed the test on their own without consultation with other staff members.

The situation outlined is not the entire picture. Skehan (1991) also raised some of the human and practical considerations that aptly apply to the foreign postgraduate proficiency testing scene in USM. He says that language testing currently is "excessively characterised by a reliance on individual endeavour, not because of determined isolationalism or a quest for personal glory, but because of a variety of almost circumstantial and institutional pressures" (Skehan, 1991:6). Indeed a main reason why the proficiency test has been open to criticism by the students is because only one individual is responsible for constructing, administering and scoring the test at any one time and the common practice has been to administer the same prepared test to the students.

The twenty first century has dawned with several major initiatives in the assessing of English proficiency for use in further study or occupational purposes. Weir (1988) developed a test of English for Academic Purposes (EAP) for a target group of aspiring students at British universities by providing a model of how empirical and theoretical work can be combined to make a test format more authentic and appropriate. Elsewhere, Olaofe (1994) designed an EAP test for students of the Ahmadu Bello University in Nigeria, using the characteristics of good EAP tests (Weir, 1993; Robinson, 1991). Skehan (1991) believes that Bachman’s (1990) model of communicative competence and performance will continue to be of pivotal importance in the years to come as it is theoretically well-founded and comprehensive and it tries to be consistent with a wide range of applied linguistic theorising.

The problems in assessing the English language proficiency of the foreign postgraduate students in USM may be examined under three headings -- use of English at the university, proficiency and assessment. Examining the use to which students are likely to put their English involves an analysis of the purposes of their study and their needs in terms of English use. The crucial question relating to proficiency is "What is proficiency?" and assessment considerations are concerned with validity, reliability, test techniques and interpretations of scores (Brown, 1996, Tierney, 1998). top

Use of English among Foreign Postgraduate Students in USM

The type of course to be followed

Foreign postgraduate students undergo various types of programmes at the Masters or PhD levels. Those who opt for taught courses (coursework) have to attend lectures and undertake much intensive reading in English to gain information and participate in tutorials, seminars and face-to-face discussions with tutors. Writing skills are also needed. Assessment is also in the form of written examinations, frequently involving the writing of short essays in limited periods of time both during and at the end of semesters.

In research and mixed mode programmes, postgraduate students may have fewer taught courses than undergraduate students but their courses may incorporate practical work and end with the writing of a minor dissertation over a stipulated period of time. In research modes, the postgraduate student may have little or no direct teaching but may have to do a lot of extensive reading, perhaps laboratory work, one-to-one discussion with a supervisor and culminate in the writing of a dissertation over a period of time, during which help with the language may be available.

Knowledge of the types of courses or programmes the foreign postgraduate students are following and the linguistic load they carry can provide a useful list of the communicative and other linguistic tasks to sample when the assessment procedures are prepared.

The disciplines

Information on the disciplines that the foreign postgraduates are following will indicate the topic areas and domains of discourse from which language samples for the tests may be stipulated. Common disciplines that the foreign postgraduate students follow are management studies, humanities, social sciences, computer sciences and pharmaceutical sciences.

Social purposes

A foreign postgraduate student coming to Malaysia, a multiracial non-native English and Malay- speaking country, will make every day social contacts. The student will communicate with fellow course mates from other countries mainly in English, and will have contact with Malaysians in connection with accommodation, travel, shopping and daily needs.

Nature of the use of English

A needs analysis along the lines discussed will lead to a consideration of the prospective students’ use of English. The theoretical models advocated by Holliday & Cooke (1983) and Hutchinson & Waters (1987) will prove useful in identifying the language needs of the students. Holliday and Cooke’s model of language needs analysis reveals the growing consciousness of the importance of ‘cultural alignment’ in the field of applied linguistics. Their model has an ongoing interaction between the observational approach and the subject of observation (the ecosystem). Hutchinson & Waters’ (1987) model stresses that when a needs analysis is carried out, both target situation needs and learning needs must be taken into account and they look at the target situation in terms of ‘necessities’, ‘lacks’ and ‘wants’. It is crucial to ensure that an analysis of needs probe into the fabric of interweaving factors that comprise the learning situation. In this light, adopting an ecological approach to needs identification will assist the researcher in compiling a comprehensive list of the possible communicative functions that are at play in the learning continuum.

Proficiency

The central problem in assessing proficiency is determining what that proficiency is. As a concept it needs to be defined both conceptually and empirically (De Avila, 1997). Language researchers openly acknowledge that there is a lack of consensus in the assessment of language proficiency.

Language proficiency tests have been developed based on a plethora of definitions and theories (Vecchio & Guerrero, 1995). Moller’s (1980) model of a proficiency box comprising areas such as linguistic systems and skills, context, overall proficiency and communicative use reformulates what ‘proficiency’ is when it is linked to use in certain specific situations.

In the 1980’s, Oller’s Unitary Competence Hypothesis exerted a dominant influence but it was supplanted by the more wide-ranging model of Canale & Swain (1980). Then, it was replaced by Bachman’s model of communicative competence and performance. The Bachman model (Bachman, 1990) has the following general structure:

The Council of Chief State School Officers’ (CCSSO, 1992) model of English language proficiency rests on the students’ ability to function in the four language skills of reading, listening, writing and speaking and the need to assess each of these four skills.

Canales’s (1994) definition of language usage (proficiency) is predicated on a socio-theoretical foundation. Canales says that language usage is:

• dynamic and contextually-based (varies depending upon the situation, status of speakers, and the topic);
• is discursive (requires connected speech); and
• requires the use of integrative skills to achieve communicative competence.

Each of these educationally driven conceptions/definitions of language proficiency share at least two critical features. First, each definition accommodates the four linguistic modalities: speaking, listening, reading and writing. Second, each definition places language within a specific context, in this case the educational setting. Consequently, an English proficiency test should utilise testing procedures that replicate – as nearly as possible-- the kinds of contextualised language processing that is used in mainstream English speaking classrooms (Vecchio & Guerrero, 1995). top

Assessment

Validity

Validity is the link between the problem of the nature of proficiency and other technical problems of assessment. Brown (1996:231) defines validity as "the degree to which a test measures what it claims, or purports, to be measuring". Teachers certainly want to base their admissions, achievement and diagnostic decisions on tests that are actually testing what they claim to test. Hence, test developers should consider the following aspects of validity if they wish to develop good language tests:

• Content validity (whether the test is a representative sample of the content of whatever the test was designed to measure)
• Criterion-related validity (how far results on the test agree with those provided by some independent and highly dependable assessment of the candidates’ ability)
• Construct validity (if it can demonstrate that it measures just the ability which it is supposed to measure)
• Face validity (if it looks as if it measures what it is supposed to measure).

Reliability

According to Vecchio & Guerrero (1995), reliability is the degree to which a test or assessment consistently measures whatever it measures. Discrete-point tests generally achieve high reliability whereas integrative tests normally possess somewhat lower reliability (Moller, 1980). Communicative performance tests at a fairly high level of proficiency tend to attain even lower reliability. To increase reliability of such tests, considerable efforts must be taken by the testers to specify explicitly the tasks to be performed, define the criterial performance against which performance is to be assessed and how scorers will be trained.

To be valid, a test must be reliable. In our efforts to make tests reliable, we must be wary of reducing their validity. There will always be some tension between reliability and validity and a tester has to balance gains in one against losses in the other (Hughes, 1989; Tierney, 1998).

Interpretation

The purpose of developing language tests, administering them and sorting through the resulting scores is to make decisions about our students. This process is sometimes called test score interpretation. Students perform the same tasks on a test but the range of performance within any one group may be very wide. The system frequently used is to assign a student’s performance to a particular level of performance band, according to definitions of individual levels. Thus, a given student may emerge with one level for his reading test performance, a different level for his written communication and yet another level for his oral communication. In short, the obtained profile may show that further learning has to take place in only one or two of the skill areas and not in all. top

Conclusion

Having discussed the problems in assessing proficiency in English among foreign postgraduate students in USM, this paper also aims to bridge the gap that exists in this proficiency test setting by putting forward some suggestions:

Firstly, developing better assessment practices within any educational setting requires more than choosing a new test or adopting a packaged informal assessment procedure. In this light, much effort must be made to carry out a comprehensive language needs analysis among the foreign postgraduate students in USM. It need not be very far ranging but it should encompass the communicative and linguistic events that will be encountered by the students. The findings can then have a direct bearing on the language content of the measures being developed and evaluated, and on the tasks to be set on the test.

Secondly, learning possibilities should be negotiated with the target group of students and stakeholders rather than imposed via standards and assessments that are preset, prescribed, or mandated (Tierney, 1998). As partners of the learning process, the foreign postgraduate students, the test developers and other stakeholders can discuss which view of proficiency can be utilised for the specific context and on the basic design of the test. The exact structure of the test would be dependent on the results of the analysis carried out on the students’ language needs.

Thirdly, instead of depending on an individual’s effort in devising and constructing the test, a team effort will yield better results. The test developers can work collaboratively on deciding issues such as test content, appropriate item formats, specific test objectives, writing test items, revising test items and administering the test.

Fourthly, it would be desirable if more raters (two or more) could be involved in assessing the students’ oral communication to increase the reliability of the test. Students’ raw scores can also be converted to descriptive levels on a scale for each of the language skills being tested. Minimum attainment levels (with consultation with course supervisors and other stakeholders of the learning situation) have to be established.

Also, a few proficiency test batteries have to be prepared so that the same test is not recycled every time. Test developers require sufficient time and resources to develop good tests and policy makers and program administrators may need to be convinced of this, since they may not be aware of how time-consuming assessment can be, particularly when it involves the construction of formal tests (Brindley, 1997).

Finally, statistical analyses can help in analysing a set of language test items. They can be helpful to testers in understanding and improving the effectiveness of item formats and content (carrying out an Item Facility analysis, Item Discrimination analysis and a distractor efficiency analysis can yield very useful information on multiple-choice items).

Although these conclusions are of necessity tentative, I believe they could form the basis for further improvements in the existing proficiency testing development at the Centre for Languages in USM.

__________________________________________________________________

Table 1: Profile of foreign postgraduate students in University Science Malaysia (USM) in the academic session 1999/2000

Table 2: Distribution of foreign postgraduate students according to schools and programmes of study in the academic session 1999/2000

References

Bachman, L. 1990. Fundamental Considerations in Language Testing. Oxford: Oxford University Press.

Brindley, G. 1997. Assessment and the Language Teacher: Trends and Transitions. The Language Teacher Online: http://langue.hyper.chubu.ac.jp. September 1997.

Brown, J.D. 1996. Testing in Language Programs. New Jersey: Prentice Hall Regents.

Canale, M. & Swain, M. 1980. Theoretical bases of communicative approaches to second Language teaching and testing. Applied Linguistics, 1(1): 1-47.

Canales, J. A. 1994. Linking Language Assessment to Classroom Practices. In R. Rodriguez,

Ramos, N & Ruiz-Escalante, J.A.(Eds) Compendium of Readings in Bilingual Education: Issues and Practices. Austin, TX: Texas Association for Bilingual Education.

Council of Chief State School Officers. 1992. Recommendations for Improving the Assessment And Monitoring of Students with Limited English Proficiency. Alexandria, VA: Council Of Chief State School Officers, Weber Design.

De Avila, E. 1997. Setting Expected Gains for Non and Limited English Proficient Students. NCBC Resource Collection Series No.8. NCBC Home Page: http://www.ncbe.gwu.edu

Holliday, A. & Cooke, T. 1983. An Ecological Approach to ESP. In Alan Waters (ed), Issues in ESP. Oxford: Pergamon Press. P124-143.

Hughes, A. 1989. Testing for Language Teachers. Cambridge: Cambridge University Press.

Hutchinson, T. & Waters, A. 1987. English for Specific Purposes: A Learning-centred Approach. Cambridge: Cambridge University Press.

Moller, A.D. 1980. English for use in further study. In John A.S. Read (ed) Directions in Language Testing. Anthology Series 9, SEAMEO Regional Language Centre: Singapore University Press.

Olaofe, I. 1994. Testing English for Academic Purposes (EAP) in Higher Education. Assessment & Evaluation in Higher Education, 19 (1): p37.

Oller, J.W. Jr. 1979. Language Tests at School: A pragmatic approach. London: Longman.

Robinson, P. 1991. ESP Today: A Practitioner’s Guide. New York: Prentice-Hall International English Language Teaching.

Skehan, P. 1991. Progress in Language Testing: the 1990’s. In Alderson and North (ed). Language Testing in the 1990s: The Communicative Legacy, p 3-20. London: Modern English Publications and The British Council.

Tierney, R. 1998. Literacy Assessment Reform: Shifting Beliefs, Principled Possibilities, and Emerging Practices. Reading Teacher, 51 (5):p374.

Vecchio, A.N. & Guerrero, M. 1995. Handbook of English Language Proficiency Tests. NCBC Online Library. December 1995. NCBC Home Page: http://www.ncbe.gwu.edu

Weir, C.J. 1988. Communicative Language Testing. Exeter Linguistics Studies No. 11: University of Exeter.

Weir, C. 1993. Understanding and Developing Language Tests. United Kingdom: Prentice Hall. top

About the author

Sarjit Kaur
School of Humanities
Universiti Sains Malaysia
Penang, Malaysia.

E-mail: sarjit@usm.my