University of Rijeka, Croatia

Keywords: Student assessment, quality teaching, competency in teaching, student learning, university staff development, University of Rijeka, higher education, case study.

Article style and source: Originally presented at ECER 98, Ljubljana, September 17-20.

Contents

• Background
• Method
• Result
• Discussion
• Conclusion
• Acknowledgements
• References

Background

The universities usually do not take a proactive position in the evaluation procedure. According to the Higher Education Law (Art.134), the testing of the necessary level of quality and efficiency of teaching, and of scientific and professional activities is to be evaluated every five years. The evaluation will be initiated by the Ministry and the evaluation process carried out by expert commissions of the Ministry, their members being appointed by the minister on the recommendation of the National Council for Higher Education. Except for self-analysis, the universities play a passive role in the process of assessment, which has strong "top-down" direction with the summative evaluation stressed.

Although universities in Croatia in most cases show little interest to improve the quality of teaching, attempts for improving university teaching and learning may be found. Beginning in 1990, the Ministry of Science funded a research project "The Conditions and Criteria for Effectiveness of University Teaching". The results (Lediç, 1990; 1992b; 1995) have shown that university professors in Croatia did not recognise the need for their development, while at the same time many of them have only a very limited knowledge of higher education teaching methods. Teaching at Croatian universities is in most cases traditional, because teachers mostly follow their own educational experience and are not given any information about more effective teaching methods. In addition, university professors have expressed doubt concerning staff development as a factor for improving teaching and learning. Motivation for staff development is weak. On the other hand, students expressed dissatisfaction with the quality of teaching. It may be concluded that university teaching in Croatia has serious problems. It is obvious that the indicators of the quality of teaching have been negative: dissatisfied students, traditional teaching, lack of an appreciation of the need for improvement, doubts concerning educational theory and practice as a mean for better teaching. This led us to the conclusion that a model for improving teaching and learning at the universities in Croatia should be proposed and applied.

In the study previously mentioned, some recent approaches to the improvement of teaching and learning at the universities were examined (TQM, reflective practitioner approach), as well as different models of staff development (Lediç, 1993; 1994). The conclusion was, that in order to improve teaching and change the institutional culture towards a more responsible attitude towards teaching, it is necessary to develop and apply an adequate university staff development model according to international standards, but adapted to Croatian traditions and circumstances. In 1995 the project "The Quality of Teaching in Higher Education" was proposed to the Republic of Croatia Ministry of Science and Technology for funding. The aim of this research is to explore internationally accepted criteria of quality concerning higher education (especially those closely connected to teaching competencies of higher education teachers), to find out the extent to which these criteria are respected in Croatian higher education, and, finally, according to these results, to develop a model of university staff development, which aims to build up a "culture of quality". The results of the 1995 project and experience from actual practice indicate that:

1. quality criteria, as well as higher education teaching practice in Croatia, differ significantly from the criteria accepted in international literature and practice;
2. the quality of teaching (seen primarily through student evaluations), is not satisfactory. content

Method

1. to explore the experts' teachers' and students' opinion about the general importance of the criteria (assessment of the relevance and contribution of each criterion to successful university teaching), and;
2. to explore the teachers' and students' opinion on the extent to which every criterion is respected in actual university teaching in Croatia (teachers were assessing their own teaching practice, and students were assessing their overall experience at university).

• Teaching objectives are clearly defined.
• Students' interests are encouraged.
• Teachers are well prepared for their teaching.
• Teachers are experts in their subject.
• Teachers are enthusiastic for their subject.
• Teachers emphasise important parts of the subject.
• Teaching methods are used which encourage active cooperation by students. Active and independent student learning is encouraged.
• Teachers respect students' individual differences.
• Teachers ask for feedback.
• Teachers respond to students' feedback.
• The assessment of students is fair and reliable.
• Individual courses are designed so as to contribute integrally to a student's subject of study.
• Books and other resources are available.
• Teachers show readiness to help.

1. the difference between "ideal" (assessment of the relevance and contribution of each criterion to successful university teaching ) when teachers' and students' opinions are compared; and,
2. the difference between "real" (assessment of the extent in which every criterion is respected in actual teaching at the university) when teachers' and students' opinions are compared.

The sample of the teachers and students was chosen at the University of Rijeka. This University incorporates: The Faculty of Economics (ECO Rijeka), The Faculty of Economics and Tourism in Pula (ECO Pula), The Faculty of Civil Engineering (CIVIL ENG), The Faculty of Hotel Management (HOT), The Faculty of Medicine (MED), The Faculty of Education in Rijeka (ED Rijeka), The Faculty of Education in Pula (ED Pula), The Maritime Faculty (MARI), The Faculty of Law (LAW), and The Faculty of Engineering (ENG).

For the teachers, a personalised covering letter accompanied every questionnaire and a self-addressed envelope was included. The questionnaires were anonymous, but the opportunity was provided to fill in name and address for future communication and collaboration. Since the questionnaires were anonymous, we did not have the opportunity to re-send them. The questionnaires were sent to every teacher from every school at the university; except that only a sample was chosen in the Faculty of Medicine, as the number of the staff in that faculty greatly outnumbered those in other faculties, and we did not wish to bias our results towards a particular discipline.

For the students we used a different approach. Our intention was to question students who attended their third year of study at the university. The third year was chosen because we assumed that these students had enough experience of university teaching. At the same time they were far enough from finishing their course work so they might be motivated to be involved in the process of university teaching improvement. With the agreement and help of the deans of every school at the university, precise appointments were made for completing the questionnaires and usually students were given about 10 minutes to fill them in. This may have resulted in some of students, who needed more time, not being in a position to think seriously about all the questions. content

Results

a) Participants

Table I shows the participation of teachers and students in the research. The percentage of the questionnaires received back from the teachers is 35%, which might be considered as sufficient for the analysis. Since the questionnaires were anonymous, we did not re-send them. The table also shows the number of interviewed students from the different schools.

SCHOOLS	ED Ri	ED Pula	MED	>ENG	CIVIL ENG	ECO Ri	LAW	MARI	HOT	ECO Pula	TOT
TOTAL SENT TEACHERS	120	44	86	72	35	44	28	46	55	33	563
TOTAL BACK TEACHERS	56	14	28	22	11	10	12	12	23	15	203
% BACK TEACHERS	47	32	33	31	31	23	43	26	42	45	35
STUDENTS ENROLLED IN THE 3RD Y.	222	105	159	63	61	331	90	116	172	130	1449
TOTAL QUEST. STU.	146	34	29	19	26	72	24	26	58	35	469
% QUEST. STUDENTS	66	32	18	30	43	22	27	22	34	27	32

TABLE I. Sample: Teachers and students

Previously mentioned results considering staff development research in Croatia show lack of interest in this problem. Bearing this in mind, we considered the response rate of 35% sufficient for analysis. The variation in response rate across the Faculties ranges from 23% (ECO Rijeka) to 47% (ED Rijeka). It might be considered that teachers from the School of Education in Rijeka are the most interested in the problem, because for many of them education is their profession, or perhaps they just responded more frequently than others because of "loyalty" for the research team coming from the same School. However, ED Rijeka should not be specially treated, because their response rate (47% ) is not significantly higher than ECO Pula (45%), LAW (43%) or HOT (42%). The lowest respond rate at ECO Rijeka (23%) could hardly be explained by the discipline, especially when compared to higher respond rate of ECO Pula.

However, we consider the student questionnaire results more significant, particularly as they point at the problem of university teaching in Croatia. The figure of 32% among students is actually not the response rate but the percentage of questioned students in the third year of their studying. Although the research team was trying to reach the students in their classrooms several times, and in some cases the questionnaires were even left for missing colleagues to be filled out at home, we were not able to get higher response rates, with extremely high variations, ranging from 18% (MED) to 66% (ED Rijeka). We have to point out again that our visits across Faculties were scheduled by the deans, who took special care that our visits were arranged at the time when most students attended classes. This serious difficulty showed us that students simply do not attend classes, which means that the students' teaching evaluation should be taken with caution. On the other hand, it is reasonable to assume that non-attenders were even less satisfied with the teaching than attenders.

b) The analysis of the criteria

The 15 criteria were presented to the teachers and to the students according to two aspects: "ideal" (assessment of the relevance and contribution of each criterion to successful university teaching ) and "real" (assessment of the extent to which each criterion is respected in actual teaching at the university). The first step in our analysis was to count the percentage for the each criterion according to the teachers and students and their assessment of "ideal" and real". The further step in the statistical analysis was ranking each criterion. These results are presented in Table II.

CRITERIA	IDEAL				REAL
	MEAN		RANK		MEAN		RANK
	TEA	STU	TEA	STU	TEA	STU	TEA	STU
1. Teaching objectives are clearly defined.	4,61	4,37	3	7,5	4,32	3,27	7	3,5
2. Students' interests are encouraged.	4,31	4,42	10	5	3,88	2,53	11	12
3. Teachers are well prepared for their teaching.	4,81	4,50	1	3	4,46	3,48	3	2
4. Teachers are experts in their subject.	4,78	4,65	2	1	4,45	3,93	4	1
5. Teachers are enthusiastic for their subject.	4,24	4,09	11	11	4,21	3,12	8,5	5,5
6. Teachers emphasise important parts of the subject.	4,41	4,39	7	6	4,44	3,27	5	3,5
7. Teaching methods are used which encourage active cooperation by students.	4,18	4,07	12	12	3,82	2,43	12	13
8. Active and independent student learning is encouraged.	4,57	4,24	5	10	4,21	2,70	8,5	8
9. Teachers respect students’ individual differences.	3,59	3,88	15	13	3,38	2,16	15	15
10. Teachers ask for feedback.	4,01	3,71	13	15	3,78	2,67	13	9,5
11. Teachers respond to students' feedback.	3,82	3,79	14	14	3,62	2,32	14	14
12. The assessment of students is fair and reliable.	4,60	4,61	4	2	4,53	2,67	2	9,5
13. Individual courses are designed so as to contribute integrally to a student's subject of study.	4,39	4,25	8	9	4,36	3,12	6	5,5
14. Books and other resources are available.	4,37	4,49	9	4	3,91	2,59	10	11
15. Teachers show readiness to help.	4,42	4,37	6	7,5	4,56	3,06	1	7

TABLE II: Teacher' and Students' differences between 'ideal' and 'real' (means and ranks) content

Discussion

Although our research has shown that the image of quality teaching according to teachers and students is almost the same, their opinions differ when the extent to which each criterion is respected in actual teaching at the university is assessed. Here teachers show that the main quality features of their actual teaching are: their readiness to help (1), fair and reliable assessment (2), and well prepared teaching (3). The least important criteria are the same as for the ideal case: to respect students' individual differences (15), respond to students' feedback (14) and to ask for feedback (13). Students think that: teachers really are experts in their subjects (1), are well prepared for their teaching (2), emphasise important parts of the subject (3.5) with clearly defined objectives (3.5). Students and teachers are in agreement regarding the least respected criteria. However, it should be noted that students do not agree with teachers in their high assessment of their readiness to help (ranked 1 for teachers and 7 for students) and fair assessment (ranked 2 for teachers and 9.5 for students). This may serve as an indication that teachers' and students' perception about the quality of "real" teaching differ.

It is significant that the criteria where the biggest difference among ranks is found are those related to students (more than to teachers or to subject matter). This leads us to the conclusion that students are, in a way, the most neglected part in the teaching.

The Mann-Whitney's U test (Table III) confirms the differences between teachers' and students' assessment of "ideal" and "real" teaching: it shows that statistically significant difference could not be found for 8 criteria, when "ideal" teaching is considered. However, when teachers' and students' assessment of "real" teaching is examined, there is statistically significant difference in every criterion.

	IDEAL			REAL
CRITERIA	RANK (TEA)	RANK (STU)	Z	RANK (TEA)	RANK (STU)	Z
1. Teaching objectives are clearly defined.	3	7,5	3,17	7	3,5	13,20
2. Students' interests are encouraged.	10	5	1,72	11	12	14,90
3. Teachers are well prepared for their teaching.	1	3	5,03	3	2	12,56
4. Teachers are experts in their subject.	2	1	2,06	4	1	7,41
5. Teachers are enthusiastic for their subject.	11	11	1,62	8,5	5,5	12,53
6. Teachers emphasise important parts of the subject.	7	6	0,82	5	3,5	13,53
7. Teaching methods are used which encourage active cooperation by students.	12	12	1,33	12	13	14,01
8. Active and independent student learning is encouraged.	5	10	4,58	8,5	8	14,92
9. Teachers respect students’ individual differences.	15	13	3,20	15	15	11,89
10. Teachers ask for feedback.	13	15	3,68	13	9,5	11,97
11. Teachers respond to students' feedback.	14	14	0,47	14	14	13,32
12. The assessment of students is fair and reliable.	4	2	0,81	2	9,5	17,05
13. Individual courses are designed so as to contribute integrally to a student’s subject of study.	8	9	1,99	6	5,5	13,69
14. Books and other resources are available.	9	4	1,96	10	11	12,44
15. Teachers show readiness to help.	6	7,5	0,94	1	7	15,28

TABLE III: Mann-Whitney U Test (Teachers' and students' assessment of "ideal" and "real"; significance level Z=1,96; P<0,05) content

Conclusions

The results of the Mann-Whitney U test show that teachers and students do not substantially differ when "ideal" teaching is assessed: the difference is statistically significant in only 7 criteria (out of 15). However, when we come to the assessment of "real" teaching, then the results are very different: there is a statistically significant difference in every criterion.

Questions concerning students' reliability in the assessment of teaching could be raised, which would prevent us from drawing firm conclusions from our results. What makes us treat the argument for taking students' assessment of "real" teaching seriously is the fact that the students and the teachers mostly agree when their vision of "ideal" teachers is examined. If we take students' assessments in this part as reliable, we may do the same when the "real" is assessed. We believe that students' assessment in this case could be understood as an indicator that teaching at this university does not adequately respond to the quality criteria, and is not favourable to students. We consider that our results must be taken as a serious warning that some actions for improving the quality of university teaching should be taken. Our study should therefore be followed by another to examine teachers' needs in the improvement of teaching and to establish a continuous process for the improvement of teaching in universities.

Finally then, we have perhaps to accept that at present staff development activities in Croatia are still in a "pre-historical" phase, but that a way forward might be to start by using staff development to reduce the gaps between "ideal" and "real". At the same time we should try to make the "ideal" closer to what is recognised as such by the international university community. content

Acknowledgements:

This study was supported by the Republic of Croatia Ministry of Science and Technology (project no.: 00913) and Research Support Scheme of the OSI/HESP, grant No.: 43/1997. Special thanks for his advice and support go to Professor Lewis Elton, Higher Education Research and Development Unit, University College London.

References:

ANDREWS, J. et al. (1996) The teaching and learning transaction in higher education: a study of excellent professors and their students, Teaching in higher education, 1, pp. 81-103.

BOOTH, A. & HYLAND, P. (1996) History in higher education (Oxford: Blackwell Publishers)

CHICKERING, A.W. et al. (1987) Seven principles for good practice in undergraduate education, AAHE Bulletin, 39, pp. 3-6.

COX, B. (1994) Practical pointers for university teachers (London: Kogan Page)

CVETEK, S. (1993) Visoksokolski kurikulum. Strategije na‹rtovanja, izvedbe in evaluacije studentskih programov (Maribor: Dialog)

LEDIÇ, J. (1990) University staff development: proposals for cross-cultural dialogue. Pedagoski rad. 45(4), pp. 438-443.

LEDIÇ, J. (1990) Pedagogic education of university teachers. International newsletter, 12, pp. 29-32.

LEDIÇ, J. (1992a) An approach to university staff development. Napredak, 133(2), pp. 195-199.

LEDIÇ, J. (1992b) Staff development for improving teaching and learning in Yugoslavia: Some ideas for a cross-cultural dialogue with West-European units for staff development and research into higher education, in: B.BERENDT & J.STARY (Eds.) Advanced study programmes for key persons and cross-cultural dialogue North-South-East-West (Frankfurt am Main: Peter Lang)

LEDIÇ, J. (1993) Towards new standards in higher education. Napredak, 134(2), pp. 187-196.

LEDIÇ, J. (1993) Towards new standards in higher education, Part 1 & Ïivotiskola, 42(2), pp. 153-160.

LEDIÇ, J. (1994) TQM - recent approach to higher education quality, Napredak, 135(4), pp. 456-465.

LEDIÇ, J. (1995) Faculty (staff) development for improving teaching and learning in higher education: Croatian experience. ERIC database, doc.no. ED 383 244, 9 May 1995.

MARENTIÂ-POÏARNIK, B. et al. (1995) Izziv raznolikosti (Nova Gorica: Educa)

McKEACHIE, W.J. (1986) Teaching tips: a guide book for the beginning college teachers. 8th.ed. (Lexington, Mass.: D.C. Heath )

PENNER, J.G. & CENTRA, J.A. (1993) Reflective faculty evaluation: enhancing teaching and determining faculty effectiveness (San Francisco: Jossey-Bass) content

About the Authors