Student perceptions of the use of online learning technology in their courses.

Author: John Kenny

pdf version

Professional development team, Learning Technology Services, RMIT University, Australia.

Keywords: Online learning, staff development, student feedback, professional development.

Article style and source: Peer reviewed, original ultiBASE publication.

Contents

• Abstract
• Introduction
• Literature Review
• Methodology
• Data Analysis
• Prior experience
• Use of tools
• Educational issues
• Online testing
• Interaction and communication-feedback
• Overall effects on learning
• Statistical Analysis
  • The experience of fulltime and part-time students
  • The overall effect on learning
  • Students willingness to do more courses online
  • Staff Development Issues
  • Technical issues
  • Access
  • Reliability
• Conclusions
• References

Abstract

As a key part of the implementation of the RMIT Teaching and Learning Strategy (1998-2000), the RMIT Distributed Learning System (DLS) was launched in 1999. Since then, the use of the DLS has grown rapidly. Student feedback data is needed to constantly improve the system to ensure the student learning experience is positive and to inform staff development programs and course designers.

This study reports on the first successful attempt to obtain a feedback from a substantial number of the student users of the DLS. 620 students responded to the questionnaire about their perceptions of the effect on their learning of using the DLS. The data was gathered using an online questionnaire accessible from the students' DLS login page. This investigation will form part of a longitudinal study of student feedback concerning the effects of online technology on student learning patterns at RMIT.

Introduction

At RMIT, the Distributed Learning System (DLS) has been in place since 1999. The DLS is a suite of web-based learning tools integrated behind a secure portal which can be used to develop and deliver online courseware. The staff decide, with guidance, which is the most appropriate tool for their needs. The DLS is centrally supported, maintained and funded. A suite of four tools was offered by the DLS at the time of this survey: BlackBoard, WebBoard, WebLearn, QM perception. Alternatively, staff could also create their own website and upload it to the DLS servers. The list of tools are commercial products except for WebLearn, which is a web-based assessment tool developed at RMIT in the faculty of Applied Science.

Several evaluations of the impact of the DLS have been carried out since it was launched, and some of these have been reported in the literature. (McNaught et al. 1999), (Kenny, 2000) and Kenny 2001). Each of these early evaluations incorporated student feedback, but, in the early days, there were only small numbers of students involved. Feedback was obtained indirectly from help desk messages, a small number of questionnaires and some focus groups. The educational questions associated with the use of the DLS technology tended to be overshadowed by the early technical issues.

As the DLS settled and technical issues became less predominant, an effort was made to reach a larger number of the students by developing an online questionnaire. The aim was to get feedback from as many students as possible, on how using the DLS had affected their learning. By September 2001, about 10 000 students were using DLS on a regular basis per month, a regular user being defined as a student accessing the DLS at least 5 times per month. This figure has increased to about 10 000 and 13 000 per month at the time of writing. back

Literature Review

The literature on student feedback with online learning tends to vary quite a bit in the contexts being studied. Brace-Govan and Clulow (2000) observed that there is a "paucity of studies about how students actually experience online learning", on the basis of this review, it was revealed that this is statement is particularly true for institution wide or longitudinal studies of student feedback. Most studies in the literature are concerned with a group of students in a particular course. They are usually comparative studies in which a course was delivered in two parallel modes, one being an online mode and the other a totally face to face mode.

One study involving a large number of students is a longitudinal study by Palmer and Bray (2001). They were concerned with the computer use habits of 325 engineering students. They conducted three surveys between 1998 and 2001. They found that student access rates to computers was consistently very high. This might be associated with the fact that, as engineering students, they are likely to have been keen users of technology. They also found that the proportion of students indicating 'Home' as the source of their internet access rose by more than 60% over the this time. The number describing themselves as regular users rose by 70 % in the same period. They did not explore student perceptions of how using computers affected their learning.

In their exploratory study, Brace-Govan and Clulow (2000) recorded student perceptions of online learning for 14 students in an undergraduate marketing subject. The students had volunteered to do the subject in a totally online mode. In particular they explored student perceptions around their expectations of learning online, the levels of communication with staff and others in the course, and how the technology affected their work patterns.

Students reported that they had volunteered to participate because of the 'novelty effect' and their expectations that the study mode would be more convenient for them. They also found that the students' perceptions were influenced by what prior experiences they had to compare. Those entering with little experience of 'online learning' expected to have less interaction with students and teachers. The students expressed approval of the organised structure of the learning materials. Some students were critical of response times to their questions in a discussion board, where they had expected to receive an immediate response.

They concluded that students found online learning was an attractive alternative to print based distance education, but it was not so popular as an alternative to face to face classes. For text dense materials, the students preferred hard copy materials over screen text. They identified a need for staff to manage students' expectations of the online learning experience.
Taley-Ongan and Gosper (2000) investigated student feedback in relation to two undergraduate courses which were delivered in face to face mode with web-supported delivery. The feedback involved 320 students from two courses over a two year period.115 students and 107 students in 1998 and 1999 from one course, and 98 students in 1999 from another course.

They found a general increase in student web-skills and positive shifts in their attitudes towards the online learning experience. Both courses originally had a two hour face to face lecture and a one hour tutorial. The one hour tutorial was replaced by a web based tutorial. The students rating of the online tutorial as "satisfactory or higher" rose from 39% in 1998 to 61% in 1999. They also noted an increase from 28% to 61% in satisfaction with the interaction with other students. The rating for the overall learning experience rose from 40% in 1998 to over 80% in 1999. In 1998 only 40% wanted to see more learning units on the web, but in 1999 this rose to about 70%. They concluded that introducing more flexible options to the courses led to this increase in satisfaction, but the results could also be due to a growth in confidence of the staff and students with the online mode.

Benson and de Zwart (2000) studied 10 volunteer law students using a web-based learning system provided by the institution to study a subject in fully online mode. The system provided a website linked to an online conferencing tool and a means to submit assignments. Three of the students subsequently withdrew.

They found that students responded well to the online subject, despite the experiencing technical problems and access difficulties due to firewalls and administrative delays. The flexibility of access to and the design of the resources were positively received by the students. Students also appreciated the contact with the subject coordinator. The students reported that the workload was heavy and difficult to manage. Expectations around the need for self-managed learning skills need to be discussed with students.

From a student support perspective, Taynton (2000) studied the personal issues of students presenting for counseling and assistance with their learning. They conclude that online learning is likely to be most effective when used in conjunction with other proven strategies such as: face to face tutorials.

Felix (2001), collected data from a total of 111 language students. The courses they were involved in came from a variety of institutions and involved a number of different teachers. The teachers in the study had been chosen especially for the high level of teaching skill.
He concluded that the web is a viable environment for language learning, especially as a support for face to face teaching. He also found that student comfort with and enjoyment of using the web increased significantly during their studies. He noted that the quality of the materials used in the study was a factor in this.

Analysing the web materials for how students perceived the usefulness of materials, the key factors were: clear and logically organised content, clear objectives, meaningful feedback and easy navigation. He reports that qualitative feedback identified a number of advantages and disadvantages. (see Table One)

 

Advantages	Disadvantages
Time flexibility Wealth of Information Reinforcement of learning Privacy Ability to repeat exercises Gaining computer literacy	Lack of practice (speaking) Distraction No interaction with peers Inadequate feedback Absence of teacher

Table One: Summary of Qualitative Feedback after Felix (2001)

Stacey and Fountain (2001) reflected on the use of online communication tools to support research students. Many of the students were located interstate and overseas. They observed a trend towards supervision of students remote from the institution and concluded that the traditional supervisory process used with face to face situations will need to be adapted to this new mode of supervision.

They recommend that familiarity with online communication tools will need to be included in induction sessions for future research students. They pointed to the need to have a strong 'social presence' within the online environment to establish trust and to make communication easier. They referred also to the difficulty of establishing discussion forums for research students as opposed to a class, due to the individual nature of their study. They proposed that the establishment of networks to enable peer and expert interactions would address this. Another key factor was the confidence of the student to be able to achieve success within this context.
In a series of case studies Kenny (2001) noted that there is likely to be a transitional period as institutions, teachers and staff make the shift to more 'online' learning. He identified underlying fears that arose in students and the need for these to be accounted for in the change process. These fears include the loss of face to face interaction and lack of skill in use of the technology or software.

He also described a problem in establishing a discussion forum for a course conducted in self-paced mode. The students who surged ahead in the course had no one with whom to discuss issues.

Aspect of the change process which students have to go through with a move to online learning were outlined in focus groups sessions. Many of the students in the online classes still had a conventional view of what teaching and learning as essentially a 'face to face' activity. Some students requested a need for more structure in the course materials. There was also an expression of difficulty in managing and organising time for online classes in comparison to normal classes and criticism of having to deal with large amounts of text on screen. The student feedback from the case studies is summarized in the table below.

Positives	Negatives
accessibility to materials convenience learning valuable IT skills works well combined with face to face sessions	technical problems fear the loss of face to face interaction, lack of skill in use of the technology or software need for more structure in course materials. difficulty in managing and organising time for online classes delivering large amounts of text on screen firewall issues

Table Two: Summary of Feedback Kenny (2001)

Arbaugh (2001) studied student satisfaction in relation to instructor "immediacy behaviours". Immediacy behaviours were defined as behaviours which help people to communicate effectively. In a normal setting, this would include verbal and non-verbal characteristics such as eye contact, smiling, demeanour, use of humour, feedback, etc. Accepting that such behaviour is linked to student motivation, he proposed that immediacy behaviours would have an effect on students involved in remote learning situations. He predicted that verbal behaviours, in particular, could be mimicked in the virtual classroom. Students doing online courses in an MBA program were surveyed.

Arbaugh (2001) concluded that the online learning environment "can in fact reduce the traditional social distance between instructor and student" and that instructor immediacy behaviours did enhance student satisfaction. Such instructor behaviours included: providing personal examples, demonstrating a sense of humour, comfort with the online experience, encouraging expression of ideas and discussion. This is consistent with the need for a "social presence" reported by Stacey and Fountain (2001). He also found that student attitudes to the software medium used to deliver the course affected satisfaction.

Leonard and Guha (2001) surveyed the attitudes of 20 volunteer Early Childhood pre-service teachers who chose to complete online versions of two courses and compared their attitudes to online learning with those of 24 students who studied in the conventional way. They found that students who did the online study were more positive about the effectiveness and potential of online learning to prepare them for teaching careers. This is not surprising as the 24 from the conventional mode class would have been only speculating about online learning. They also noted that the student volunteers had a high level of technical skill to begin with. back

Methodology

Harvey (1998, p.12) identified qualitative evaluation techniques (focus groups, interviews, etc.) as particularly suitable for exploratory studies and to identify possible issues for further investigation. Quantitative studies lend themselves well to the identification of patterns and trends and the analysis of large amounts of data. Since several evaluation activities had been carried out prior to this survey, many of the key issues were already identified. As in the case of "Grounded Theory" proposed by Glaser and Strauss (1967), issues had emerged from the earlier data and a direct bearing on the framing of the questions for the survey.

The likelihood of receiving a large number of responses was another reason that a quantitative survey was preferred in this study. It was felt that students would be more likely to complete simple response questions than extended response questions. For these reasons, the survey was designed using mainly closed question types such as 'Likert' scale questions. This would facilitate the analysis of the data and enable the exploration of possible relationships between the variables. To ensure that some qualitative data could still be collected, a free text response question was added at the end of the survey. It is planned that the survey may be used over several years to collect some longitudinal data.

The questionnaire used in this study was made available to students between April 9th and June 14th 2001, via a link on the DLS login page , accompanied by a brief message inviting students to complete the survey. Any student using the DLS was able to fill it in on a voluntary basis. Anonymity was assured by providing generic login information.

The Likert questions presented students with a set of statements for which they had to choose a response. Each statement asked the students to select from six possible responses: TA= Totally Agree, A= Agree, U= Undecided, D= Disagree, TD= Totally Disagree NA= Not Applicable.

The survey was designed to get feedback on a range of issues concerning the learning experience of the students using the DLS. The questions fell into three categories:

1. Background information
2. Educational issues
• flexibility of the learning,
• opportunities for interaction and communication,
• assessment experiences, administration,
• access to learning resources
• overall affects on learning as perceived by the students
3. Technical issues
   • login and access
     

The aim of this questionnaire was to gain an overview of the range of student concerns when using the DLS and to identify trends and issues. Most students were able to complete the questionnaire within ten minutes. back

Data Analysis

A note about terminology. Recently RMIT terminology changed to comply with a new computerised student management system. In the new terminology, a 'subject' is now called a 'course' and a 'course' is now referred to as a 'program'. For example, "Mathematics 101" would now be called now a course and a "Bachelor of Science" would now be referred to as a program. The new terminology is applied consistently by the author, but in some student feedback quoted, the old terminology is used.

A total of 620 students responded to the survey and 431 of these submitted a response to the extended answer question.

Students were asked to give the code of their online course, but many of them did not know it, or gave an incomplete or obviously erroneous code. Several students offered a number of codes. The course data received is therefore not completely clear on this point. For these reasons, only complete and obviously correct codes were counted. An estimate of the number of courses represented in the data, based on the restrictions mentioned above, is approximately 150. This includes courses from each of the seven faculties at the University.

Table three shows the data for background information on the students. The questionnaire did not collect data on gender or age. Future versions will include these variables to bring it into line with other studies in the literature.

RMIT is a dual sector institution in that it has both a University sector and a Technical and Further Education (TAFE) sector. The DLS is available for staff and students from both sectors. Higher Education (University) students accounted for 94% of the responses and Technical and Further Education (TAFE) students only 6%. This reflects the situation at the time at RMIT, where there had been little use of the DLS by TAFE staff. 79% of the students were full-time. back

Prior experience

When asked to chose a description of their prior experience of using online activities and materials as a part of their program, 45% of the students selected "a great deal" or "quite a lot" as their response. Combined with the 24% who selected "some" as their response, this meant that 69% of the students has had some experience of using online materials as a part of their progra m.

		Number	%		Number	%
3. Are you a TAFE or Higher Education student?	TAFE	35	6%	Higher Ed	549	94%
4. Are you full time or part time?	Full-time	462	79%	Part-Time	120	21%
5. Up to now, how much experience have you had using online activities and materials provided as part of your program?		None 87 15%	A Little 92 16%	Some 137 24%	Quite a lot 181 31%		A great deal 14%

Table Three. Student responses to background questions.

Use of tools

When asked to select from the list which one or more of the software tools available in the DLS were used in their course, students returned the figures in table four.

Table Four: Use of the DLS Tools

The BlackBoard is by far the most used tool as indicated by these results. This corresponds with the DLS monthly user reports which put usage of Blackboard at around 90% but the user figures for the other tools is well above that figures from the user data, all of which are below 10%. The figure for BlackBoard probably reflects the fact that it is the tool which has been supported the most with central training resources and is the most versatile of the tools offered in the DLS. back

Educational issues

In the open response question, the students' comments about using the DLS for the learning were mainly positive. They suggested improvements such as: the need to make the learning materials more flexible to allow more independent learning to occur. There was also strong support among students for the DLS to be used more extensively to support learning.

In the discussion that follows, "Agree" refers to the students who selected "totally agree" or "agree" in response to a question. "Disagree" refers to the students who selected "totally disagree" or "disagree".

 

Question	TA	A	U	D	TD	NA
7. Using the DLS made it easy to work at my own pace.	104 18%	256 44%	84 14.5%	83 14.5%	25 4.5%	26 4.5%
8. Using the DLS reduced the number of time I had to travel to campus.	95 16.5%	173 30%	49 8.5%	121 21%	95 16.5%	42 7%
10. Using the DLS made it easy to discipline myself to complete work.	32 6%	130 22%	134 23%	129 22%	102 18%	48 8%

Table Six: Results for questions on flexibility

Some students were critical of lecture notes going up too late. This restricted their ability to prepare or pre-read the information, particularly if they had limited access to a computer.

Lecture notes were not accessible on time - I could not work ahead when time was available.

A number of students questioned the motives behind the move to online learning expressing underlying fears about the loss of face to face sessions. 40% felt that it was not easy to discipline themselves to complete work and 23% were undecided about this. back

Online testing

Online testing functionality is provided by three of the DLS tools: BlackBoard, QM Perception and WebLearn. The results in Table Seven indicate that the question was "Not Applicable" in 30% of the cases. This could mean that up to 70% of the students experienced online testing as a part of their course. Comparison with the system data on this figure, where the online testing tools are used by between 5% and 15 % of the students, would indicate that this sample of students was skewed towards those who use the system regularly to complete online testing activities.

Using the online testing functions in the DLS is one means of promoting independent study for students by the provision of self-correcting tests with feedback. 60% of the students who experienced online testing agreed that the tests, with feedback, helped them to learn.

Table Seven: Results for question on online testing

There was some criticism when the online testing did not provide students with the answers and/or feedback on the questions they got wrong:

Generally I am very impressed with the online service as it enables you to work at your own pace, and progress forward if you wish. My only criticism is the tests - I feel that you are not really able to judge your learning progress, as you cannot find out results of your test, and where you went wrong. If this was able to be done, then I would say the system is great.

Students also commented on the design of the questions:

Marking on quizzes did not give any indication as to why it was incorrect - suggest giving examples of similar (but different) questions with correct resolutions (hotlink?). This was very important as the quizzes were often significantly more complex than the course material. …Overall, I really liked the concept as it saved me an enormous amount of travel time. back

Interaction and communication-feedback

Table Eight shows the results for questions related to interaction with others and feedback on their progress. As the question of feedback was raised in the previous section it is an important issue from the perspective of the students:

This absolute lack of feedback from the lecturers is a big problem, which must be fixed or the course will fail. ON THE POSITVE SIDE.....THE NOTES AND REFERENCES PROVIDED ARE EXCELLENT AND I DO ENJOY THE COURSE,BUT ARE (sic) CRAVING FOR SOME INTERACTION WITH THE LECTURERS.

The variation in feedback as perceived by the students is indicated below. In question 13, 35% agreed but 37% disagreed that the DLS enabled them to get prompt feedback on their progress.

Table Eight: Questions about interaction and communication.

While for question 11, 48% of students agreed that it was easier to contact their lecturer/tutor 28% disagreed, In response to question 12, only 29% agreed using the DLS made it was easier to discuss difficulties with other students. In question 13, the fact that 57% selected "Not Applicable" and 37% actually disagreed the they could get prompt feedback by using the DLS indicates that this is a key area for improvement. The DLS certainly enables feedback to be given promptly, but it would seem that from these figures, it is not used well enough for this purpose by a number of staff.

Using the online system is alot different to what I expected, coming into this course. I had never expected that most of our work is done online. A great improvement which would help a great deal would be more feedback on assignments and tests. I feel I am not learning to the best of my ability in relation to assignments and tests, as there is no feedback. I never know what answers I did get right and those wrong. Knowing would help in a great way as then I would know what I can improve on.

Active involvement by teaching staff in the website, by means of timely feedback, reports on progress and interaction are important aspects for satisfaction of students in the online courses.

Question 25 asked the students to indicate the sort of online activities in which they were involved while doing their course. They were given a number of responses from which they could select one or more answers. The results are summarized in Table Nine.

Table Nine: Student choices for the activities types in their online courses.

Five of the top six responses were concerned with using the DLS to access information. Accessing lecture notes and course management information such as course guides and documents as well as announcements predominated the reasons given for accessing the DLS. The more interactive communication aspects were selected in only 30% of cases. Other functionality available through the DLS, such as submitting assignments electronically or using the web to research were also low at 25% and 22% respectively. This indicates a predomination at this stage of using the technology as a delivery mechanism for course materials and information.

In relation to obtaining documentation from the web, there were a considerable number of negative comments about courses which provided material with large amounts of text online which students would usually print off:

An overall comment: the system is shifting the cost burden of providing printed documentation to students … by requiring them to print all their own lecture and tutorial notes. This should be explained prior to enrolment because not everyone has ready access to computers and reliable printers to print large documents (it is not tolerated by all workplaces).back

Overall effects on learning

As indicated in Table Ten, for question 19, a clear majority of the students (63%) said they would be happy to do other courses using the DLS based on their experience in their current course. In question18, 55% agreed that having the course on the DLS helped them to learn, but 20% were undecided and 24% disagreed.

Table Ten: Overall effects on learning

That 44% were undecided or disagreed with the statement question 18 indicates that there is room to improve in this area. The results of question 19 discrepancy could indicate that students are generally positive about the experience.

Exploring the student learning experience further, Question 24 asked students to select at least three from a range of adjectives and phrases, to describe their online learning experience. The results are summarized in Table Eleven below.

The top three selections are related to the convenience aspects of using the DLS rather than its ability to add value to the learning experience. 68% of the respondents chose the phrase "Accessible off campus" to describe their course while 53% found it "Useful" and 48% chose the term "Flexible".

However 40% also selected the term "frustrating" in relation to their learning experience using the DLS. The low figure for enjoyment (11%) is a concern, but whether this is significantly different to the figure for classes not using the DLS is not known.

Table Eleven: Adjectives selected by students to describe their online learning experience.

There is a noticeable difference between the 31% who selected "Helped my learning" with the figure in question 18 where 55% either "agreed or totally agreed" with the statement that the DLS helped them to learn. To explore this information more closely, the responses in Table Eleven were matched to responses to the statement of question 18- Overall, having the course in the DLS helped me to learn" (see Table Ten), and the results are shown in Table Twelve.

Table Twelve: Linking “Overall effect on learning” to chosen adjectives.

64% of the students who were positive about the effect of the DLS on their learning (question 18) described their experience of using the DLS as "Useful". A surprising result is that 85% of those who selected "Undecided" also selected "Useful". This might indicate that a lot of students can see the potential of the online technology but that this does not necessarily match with their experience, a stance which was reflected in the responses to the free text question.

40% who were undecided about the effect of the DLS on their learning also selected "Frustrating" to describe their experience of using the DLS. Of those who responded negatively to question 18, 54% selected "Hard to access" and 49% selected "Lonely".

Student comments ranged across the spectrum, but, in general, attitudes to using the DLS were in favour. Quite a lot of students urged a greater use of the DLS. They found it annoying that only some of their courses were online and others were not.

I believe that the system is absolutely terrific it offers me the flexability to stay at home rather then travel the one and a half hours to uni which it would normally take me for every class. I also believe that RMIT in accordance with its flexible learning policy should make it mandatory for all subjects to be made availably through the online system....

However, there is a body of students who do not like the change:

Personally, I hate learning off the Internet and I hope that this does not become the way of the future. I need to hear and see a lecturer explaining concepts. I also do not have adequate Internet facilities, nor enough money to pay for the excessive Internet hours which I would need to clock up. back

Statistical Analysis

The Likert questions used enable the data to be statistically analysed for possible relationships using a chi-square analysis. For ease of analysis, the Likert question responses were reduced from six to three categories. In most cases, the NA category was left out of the calculations.

This reduced the number of degrees of freedom, but ensured there was sufficient data in each category for the statistical analysis to be valid. Various hypotheses were then tested to determine if there were any statistically significant relationships between certain variables. The relationships considered are listed in the discussion below. back

The experience of fulltime and part-time students

The responses of full-time and part-time students were compared to determine if there was any significant difference in their responses to three questions. The results are summarized in Table

Table Thirteen: Chi-squared results for three relationships around full-time and part-time students

At the 5% level, no difference was found between how full-time or part-time students perceived the effects of the DLS on their learning or on their willingness to do another course using the DLS in future. However, there is a clear difference in how they described their experience with using online learning activities. Full-time students were more likely to describe themselves as having had "A great deal" or "Quite a lot " of prior experience with online learning. back

The overall effect on learning

A number of other variables were related to student selections for the overall effects on learning. Table Fourteen summarises the results.

Table Fourteen: results exploring relationships to the overall effect on learning

Very strong relationships were found in each case, with those students who agreed that the DLS helped them to learn also being more likely to agree that:

• they could contact their lecturer
• work at their own pace
• access library resources
• receive prompt feedback
• had enough training to use the DLS. back
  

Students willingness to do more courses online

Table Fifteen summarises the results for how the willingness of students to do more course online using the DLS related to a range of other variables.

Table Fifteen: results exploring relationships to the willingness to do more courses

Again very strong relationships were found with those who were willing to do more courses, also more likely to agree that their prior experience levels were "A great deal" or Quite a lot"

• their perceptions of receiving prompt feedback;
• the DLS reduced their need to travel to campus;
• the DLS made it easier to contact with the lecturer/tutor
• they could work at their own pace;
• the DLS provided opportunities to share and discuss with other students. back
  

Staff Development Issues

Some students criticised the lack of consistency in layout of the materials and also the skills of some lecturers to use the DLS tools effectively. Also a number of students urged staff to make a greater use of the potential of the tools in the DLS.

Each lecturer has set up their subject differently on the system, and uses it to a different extent and for different purposes. RMIT needs to enforce a policy of consistency amongst lecturers on how they use the system, what they call documents and where they are placed on the system. … Many features were also not used - eg putting marks online, establishing pages for sharing documents by groups etc. These would all be beneficial. …

Tools such as BlackBoard offer the advantage that staff can quickly upload learning materials to go online. The question of how to structure and organise these learning materials within is a more difficult one.

I … did cc070 online. This subject was very well designed and applicable to the web. It made study easy and flexible. In contrast XX111 and XX112 (codes changed) have not been designed for online learning, they are just a bunch of notes plonked on the Internet.
The ability to develop meaningful learning activities and to acquire the skills to effectively facilitate in the online environment will require concerted professional development.

The ability to develop meaningful learning activities and to acquire the skills to effectively facilitate in the online environment will require concerted professional development.

The biggest thing that could be done is to train lecturers on how to use the DLS to its full potential. I had four or five subjects on the learning hub and only one of those was used properly and regularly by the lecturers. I think the reason for this is that they simply don't know what all the features are and how to utilise them effectively.

The results indicate that the DLS is still largely being used as a means of delivering content to students. The comments on feedback above indicate that the ability of the DLS to enhance communication between staff and students and between students is underutilized. Many staff may need clearer professional development on how they to structure their site in the DLS and to interact with students online.

It is likely that the concerns of students would be considerably lessened by ensuring appropriate clear guidance for lecturers. Courses which contain well structured materials, higher levels of interaction and prompt feedback on queries are more appealing to students.

These comments point to the fact that many lecturers will start by using only the basic functionality to deliver information and to develop their own confidence. The more interactive and learning activities require a greater level of educational re-design of teaching materials and changes in teaching practice which will not happen without professional development support. back

Technical issues

Technical and access problems were the biggest area of comment by the students in the open response comments. There was confusion around the login procedure and a lot of frustration with having multiple logins to access resources. Many students also complained about the difficulty of access in some computer labs.

The password system is a pain. At home its ok because I can bookmark but on RMIT computers, typing 3 passwords and usernames for access is a nuisance. back

Access

Table Sixteen shows the regular access points reported by the students. It is significant that 82% of students accessed their courses from home on a regular basis. This supports the view that students find the convenience a positive,

Table Sixteen: Student access points for the DLS.

Reliability

Table seventeen summarise the results for question 23. Overall the majority of students (59%) felt that the DLS was reliable and easy to access while (28%) disagreed (Table Sixteen). However, the students who disagreed were often very adamant and forceful in their opinions, which indicates they had very bad experiences and/or have considerable fear and resistance to the change to online delivery.

Table Seventeen: Student responses to the reliability and user friendliness of the DLS.

A lot of the negative comments were fuelled by technical problems, which seemed to happen earlier in the semester. This would have undermined the confidence of the students in the system. The access problems can seriously affect the quality of the experience of the learners. Several students commented on improved reliability later in the semester.

At the begining of semester, the system was not reliable, not well put together, the support was poor and the reponse to complaints inadequate.Performance was totally inadequate.Lack of ownership and accountability between departments was a big problem. As the semester progressed the system appeared to become reliable. It is totally crucial the sytem is reliable from day 1… back

Support

Table Eighteen gives the results to two question concerned with the support received by students. Some attention needs to be paid also to student induction and support to use the DLS. 32% felt that they had not had sufficient preparation to use the DLS.

Table Eighteen: Student responses to training and support of the DLS.

At RMIT, induction is largely the responsibility of the lecturer of a course, but in view of the previous comments criticizing the skill and understanding of the DLS by the lecturers, there is an obvious need for more support from central services.

I also think that it would be good having introductory activities that require you to navigate around the learning site ensuring you become familiar with the site and its resources before starting introductory activities for the course.

The results also show dissatisfaction with the support desk operation, only 18.5% agreed it was a helpful service, 22% disagreed and 26% were undecided. Some commented on the delays in getting a response. There was found to be a strong relationship between the satisfaction with support offered and the students' willingness to do more courses as well as their perceived overall effect on their learning. back

Conclusions

There are a number of themes running through the literature which are also supported in the results of this study. There is a strong relationship between students' confidence and familiarity with online technology and their satisfaction with online courses. Students are also more positive about courses which are designed to be flexible, are clear and easy to follow, allow them to move at their own pace, provide prompt feedback and enable contact with lecturers and other students. The importance of a 'staff social presence', in the form of interaction with and feedback to students is also highlighted as an important factor in student satisfaction with their online experience.

However, some students are clearly suspicious of a perceived trend to reduce face to face learning situations. An assumption that students possess the independent learning and time management skills which maybe associated with online delivery is also to be contested. It is an area where some students will need support. This is particularly true for students whose confidence in the use of online technology is limited. Clearly any move to online learning involves consideration of change management issues around student expectations of about their learning. Their skills, confidence and the understanding of how online learning benefits them will have a bearing on their attitude to the change.

Students' confidence with the technology seems to develop with their experience during a course so much so that they are generally willing to do more courses with online components. Staff deciding to incorporate online learning will need to think about how to manage student expectations and how to induct them into the use of the technology. Obviously some understanding of the student population for a course will be an important component of the course design. Despite students reporting poor online experiences due to technical issues, lack of skills of the facilitator and/or poorly design materials, there seems to be a general willingness by students to engage with the technology. Students seem to appreciate the flexibility and convenience offered by being able to access their materials at a time and place that suits them.

This seems to support the argument for phasing in the use of technology with an existing course so that students can adjust to the use of online technologies. The more prior experience students have with online technologies, the more open they are to it as a part of their course and the more ready they are to extending its use.

Staff skills and understanding of the online learning environment and their effective use of the tools is a key factor in student satisfaction. These results indicate that staff tend to initially use the online environment as a means to deliver course information and materials. Many lecturers/teachers start by using only the basic online functionality to deliver course information, lecture notes etc. This is approach fits well with the stages of adopting an innovation, as outlined by Rogers (1995), in which staff will first try to make sense of the technology. However, while using the technology solely to deliver course content is useful, it may not encourage, on its own, sufficient regular use to build confidence.

Course management tools enable staff to easily upload course content to the web. While this is advantageous at one level, the ability to choose the most appropriate medium for content is also important. A trend to deliver "text-dense" content online was not viewed favourably by students. It was seen as a form of cost shifting, as they invariably tended to print off the materials anyway. Print must still be viewed as the most appropriate medium for such materials, possibly with electronic versions as a back-up.

Staff development which simply addresses training in the mechanics of online tools is insufficient to produce satisfactory online learning activities. Staff developing online learning for the first time need, in addition to the basic training, some supported professional development as they think through how the online tools might add value to their courses and how to best structure the course materials so that they are meaningful to students and have a degree of consistency with other online courses which students might encounter.

Usually there will be a phase of transition. Using the online environment to support face to face teaching without making other significant changes to the course is a common approach adopted. Even at this beginning stage, there needs to be some careful thought about how to incorporate the online environment into a course. Particularly in the situation where staff and / or students are inexperienced users, encouraging some level of regular use would help to build the confidence and legitimise the use of the technology as an integral part of a course.

Regular access to the online environment might be encouraged through the inclusion of relatively simple strategies in addition to the provision of access to content and course information. Such strategies might include: using the technology to post regular announcements or email students, providing samples of solutions to online assessments, allow electronic submission of assignment work, posting of an FAQ section in a discussion board, creation and use of self-help tests with feedback, etc. Staff should aim to create some level of "social presence" even at these rudimentary levels to encourage students to use the site and enhance their online experience.

At the next level, the development of quality, more interactive learning activities which integrate the use of online technology requires more careful educational design. The development of learning activities which include for example, interactive discussion boards, requires a greater level of educational rethinking, increased online management skills and change in teaching practice. This is unlikely to happen without direct professional development support of some kind. This is best done, in the experience of RMIT, with a centrally (or locally) supported mentoring or action learning activities, as staff work through the development process (Kenny, Quealy andYoung, 2002).

It is likely that the concerns of students would be considerably lessened by ensuring appropriate clear guidance for staff. Courses which contain well structured materials, higher levels of interaction and prompt feedback on queries are more appealing to students. back

References

Arbaugh, J. B. (2001). “How instructor immediacy behaviors affect student satisfaction and learning in web-based courses.” Business Communication Quarterly. 64(4): 42-54.

Brace-Govan, J. C., V. (2000). “Varying expectations of online students and the implications for teachers: Findings from a journal study.” Distance Education 21(1): 118-135.

Benson, R., and de Zwart, M. (2000). The Experience of Online Learning: Evaluating the Effectivenenss of an Innovation in Web-based Legal Education. Learning to Choose: Choosing to Learn. Proceedings of the 17th Annual ASCILITE Conference., Coffs Harbour, NSW, Australia., Southern Cross University Press. Pages 425-434.

Felix, U. (2001). “A multivariate analysis of students' perspectives of web based learning.” Australian Journal of Educational Technology 17(1): 21-36.

Glaser, B.G. and Strauss A.L.(1967). The Discovery of Grounded Theory. New York : Aldine Publishing. Company. New York.

Harvey, J. (Ed.), (1998). Evaluation Cookbook. Learning Technology Dissemination Initiative. The Scottish Higher Education Finding Council.  http://www.warwick.ac.uk/ETS/Resources/cookbook.pdf

Kenny, Quealy and Young (2002). RMIT ICT DLS Competency Framework - A basis for effective staff development. UltiBase. November 2002. RMIT Unversity. http://ultibase.rmit.edu.au/Articles/nov02/kenny1.htm

Kenny, J. (2001). Implementing and Institutional Online Learning Some Illustrative Case Studies at RMIT. In ed. Wallace, M., Ellis, A and Newton, D. Proceedings of the Moving Online II Conference. September 2001. Southern Cross University. http://www.scu.edu.au/schools/sawd/moconf/

Kenny, J. (2000). Evaluation report on the operation of the RMIT Distributed Learning System. http://www.online.rmit.edu.au/data/community/DLSeval2.rtf

Leonard, J., and Guha, S. (2001). “Education at the crossroads: Online teaching and students' perspectives on distance learning.” Journal of Research on Technology in Education 34(1): 51-57.

McNaught, C., Kenny, J., Kennedy, P., & Lord, R. (1999). Developing and evaluating a university-wide online Distributed Learning System: The experience at RMIT University, Educational Technology and Society, 2 (4) October 1999. <http://ifets.massey.ac.nz/periodical/vol_4_99/mcnaught.html>

Palmer, S. R., and Bray, S. L. (2001). “Longitudinal study of computer usage in flexible engineering education.” Australian Journal of Educational Technology 17(3): 313-329.

Rogers (1995), E.M. (1983). Diffusion of Innovations. 4^th Edition. The Free Press, Simon & Schuster, New York.

Stacey, E., and Fountain, W. (2001). Student and supervisor perspectives in a computer-mediated research relationship. Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education., Melbourne, Australia, University of Melbourne. Pages 519-528.

Talay-Ongan, A., and Gosper, M. (2000). Approaching Student Managed Learning. Proceedings of the 17th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education., Lismore, NSW, Australia., Southern Cross University Press. Pages 595-603.

Taynton (2000). Online Learning-A Student Perspective. Proceedings of the 17th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education., Lismore, NSW, Australia., Southern Cross University Press. Pages 125-134. back

About the authors

John Kenny
Professional development team
Learning Technology Services
RMIT University
Melbourne, Australia

Email: john.kenny@rmit.edu.au