Evaluating the Usability and Accessibility of LMS at King Saud University

Abstract

LMS is a type of e-learning method, which has become an important tool for education in modern institutions of higher education. LMS is known to be complex environments that make use of several available technologies and techniques. Although several LMS are universal, different institutions of higher learning prefer to develop and use their systems. For instance, in Saudi Arabia, the national center for E-learning and distance learning (NCeDL) has tested some LMSs such as JUSUR and others.

Currently, a variety of open-source or commercial LMS is available, providing institutions of higher learning with a wide range of choices and features. In addition, there are several types of LMS available for use. This research study will attempt to answer the question “is buying a Blackboard Vista license worth it? The proposed study will evaluate the usability and accessibility of Blackboard LMS at King Saud University, Riyadh. The study will be qualitative research that aims at describing the phenomenon in question “the level of usability and accessibility of Blackboard LMS at KSU”. In general, the study indicates that the majority of students are satisfied with the system.

Introduction

Learning management systems (LMS) have become the main platforms for the delivery of knowledge in most course-related activities like discussions, presentations, and quizzes. LMS is a type of e-learning method, which has become an important tool for education in modern institutions of higher education (Zemsky & Massy, 2009). LMS is known to be complex environments that make use of several available technologies and techniques. Although several LMS are universal, different institutions of higher learning prefer to develop and use their systems (Pearson & Bohman, 2009). For instance, in Saudi Arabia, the national center for E-learning and distance learning (NCeDL) has tested some LMSs such as JUSUR and others.

Currently, a variety of open-source or commercial LMS is available, providing institutions of higher learning with a wide range of choices and features. In addition, there are several types of LMS available for use. For instance, general LMS have useful tools that make it easy to create and manage courses. In addition, collaborative learning support systems, institution resources management systems, test-authoring systems, and virtual classroom management systems are widely used in modern learning.

King Saud University (KSU) has adopted Blackboard Vista, a type of modern LMS that provides faculties and departments in institutions of higher education with a powerful and user-friendly set of tools in teaching courses. For instance, students have the opportunity to access the system’s sites anywhere and anytime, allowing them to access course content that their faculties and departments keep positing. For instance, students easily access notes, lecture materials, links to important materials, exams, and other materials.

Problem statement

There are many cases of frustrations and disappointments associated with the use of LMS in most institutions due to poor features, the ineffectiveness of evaluation procedures, poor pedagogical guidance, and issues related to the ineffective usability of the systems (Nielsen, 2005). In terms of their usability, it has been established that LMS has several particularities that are not common in other Web-based systems. In addition, past research studies on MLS application in higher education have only concentrated on determining issues related to technical features and technical issues (Zaharias, 2004). This means that there is little knowledge on issues related to usability and accessibility of LMS in institutions of higher learning, especially where individual institutions decide to develop and use their systems. At KSU, buying a blackboard Vista license is arguably an important step towards developing effective learning and delivery of knowledge. However, with little knowledge of the usability and accessibility of LMS at the university, there is a need to carry out research studies. This research study will attempt to answer the question “is buying a Blackboard Vista license worth it?

Research questions

1. What is the level of usability and accessibility of LMS at KSU?
2. What is the effectiveness of LMS at KSU?
3. Is buying and applying for Black Board Vista at KSU worth it?
4. To evaluate students usage of blackboard

Objectives

1. To evaluate students usage of blackboard
2. To see what are the most useful component for students
3. To see what are the most useful component for faculties

Is there any statically significant difference between users referred to gender or majors?

Literature Review

Several studies have shown that LMS has enhanced education in various institutions, especially in universities and colleges. However, some studies have revealed that LMS can pose serious problems associated with accessibility for both students and course facilitators. A study by Martin et al (2008) has shown that some forms of LMS tools such as Chat, Discussions, and Wikis have proved to be more problematic.

Inversini et al (2006) have shown that there is a correlation between the effect of LMS’s outcomes and the system’s usability. Therefore, perceiving LMS from their usability perspective has been shown that these systems have some specific particularities not common in other Web sites. For instance, an LMS is an out-of-box product, implying that each institution has the opportunity to adopt an already existing system rather than create a new version (Pierotti, 2010). Secondly, each LMS has to be very flexible to meet a variety of needs, including teaching, collaborative, and learning styles in various institutions (Nielsen, 1992). Thirdly, LMS has a high level of customization, offering a wide range of options for configuration at all levels (Sherman, 2009). LMSs are also able to capture their audience because students and instructors have no option rather learn to adopt the specific tool adopted by their institutions. LMSs are also able to comply with the education standards set by the institutional control bodies and authorities.

In an attempt to evaluate the usability and accessibility of the existing LMSs, many studies have produced a range of information (Melton, 2006). Inversini et al (2006) did a comparative study on four different LMSs, comparing between two open-source and two commercial LMS. In addition, a study by Martin et al (2008) evaluated Moodle, dotLRN, and Sakai using heuristics evaluation techniques (Kruse, 2010).

All these studies proved that there are several problems associated with the usability and accessibility of LMSs, including the open-source and commercial varieties. In Saudi Arabia, a study by Al-Khalifa (2012) was conducted at KSU to evaluate the usability of JUSUR LMS. JUSUR is a type of LMS that was created and adopted by the National Centre for E-learning and Distance Learning (NCeDL). Al-Khalifa used satisfaction surveys, which measured user satisfaction with the LMS with two questionnaires (Meiselwitz & Sadera, 2008). According to the results, respondents say the system is friendly. However, they are sometimes dissatisfied with the system because it often suffers from functional and technical problems, causing it to fail.

Methodology

Study design

The proposed study will evaluate the usability and accessibility of Blackboard LMS at King Saud University, Riyadh. The study will be qualitative research that aims at describing the phenomenon in question “the level of usability and accessibility of Blackboard LMS at KSU”.

Study area and population for the study

The study will take place in the first semester of the 2013/2014 academic year. The researcher will target 200 students in Web Design and applications engineering, which falls under the department of information technology. These students will participate in answering questions for evaluation. The researcher aims at including students whose average age is 21 years for both males and females. In addition, the students will exclusively be undergraduates taking their first degree.

Students included in the study will have used Blackboard LMS during the semester to execute tasks such as obtaining course materials, taking and submitting assignments, asking questions, sharing information, completing online and real-time exams (quizzes), and checking departmental announcements.

The study will adopt two standard questions in evaluating the usability and accessibility of Blackboard LMS at KSU:

1. The system usability scale mature question; contains 10 statements, which was developed by Brooke (1986). Brooke’s question is very robust and has been used in various studies, giving successful results.
2. A researcher-deigned questionnaire based on usability and user satisfaction questionnaire by Zins et al (2004) and web learning environment instrument by Chang (1999).

Therefore, the researcher will assess several items using the two types of questionnaire- functionality, satisfaction, design/layout, outcome & future, and level of usefulness. In addition, the researcher will use the questionnaire to target user satisfaction as well as the usefulness of the LMS.

Results of the study (anticipated- makeup)

The study expects to use a 5-scale Likert scale ranging from “I strongly agree” (5 points) to “I strongly Disagree” (0 points) in rating the response given by the participants towards each question on the questionnaires.

It is also expected that out of the 200 students who will participate in the study, 180 will return valid questionnaires with honest answers. Thus, N=180. Here are the expected results (makeup):

Results for SUS question

Findings

In general, the study indicates that the majority of students are satisfied with the system. More than 77% of the students reported positive responses on all the items targeted by the researcher. Some of the main areas of research that show the high rate of response for the system are usability, ability to learn, outcomes, and ease of use. In all these areas, the scores given for positive response were above 70% on the Linkert scale. In addition, it was found that the future and outcomes of the system are two important areas that students gave positive responses. This area had more than 80% positive answers as shown above. According to the findings, students believe that the system will have improved their grades by the end of the semester. It was also found that most students were able to learn the system with ease, with only a few reporting that they had to ask someone to assist them or assist some of their colleagues.

The study also shows that the majority of the students, in fact, more than 80%, were able to recover from errors they made when using the system. They reported that they had made an error at least once in the semester, but the time used to recover from the error was amazingly reduced (Al-Khalifa, 2012). Moreover, more than 75% of the responses indicate that the system was able to return error messages that were clear, understandable, and easy to follow.

However, it is worth noting that the study has revealed several weaknesses with the usability and accessibility of the system. For instance, more than 85% of the responses indicate that the system suffers from functional and technical problems, which make it difficult to use in learning.

Conclusion and Recommendations

Since most of the responses indicate that Blackboard is an effective LMS at the university, it is strongly recommended that the university continue using the system. However, it is also recommended that more efforts be applied in ensuring that the system does not suffer from technical and functional problems. It is recommended that the university consider increasing the number of qualified system administrators. In addition, alternative internet servers are needed to host the system once the main server experiences problems. It is also recommended that the system be improved to enhance student communication with each other to allow enough room for information sharing.

References

Al-Khalifa, H. (2008). JUSUR: The Saudi learning management system. Dubai, UAE: Middle-East Education Forum.

Al-Khalifa, H. S (2012). A First Step in Evaluating the Usability of JUSUR Learning Management System. Riyadh, KSA: KSU

Chang, V. (1999). Evaluating the effectiveness of online learning using a new web based learning instrument. Retrieved from

Inversini, A., Botturi, L., & Triacca, L. (2006). Evaluating LMS Usability for Enhanced eLearning Experience. Chesapeake, VA: AACE

Kruse, K. (2010). e-Learning and the Neglect of User Interface Design. Web.

Martin, L., et al. (2008). Usability in e-Learning Platforms: heuristics comparison between Moodle, Sakai and dotLRN. Agia Napa, Cyprus: ECEL.

Meiselwitz, G., & Sadera, W. A. (2008). Investigating the Connection between Usability and Learning Outcomes in Online Learning Environments. MERLOT Journal of Online Learning and Teaching, 4(2), 145-151.

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. Monterey, CA: Cengage

Nielsen, J. (2005). Heuristic Evaluation. Web.

Pearson & P., & Bohman, P. (2009). Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2006. Web.

Pierotti, D. (2010). Heuristic Evaluation – A System Checklist. Web.

Sherman, P. (2009). UPA Salary Survey. New York, NY: UPA

Zaharias, P. (2004). Usability and e-Learning: The road towards integration. ACM eLearn Magazine, 204(6), 223-235.

Zemsky, R., Massy W.F. (2009). Thwarted Innovation What Happened to e-learning and Why. Web.

Zins, A.H., et al. (2004). An Experimental Usability Test for different Destination Recommender Systems. ENTER: Cairo.