Abstract
The development of digital libraries has changed the handling and access to information. Using such a library involves a computer-human interface as well as commands and search strategies to retrieve information. The purpose of this study was to evaluate a digital library in an institution of higher learning that serves approximately 5000 students. Quantitative and qualitative data were collected from a total of 206 participants through structured interviews, popup questionnaires and transactional log analysis.
Descriptive statistics and thematic analysis were used for quantitative and qualitative data. Online journals were the most commonly used resources while reference resources were least used. The usability and information retrieval capacity of the library was good. However, there was a need to improve the user interface of the digital library, create more awareness and subscribe to more online journals to meet the information requirements of the users.
Introduction
The library together with the information sector has undergone tremendous changes in recent years. These changes involve the collection and arrangement of information. Consequently, libraries can offer their services without confinement to geographical borders. Advances in information and communication technologies have revolutionised the type, subject matter, design and perception of libraries (Harris 2017). Hence, the progression of digital libraries originated from the amalgamation of technology with conventional library tools to satisfy the increasing information needs of users. At present, most libraries all over the world are moving towards the ‘digital’ mode. A digital library can be defined as a collection of digital objects such as text, images video or audio, which can be accessed through the internet or CD-ROM (compact disk read-only memory) (Lyman 2017).
The creation of digital libraries is a big investment considering the financial, technological and human resources required. Therefore, it is important to conduct regular appraisals to determine whether the intended objectives of the library are attained. Evaluation guides the decision-making process of digital library developers as well as other stakeholders who are directly or directly involved with the library. Evaluation is delineated as the methodical procedure of establishing the advantages,
importance and usefulness of something (Pinfield 2017). In the context of a digital library, evaluation can be regarded as the process of finding out whether the original objectives that led to the establishment of the system have been achieved. This process is often confused with an assessment. The key difference is that evaluation concentrates on factors such as systems, services and products while assessment concentrates on consumers, their attitudes, abilities and other related attributes (Wynne et al. 2016). Therefore, performance assessment can be incorporated into the evaluation structure. Nevertheless, both processes enable the making of sound decisions.
Before, conducting an evaluation, it is important to consider the following areas during the planning stages. First, the main purpose of the evaluation should be ascertained together with specific attributes to be appraised. The process and timing of the evaluation should also be considered. Digital libraries are multifarious structures.
Thus, evaluation approaches and metrics differ based on whether the digital libraries are seen as information systems, institutions, collections, new technologies or new services (Lamb 2017). The purpose of this paper is to evaluate a digital library in an institution of higher learning.
Forms of Evaluation
Evaluation can be done at different stages of the development of a digital library. Four forms of evaluation are explained herein. They include formative, summative, iterative and comparative evaluations.
Formative evaluation is commonly conducted during the initial stages of a project (Stefl-Mabry 2018). For instance, before starting a digital library project, it is necessary to find out the information needs of the target users to determine whether or not a digital library should be established. Thus, a formative evaluation is akin to conducting a market survey before introducing a commodity. The findings of the evaluation can guide developers to include certain aspects into the structure of the digital library or implement corrective measures in the early phases of a project. Additionally, a formative evaluation provides baseline data that can be used in subsequent stages of evaluation to determine whether the project has achieved some of its intended uses.
Summative evaluation is done at the end of a project to ascertain whether the original targets leading to the initiation of the project have been met or not (Pinfield 2017). Therefore, the focus of this evaluation is the outcome of an initiative. Iterative evaluations encompass short-term appraisals that are performed in the course of a project. They act as ‘in-between’ assessments that help to ensure that the project is on the right track (Tank, Maradiya & Bhatt 2017). These evaluations can be done as many times as possible in the course of the project.
Comparative evaluations are complete appraisals performed using
formats that can be contrasted across similar systems. In other words, they can be used as benchmarking processes to determine the value of a digital library (Campbell 2018). Similar systems are compared in a comparative evaluation. An example is comparing various digital library platforms across several institutions of higher learning.
Digital Library Evaluation Framework
Digital library evaluation frameworks enable the evaluator to conduct a detailed and logical assessment. However, before designing such a framework, it is necessary to point out the key constituents that typify the scope of the digital library environment. Three major components in the domain of a digital library are users, content and technology (Agosti, Ferro & Silvello 2016). Service is a valuable parameter that can be investigated. However, it often falls under content. These three parameters should be examined in detail to develop an effective framework for the evaluation of a digital library.
Users form the most critical entity in the information chain regardless of the library platform (whether it is a digital or conventional library). It is important to verify the target users and their information needs to appraise a digital library effectively. To understand the users fully, four points are important. The identity of the users should be ascertained, for example, students, researchers or professionals. Their information-seeking behaviour should also be determined. The type of information required needs to be clarified and this has to do with specific subject areas. Lastly, the purpose of the information should be identified. In summary, four questions to ask regarding the users of a digital library are “who”, “when”, “what” and “why”.
Contains refers to what kind of information is available in a digital library. The collection may differ based on the objectives of the library. For instance, the contents of an academic library may differ from the components of a professional library. The primary objects may be reports, books or journal articles, whereas secondary data could be metadata schemes or bibliographic descriptions. Various formats can be used to present the data, for instance, video, text or audio. When planning an evaluation, the type of content (audio, text or video), metadata schemes (indexing, citation, thesaurus and bibliographic arrangement) and quality of content (pertinence and subject coverage) should be considered.
Technology refers to the aggregate of skills, techniques and processes used in the development of commodities or services. Technological matters that are factored in digital libraries include user interface, management of access, document technology and system structure (Lyman 2017). The user interface takes care of diverse options that a digital library offers to its users and the ease of content access. A system ought to have efficient triangulation tools and recovery techniques to aid in the access of information. In contrast, system structure entails the structural design of the system, for example, protocols, database and middleware used in developing the platform.
Matters concerning the depiction of documents are considered in document technology, including format and model. Model denotes the conjectural features of a document such as semantic content, hyperlinked logical structure and external features. In contrast, format identifies the core document depiction such as rich text format (RTF), PDF and DOC (Fenlon et al. 2016).
Research Focus
The digital library at an institution of higher learning was unveiled in 2018 December. Its objective was to provide access to various resources offered by the library through a single window on campus and remotely. The architecture of the digital library included a host of hardware and software. Four different servers supported the main one. They included a server for web OPAC, another one for the institutional digital depository, a third server for databases based on hard-disk storage and a fourth CD mirror server for audio-visual materials stored on CD or DVD. The library was also linked to a virtual private network to facilitate access for external users (remote use).
The key contents of the digital library included information about the library, e-books, online journals, institutional repository, web, online and offline databases and CD/DVD based training tools. Internet protocol (IP) facilitated access to online resources. Therefore, there was no need to log in to find individual resources.
The digital library has about 5000 users, including research scholars, faculty members, undergraduate and postgraduate students. Approximately 80% of users log on to the digital library through the institution’s intranet. However, the remaining 20% accesses the services of the library remotely through a virtual private network (VPN) server. The focus of this study is to evaluate the digital library in terms of patterns of use, usability and information retrieval. These three forms of evaluation are explained further under this section.
Usability Evaluation
Usability inquiry testing was done. This type of evaluation entails appraising the usability of a digital library when target users are performing normal day to day tasks instead of evaluator-assigned tasks. This mode of evaluation is useful when trying to collect information concerning the needs, likes and dislikes of the users (Sánchez-Gálvez & Fernández-Luna 2015). Several approaches can be used in this regard, including focus groups, interviews, questionnaires and field observations.
Information Retrieval
People seek information for various purposes. When evaluating of a digital library, information retrieval refers to finding the information that is being sought by the user. The retrieval of information in the context of a single digital library is a multifarious process entailing aspects such as cataloguing, metadata and indexing. The convolution of a digital library is proportional to the number of aspects (indexing and cataloguing) that are effective at the same time when a user searches for data across various collections that apply diverse metadata systems (Gaona-García, Martin-Moncunill & Montenegro-Marin 2017). Nevertheless, details regarding these intricacies are of no use to the library users.
Their main concern is being able to find information efficiently and effectively. Thus, information retrieval evaluation is dual, user-focused and systems-oriented. Information retrieval evaluation from a user’s standpoint is determining the effectiveness with which a user’s search for information satisfies their interests or needs. In contrast, information retrieval evaluation from a systems perspective ascertains the usefulness and efficiency of the retrieval system, which is a core objective of all digital libraries.
In user-focused evaluation, the emphasis is on the user’s experience with the information recovery tools provided by the digital library (Cabrerizo et al. 2015). A digital library is of little value to users if they cannot find the information they desire effectively, notwithstanding the quality of information or sophistication of its technology. Therefore, when conducting a user-focused evaluation, it is necessary to determine the performance of the information retrieval system with respect to the users’ interests, requirements and anticipations.
The main challenge faced by most digital libraries is the storing, configuration and recovery of its contents (Places et al. 2016). Therefore, they aim to possess information retrieval systems that permit users to find specific items effectively in the shortest time possible. Assessing the information retrieval potential of a digital library provides valuable information that may guide future decisions concerning the hypothetical and practical modules of the library to optimise the efficacy of user searches.
Other aspects to be considered in the evaluation information retrieval are precision and recall. Precision can be described as the fraction of retrieved documents that satisfy the search requirement (are pertinent to the information being sought by the user). Conversely, recall is the fraction of relevant documents that are recovered from the assortment of all appropriate files. The estimation of recall is more convoluted than precision because the cataloguing of most digital libraries does not allow the identification of all the potentially relevant documents. This study focused on information retrieval from the user’s standpoint with a focus on precision.
Research Method
A formal evaluation was done a year following the establishment of a digital library in an institution of higher learning. Qualitative and quantitative data were obtained from the users. Different methods were applied to various categories of users. For example, questionnaires were used to collect information from students, whereas informal interviews were used to make inquiries from faculty. Transaction log analysis was used to collect quantitative data from all users. Through this mixed-method approach, it was possible to determine the information about the usage patterns of different groups as well as the usability and information retrieval potential of the digital library.
Sample
A sample of 206 participants took part in the study. Out of this number, 200 were students at undergraduate and postgraduate levels, whereas the remaining 6 were faculty. The students were chosen through random sampling as determined by a computer algorithm. The 6 members of faculty were identified through systematic sampling. Faculty members received letters inviting them to take part in the study. The letters contained a brief description of the study, its objectives and the expected duration of the study. The letter also contained an informed consent form that participants were expected to complete to verify their participation in the study. A copy of the invitation letter (Appendix A) and the informed consent form (Appendix B) are included in the appendices.
Informal Interviews with Faculty
The chosen faculty members were interviewed for about 30 minutes. Individual interviews were conducted using a predetermined set of questions as indicated in Appendix D. The informal interviews entailed questioning the user, recording their responses, transcribing the interviews before performing data analysis. Structured interviews were chosen for this study to minimise ambiguity and narrow down the responses to specific areas of study that were targeted by the researcher.
Questionnaires
Surveys are among the old-fashioned ways that libraries use to collect data. The most common form of carrying out surveys is via questionnaires. When designing the questionnaires, questions should be selected carefully to capture the research objectives. Two types of questions can be used: closed or open-ended. However, data from closed-ended questions are easier to analyse than information from open-ended ones. Nonetheless, open-ended questions allow the respondent to express themselves fully by providing additional explanations, which enriches the quality of data (Oosterveld, Vorst & Smits 2019).
Questionnaires are the most common instruments used in general evaluations. However, they have limited utility in usability evaluations. Currently, surveys can be conducted using web-based. Such data can easily be interpreted and analysed using various software tools (Stoet 2017). Statistical analyses may also be needed to make inferences of the resultant data. Popup questionnaires, however, are useful in gauging the usability of a digital library.
They are programmed to emerge when a user does something out of the blue in a digital library. In some cases, a short on-screen questionnaire regarding usability matters may be instigated after a specified duration of time or when leaving the library. It is also possible to email the questionnaires to users if identification protocols are needed to access the library because they would provide the users’ email addresses. Overall, questionnaires used in usability testing need to be as brief as possible notwithstanding their mode of presentation (Sánchez-Gálvez & Fernández-Luna 2015). Brevity encourages the users to complete them without feeling that they are wasting their time.
They should also be clear and unambiguous to yield accurate responses. A researcher can consider including incentives for prospective users to take part, especially if very many responses are required (Stoet 2017). Some of the motivations that can be used include entries into draws to win a prize or coupons for online shopping in specified stores. In this study, a popup questionnaire containing a prompt to redirect the user to a longer questionnaire was used (Appendix C).
Transactional Log Analysis
Transaction log analysis is a common approach in the evaluation of digital libraries. It was originally designed to collect quantitative data for the appraisal of OPAC libraries (Arshad & Ameen 2015). It has since been adopted for the evaluation of other types of digital libraries. Transactional log analysis is an effective tool when applied alongside other assessment tools. It provides valuable information such as the people who use digital libraries, the specific resources used, how long the library is used among other parameters.
Transactional logs can also generate information such as frequency and sequence of feature use, response times of the system, hit rates, location of users, error rates and the number of transactions per use (Siguenza-Guzman et al. 2015). In this study, transactional log analysis was used to determine information regarding patterns of use of the digital library. This information was also partly captured by the questionnaires.
Data Collection
A user questionnaire popped up on the screen after an individual had interacted with the digital library for at least 30 minutes. The popup questionnaire was brief and had only 2 questions. However, it contained a link to redirect the user to a longer questionnaire. The system was set to produce the popup questionnaires for 14 days until a total of about 200 users had completed the longer survey. Appendix C shows a copy of the questionnaire (popup and longer questionnaire). Evaluation parameters that were covered in the questionnaires were usability and information retrieval. This form of questionnaire has previously produced acceptable data supporting the reliability and validity of web-based questionnaires (Tella 2015).
Limitations
The first limitation of the study is the number of participants, which was relatively small. The 206 participants formed about 4.12% of the entire population of the digital library users. A larger sample would have been desirable though it was not possible due to financial, technical and time constraints. The second limitation was related to the various perspectives of the participants in this appraisal. Some of the partakers were postgraduate students while other were undergraduate students. The information needs of these two groups of students have been shown to differ significantly due to the scope of their studies. The questionnaires did not capture the input of faculty members. These different perspectives should be considered when analysing the outcomes of the evaluation.
Logistics
The author coordinated the execution of this evaluation plan, which consisted of planning, collection of data and data handling. The institution’s assistant librarian, who was in charge of the digital library project, was the primary point of contact. All data were processed, analysed, construed and reported by the author. All reports were conveyed to the digital library’s project manager and members of the development team. The evaluation outcomes were further disseminated to other stakeholders such as the administrators of the learning institution.
Data Analysis
Quantitative data obtained through questionnaires and transactional log analysis were recorded in an Excel spreadsheet for further analysis. Deductions were made by further categorising the responses into three main groups. For example, findings on ‘strongly agree’ and ‘agree’ were combined to mean ‘agreeing with the statement’, whereas ‘strongly disagree’ and ‘disagree’ were combined to mean ‘disagreeing with the statement’. The ensuing data were summarised using descriptive statistics such as means and percentages. Thematic analysis was used to analyse qualitative data from the structured interviews.
Findings and Discussions
Findings
About 47% of users who interacted with the system for more than 30 minutes completed the popup questionnaire. Out of this number, 32% completed the longer survey. Out of the complete responses, 43% were from female students while 57% were from male students (Figure 1). The majority of the participants were in their third year of study (25%) whereas the lowest proportion of partakers were in their first year of study (13%).
Postgraduate students consisted 22% of the subjects (Figure 2). Figure 3 showed that about half of the students used the digital library to find information about specific research topics. Journals and e-journals were the most used resources at 42% while the least used facility was the reference resources at less than 5% (Figure 4). This observation was also reiterated by the transactional log analysis in Table 1.
The usability evaluation (Figure 5) showed that 75% of users could accomplish all their tasks with ease, 50% could access information fast and effectively, whereas 60% did not need guidelines to use the library. Furthermore, 60% of users could recommend the library to their fellow students. However, 60% of the participants agreed that the system needed some improvements.
The information retrieval assessment showed that most users agreed about the system’s ability to yield results based on search criteria, retrieval of relevant data within a short time and agreement between search criteria and users’ expectations (Figure 6). More than 50% of the users had access to materials that were pertinent to their study areas and were satisfied with the system’s ability to retrieve information. However, about 50% of users agreed that the system was inconsistent and had difficulties remembering all information retrieval steps. 60% of the users reported that the system was unreliable in terms of providing information based on their needs.
Table 1: Patterns of use as shown by the transactional log analysis.
Thematic analysis from the interview involving members of faculty revealed 3 main themes: low awareness, underutilisation and satisfactory usability. They agreed that there had been substantial changes in the quality of work submitted by their students since the development of the digital library. The usability of the digital library was satisfactory as it was relatively easy to access information. However, it was apparent that there were low awareness levels about different resources that could be found in the digital library. Therefore, most users preferred to use the same resources frequently instead of other reserves of a similar nature.
Discussion
Usability, in the perspective of digital libraries, can be described as the efficacy with which people can access and use the wherewithal of a library successfully. A human-computer interface is commonly used in digital libraries. Thus, the interface and functionality are the most crucial aspects of digital libraries that should be evaluated and enhanced. According to Iqbal and Ullah (2016), the usability of computer platforms is determined by five attributes as determined by the user. They include ease of learning and use, the fast accomplishment of tasks, low error rates, user satisfaction and user retention.
Therefore, when creating digital libraries, the developer should ensure that accessing resources in the library is instinctive and easy. It should take the shortest time to find the needed resources. Additionally, errors of omission and commission should be minimised. Errors of omission entail the failure to find the user’s requirements, while errors of commission involve retrieving the wrong findings. A user should also be able to learn and understand the functions and navigational organisation of the library with the lowest level of cognitive overload. This means that a user should dedicate their thoughts and focus on impending tasks such as reviewing information to match precise needs instead of figuring out how to navigate the system.
Challenges that may arise in the usage of digital libraries include general unfamiliarity with computers, inadequate knowledge about search strategies and inexperience with the functionalities of the library (Jabeen et al. 2017). Usability issues are not faced by new users only. Frequent patrons may also face difficulties if new features are introduced without their knowledge. Therefore, these changes should be communicated to users once implemented. Users should also be informed that the modifications are meant to enhance their experience.
Conclusion and Recommendations
Conclusions
Digital libraries have transformed the idea of information services by reaching out to many users without temporal and spatial limitations. The advent of open-source software for online library platforms has also improved digital library technologies. However, new inventions are created each passing day. Therefore, it is necessary to conduct regular evaluations to determine whether appropriate developments are being incorporated. This study showed that the usability and information retrieval capability of the institution’s digital library were good. Nonetheless, the usage patterns showed that more 3rd and 4th years students, as well as those pursuing postgraduate studies, used the digital library than those in their initial years of study. Furthermore, users faced certain challenges that need to be addressed to enhance the usability and information retrieval of the library.
Recommendations
Based on the findings, four main recommendations were made. A collection development policy should be created to subscribe to as many online journals as possible given that this resource was most used. The user interface of the digital library should be redesigned to simplify it and ease the navigation process. User education should be done to enhance the usage of different available databases. More awareness should be created about the existence of the digital library, available resources and benefits of the facility.
Reference List
Agosti, M, Ferro, N & Silvello, G 2016, ‘Digital library interoperability at high level of abstraction’, Future Generation Computer Systems, vol. 55, pp. 129-146.
Arshad, A & Ameen, K 2015, ‘Usage patterns of Punjab University Library website: a transactional log analysis study’, The Electronic Library, vol. 33, no. 1, pp. 65-74.
Cabrerizo, FJ, Morente-Molinera, JA, Pérez, IJ, López-Gijón, J & Herrera-Viedma, E 2015, ‘A decision support system to develop a quality management in academic digital libraries’, Information Sciences, vol. 323, pp. 48-58.
Campbell, HM 2018, A comparative evaluation of three digital libraries. Web.
Fenlon, K, Wood, LC, Downie, JS, Han, R & Kinnaman, AO 2016, ‘Toward accessible course content: challenges and opportunities for libraries and information systems’, in Proceedings of the 79th ASIS&T Annual Meeting: Creating Knowledge, Enhancing Lives through Information & Technology, Copenhagen, American Society for Information Science, pp. 1-10.
Gaona-García, PA, Martin-Moncunill, D & Montenegro-Marin, CE 2017, ‘Trends and challenges of visual search interfaces in digital libraries and repositories’, The Electronic Library, vol. 35, no. 1, pp. 69-98.
Harris, MW 2017, What’s a digital library? . Web.
Iqbal, M & Ullah, A 2016, ‘Qualitative usability evaluation of databases: a case study’, Library Hi Tech News, vol. 33, no. 3, pp. 8-10.
Jabeen, M, Qinjian, Y, Yihan, Z, Jabeen, M & Imran, M 2017, ‘Usability study of digital libraries: an analysis of user perception, satisfaction, challenges, and opportunities at university libraries of Nanjing, China’, Library Collections, Acquisitions, & Technical Services, vol. 40, no. 1-2, pp. 58-69.
Lamb, A 2017, ‘Debunking the librarian ‘gene’: designing online information literacy instruction for incoming library science students’, Journal of Education for Library and Information Science, vol. 58, no. 1, pp. 15-26.
Lyman, P 2017, ‘What is a digital library? Technology, intellectual property, and the public interest’, in SR Graubard & P LeClerc (eds), Books, bricks and bytes, Routledge, New York, NY, pp. 1-34.
Oosterveld, P, Vorst, HC & Smits, N 2019, ‘Methods for questionnaire design: a taxonomy linking procedures to test goals’, Quality of Life Research, vol. 28, no. 9, pp. 2501-2512.
Pinfield, S 2017, ‘eLib in retrospect: a national strategy for digital library development in the 1990s,’ in J Andrews (ed.), Digital libraries, Routledge, New York, NY, pp. 19-34.
Places, ÁS, Fariña, A, Luaces, MR, Pedreira, Ó & Seco, D 2016, ‘A workflow management system to feed digital libraries: proposal and case study’, Multimedia Tools and Applications, vol. 75, no. 7, pp. 3843-3877.
Sánchez-Gálvez, LA & Fernández-Luna, JM 2015, ‘A usability evaluation methodology of digital library’, Service Computation, vol. 12, no. 2, pp. 23-28.
Siguenza-Guzman, L, Saquicela, V, Avila-Ordóñez, E, Vandewalle, J, & Cattrysse, D 2015, ‘Literature review of data mining applications in academic libraries’, The Journal of Academic Librarianship, vol. 41, no. 4, pp. 499-510.
Stefl-Mabry, J 2018, ‘Documenting evidence of practice: the power of formative assessment’, Knowledge Quest, vol. 46, no. 3, pp. 50-57.
Stoet, G 2017, ‘PsyToolkit: a novel web-based method for running online questionnaires and reaction-time experiments’, Teaching of Psychology, vol. 44, no. 1, pp. 24-31.
Tank, SD, Maradiya, M & Bhatt, R 2017, Blended librarianship for academic libraries in digital era theory and practice. Web.
Tella, A 2015, ‘Electronic and paper-based data collection methods in library and information science research: a comparative analyses’, New Library World, vol. 116, no. 9-10, pp. 588-609.
Wynne, B, Dixon, S, Donohue, N & Rowlands, I 2016, ‘Changing the library brand: a case study’, New Review of Academic Librarianship, vol. 22, no. 2-3, pp. 337-349.
Appendices
Appendix : Questionnaire
SECTION A: Popup Questionnaire
Are you enjoying the online library platform?
Are you willing to share your experiences with the online library?
Your feedback will help improve your online experience.
Please let us know more about your experiences by completing a short survey that is available at this link.
SECTION B: Long Questionnaire
Please mark with X where appropriate
Gender
Year of Study
What do you use the online library for?
Which one of the following resources do you use most on the online platform?
Usability Evaluation of the Digital Library
Please mark with X where appropriate
Please indicate the extent to which you agree with the following statements on a scale of 1 to 5 (1= Strongly disagree, 2= Disagree, 3= Neutral, 4= Agree, 5= Strongly agree).
Information Retrieval on the Digital Library Platform
Please indicate the extent to which you agree with the following statements on a scale of 1 to 5 (1= Strongly disagree, 2= Disagree, 3= Neutral, 4= Agree, 5= Strongly agree).
Thank you for your participation.
Appendix : Interview Schedule for Faculty
Questions
- How has been your experience with the digital platform of the library?
- Have you noted any changes in students’ performance or use of academic resources since the inception of the digital library?
- Do you think digital resources are utilised adequately by students and faculty? Why?
- What problems/ challenges have you encountered so far?
- What do you think can be done to improve the user interface of the digital library?
- What are your suggestions to enhance searching for e-books, e-journals and research papers?
Thank you for your participation.