Keywords

Computerized template, Electronic health record, Technology Acceptance Model, Customization, Provider satisfaction

 

Authors

  1. GARDNER, CARRIE LEE DNP, FNP-BC
  2. PEARCE, PATRICIA F. PhD, MPH, FNP-BC, FAANP

Abstract

Since 2004, increasing importance has been placed on the adoption of electronic medical records by healthcare providers for documentation of patient care. Recent federal regulations have shifted the focus from adoption alone to meaningful use of an electronic medical record system. As proposed by the Technology Acceptance Model, the behavioral intention to use technology is determined by the person's attitude toward usage. The purpose of this quality improvement project was to devise and implement customized templates into an existent electronic medical record system in a single clinic and measure the satisfaction of the clinic providers with the system before and after implementation. Provider satisfaction with the electronic medical record system was evaluated prior to and following template implementation using the current version 7.0 of the Questionnaire for User Interaction Satisfaction tool. Provider comments and improvement in the Questionnaire for User Interaction Satisfaction levels of rankings following template implementation indicated a positive perspective by the providers in regard to the templates and customization of the system.

 

Article Content

SIGNIFICANCE AND BACKGROUND

Health information technology (HIT) and its impact on the delivery of patient care is a subject widely discussed by clinicians, policy makers, and healthcare leaders in the US. The use of HIT has the potential to drastically impact the provision of healthcare to consumers by providing better access to complete and accurate information at the point of care.1 The most widely recognized form of HIT is the electronic health/medical record. In the Report to the Office of the National Coordinator for Health Information Technology (ONC) on Defining Key Health Information Technology Terms, the National Alliance for Health Information Technology2 defines an electronic medical record as "an electronic record of health-related information on an individual that can be created, gathered, managed, and consulted by authorized clinicians and staff within one healthcare organization."2 However, as reported by Hayrinen et al,3 further refinement of definitions, content, and use of electronic medical/health records must occur for standardization of health information systems internationally.

 

Increasing importance has been placed on the adoption of electronic medical/health records in practice environments in the US since 2004 when the federal government set an overall goal for the majority of Americans to have electronic health records by the year 2014.2 By this same executive order, the ONC was also established, but not funded. In 2009, the American Recovery and Reinvestment Act (ARRA), a $787 billion economic stimulus package, became law.4 Of the $150 billion that the ARRA allocates toward healthcare, $19.2 billion is specifically allocated for HIT.5 The Health Information Technology for Economic and Clinical Health (HITECH) Act is the portion of the ARRA bill that aims to accelerate the adoption and usage of electronic medical records by practitioners.6

 

However, providers must move beyond simple adoption of electronic records and into effective utilization of electronic medical records. There are varying degrees of utilization of electronic medical records and controversy over whether the systems are being utilized in a meaningful way. DesRoches and colleagues7 surveyed 4484 US physicians who provided direct patient care and yielded a response from 2758 survey participants, representing a 62% response rate. Of those who responded to the survey, 17% reported using an electronic medical record system in their clinical practice. However, of this 17%, only 4% of physicians reported having a fully functional electronic medical record operational in their practice. The survey defined a fully functional system as one capable of handling clinical and demographic patient data, viewing and managing laboratory and imaging results, managing order entry including electronic prescriptions, and clinical decision-making support. DesRoches and colleagues'7 report serves to strengthen the importance of actual utilization of systems rather than the simple adoption of a system.

 

In 2010, the Centers for Medicare & Medicaid Services (CMS)8 in conjunction with the ONC announced the final regulations and rules outlining guidelines for meaningful use of electronic medical records and standards for certified electronic health record technology. As defined by Blumenthal and Tavenner,9 meaningful use encompasses the use of electronic health records by providers to achieve significant improvements in care for patient populations. In response to the call for the adoption and meaningful use of electronic medical records in patient care, CMS8 has developed a set of criteria progressing through three stages outlining the meaningful use of a system. In 2011, CMS began providing incentive payments to eligible professionals for attestation of meaningful use of an electronic medical record system. Participation for incentive payments is not mandatory. However, Medicare payment adjustments, in the form of deductions, will begin for eligible professionals not demonstrating meaningful use as outlined by CMS in the year 2015. In order to meet the criteria for meaningful use of a system in stage 1, the eligible entity must demonstrate attainment of a set of core measures or objectives. The objectives and criteria for stage 1 meaningful use focus on electronically capturing information about patients, in order to track clinical conditions, coordinate care, and promote reporting of quality indicator measures. For stage 2, CMS intends objectives to expand on stage 1 criteria and to integrate further coordination of services through electronic record systems. Stage 3 criteria will expand on the previous two stages and focus on achieving improvement in quality and safety of patient care by focusing on decision support for providers. Therefore, healthcare providers should not only adopt electronic medical/health records in patient care, but also use these records meaningfully and to the utmost potential of the computer system for integration of improved quality and safety in patient care.8 Even with the heavy emphasis placed on the adoption, implementation, and utilization of electronic health/medical records, widespread acceptance has been very slow. Although improvements in the quality and efficiency of patient care have been demonstrated with the use of electronic medical records, many providers are reluctant to utilize the technology.10

 

DEVELOPMENTAL FRAMEWORK

The Technology Acceptance Model (TAM) provides a framework to help guide providers and managers in healthcare regarding variables that affect whether a computerized system or another type of technology will actually be used.11 The TAM was developed in the late 1980s to provide a better understanding of user response to computers and technology.11 In light of the disappearance of many technical barriers, Davis et al11 sought to explain and define why users of technology are willing or unwilling to accept and use technology. The TAM entails several components: behavior intention (motivation, intention, attitude), usage, perceived ease of use, and usefulness. The TAM illustrates that the use of computers or information technology is determined by the behavioral intention to use, which in turn is determined by the person's attitude toward usage. System adoption can be more easily achieved when user attitudes toward and intention to use the technology are positive.

 

Customization and End-User Satisfaction

Perceived ease of use has the strongest impact on perceived usefulness and in turn on the attitude toward use of electronic health records. Additionally, physician/provider involvement has a significant effect on perceived ease of use of a system.12 By allowing providers to participate in the development process of HIT, a positive feeling toward the system can be cultivated along with a sense of ownership of the process.13 During implementation of an electronic medical record, the impact on provider workflow must be taken into account. Provider input should be obtained, and open and honest communication is crucial for successful implementation of changes to the system.14

 

As reported by Morton and Wiedenbeck,12 there is an overwhelming need for flexible customizable electronic medical records products and technology. Small customization changes to a system can have a significant impact on provider usage of electronic medical record systems.15 Lack of customizability has been cited in the literature as a major barrier to the acceptance of electronic medical records by providers.16 Interface designs and development of electronic medical record features, such as templates, that support clinic workflow and usage can impact the success of an electronic medical record system.17 Providers within a practice are in a unique position to adapt interfaces and create templates based on personal preferences and practice demographics to meet the needs of specific patient populations.18 Template use in documentation has also been shown to improve the overall quality and thoroughness of both written and computerized documentation.19,20

 

PROCESS FOUNDATION

As outlined by Langley and colleagues,21 the Model for Improvement and the Plan-Do-Study-Act (PDSA) tool provide a two-phase framework to facilitate change and improvement in an organization. During implementation of the Model for Improvement, users are guided to identify the desired change, whether this change will be an improvement, and the actions that must be undertaken to facilitate the change and result in the improvement. Once the need for change and the steps for change have been identified, the PDSA cycle should be implemented, which completes the Model for Improvement. Through the PDSA cycle, trial and learning methodology are utilized, building knowledge to facilitate change in a system.21 The Model for Improvement and the PDSA cycle allow those involved in the change process to set goals for change, to evaluate the process and results, and to modify the process as needed for a particular system.

 

This project entailed a change process in a practice setting, and to facilitate this process, time to plan and implement the protocol was allowed. Staff and providers were educated regarding the project so that they possessed the necessary skills for implementation.22 Staff and provider ownership of the project was fostered by allowing discussion of strategies to help the participants to construct a new vision of improved satisfaction and usage of the system.23 Furthermore, all providers and staff involved were asked to identify personal and office-wide facilitators and barriers to the process. Responsibilities were outlined prior to implementation so that everyone involved was aware of his/her specific individual responsibilities, thereby decreasing confusion and dissension among the providers and staff.

 

PURPOSE

A small, privately owned internal medicine practice in the city of Troy, located in Pike County in southeastern Alabama, implemented an electronic medical record system in April 2009. The goal of the providers in the practice was to achieve meaningful use of the electronic medical record system as defined by CMS. Providers in the practice were already utilizing the electronic medical record system. However, satisfaction with the product was lacking, resulting in minimal use of many of the system's capabilities. Providers were typing narrative notes into the system rather than developing and implementing customized templates to improve the time and ease of documentation. Use of the narrative style of documentation diminished the ability to track quality indicators.

 

The purpose of this quality assurance project was to devise and implement customized documentation templates into an existing electronic medical record system and to measure the provider satisfaction with the system before and after implementation.

 

SETTING AND SAMPLE

Setting

Pike County is approximately 50 miles from Montgomery. The majority of the patients seen in the practice have private insurance or primary Medicare. Primary Medicaid is not accepted by the practice, and patients without insurance must pay at the time of service. The most frequently billed International Classification of Diseases, Ninth Revision (ICD-9) codes by the providers at the practice for the last year were hypertension, hyperlipidemia, diabetes, sinusitis/bronchitis, and low-back pain (Office Manager, personal communication, July 9, 2011).

 

Sample

The participants in this project were the three providers in the practice: one physician, one nurse practitioner, and one physician assistant. Participants were identified based on their positions at the practice and their interaction with the electronic medical record system. They were all aware of the need for more effective use of the electronic record, were positive about moving into more effective use and evaluation of use, and were highly motivated to participate in the project. No incentives were provided for the participants. Patient confidentiality was maintained through privacy standards that were already in place at the practice in compliance with HIPAA regulations.

 

DESIGN

This project was a descriptive, quality improvement project utilizing a process orientation foundation of the Model for Improvement and the PDSA tool,21 with evaluation through electronic survey technique. Provider satisfaction with the electronic medical record system was assessed at the onset of the project and following implementation of the templates.

 

PROCEDURES

Approval for this project was obtained from the University of Alabama at Birmingham institutional review board. Approval to conduct the study was also obtained from the internal medicine clinic that served as the project site. The project was explained to the participants, along with an explanation of risks, benefits, alternatives, withdrawal, and confidentiality. Following this explanation, participants were asked to sign an informed consent document.

 

Planning and Development

Prior to any discussion regarding change or template implementation, the current version 7.0 of the Questionnaire for User Interaction Satisfaction (QUIS) tool was administered to the providers utilizing the electronic medical record system. The providers then met and discussed the need for change in the current electronic medical record system. Overwhelmingly, the providers wanted customization of documentation through the electronic medical record system without having to utilize narrative charting. Guided by the TAM, providers discussed perceived facilitators and barriers to the change process along with external factors affecting the perceived ease of use and perceived usefulness of the system. Following discussion regarding the change process in the practice, a query of the top five ICD-9 codes charged for the last year was performed. The top five ICD-9 codes were utilized because these are the most frequently used ICD-9 codes and crossed all providers and patients in the practice. The codes guided the providers in deciding which disease processes linked with the ICD-9 codes should be the focus of the templates to be designed.

 

Template Configuration

The existing electronic medical record system allows structured and free-text narrative entry of information regarding patient care. Sections for documentation of a patient visit including constitutionals, medical history, family history, review of systems, physical examination, assessment/diagnosis, planning, and prescriptions are listed in a tabular format on the computer screen. The system has preprogrammed generic templates under review of systems, physical examination, and orders available to providers for documentation. The providers in this practice were mainly using the free-text or narrative option during documentation due to the fact that the documentation and wording of the information found in the structured templates in the program did not meet individual documentation preferences. Each of the providers had individual documentation style preferences, so rather than editing the preprogrammed templates, narrative charting seemed to be the best choice for meeting documentation needs.

 

During the planning phase of the PDSA cycle, providers identified the information to be included in each of the customized templates to be developed. Through scheduled biweekly provider meetings, all of the templates were developed over a 2-week time frame. Initially, the three providers and the office manager met and discussed the disease processes and general specifications for the templates. It was decided that templates would be developed for a review of systems, physical examination, and order/planning set for each disease process. The providers decided to combine sinusitis and bronchitis into one template that focused on the upper respiratory tract system and to combine hypertension and hyperlipidemia into one template that focused on the cardiovascular system. Individual templates were developed for diabetes and low-back pain. During the planning phase, the providers also decided to develop a template for a general well-visit review of systems and a general well-visit examination. Providers worked individually on the templates and then reported personal preferences to the group. The medical technology vendor was also consulted in these meetings to ensure that the templates met the logistical requirements of the system. The medical technology vendor is a Certification Commission for Health Information Technology and Drummond-certified provider of HIT.

 

The customized templates gave providers access to a review of systems, a physical examination, and order/planning templates for specific disease processes based on provider documentation preferences. The new templates allowed providers to easily document areas such as normal examination and review-of-system findings and patient education based on specific personal preferences by point and click without having to type the information on each individual patient. Order sets for medications and laboratory work commonly utilized in the treatment of specific disease processes were developed to allow easier and more timely documentation of commonly used treatment regimens. Quality indicator measures and disease-specific follow-up items were also embedded into the templates so that providers are now prompted to address them.

 

Programming

Once the templates were developed, information was programmed into the electronic documentation portals of medical record system for each provider by the medical technology vendor. During the programming portion of the implementation process, the providers developed interest in on-site programming to decrease the amount of time it took to implement changes. Staff from the electronic medical record company provided a training session on programming logistics and instructed providers on how to change personal settings within the electronic medical record system. Following implementation of the templates, each provider then made changes to the templates based on personal documentation and treatment preferences. Once the providers had all of the templates in the electronic medical record system, they were trained on the use of the templates and practiced template use through training accounts within the system prior to the documentation of actual patient care.

 

Implementation

As recommended by the PDSA cycle, once planning has occurred, then implementation and analysis of the plan should occur. Following the programming of the system and training of the providers, the templates were implemented into patient care activities at the clinic. Throughout the implementation process, providers continued to modify the templates based on personal preferences. Each provider utilized the templates for 20 patient encounters.

 

Evaluation

Evaluation of the process was ongoing during the development and programming phases, with a more formalized evaluation of the provider satisfaction with the system, using the QUIS as a pre-evaluation and postevaluation tool. Focus of the QUIS was to determine user perceptions of the computer system and the interface-specific points of the newly instituted templates.

 

PROCESS EVALUATION

Facilitators and barriers to the change process were addressed through the use of the TAM. Prior to beginning the planning process for the templates, providers discussed the benefits of changing the current documentation practices. The providers identified factors that were influencing their perceived usefulness of the system and perceived ease of use of the system. Factors such as lack of knowledge regarding making changes to the system, programming, and system capabilities were identified as barriers to the change process. Providers were unaware that order sets could be added and changes could be made to tailor the system to individual preferences. Providers also viewed the process of programming changes into the system as complex and time consuming rather than a simple operation that could take place on-site. Lack of time for planning and implementing the change was also identified by the providers as a barrier. Training on system use and scheduled meeting times helped to address these particular barriers. Once providers realized that changes to the system could be made on-site in a timely manner, their perceived ease of use led to improved behavioral intention to use and actual use. The providers and staff of the office were motivated to improve satisfaction with use of the electronic medical record system. The providers wanted to improve their use of the system and also the ease of documentation. Financial incentives offered through the ARRA also served as facilitators to the adoption of measures to improve the usage of the medical record system and quality indicator reporting.

 

PROVIDER SATISFACTION EVALUATION INSTRUMENT

Provider satisfaction regarding system use was initially evaluated prior to the planning phase of the process and then following the implementation of templates into the system. User satisfaction with the system was evaluated using the current version 7.0 of the QUIS tool.24 Developed by a team of researchers at the University of Maryland, the QUIS was designed to subjectively assess the satisfaction of users of computer applications regarding the human-computer interface.24 The QUIS short form is divided into 12 parts. Parts 1 and 2 of the form focus on demographic information and past computer experience. The remaining parts measure the overall user reactions to the computer system along with the reactions to interface-specific factors along a 9-point Likert scale. The questionnaire is designed for individualized configuration so users can include only the areas of interest for a specific system.24 The overall reliability of the QUIS 7.0 is high, with a Cronbach's [alpha] of .95.25 Parts 1 to 7 of the QUIS short form were used for the project. Parts 8 to 12 were deemed not applicable to the current project because of the focus on the following topics: technical manuals and online help, online tutorials, multimedia, teleconferencing, and software installation.

 

QUESTIONNAIRE FOR USER INTERACTION SATISFACTION DATA ANALYSIS

Data analysis included a descriptive comparison between provider responses to the QUIS prior to and following implementation of templates into the electronic medical record. The three participants in the project were the providers at the practice. According to the data obtained from the administration of the QUIS, all three of the participants had worked with this particular system for more than 2 years but less than 3 years and spent more than 10 hours per week utilizing the system. All three participants had previous personal experience and familiarity with the following: computer terminals, personal computers, laptop computers, color monitors, touch screen, floppy drives, CD-ROM drives, keyboards, mouse devices, modems, joysticks, scanners, computer games, e-mail technology, and Internet.

 

Descriptive statistics were calculated for the QUIS items that were reported along a 9-point Likert scale. The Likert scale points range from one, as worst, to nine as the best. The first portion of the QUIS focuses on overall reactions of the participant to the computer system. All three provider ratings of the overall computer system were favorable (rank 6-9), with preimplementation only including the ratings of 6 to 8, and postimplementation including 7 to 9. These include items targeting terrible to wonderful, frustrating to satisfying, difficult to easy, and rigid to flexible (Table 1).

  
Table 1 - Click to enlarge in new windowTable 1 Overall User Reactions to the Electronic Medical Record System Before and After Implementation of Customized Templates

The QUIS also allows the participant to rank his/her reaction regarding specific functions such as screen layouts, terminology, and learning aspects. Participants were asked to rank reactions to the helpfulness of screen layouts, terminology usage, and the degree to which terminology related to the work the user was doing, and for all these areas, rankings were from 8 to 9 prior to and following template implementation. The issues of learning the system, exploration of features by trial and error, and straightforward task performance showed the most substantial leap from before to after implementation, with rankings from 5 to 7 at preimplementation to 7 to 9 following implementation (Table 2). Because of the limited number of providers in this setting, no further analysis can be done. Although these are small changes, they are favorable and positive.

  
Table 2 - Click to enlarge in new windowTable 2 QUIS Item Responses Related to Learning Before and After Implementation of Customized Templates

Participants were also allowed to provide comments about the electronic medical record system and their reactions. No comments were provided during the administration of the QUIS prior to the implementation of templates. However, following implementation of templates, providers made the following comments:

 

* We now have templates that were added that are very helpful with our most used chief complaints. The templates were designed based on chief complaint. They make charting easier and quicker than before.

 

* The templates are very useful and have changed the way I document.

 

* Templates helped me to have consistent terminology throughout my documentation.

 

 

DISCUSSION

This was a quality assurance project in a single clinic with the providers in the clinic as participants. Movement in rankings from a level of 5 or 6 to the 7-to-9 category overall, on a scale for which 9 is the highest rating possible, demonstrates a positive perspective in how these providers viewed the electronic record templates. Clearly, there was more satisfaction following the design and implementation of the project. Comments on the postimplementation survey also indicated that the providers were pleased with the implementation of templates. Of particular interest, provider responses indicated a marked improvement in their ratings in term of learning. Prior to template implementation, providers had low response scores in the areas of ease of learning to operate the system, exploration of features by trial and error, and the manner in which tasks could be performed. Following the implementation of templates, response scores improved in these areas. Through the process of template implementation, providers actually improved their knowledge of system capabilities and the process of programming system changes.

 

Increasing emphasis has been placed on the adoption and use of electronic medical records since 2004.2 Focus has shifted from simple adoption to actual meaningful use of the system over the last several years. The purpose of this project was to evaluate the provider satisfaction prior to and following template implementation into the electronic medical record system of a small, privately owned internal medicine practice. As outlined by the TAM,11 a person's attitude about technology use can influence actual system use. This project indicates that provider satisfaction and reactions to a system's capabilities were affected positively by the addition of templates to the electronic medical record system. Further research in the area of template addition and its link to electronic medical record user satisfaction could provide valuable information regarding steps to improve actual meaningful use of a system.

 

REFERENCES

 

1. US Department of Health and Human Services. Electronic health records and meaningful use. 2012. http://healthit.hhs.gov/portal/server.pt/community/healthit_hhs_gov__meaningful_. Accessed April 2011. [Context Link]

 

2. The National Alliance for Health Information Technology. The National Alliance for Health Information Technology report to the Office of the National Coordinator for Health Information Technoloy on defining key health information technology terms. Washington, DC: The National Alliance for Health Information Technology; 2008. [Context Link]

 

3. Hayrinen K, Saranto K, Nykanen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform. 2008; 77 (5): 291-304. [Context Link]

 

4. US Congress. American Recovery and Reinvestment Act of 2009. 111th Congress. 2009. [Context Link]

 

5. Steinbrook R. Health care and the American Recovery and Reinvestment Act. N Engl J Med. 2009; 360 (11): 1057-1060. [Context Link]

 

6. US Department of Health and Human Services. HITECH programs. 2012. http://healthit.hhs.gov/portal/server.pt/community/healthit_hhs_gov__hitech_prog. Accessed April 2011. [Context Link]

 

7. DesRoches CM, Campbell EG, Rao SR, et al.. Electronic health records in ambulatory care-a national survey of physicians. N Engl J Med. 2008; 359 (1): 50-60. [Context Link]

 

8. Centers for Medicare & Medicaid Services (CMS). EHR incentive programs. 2012. http://www.cms.gov. Accessed April 2011. [Context Link]

 

9. Blumenthal D, Tavenner M. The "meaningful use" regulation for electronic health records. N Engl J Med. 2010; 363 (6): 501-504. [Context Link]

 

10. Chaudhry B, Wang J, Wu S, et al.. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006; 144 (10): 742-752. [Context Link]

 

11. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models. Manag Sci. 1989; 35 (8): 982-1003. [Context Link]

 

12. Morton ME, Wiedenbeck S. A framework for predicting EHR adoption attitudes: a physician survey. Perspect Health Inf Manag. 2009; 6: 1a. [Context Link]

 

13. Pare G, Sicotte C, Jacques H. The effects of creating psychological ownership on physicians' acceptance of clinical information systems. J Am Med Inform Assoc. 2006; 13 (2): 197-205. [Context Link]

 

14. Dagroso D, Williams PD, Chesney JD, Lee MM, Theoharis E, Enberg RN. Implementation of an obstetrics EMR module: overcoming user dissatisfaction. J Healthc Inf Manag. 2007; 21 (1): 87-94. [Context Link]

 

15. Bennett KJ, Steen C. Electronic medical record customization and the impact upon chart completion rates. Fam Med. 2010; 42 (5): 338-342. [Context Link]

 

16. Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res. 2010; 10: 231. [Context Link]

 

17. Lau F, Price M, Boyd J, Partridge C, Bell H, Raworth R. Impact of electronic medical record on physician practice in office settings: a systematic review. BMC Med Inform Decis Mak. 2012; 12: 10. [Context Link]

 

18. Rosenthal BJ, Spiegel DM. Computer templates in chronic kidney disease care: a tool for improving patient management. Adv Chronic Kidney Dis. 2008; 15 (1): 37-41. [Context Link]

 

19. Fung CH. Computerized condition-specific templates for improving care of geriatric syndromes in a primary care setting. J Gen Intern Med. 2006; 21 (9): 989-994. [Context Link]

 

20. Rose EA, Deshikachar AM, Schwartz KL, Severson RK. Use of a template to improve documentation and coding. Fam Med. 2001; 33 (7): 516-521. [Context Link]

 

21. Langley GJ, Moen RD, Nolan TW, Norman CL, Provost LP, eds. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd ed. San Francisco, CA: Jossey-Bass; 2009. [Context Link]

 

22. Borkowski N, ed Organizational Behavior in Healthcare. 2nd ed. Sudbury, MA: Jones and Bartlett; 2011. [Context Link]

 

23. Lorenzi NM, Riley RT. Managing change: an overview. J Am Med Inform Assoc. 2000; 7 (2): 116-124. [Context Link]

 

24. University of Maryland Office of Technology Commercialization. Questionnaire for user interaction satisfaction. 2012. http://lap.umd.edu/quis/. Accessed April 2011. [Context Link]

 

25. Harper B, Slaughter L, Norman K. Questionnaire Administration Via the WWW: A Validation and Reliability Study for User Satisfaction Questionnaire. Toronto, ON, Canada: WebNet97, Association for the Advancement of COmputing in Education; 1997. [Context Link]