The recommendations proposed by the Institute of Medicine's (IOM's) 2010 report The Future of Nursing: Leading Change, Advancing Health and the 2010 Carnegie Foundation's study Educating Nurses: A Call for Radical Transformation will require an unprecedented collaboration of registered professional nurses at the national, state, and local levels.1,2 The report and study call for nursing associations and educators to expand opportunities for members, including opportunities for RNs in professional development, professional advocacy, and entrepreneurship. Professional Web sites have common characteristics, such as being well designed, providing content specific to user needs, and putting the content in a language the end users understand.3 The American Nurses Association (ANA) and its constituent state nurses associations use Web sites in order to educate professional nurses about policy, ethics, credentialing, and health and safety. The purpose of this study was to evaluate design of a sample of state nurses association Web sites that are part of the ANA. We incorporated heuristic criteria to assess 27 nursing association Web sites including expert evaluation by five graduate PhD nursing students and two PhD nursing faculty using systematic usability assessments that included heuristic and ethical measures for examining Web sites.

BACKGROUND

The ANA is the professional association for nursing in the US. In 1901, New York was the first state to form a nurses association. Shortly after, in 1903, New York passed the first nursing practice act that created the title of registered nurse.4 A group of state nurses associations came together in 1911 to form what would become the ANA.4 The ANA is a professional association that represents the interests of approximately 3 million RNs in the US through constituent and state nurses associations. The ANA is dedicated to advancing the nursing profession and advocates at the state and federal level for legislation that protects the public and promotes nursing. The goal of the ANA is to ensure competent, well-qualified nurses serve the public, promote healthy work environments, and support a positive public image of nursing.5 Each state nurses association and the ANA provide Web sites to share essential information.

Internet use has increased greatly over the past 20 years. Sixty-eight percent of nurses use the Internet daily, and 72% use e-mail daily.6 Only 1.3% of nurses reported never using the Internet at all.6 More than 80% of nurses report using the Internet to obtain healthcare information.7 Nurses who were most likely to use the Internet for information had recently graduated from a nursing program, were in positions of authority, held multiple positions, were in positions that required ongoing research, had been employed fewer than 5 years, or had subscribed to a journal in the last 12 months.8 In addition, nurses whose work associations provided access to current information, Internet access, and adequate orientation to meet learning needs had greater odds of Internet use.8

State nurses associations may derive several benefits from well-designed, user-focused Web sites. By definition, the goal of any association's Web site is to turn visitors into consumers.9 Because the goal of each state nurses association is to represent, promote, and advocate for its nurse members, the association's Web site should show members how the groups are meeting those goals. Likewise, the Web sites should encourage nurse visitors to become members and subsequently keep members informed so they are effective advocates for the nursing profession. Other benefits include bringing services to nurses previously excluded from receiving the services due to geographic barriers, creating interactive communication with members, engaging in interactive research, and outsourcing activities such as fundraising, event planning, and accounting services.10 One of the biggest benefits is reduced mailing and printing costs by sending informational materials at a very low cost over the Internet.10

The ability to engage visitors is a reflection of a Web site's usability. Usability refers to the ease with which the user can attain, learn, and initially use a Web site. Usability of a Web site includes the quality of the consumers' experience and their satisfaction. The focus of a usability assessment is to determine whether a Web site is user centered and to determine if it meets the needs of users. A Web site's usability influences users' satisfaction and ultimately their intention to use a Web site.11 The concept of usability includes important elements integral to user satisfaction with a Web site such as "ease of learning, efficiency, memorability, error frequency and severity, and subjective satisfaction."12 Usability and effectiveness evaluation can locate problems that might negatively affect the goals and strategies of associations.

Web site usability is also a key factor in a user's perception and impression of an organization.13,14 Usability of a Web site may entice a consumer to want to join the association, remain active with the association, and continue to visit the Web site. As Nielsen15 states, "if a Web site is hard to use, people leave it." For nurses associations' Web sites, ineffective usability means risking missed opportunities to recruit new members, potential loss of existing members, and potential obstacles to meeting the IOM's recommendations.

One way to address the key components of usability is to use heuristic evaluation. Heuristics are agreed upon principles that deem a Web site effective and usable. Using heuristics to evaluate a Web site entails examining a Web site's interface and assessing its compliance with a set of designated pertinent heuristics.16 Heuristic evaluation can identify 60% of usability problems compared with 30% for user testing.17 Heuristic evaluations offer the ability to discover high-level structural problems and address the root cause of usability problems, which helps to improve the use of Web sites.17 Heuristic evaluation is a cost-effective method of providing important feedback to designers and Web site managers, as well as to the organization.

METHODS

Sample

State Web site selection. The continental US was divided into five geographic regions, and a purposive sample of 27 state nurses associations was selected based on geographic distribution and the total number of employed RNs in 201118 (Figure 1; Table 1). We could not locate a state where an association's Web site served a small RN population in the South.

Table 1 State Characteristics by Region and RN Population

FIGURE 1. Geographic distribution and the total number of employed RNs in 2011.

Once the state nurses associations were determined, the Web site review began by exploring two methods to access the Web sites. Initially, we accessed the state nurses association sites through the ANA's official Web site, http://www.ana.org. At the ANA site, access to the state nurses association was obtained through the link, "Find your state." An image of the US appears, and a state could be chosen by clicking on the identified state(s) for the review. Once the state was selected, an ANA's state association communication page would open. Selecting the state link allowed access to the state Web site.

We also explored a second method of visiting state nurses association Web sites to assess whether a starting point that was not the ANA Web site made a difference to the destination site. The Google search engine (Google, Mountain View, CA) was used to search for and navigate to each state association. Results of this investigation yielded different outcomes. For example, the first results for Ohio and Wisconsin state nurses associations were the true state nurses association sites. For Michigan, however, the first result was not the Michigan State Nurses Association but rather the collective bargaining unit for Michigan nurses. A similar event occurred during the search for the California state nurses association Web site. As a result of this initial exploration, all reviewers used the ANA Web site for access to the state Web sites.

Measures

Usability of the Web sites was evaluated using heuristic and ethical principles. Each set of criteria is described below.

HEURISTIC PRINCIPLES

While there are several methods for heuristic evaluation, the most often utilized is based on the 10 principles for user interface design developed by Jakob Nielsen.16 Nielsen's 10 heuristic principles include the following:

1. Visibility of system status. The system should always keep the user informed about what is going on, through appropriate feedback within reasonable time.

2. Match between systems and the real world. The system should speak the user's language, with words, phrases, and concepts familiar to the user, rather than system-oriented terms. It should follow real-world conventions, making information appear in a natural and logical order.

3. User control and freedom. Users should be free to select and sequence tasks (when appropriate), rather than having the system do this for them. Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Users should make their own decisions (with clear information) regarding the costs of exiting current work. The system should support undo and redo.

4. Consistency and standards. Users should not have to wonder whether different words, situations, or actions mean the same thing. The system should follow platform conventions.

5. Help users recognize, diagnose, and recover from errors. Error messages should be expressed in plain language (no codes).

6. Error prevention. The system should have a careful design that prevents a problem from occurring.

7. Recognition rather than recall. The system should make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.

8. Flexibility and efficiency of use. Accelerators-unseen by the novice user-may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. The system should allow users to tailor frequent actions and provide alternative means of access and operation for users who differ from the "average" user in terms of physical or cognitive ability, culture, language, and so on.

9. Aesthetic and minimalist design. Dialogues should not contain information that is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.

10. Help and documentation. Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, be focused on the user's task, list concrete steps to be carried out, and not be too large.

ETHICAL PRINCIPLES: HEALTH ON THE NET FOUNDATION CODE

The Health on the Net Foundation (HONF) has published a widely accepted ethical code for health information Web sites to follow.19 Associations can also request certification from the HONF. This certification advises consumers that the organization is cognizant of the need to meet certain standards when providing healthcare information. The HONF ethical code consists of eight principles or ethical criteria that can be used to assess ethical principles designed into Web sites.19 For example, to meet these ethical design principles, Web site developers must identify person(s) responsible for operation and development of Web site content. Another ethical criterion is met if the Web site designers clearly mention the intended audience on the Web site.

Table 2 provides a list of the heuristic and ethical principles and the total number of criteria evaluated for each principle type. Some specific criteria for each principle type are also listed in Table 2. A total of 68 criteria were used for this analysis.

Table 2 Heuristic Frequencies for 27 State Nursing Web Sites

DATA COLLECTION

Training of Data Collectors

Heuristic evaluations improve when content experts are involved in the evaluation process.20 A "double expert" is a person with both a usability background and professional experience in a specific area, in this case, nursing.17 Double experts are desirable because double experts are able to find 2.7 times more usability problems than single experts or novice users.16 In this study, five nursing PhD students (A.A., V.L., D.P., M.S., E.B.S.) were trained in the heuristic and ethical principles of design and evaluated usability of these professional nursing Web sites using these principles.

One investigator (G.A.) with expertise in heuristic evaluation provided an in-depth verbal review of the heuristic criteria to be used in the evaluation to all members of the research team. Two investigators (G.A. and B.J.W.) then independently reviewed two state Web sites, compared ratings, and came to a consensus agreement by openly discussing differences in the interpretation of the criteria, oftentimes reviewing Web sites a second time with criteria during discussions. These two investigators then reviewed all ratings by the remaining investigators, reaching at least 85% agreement on the review; corrections were made as consensus was reached.

Each Web site was evaluated using a checklist based on the 10 usability principles proposed by Nielsen and the eight HONF criteria (Table 2). For each Web site evaluation, these principles were explored in depth to identify existing design features and to describe any usability issues encountered. On each Web site, all components of the navigation toolbar were visited including all underlying links. When possible, forms encountered on the Web pages were completed both correctly and incorrectly to elicit intentional error messages. External links on each site were visited to identify any possible inactive links. The evaluator attempted to drill down into the Web site as far as possible by clicking all hyperlinks until a dead end was reached or an error occurred. During the evaluation, each characteristic was documented as either a 1 for yes or a 0 for no. Comments detailing any significant findings were recorded on the tool during the evaluation, for example, notes about general aesthetics, titles, tabs, links, and errors to assist during the next evaluation step. The total number of criteria met and violated was analyzed for each state's Web site to determine which heuristic and ethical principles were met or unmet. Reviewers documented their findings on an Excel spreadsheet (Microsoft, Redmond, WA) using criteria with specified definitions as a guide to categorize their findings.

ANALYSIS

Analyses consisted of descriptive statistics identifying the percentage of criteria met for each heuristic and ethical principle. For each of the principles, identified the widest range of values across all 27 states was identified. This range of information was used to identify heuristic and principal criteria with the most variability in usability measures within all states in the sample (Table 2). Additionally, percentile frequencies were tallied for each state by region and by state size by RN population. This analysis helped to determine the mean frequency range for heuristic and ethical criteria by region and by state size by RN population (Table 3).

Table 3 Heuristic Scores: Percentile Frequencies for State Nursing Web Sites by Region and RN Population

RESULTS

The extent to which individual states met heuristic and ethical criteria ranged widely from one state that met 0% of the criteria for "help and documentation" to states that met more than 92% of criteria for "visibility of system status" and "aesthetic and minimalist design." Criteria that showed the most variability across states are listed in Table 2. Results of the review for each criterion are described in detail including the range of positive outcomes and potential areas for Web site improvement.

Criterion 1: Visibility of System Status

Within the heuristic principle "visibility of system status," 25 of 27 (93%) states used menu naming terminology that reviewers indicated was consistent with the user's task domain, in this case, nursing. Other positive findings by reviewers that met this principle were that most state nurses associations provided a consistent location of menu instructions and prompts within each Web page menu. Additionally, 85% of states displayed Web pages with a title or header on the Web page to describe screen contents.

In contrast to these findings, areas for improvement of state nurses association Web sites were that only two of 27 (7%) states in this study had links that changed colors after a link had been selected. A convention established by usability principles typically applied to page design, advises that links change from blue to purple after the link has been selected by a user. Some states used this convention, but altered the colors. For example, some Web sites used blue links, but they changed to green after they were visited. In all of these cases, background colors on the Web page were critical to be able to see links to make navigational choices. Additionally, only 18% of the Web sites used interactive features such as progress bars that kept users informed if there was an observable delay in processing, such as when a Web page or document is downloading. A conventional tool used on many Web pages is a highlighted circular object that appears to be rotating as documents are downloaded.

Criterion 2: Match Between System and Real World

A high percentage of state nurses association Web sites met these criteria. Reviewers indicated that nearly 89% of the Web sites used a very logical approach to order their menu choices. However, embedded within these menus, 33% (nine of 27) of state associations frequently used technical jargon in the form of acronyms, abbreviations, and technical terminology that made text difficult to read. Similarly, 30% of the state associations seemed to use a more informal dialogue incorporating some slang rather than using a more formal style to describe resources on the Web page.

Criterion 3: User Control and Freedom

There was a wide range of criteria met under the heuristic "user control and freedom." Very few sites (7%) had transactional sequences that allowed forms with repetitive fields to be autofilled, such as home address or state. However, a few more state associations (41%) integrated dropdown menu lists to support transactional sequences that allowed users to move the cursor to the correct state. In most cases, these dropdown menu lists were lengthy and difficult to navigate, so some Web sites were designed to enable users to type a mnemonic code corresponding to a state abbreviation that autofilled the state. Only 29% of nursing Web sites supported an undo function to avoid exiting a Web page accidentally that could result in a loss of user information and time. Approximately 22% of sites used clearly marked exits to support users who navigated to an unexpected area within the Web site. Users were free to select and sequence a number of tasks in these Web sites. For example, 70% of the states provided a send or submit button to users that was a signal from the user indicating that a task was ready for completion. Additionally, nearly 90% of the state Web sites allowed users to move forward and backward between fields and Web pages during navigation.

Criterion 4: Consistency and Standards

A high percentage of association Web sites were consistently organized with important information placed in the same location across Web pages. Field labels in forms were highly visible to facilitate better consumer transactions, meaning that labels were located to the left and above list fields. To improve readability and consistency of appearance, 21 associations' Web sites minimized background and foreground colors, using no more than four to seven colors on a Web page.

Criterion 5: Help Users Recognize, Diagnose, and Recover From Errors

Our reviewers encountered a moderate number of errors. Reviewers did attempt to break the system and create as many errors as possible during interactions. Reviewers encountered errors on 59% of association Web sites during reviews. The error messages received at the time errors were encountered did not always provide clear information about what had happened to cause the error. Unclear messages about the status of error occurred 26% of the time.

Criterion 6: Error Prevention

Many associations did use design principles in their Web sites to attempt to prevent errors. Error prevention is enhanced by making the design intuitive to the user, so that actions taken during human computer interactions have anticipated results. For example, 85% of the associations used menu choices that reviewers thought were logical and consistently placed, which helped to meet the expectations of the users so that no unexpected events happen during interactions. Reviewers indicated that 63% of the Web sites were intuitively designed. Additionally, many Web sites (67%) prevented errors by providing alternative selection methods, such as dropdown lists, rather than a direct method of entering data.

Criterion 7: Recognition Rather Than Recall

A wide range of design features was used across the association Web sites to meet this criterion. As stated previously, approximately 7% of the associations incorporated reminders such as links that change color to help users know where they had been within the Web site. Lack of accepted conventions causes a greater memory load for users and can reduce satisfaction with the usability of the Web site. Conversely, prompts and messages on the Web sites were placed in a location where reviewers were likely to be looking at the screen, 89% of the time. Similarly, 85% of the associations used white space and visual cues to distinguish the main parts of the Web site, separating columns, tables, and instructions. Designing the Web site this way can increase efficiency and user satisfaction during interactions by preventing unnecessary searching.

Associations used a wide array of design features to increase visibility and easier retrieval of information to decrease use of memory. Some included using search boxes that allowed for a sufficient amount of characters to facilitate better searching (67%), distinguishing between most recent and best nursing content from the rest of the Web site content (67%), and the use of light, bright colors that saturate and emphasize important content versus darker, duller colors that deemphasize content that is less important (82%).

Criterion 8: Flexibility and Efficiency of Use

These design features are important to help support a wide range of Web site users from novice to expert. Reviewers indicated that associations did support access by novice and expert users in 67% of the Web sites. This access was enhanced by flexible designs that enabled users to move back and forth between Web pages, thereby improving navigational ability without the fear of exiting the Web site accidentally during interactions. However, encounters where transactions required forms to be filled out on an association's Web site could create complications for different user types. Novice users who might not be prepared to complete all the information requested in a form during initial interactions may have to enter and leave a site multiple times. Subsequently, information can be lost if a form does not autofill or information entered by the user is not saved at the time the user exits. Few of the association Web sites (7%) used forms that appeared to autofill or save information after it had been entered.

Criterion 9: Aesthetic and Minimalist Design

According to reviewers, more than half of the sites provided relevant information that was important to nurses and was essential for making decisions (56%). However, some Web sites had no information available at important interaction points. For example, Figure 2 illustrates an anonymous picture of one state Web site with a tab labeled public policy, an important area for nurses to know about and reference; however, there is no information under the tab. This lack of content reduces credibility and impact of the Web site. Aesthetic appeal of these association Web sites was improved by using colorful and distinctive icons that stood out from the background so they were easily identified by the users (74%). Web site designers avoided competition with nonessential information by using short, simple, clear, and distinctive titles to identify Web pages and information, and that were familiar to users (78%); at the same time, menu titles, even though short, were long enough to communicate effectively what information was contained in the Web page (93%).

FIGURE 2. State nurses association Web site example.

Criterion 10: Help and Documentation

Information that supported user interactions through help and documentation information was limited in these associations' Web pages. Reviewers indicated that only nine of the 27 Web sites had a help or frequently asked questions (FAQs) section to support user interactions. Of the Web sites that made help and documentation available, access to this information had limited visibility so that users had to search to find the help or FAQ information. Once the help or FAQ information was discovered, reviewers indicated only 11% of the Web sites offered documentation and help that supported the sequence of user actions. In only seven Web sites were documentation and help provided to users who encountered ambiguous menu items with unclear information. The visual layout for these Web sites was thought to be well designed (63%), and information was thought to be accurate, complete, and understandable in slightly more than half of the Web sites (59%).

Criterion 11: Ethical Principles From the Health on the Net Foundation

There was some diversity meeting ethical criteria for design across associations' Web sites. For instance, reviewers found only five sites that provided a date when the Web site was last modified. Nearly 60% of the associations provided a valid e-mail address for a Webmaster or a link to a valid contact if users had questions about the Web site. This could have implications for users trying to determine if information is up to date and/or accurate. Conversely, the intended mission statements for the Web site and intended audience were highly visible by the reviewers (78%). Most sites (74%) contained information from external sources, and more than half of the sites (56%) contained advertising from external sources. Only 30% of the Web sites contained a privacy, confidentiality, or policy statement regarding e-mail addresses or other information posted on Web sites.

In the final analysis, the percentage of heuristic and ethical criteria met within sites were compared by size of employed RN population and geographic region. As shown in Table 3, the states with the largest number of employed RNs in the sample performed the best at meeting criteria; the Midcentral region performed the best when compared with other regions.

There was wide variability across states in the percentage of criteria met and the types of criteria met. Individual state scores ranged from a low of 26% (one Western and one Midwestern state) up to 75 % (one Midwestern state). Meeting criteria did not vary greatly by state size (ie, number of employed RNs in the state) but ranged from a low of 48.8% in Western states up to 63.8% in Midcentral states.

DISCUSSION AND RECOMMENDATIONS

Heuristic evaluation of a Web site will discover usability problems in a quick and efficient manner. Incorporating double experts, doctoral nursing students who have been trained to evaluate usability, adds credibility to the results of this study of state nurses association Web sites. As nurses, the evaluators understand what to expect and have experience with their own state association Web sites. An expert evaluator will find more usablilty problems than a novice evaluator. Recognized and accepted heuristic and ethical principles on Web site design provided a mechanism for the first peer-reviewed evaluation of state nurses association Web sites to be completed. The feedback provided can be used to improve each state association's Web site usabilility.

There are opportunities for most of the nurses associations sampled to make improvements in their Web sites' usability by using best practices for Web site design by focusing on aesthetic appeal, enhancing visual representation of important information, and focusing on designing for maximizing user experiences.21 The suggested improvements would be simple yet make an impact on a first-time visitor's impression of the associations. Small changes such as adding internal navigation and tracking features, as well as providing more details through help and FAQ documentation, could potentially attract users. A common evidence-based method for evaluating usability and Web site design is to consider if the Web sites are effective at supporting user needs, if they are efficiently designed, and if they satisfy consumers.22,23

Effectiveness

Effectiveness of a Web site enables a user to complete a task and enhances the usefulness of a Web site to the user. Criteria that can enhance effectiveness of a Web site provide a user with adequate controls and freedom to move about the Web site without encumbrances. Error prevention strategies are also crucial. The state nurses association Web sites were designed well enough that few errors were made by reviewers. Specifically, by providing logical and consistent placement of information on the Web page, state nurses associations enabled users to detect the few problems encountered.

One area to improve effectiveness of these Web sites is to provide better mechanisms for information to be automatically populated in specific fields required for the community to complete. Furthermore, if transactional forms were automated more fully, transaction processes could be improved in ways that would limit the number of errors that could potentially be made by Web site users. One limitation of this study was that our reviewers were not members of the state associations for the Web sites reviewed. We purposely assigned reviewers who were not members of the selected state nurses associations to minimize bias during the review. Therefore, reviewers had some difficulty getting into Web pages that required forms to be filled out, especially in Web pages that were members-only Web pages.

Another area to improve effectiveness of these Web sites is to develop better help and documentation systems to support users. In this sample, about one-fourth of the Web sites had some help and documentation available, but users had to search to find it. Users who do not know where to search for information on a Web site or find that information on a Web site is cryptic often report negative experiences. Designing resources such as help and documentation instructions or FAQ documents provide valuable information to users who want to find information, although they do not know the Web site very well. This prevents unnecessary searching that could entice a user to leave a Web site altogether out of frustration.

Efficiency

Efficiency is measured by a user's ability to conduct business with minimal effort. Many of the Web sites reviewed in this analysis used aesthetically pleasing design and color to enhance critical information. Interactive icons were visible to users and were enhanced to limit blending with common background colors. Using color to identify important sections of a Web site or to draw the eyes of a user to specific information is a critical design element that can increase efficiency and improve interactive experiences. Colors can evoke important emotions in a user, which can either leave a user feeling overwhelmed or can help him/her to navigate to the most important message that an association wishes to communicate.

Web sites reviewed in this study would benefit from enhancing visibility of the Web sites status as the user moves through various Web pages. For example, very few of these Web sites used conventional functions such as having links change colors when a section of a Web site is visited. Some of the Web sites incorporated the conventional function but altered the way it was used, which required more thought by the user. Bypassing conventional mechanisms can become a problem when there is depth of information within a Web site requiring complex navigation and multiple clicks to get to vital information especially for novice users who are exploring Web sites.

Satisfaction

User satisfaction with Web site design was not measured as an outcome of this study. This would be an important and relevant research question for leaders of state nurses associations to consider as they evaluate whether association Web sites are enhancing membership efforts. Certainly, the effectiveness and efficiency issues that were identified would affect user satisfaction as members or potential members interacted with state association Web sites. This type of satisfaction measure could be proxy for determining how satisfied members are with the services provided by state associations they belong to as well. Greater satisfaction and usability could improve members' activity levels in state association Web sites.

CONCLUSION

State nurses association Web sites provide a window into the nursing profession that can enhance current membership experiences. State nurses associations have a great opportunity to use their Web sites to promote the profession of nursing, ethics, and safety and health. Evaluating usability of state nursing association Web sites using recommended heuristic and ethical principles could help identify opportunities to improve member interactions during Web site visits. State associations have a wide range of usability issues including limited amount of help documentation and FAQs to help users during interactions and frequent use of minimalist designs. Taking advantage of better designs and usability of state association Web sites could contribute to recommendations made in the IOM 2010 report on the future of nursing and the Carnegie report on educating nurses.

REFERENCES