My HealtheVet Summative Usability Test / VE Non-Sub Change Request_WH HotlineCallSurvey Gender

MyHealtheVet & My VA Health Journal Joint Study

OMB No. 2900-0770
Estimated Burden: 60 minutes

Expiration Date: 9/30/2020

The Paperwork Reduction Act of 1995: This information is collected in accordance with section 3507 of the Paperwork Reduction Act of 1995. Accordingly, we may not conduct or sponsor and you are not required to respond to, a collection of information unless it displays a valid OMB number. We anticipate that the time expended by all individuals who complete this survey will average 60 minutes. This includes the time it will take to follow instructions, gather the necessary facts and respond to questions asked. Customer satisfaction is used to gauge customer perceptions of VA services as well as customer expectations and desires. The results of this telephone/mail survey will lead to improvements in the quality of service delivery by helping to achieve services. Participation in this survey is voluntary and failure to respond will have no impact on benefits to which you may be entitled.

My HealtheVet & My VA Health Journal Joint Study Plan

Veterans Health Administration

Human Factors Engineering

April 18, 2018

Table of Contents

Introduction 3

Study Details 3

Application Description 3

Study Objectives 3

Method Overview 4

Study Design 4

Participant Recruitment, Screening & Scheduling 5

Tasks 5

Findings and Recommendations 7

Preliminary Schedule 8

Assumptions & Constraints 9

Appendix A: Usability Testing Materials 10

Appendix B: Session Test Scripts 11

Appendix C: Human Factors Engineering Survey 16

Appendix D: HFE System for Ranking Usability Findings 18

Introduction

Study Details

Study Author(s): William Plew ([email protected]), VHA OHI HFE

Robert Gluck ([email protected]), ArcSource Group

HF Point of Contact: Nancy Wilck ([email protected]), VHA OHI HFE

Application: My HealtheVet & My VA Health Journal

Study Sponsor: Connected Health

Developer POC: Jeff Sartori, ([email protected]), VHA Connected Health

Devices(s): Windows 7 Laptop using Google Chrome

Application Description

The My VA Health Journal App (MVAHJ) shares a number of self-entered data modules which overlap with existing modules of MyHealtheVet (MHV). These modules serve as repositories of self-entered health information; however, only a portion of MVAHJ’s information is shareable with a provider. None of the MHV information is shareable. This evaluation will look at Veteran end-users’ perceived usefulness, efficiency, and effectiveness of the modules of MVAHJ and MHV.

Study Objectives

The following objectives have been determined by Human Factors Engineering (HFE) and the Program Office:

• Collect qualitative feedback on the perceived usefulness of the self-entered health data modules in MHV and MVAHJ.

• Perform an AB Test between the two areas, having Veterans enter health data into both to determine the modules’ effectiveness and efficiency.

HFE will recruit 18 participants both familiar and unfamiliar with MHV. Familiarity with MVAHJ is not a criterion, as the app is not available to the public at this time. HFE will record the interviews via WebEx and then code the qualitative findings to identify similar trends across Veteran interviews. After this study, HFE expects to have a set of recommendations based around user perceptions and steps for a potential follow-up study.

Method Overview

Study Design

This study will employ an AB test methodology, which involves users completing similar health data entry tasks in MHV and MVAHJ. The order of introduction of MHV or MVAHJ tasks will be counterbalanced to avoid any potential learning bias. Qualitative surveys will be included after each task and at the end of the study to assist with fulfilling the first objective. The qualitative feedback obtained via the surveys, in addition to the quantitative findings obtained during the AB test, will allow HFE to develop a set of findings and recommendations.

To evaluate the two interfaces, we will conduct 18, 60-minute sessions. The participants will all be recruited via an HFE recruiting vendor. The participants will connect to the facilitator’s computer via WebEx and take control of the facilitator’s device to complete tasks in both interfaces. Each session will proceed according to the following agenda:

• Greeting and instructions about the test session (includes verbal consent to being recorded).

• Opening demographics questionnaire.

• Task completion (see Tasks section below).

• Closing questionnaire.

• Follow-up questioning by representatives from the Study Sponsor and/or subject matter experts (if time allows).

• Closing.

A follow-up survey will be sent to each participant to gather feedback on how the HFE Office conducted the session.

Representatives from the study sponsor and/or technical subject matter experts are encouraged to attend the sessions, but should be silent observers to avoid any kind of influence on the participant’s behavior (including coaching or disagreement with the participant). This is critical in order to capture the required task success metrics to verify usability. The following are guidelines for observers:

• Please do not participate in the discussion unless the moderator has invited you.

• Do not interrupt the participant discussion.

• During the discussion, refrain from explaining the website/app.

• Do not tell the participants that they are wrong (this may discourage them from sharing their feedback).

• Once invited by the moderator, you can ask the participants to repeat a statement (to better understand what they said) or follow up/expand on a comment and/or observation.

Each session recording will be analyzed post-session by an HFE team member.

Participant Recruitment, Screening & Scheduling

HFE will work with the study sponsor to establish the demographics associated with people who would make use of either MHV or the MVAHJ app. The resulting recruiting criteria will be shared with the contracted recruitment vendor. Recruited participants will be scheduled by the recruitment vendor after HFE and the study sponsor agree upon the schedule in the corresponding Preliminary Schedule.

Tasks

The participants will be asked to carry out fictional but realistic tasks within MHV and MVAHJ. Tasks have been written in coordination with the study sponsor to fulfill the objectives of the study.

Task 1: Allergies

Task Description: Please add a new self-entered allergy for walnuts. You had a moderate reaction to them, but did not require medical attention. This allergy was discovered on 4/30/2018.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 2: Medications

Task Description: On 4/28/2018, you started taking one 100mg aspirin tablet daily to help with heart health. Please add this into your Medications list.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 3: Goals

Task Description: 3 months from today, you’ll be attending a wedding. You want to set a goal to lose 30 lbs. by the time of the wedding. Go ahead and enter the goal into your tracker.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 4: Activity Journal

Task Description: To help lose weight for the wedding, you’ve decided that every day you’re going to spend 30 minutes going for a walk before your morning meetings. While it will be light exercise, you’re hoping this will help with your goal. Please enter this activity.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 5: Personal Trackers

Task Description: Along with the exercise, you plan on using the app to track your weight loss. Please enter a weight for today of 200 lbs.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 6: Family Health History

Task Description: Your aunt on your father’s side was recently diagnosed with type 2 diabetes, and you want to update your family medical history to show this information.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Findings and Recommendations

The final report will include detailed findings that represent the observations, interpretations, impact, and recommendations from the usability professionals involved with the study. Findings will be primarily qualitative, with some quantitative statistics. Each usability issue will have an assigned ranking and associated recommendations for improvements. Screen captures and/or video clips will be included to contribute to the explanation of the issues found. See Appendix D for a discussion on usability rankings.

User Performance Measures

The table below outlines the user performance measures that will be reported as part of this study. Please note that, because of the small sample size, HFE will not be able to infer statistical significance. Detailed performance data, for each task and each participant, will be provided in the form of an Excel workbook, along with the final PowerPoint report.

Effectiveness
Task Success	Ratio of Task Success (# of successes/# of attempts)
Task Errors	Ratio of Errors (# of errors/# of attempts)
Error Count	Total # of Unique Errors
Efficiency
Task Times	Mean Time (in minutes or seconds) of Successful Attempts (per Task)
Satisfaction
Task Satisfaction	Mean Response to Post Task Satisfaction Questionnaires (per Task)

Table 1: User Performance Measures

Preliminary Schedule

The following timeframes are provided to plan, execute, and report on the study. A complete session schedule will be included in the final report.

Task	Responsible Party (ies)	Date
Develop Study Plan	By April 18	HFE Team
Recruit Participants (18 total)	By April 27	HFE Team
Dry Run	April 30	HFE Team/Jeff
Conduct Study	Start May 1**	HFE Team
Begin Analyzing Data and Developing Findings	Start May 14	HFE Team
Deliver Draft Report & Briefing to Sponsors	May 30	HFE Team
Final Report Delivery to Sponsors (with any necessary revisions)	May 31	HFE Team

*Dates are tentative and may change based on scheduling and other constraints.

** 5 days must be waited prior to OMB submission before the study can commence.

This study plan includes draft scenarios and success criteria based on HFE’s understanding of MHV and the MVAHJ app. Upon delivery of the study plan, HFE will work with the study sponsor and developer POC to ensure that the scenarios and success criteria are appropriate and meet study objectives. Following finalization of the scenarios, HFE will complete the configuration files and other materials needed to conduct the sessions.

The dry run will consist of a working session with the study sponsor to ensure that the test materials (including scenarios/tasks), technical configuration, and roles of all members of the study team (including the moderator, technical support team member and observers) do not require revisions prior to the first session with a participant. In order to proceed with this session, the following are required:

• Complete scenarios/tasks (prepared by the study sponsor, consolidated by HFE and agreed upon by the study sponsor).

• Preparation of a Morae configuration file for each session (prepared by HFE using scenarios and tasks prepared by the study sponsor. This will be used in the background by a notetaker or facilitator to enhance post-study analysis).

Complete study materials can be found in Appendix A: Usability Testing Materials.

Assumptions & Constraints

Several assumptions and dependencies have been determined:

• Test sessions will be held remotely using a web conferencing software, such as WebEx.

• Stable and consistent test environments and test data with credentials exclusive to HFE.

  • HFE is provided with credentials, limited solely to HFE, for the duration of the testing.

  • Business owner agrees that the application will not be modified from the first test session through the last test session.

• An SUS test will not be performed, as participants may be confused on which interface they are being asked about, potentially skewing the value of this test.

0. 
   
   
   

Appendix A: Usability Testing Materials

The following study materials include:

• Session Checklist

• Session Agenda

Session Checklist

Prior to all sessions:

• Ensure test configuration is running properly (including recording equipment).

Prior to session:

• Confirm schedule with participant (within 24 hours of session).

• Ensure WebEx is running and computer running Morae Recorder has the Presenter Role.

• Have links ready to chat to participant:

• https://www.myhealth.va.gov/mhv-portal-web/home

• https://vet-test.mobilehealth.va.gov/my-va-health/

After each session:

• Distribute the Morae recording.

• Compile markers into the Morae recording.

Session Agenda

Time	Item
5 min	Introduction Greet the participant. Read introductory script. Consent to Record & Begin Morae and WebEx Recording. Opening Questionnaire
50 min	Task Completion Read pre-task briefing before participant begins tasks.
8 min	Task 1: Allergies
8 min	Task 2: Medications
8 min	Task 3: Goals
8 min	Task 4: Activity Tracker
8 min	Task 5: Personal Tracker
8 min	Task 6: Family Health History
5 min	Closing Questionnaire Within Morae
5 min	Closing Thank the participant for their time and cooperation. Ask the participant to complete the HFE survey. Ask for any final questions.

Appendix B: Session Test Scripts

Introduction

I will be reading from a script to standardize the session for all participants.

I’d like to start by thanking you for participating in this study today. We rely heavily on volunteers such as you to make VA’s systems better for Veterans. I am your facilitator, <insert moderator name>, from Human Factors Engineering. We also have <insert names> who will be observing today and helping with the technology. They may also be taking notes. If we have extra time at the end of the session, I may open the call for questions from these people. But, more than likely, you will only be interacting with me today.

Our session today will last no longer than 60 minutes. There are three sections to the testing today. First, we will begin with a brief opening questionnaire to collect demographics information. Then, we will watch while you complete fictional but realistic tasks that test specific functions of both My HealtheVet and the My VA Heath Journal App. We will wrap up with a closing questionnaire designed to gather your impressions on the usability and utility of My HealtheVet and the My VA Heath Journal App.

Your participation is completely voluntary, and you may withdraw at any time. All the information that you provide will be kept confidential, and your name will not be associated with your use of the system today or any feedback that you provide, verbally or via on-screen questionnaires. To compile your use of the system and your feedback on it with other participants in this study, we would like to record the session. Do we have your permission to record your voice and screen interactions?

OK, we will start recording now. <Start recorders>

Now that we are recording, for our records, can you verbally confirm that we have your permission to record your voice and screen interactions?

Do you have any questions or concerns before we get started?

Opening Questionnaire

The following questions are included in the questionnaire presented via Morae.

We will begin with some background questions about you. Please take your time and if you have any questions you are welcome to ask me for help.

You should see a questionnaire on the screen. Please complete it.

1. Gender: □ Male □ Female

2. Age Range: □ 20s □ 30s □ 40s □ 50s □ 60s □ 70s or older

3. How often do you use MHV? □ Daily □ Weekly □ Monthly □ Rarely or Never

4. When it comes to technology, what best describes you?

1. I am skeptical of new technologies and use them only when I have to do so.

2. I am usually one of the last people I know to use new technologies.

3. I usually use new technologies when most people I know do.

4. I like new technologies and use them before most people I know.

5. I love new technologies and am among the first to experiment with and use them.

5. When it comes to self-entered information in general (journaling) how frequently do you keep updates? □ Daily □ Weekly □ Monthly □ Rarely or Never

6. Have you ever participated in a usability session before?

All right, at this point we’ll move onto the first set of screens.

Task Completion

I want to explain that we are interested in uncovering your perception of the two systems you’ll be interacting with today. In other words, we are interested in seeing how you complete the tasks and finding out if there are system elements or functions that are frustrating or confusing, or alternatively elements that you like or find pleasing. As you complete the tasks, please let us know if you find anything hard to follow or if you’re not sure what to do next. There are no wrong answers to any of the questions.

We appreciate any comments and ask that you “think out loud” as you are presented with the screens. Basically, we’d like any thought that comes into your head to come out of your mouth. For example, if I were driving to work I might say as I get in my car, “I’m opening the car door and sitting down, my seat’s really hot from sitting in the sun…I should put it in the shade tomorrow. I’m putting on my seat belt. I’m almost out of gas, I need to remember to get some tonight on my way home.” Hearing this type of narration helps us know your concerns, any issues that might come up in the process and why you’re going through the process the way that you do.

As you proceed through the tasks, you may have questions. Please feel free to ask whatever comes to mind, but I will refrain from assisting directly with a task unless you are sure you're stuck and cannot continue. Please be aware that due to the technical setup of the test sessions, there may appear to be some performance latency-the app might move a little slower than it normally would. If it becomes particularly bad, please let me know.

I’m going to read aloud each task for you before you start and ask if you understand the scenario. If you don’t, please feel free to ask questions to clarify. This is important because we are not testing whether you understand the task. When you confirm that you understand the task, you can simply select “Start Task” When you feel like you have completed the task, you’ll be able to click the “End Task” button above and then you’ll have a question regarding how difficult you felt it was to accomplish the task. If you need a reminder at any point of what the task is, just ask and I’ll read it to you again. There are 12 tasks in this session.

Do you have any questions about how the set-up works? Do you have any questions before we start with the first task?

Tasks

Task 1: Allergies

Task Description: Please add a new self-entered allergy for walnuts. You had a moderate reaction to them but did not require medical attention. This allergy was discovered on 4/30/2018.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 2: Medications

Task Description: On 4/28/2018, you started taking one 100mg aspirin tablet daily to help with heart health. Please add this into your Medications list.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 3: Goals

Task Description: 3 months from today, you’ll be attending a wedding. You want to set a goal to lose 30 lbs. by the time of the wedding. Go ahead and enter the goal into your tracker.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 4: Activity Journal

Task Description: To help lose weight for the wedding, you’ve decided that every day you’re going to spend 30 minutes going for a walk before your morning meetings. While it will be light exercise, you’re hoping this will help with your goal. Please enter this activity.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 5: Personal Trackers

Task Description: Along with the exercise, you plan on using the app to track your weight loss. Please enter a weight for today of 200 lbs.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Task 6: Family Health History

Task Description: Your aunt on your father’s side was recently diagnosed with type 2 diabetes, and you want to update your family medical history to show this information.

Moderator Question: How would you rate the difficulty of this task on a scale of 1 to 5, where 1 is very easy and 5 is very difficult?

*After each interface*

Moderator Question: Which interface allowed you to complete the task more efficiently?

Moderator Question: Which of the two interfaces did you prefer for completing the task and why?

Closing Questionnaire

Please hit next after you finish the first page of the questionnaire. You should now see a second questionnaire on the screen. Please complete the questionnaire.

• What was your overall impression of the interface in MHV?

• Did you find any parts of the MHV interface confusing?

• Yes

• No

If yes, which parts?

• How likely will you be to use MHV for entering self-health data in the future?

□ Definitely □ Likely □ Neutral □ Not Likely □ Definitely Not

• What was your overall impression of the interface in MVAHJ?

• Did you find any parts of the MVAHJ interface confusing?

• Yes

• No

If yes, which parts?

• How likely will you be to use the MVAHJ for entering self-health data in the future?

□ Definitely □ Likely □ Neutral □ Not Likely □ Definitely Not

• Were there any features that you were surprised to see?

• Which of these interfaces would you recommend to other Veterans? □ MVH □ MVAHJ □ Neither

• If you had to pick between MVH or MVAHJ which would you prefer? □ MVH □ MVAHJ

Closing

This concludes our session today. We will be combining your feedback with that of other people who participate in this study. When all the sessions are complete, we will be delivering a final report based on the combined feedback.

Once again, thank you for participating today. Your feedback is invaluable in helping us to evaluate both MHV and the My VA Health Journal. If you have any additional feedback after our session today that you would like me to consider for inclusion in the final report, please feel free to email me.

I would like to ask you one more favor before we hang up. Later today you will receive an email from me with links to one more questionnaire. To continually improve how Human Factors Engineering conducts studies, we would like your feedback on the session carried out by our team today, including my facilitation. The email will include a link to a questionnaire to gather that feedback. It is very short, including only a handful of questions, and should only take a couple of minutes. There will be a code in the email that you should use in the questionnaire to differentiate this study from other studies currently underway by Human Factors Engineering.

Do you have any remaining questions or comments for me today?

Thank you again and have a fantastic day!

Appendix C: Human Factors Engineering Survey

HFE Feedback Survey

The survey included in this appendix was sent to each participant after the session was completed via follow up email. The email included the link to the Survey Monkey: https://www.surveymonkey.com/r/9XVVRT6.

Thank you for your participation in our study. It was a pleasure to work with you, and we really appreciate your time.

To continually improve how studies are conducted by Human Factors Engineering, we would like your feedback on the recent test session carried out by the team.

• Please enter the session identifier provided to you by your Human Factors Engineering facilitator.

• The Human Factors Engineering team sufficiently explained the purpose of the session and my role as a study participant.

• Strongly Agree

• Agree

• Neutral

• Disagree

• Strongly Disagree

• I feel the Human Factors Engineering team considered and understood the input I provided during this session.

• Strongly Agree

• Agree

• Neutral

• Disagree

• Strongly Disagree

• The Human Factors Engineering team planned and carried out this session so that my time was well utilized.

• Strongly Agree

• Agree

• Neutral

• Disagree

• Strongly Disagree

• Overall, I am highly satisfied with how this session was carried out by the Human Factors Engineering team.

• Strongly Agree

• Agree

• Neutral

• Disagree

• Strongly Disagree

• We are dedicated to providing you the best possible experience. If you have identified areas where we need to improve, could we contact you to discuss further? This is completely voluntary, and if you are willing to be contacted, please include your contact information in the comment box below. Thank you.

Appendix D: HFE System for Ranking Usability Findings

Human Factors Engineering (HFE) evaluates Health Information Technology (HIT) products’ strengths and weaknesses to materially improve their usability. We call these strengths and weaknesses findings. When a finding is related to a weakness in usability, HFE works with development teams to resolve or mitigate the issue. Since many evaluations uncover multiple weaknesses, development teams cannot always address all findings. HFE has developed a ranking system to help business owners prioritize resolution of the findings.

The Ranking System

HFE adheres to the following ranking system to report on the severity of usability findings:

Ranking	Definition	Recommended Priority for Resolution
Minor	One of more of the following: Causes user hesitation, confusion, or slight irritation. Impedes task completion or decreases efficiency but does not cause task failure. Presents small likelihood that the credibility of the VA HIT product will be diminished.	Consider resolving this issue.
Moderate	One or more of the following: Causes occasional task failure after which recovery is possible. Causes user delays and/or moderate dissatisfaction, but some users are able to recover in order to complete the task. Expected to negatively impact use, possibly leading to dissatisfaction at a level that users might opt to discontinue use. May diminish the credibility and/or reputation of the VA product.	Give high priority to resolving this issue.
Serious	All of the following: Causes frequent task failure or occasional task failure from which recovery is not possible. Causes extreme user irritation and/or task abandonment. Likely to diminish the credibility or reputation of the VA product. Or: Causes system/sub-system failure (i.e., produces system error or “crash”).	Give highest priority to resolving this issue prior to further product testing or release. HFE recommends resolution or mitigation for serious usability issues before deploying products.
Strength	Any findings related to strengths in the system.	Optional.
Suggestion	Finding is purely an unsolicited suggestion from a participant or the study team. Finding is not related to an identified strength or weakness in the system.	Optional

• HFE Usability Findings Ranking System

How to Employ the System

Regardless of the usability evaluation method employed, HFE categorizes all weaknesses by the above ranking system.

Frequency of Occurrence

Frequency should always be reported separately from severity because a serious problem may be experienced by a low number of users but could lead to disastrous consequences. However, development teams may choose to use frequency as a means of determining priority for resolution, especially if the resolution is associated with a low level of effort.

• HFE delivers a Problem by Participant Matrix with [end user] usability testing reports to provide frequency information. When a heuristic inspection method is used, frequency might be estimated; however, frequency of occurrence is best tabulated directly from end user experience (i.e., [end user] usability testing, field testing, flurry data).

Reporting “Bugs” and Content Issues

Occasionally, a product under evaluation may prove defective, producing system errors or including grammatical errors or content that is not written to Plain Language standards. This is especially likely to occur when HFE is working with development teams early in the development lifecycle (which is preferable for producing the most usable end product). If/when defects are identified that result in a system error or “crash”, these issues are reported with other usability issues as “Serious”. This ranking is appropriate because these types of defects will cause task failure, extreme irritation (or task abandonment) and will diminish the credibility or reputation of the product.

Content issues and defects not resulting in a system error or “crash” may be grouped together as children of a “parent issue”. The parent issue will be marked as “Serious” if the combined experience diminishes the credibility or reputation of the VA. In this manner, HFE will assist development teams in prioritizing and resolving defects and content issues.

Consistency in Ranking

It is not uncommon for practitioners to find different weaknesses and/or disagree on the ranking of weaknesses that they collectively identify. This is due to a variety of factors, including professional experience, minor differences in methods, how tasks are defined and the users involved (or not) in the method. The system is designed to aid in consensus by providing a standardized set of definitions by which findings are ranked. In addition to these definitions, the following are examples, developed as a result of an HFE group Heuristic Evaluation performed for the purposes of clarifying this system:

Ranking	Example
Minor	Use of “Click here for more” to take user to an external link.
Moderate	Inconsistent access to app navigation (e.g., menu button alternates between the right and left side, depending on page).
Serious	Blank page in app. Broken [external] web link (e.g., link has changed). Inaccessible web link (e.g., link is behind firewall, but app user is not). Use of language that is not easily comprehended by end users

• Examples of HFE Usability Findings Rankings

Regardless of definition and/or example, practitioners are expected to collaborate in order to reach consensus when variability is present. As a best practice, HFE performs peer reviews on all reports. As such, at least two practitioners review and agree on all findings and their severity as reported to product teams.

Page 18 of 20