NIST, ITL, EHR Usability Testing - Part 4 - Supporting Statement

OMB Control No. – 0693-0043 – NIST Generic Clearance for Usability Data Collections

Project title: Study of Electronic Health Record Usability.

PART 4 – Usability testing of commercial EHR products

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

The Information Technology Laboratory (ITL) of the National Institute of Standards and Technology (NIST) will be working with participants consisting of clinical staff (e.g., physicians, nurses and radiology technicians) and non-clinical staff (e.g., hospital unit clerks) at study sites who interact directly with electronic health record (EHR) systems as part of their professional duties at the research site. We will select research sites based on the EHR system that they use, their geographic location, and their availability to participate in the research study. Specifically, we aim to recruit a sample of research sites that use a variety of EHR vendors and at locations within close proximity to the research team’s locations. Participants will be recruited and reminded about participation opportunities via standardized mass communication: e-mail distribution lists and recruitment posters. Copies of the e-mail invitation and recruitment posters have been uploaded into ROCIS. Participation in this study will be open to all individuals, regardless of race, ethnicity, gender, or other personal characteristics. The aim of the recruitment strategy is inclusiveness and no EHR user will be denied the opportunity to participate as long as enrollment is open. Once OMB approval is obtained, the enrollment period will run through May 27, 2013. This group is appropriate to survey because they will provide us with the necessary information to understand how individuals across a variety of professions and institutions use EHR systems.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The survey tool was developed by the contractors and subcontractors in consultation with the team from the NIST. Key personnel from the contracting organization, Wiklund Research & Design (WR&D), developed the first draft of the survey. WR&D sent the survey via email to the subcontracting organizations (Vanderbilt University, Design Science and University of Wisconsin) for review and comments. All subcontractors were given approximately two weeks to submit comments to WR&D. WR&D incorporated the subcontractors’ suggestions for improvement and provided NIST with a draft for feedback. The final draft of the survey reflects the feedback that WR&D received from NIST. Notably, all of the suggestions for improvement were incorporated into the document.

While WR&D did not conduct formal pre-testing of the survey questions and methodology, several of the subcontractors are clinicians who interact directly with EHR systems and could assess the appropriateness of the survey as qualified participants. Moreover, the contractors and subcontractors developed the survey based on materials from previous usability studies that had proved to be an effective means to obtain information about users’ interactions with health information technology (HIT).

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

Participants that have used EHR systems will be recruited to participate in the collection of information via email and recruitment posters. The research team (Wiklund Research & Design and members of the subcontracting organizations) will conduct interviews during the usability test sessions to receive feedback from participants about their interactions with the EHR systems. The participants will perform several tasks with the EHR system during the usability test session. The researchers will ask participants for their feedback after each task and at the end of the test session (filename: Usability test questions). Moreover, the researchers will collect demographic information about participants at the beginning of the usability test session. The usability test session will last about 90 minutes. The researchers will video record the session and take notes. NO identifying information about EHR users or patients, or patients’ protected health information (PHI) will be captured.

Data Management: Usability testing will be video recorded. Once the video has been reviewed and analyzed, the video recording will be destroyed. The researchers’ notes will be stored in a secure electronic database. All usability testing data will undergo a cleaning process to ensure both respondent and affiliated institution anonymity and confidentiality. For example, participant’s narrative responses will be reviewed and edited by authorized research project staff to remove any participant identifying text.

NIST will not have access to any of the raw data. Instead, the prime- and sub-contractors’ will analyze the accumulated data to create Summary Reports that are completely devoid of identifying information or characteristics. Only the Summary Reports will be provided to anyone outside the research team. Therefore, NIST will never possess any information that is traceable to individual participants, specific commercial vendors, or specific healthcare entities. Data will be received and kept by the prime contractor, Wiklund Research & Design.

The research team (Wiklund Research & Design and members of the subcontracting organizations) expects that approximately 90 % of recruited participants will attend his/her scheduled usability test session. We will attempt to improve the response rate by recruiting 10 % more participants than our target sample size at each research site. We will also attempt to improve the response rate by sending reminder emails and asking the department managers within the research sites to verbally remind their staff about the research opportunity.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

The research team (Wiklund Research & Design and members of the subcontracting organizations) will collate and integrate the data collected from the survey instrument with the data from observing participants interactions with the EHRs during the usability test sessions. We will analyze the results by looking for trends in the data overall and across different clinical roles (e.g., physician, nurse, resident, administrative secretary) and departments/facility types (e.g., intensive care unit, clinic, emergency room). Based on the observed data trends, we will develop a narrative description of user characteristics, use environments, EHR-related tasks, functions, goals, and best practices. We will measure associations between variables (e.g., between amount of EHR use and satisfaction with EHR). The findings yielded from this survey will help inform continuing research, guidelines, and policy. The results will generalize to the population to the extent a representative sample can be obtained. In cases where the data cannot be generalized to the entire EHR user population, they may be generalized to specific EHR user groups (e.g., most common tasks performed by EHR users in intensive care units).

The research team will analyze the raw numeric and narrative data to create summary reports that are completely devoid of identifying information or characteristics. The research team will provide only the summary reports to NIST. The research team and NIST will use the summary reports to develop a final report on the usability of EHRs that will be publically available.

3