Supporting Statement A

Health Resources and Services Administration

SUPPORTING STATEMENT

National Practitioner Data Bank (NPDB) Usability Surveys

1. Justification

1. Circumstances of Information Collection

The Health Resources and Services Administration (HRSA) currently has approval under a generic clearance, Office of Management and Budget (OMB) Control No. 0915-0212, to conduct customer satisfaction surveys and focus groups. This collection of information will help fulfill the requirements of:

1. Executive Order 12862, “Setting Customer Service Standards,” which directs agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector;

2. The Paperwork Reduction Act (PRA) of 1995, which is designed to reduce the total amount of paperwork burden the Federal government imposes on private businesses and citizens;

3. Federal security requirements established by the National Institute of Standards and Technology, the Federal Information Security Management Act, and HRSA; and

4. The statutes and regulations that govern and maintain NPDB operations include:

   1. Title IV of Public Law 99-660, Health Care Quality Improvement Act (HCQIA) of 1986,

   2. Section 1921 of the Social Security Act,

   3. Section 1128E of the Social Security Act,

   4. Section 6403 of the Patient Protection and Affordable Care Act of 2010

The NPDB regulations implementing these laws are codified at 45 CFR Part 60.

HRSA’s Division of Practitioner Data Bank (DPDB) manages the NPDB and seeks approval to conduct usability testing to improve the NPDB’s password-protected, restricted data system. Specifically HRSA is requesting OMB approval of voluntary usability test surveys under HRSA’s generic clearance.

Note: As used in this document, the term “NPDB system” includes both the password-protected data system and the public website, found at www.npdb.hrsa.gov. The password-protected data system is also known as the IQRS, or Integrated Query and Response System.

The NPDB is a web-based repository of reports containing information on medical malpractice payments and certain adverse actions related to health care practitioners, providers, and suppliers. The NPDB assists in promoting quality health care and deterring fraud and abuse within health care delivery systems. When changes are made to the NPDB system, usability testing is needed to ensure that the system is user-friendly, minimizes user burden, facilitates compliance, and maximizes return on investment. In addition, usability testing helps to identify errors or bugs that may be present in the system so that they can be fixed before enhancements are put into production.

The International Organization for Standardization defines “usability” as the extent to which a product can be used by customers to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

• effectiveness: the accuracy and completeness with which specified users can achieve specified goals in particular environments

• efficiency: the resources expended in relation to the accuracy and completeness of goals achieved

• satisfaction: the comfort and acceptability of the work system to its users and other people affected by its use ISO 9241-11

Executive Order 12862 directs agencies that “provide significant services directly to the public” to “survey customers to determine the kind and quality of services they want and their level of satisfaction with existing services.” The overarching objective of surveying users of the IQRS prior to putting changes and enhancements into production is to ensure the usability of the NPDB system and to identify system bugs or issues that would hinder usability.

2. Purpose and Use of the Information

The purpose of this data collection effort is to ensure that the NPDB system and any future enhancements are user-friendly. Data obtained from this effort will be used by DPDB and its contractor(s) to identify strengths and weaknesses in the NPDB system and to identify issues that need to be remedied to allow for ease of use.

DPDB or its contractor(s) will solicit voluntary participation among current or prospective NPDB users in usability testing. The IQRS is not open to the general public, and by statute and regulation only certain entities are permitted to use it. If a respondent agrees to participate, the usability testing will be conducted online (e.g., WebEx). In rare cases, respondents may be asked to participate in person during DPDB-attended conferences (e.g., the National Association of Medical Staff Services). Respondents will be asked to perform specific tasks related to the purpose of the usability test. For example, if the usability test is about a system enhancement to improve the reporting process, respondents may be asked to enter a report into a pre-production testing environment. After the usability test is complete, respondents will be asked a series of questions about their experience.

To obtain information about the user’s experience with the NPDB system, the NPDB proposes to use a survey that consist of the System Usability Scale (SUS),¹ a standardized, 10-item Likert scale of system usability developed by John Brooke at Digital Equipment Corp., and a series of open-ended questions. As a standardized instrument, SUS is used in its current, copyrighted form. Through standardized scoring, SUS will provide DPDB with quantitative information about how usable the system is for a range of domains: efficiency, effectiveness, and satisfaction.

Because SUS measures usability quantitatively, it is not used for diagnosing specific system issues. Therefore, open-ended questions will provide qualitative information about strengths and weaknesses of the IQRS to assist in diagnosing the NPDB system. Additional questions may be

asked for probing purposes. Questions related to IQRS are presented on the Usability Testing Survey-NPDB System form.

Data gathered from the usability surveys will be used only for internal purposes; no data will be disseminated outside DPDB or HRSA.

If this data is not collected, DPDB will have no knowledge of how user-friendly the NPDB system is for its customers, will not be able to effectively minimize user burdens and facilitate compliance, and will not be able to determine the return on investment to implement enhancements to the system. Based on anecdotal feedback from some users in the past, difficulty with using the system (e.g., entering a report) has resulted in entities not filing required reports. Collecting usability information will allow DPDB to continually enhance the system so that all NPDB users can utilize the system with ease, which will increase overall use and, thus, the value of the NPDB to the U.S. health care system.

3. Use of Improved Information Technology

The usability tests will be conducted online (e.g., WebEx) to allow for respondents to schedule the test at a time and place convenient for them. Usability testing also may be done in-person at national health-related conferences that DPDB attends. (The test would still be conducted on a computer in a pre-production environment.) The survey will likely be given by DPDB staff or its contractor(s) either in person or via telephone. However, DPDB also may use technology, such as a web-based survey platform, to administer the survey. Whether oral, electronic, or written, the OMB control number will be either read to the respondent or clearly visible on the survey.

4. Efforts to Avoid Duplication

This data collection is not duplicative of other DPDB information collection efforts. It will be conducted during specific instances around IQRS enhancements or changes.

5. Involvement of Small Entities

The survey will not have a significant impact on small businesses or other small entities.

6. Consequences if Information is Collected Less Frequently

DPDB proposes to conduct usability testing for all major enhancements as necessary. Consequences of not conducting usability testing at all, or less frequently (e.g., not for each enhancement), could mean that enhancements would not be user-friendly. DPDB would have no knowledge of a user’s experience with the system and could design enhancements that are not user-friendly or that contain issues that hinder its use. In addition, DPDB would not be able to effectively minimize user burden, facilitate compliance, or determine return on the investment of its implementations.

7. Consistency With the Guidelines in 5 CFR § 1320.5(d)(2)

The survey will be implemented in a manner fully consistent with 5 CFR § 1320.5(d)(2).

8. Consultation Outside the Agency

In accordance with 5 CFR 1320.8(d), on November, 13, 2017, a 60 day notice was published in the Federal Register for HRSA’s generic clearance, OMB Control No. 0915-0212 (Vol. 82, No. 217, pp. 52308-09). No public comments were received.

9. Remuneration of Respondents

Respondents in usability testing will not be provided with payments or gifts.

10. Assurance of Confidentiality

Respondent confidentiality cannot be assured (i.e., the test moderator and survey administrator will know who participated). However, the survey does not collect any personal information. Data will be kept private to the extent allowed by law.

11. Questions of a Sensitive Nature

There are no questions of a sensitive nature on the survey.

12. Estimates of Annualized Hour Burden

Respondents:

We estimate that up to 200 respondents will participate in usability testing during a 12-month period. Respondents will be solicited via phone or email from current or prospective NPDB users. Respondents may participate in more than one usability test during a 12-month period; however, each test is a unique response and is counted as such. Each usability test is approximately one hour in duration, which includes time for the survey. This burden information is based on past usability tests and surveys conducted by DPDB for system enhancements.²

Annual burden estimates:

Type of Collection	Number of Respondents	Responses per Respondent	Total Responses	Hours per Respondent	Total Burden Hours	Wage Rate	Total Hour Cost
Usability Testing - NPDB System	200	1	200	1	200	$58/hr3	$11,600
Total	200		200		200	$58/hr	$11,600

We estimate that up to 200 respondents will participate in usability testing, at a cost not to exceed $11,600. Since many of the NPDB users are state or local government agencies, we obtained a median state government worker hourly rate (rounded) for Management Occupations ($58.44) to calculate the cost.

Planned frequency of information collection:

Usability testing generally will occur quarterly. Testing may occur more or less frequently depending on the number of enhancements made to the NPDB system; however, the total number of respondents will not exceed 200 in a 12-month period.

13. Estimates of Annualized Cost Burden to Respondents

The only associated cost to respondents is their time to participate in testing and providing the requested information.

14. Estimates of Annualized Cost to the Government

DPDB staff or its contractor(s) will conduct usability testing. The total cost to the Federal government for usability testing in a 12-month period is $125,863.90. This cost is already covered within an existing contract and is not an additional cost. DPDB plans to conduct usability testing indefinitely and will resubmit its plans for OMB clearance prior to the expiration of the current HRSA generic clearance. The government cost estimate was calculated as follows:

Total cost to Federal Government: $125,863.90

• Contract Support Costs - $118,485.90

  • Human Factors Engineer ($138.58) x 855 hrs/12-month period = $118,485.90

• DPDB Staff Costs - $7,378.00

  • GS-14 level $65.88/hr) x 30 hrs/12-month period = $1,976

  • GS-13 level $55.75/hr) x 80 hrs/12-month period = $4,460

  • GS-12 level $47.12/hr) x 20 hrs/12-month period= $942

15. Change in Burden

Not applicable. This is a new activity under HRSA’s generic clearance and will be included in the total burden currently approved by OMB under OMB Control No. 0915-0212.

16. Plans for Analysis and Timetable of Key Activities

The usability testing process typically takes 1-2 months to complete and includes the following key activities:

(1) Develop a usability test plan. Determine the scope of the product being tested, the concerns, questions and goals for the test, the number and type of participants to be tested, location, methods, and equipment to be used.

(2) Compose the scenarios and tasks to be completed for the evaluation. Develop post-task and post-test feedback questions.  In most cases post-test feedback will include the System Usability Scale (SUS).

(3) Develop the materials or product required to complete the evaluation with the identified scenarios and tasks. Build a high-fidelity prototype that facilitates the user’s completion of all scenarios and tasks if the scope requires simulation of realistic user interactions.

(4) Recruit respondents to participate.  Once a respondent agrees to participate in usability testing, a time is scheduled to conduct the session. If required, the product and materials for the session are tailored to simulate each participant’s environment. Materials the participant will require for the evaluation session are sent prior to their scheduled session.

(5) Conduct evaluation sessions, document observations and collect feedback. This includes post-task and post-test feedback, direct open feedback from participants and observations of behaviors during task completion in the session.

(6) Analyze the observations and feedback to determine findings and recommendations. This includes scoring for the SUS, compilation of all post-task and post-test feedback and identification of themes or issues emerging from the responses.

17. Exemption for Display of Expiration Date

No exemption is being requested. The expiration date will be displayed.

18. Certifications

This information collection activity will comply with the requirements in 5 CFR § 1320.9.

1 U.S. Department of Health and Human Services. (n.d.). System usability scale (SUS). http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

2 Based on prior information: approximately 10 participants in 2018 spent, on average, one hour in usability testing.

3 https://www.bls.gov/oes/current/naics4_999200.htm