Supporting Statment

Health Resources and Services Administration

SUPPORTING STATEMENT

DPDB Usability Studies

A. Justification

1. Circumstances of Information Collection

The Health Resources and Services Administration (HRSA) currently has approval under a generic clearance, Office of Management and Budget (OMB) Control No. 0915-0212, to conduct customer satisfaction surveys and focus groups. This collection of information will help fulfill the requirements of:

1. Executive Order 12862, “Setting Customer Service Standards,” which directs agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector;

2. The Paperwork Reduction Act (PRA) of 1995, which is designed to reduce the total amount of paperwork burden the Federal Government imposes on private businesses and citizens;

3. Federal security requirements established by the National Institute of Standards and Technology, the Federal Information Security Management Act, and HRSA; and

4. The laws governing the National Practitioner Data Bank (NPDB):

   1. Title IV of the Health Care Quality Improvement Act of 1986 (HCQIA);

   2. Section 1921 of the Social Security Act (SSA); and

   3. Section 1128E of the SSA.

This is a request for OMB approval of a quantitative and qualitative voluntary usability test survey under HRSA’s generic clearance. In an effort to improve the usability of the NPDB’s password-protected, restricted data system – known as the Integrated Query and Response System, or IQRS – and its public website, the Division of Practitioner Data Banks (DPDB), which manages the NPDB, seeks approval to conduct usability testing. Note that the term “NPDB system” used in this document includes both the IQRS and the public website, found at www.npdb.hrsa.gov.

The NPDB is a federally mandated repository of disciplinary actions, medical malpractice payment reports, and criminal convictions, among other actions, taken against health care practitioners, providers, suppliers, and entities. The aim of the NPDB is to promote health care quality and reduce fraud and abuse. When changes are made to the NPDB system, usability testing is needed to ensure that the system is user-friendly, minimizes user burden, facilitates compliance, and maximizes return on investment. In addition, usability testing helps to identify errors or bugs that may be present in the system so that they can be fixed before enhancements are put into production.

In addition to the concerns outlined in the preceding paragraph, DPDB also wants to be certain that its public website includes information that customers need. DPDB is preparing a redesign of the public website, and usability testing will be important before the new website is rolled out to the public at large.

The International Organization for Standardization defines “usability” as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

• effectiveness: the accuracy and completeness with which specified users can achieve specified goals in particular environments.

• efficiency: the resources expended in relation to the accuracy and completeness of goals achieved.

• satisfaction: the comfort and acceptability of the work system to its users and other people affected by its use.

Executive Order 12862 directs agencies that “provide significant services directly to the public” to “survey customers to determine the kind and quality of services they want and their level of satisfaction with existing services.” The overarching objective of surveying users of the IQRS and website prior to putting changes and enhancements into production is to ensure the usability of the NPDB system and to identify system bugs or issues that would hinder usability.

2. Purpose and Use of the Information

The purpose of this data collection effort is to ensure that the NPDB system and any future enhancements are user-friendly. Data obtained from this effort will be used by DPDB and its contractor(s) to identify strengths and weaknesses in the NPDB system and to identify issues that need to be remedied to allow for ease of use.

DPDB or its contractor(s) will solicit voluntary participation among current or prospective NPDB users to participate in usability testing. (The IQRS is not open to the general public, and by statute and regulation only certain entities are permitted to use it.) If a respondent agrees to participate, the usability testing will be conducted online (e.g., WebEx). In rare cases, respondents may be asked to participate in person during DPDB-attended conferences, such as the National Association Medical Staff Services. Respondents will be asked to undertake specific tasks related to the purpose of the usability test. For example, if the usability test is about a system enhancement to improve the reporting process, respondents may be asked to enter a report into a pre-production testing environment. After the usability test is complete, respondents will be asked a series of questions about their experience.

To obtain information about the user’s experience with the NPDB system, DPDB proposes to use a survey that consists of the System Usability Scale (SUS),¹ a standardized, 10-item Likert scale of system usability developed by John Brooke at Digital Equipment Corp., and a series of open-ended questions. As a standardized instrument, SUS is used in its current, copyrighted

form. Through standardized scoring, SUS will provide DPDB with quantitative information about how usable the system is for a range of domains: efficiency, effectiveness, and satisfaction.

Because SUS measures usability quantitatively, it is not used for diagnosing specific system issues. Therefore, open-ended questions will provide specific qualitative information about strengths and weaknesses of the IQRS or website to assist in diagnosing the NPDB system. Additional questions may be asked for probing purposes. Questions related to IQRS are presented on the Usability Testing Survey-IQRS Form. Questions specific to usability testing for the website are on the Usability Testing Survey-Website Form. Both forms are found at the end of this Supporting Statement. The primary difference between the two proposed forms is that the Usability Testing Survey-Website Form includes questions, called the Five-Second Test, that seek to understand a user’s website experience after looking at a website for 5 seconds.

Data gathered from the usability surveys will be used only for internal purposes; no data will be disseminated outside DPDB or HRSA.

If this data is not be collected, DPDB will have no knowledge of how user-friendly the NPDB system is for its customers, will not be able to effectively minimize user burdens and facilitate compliance, and will not be able to determine return on the investment to implement enhancements to the system. Based on anecdotal feedback from some users in the past, difficulty with using the system (e.g., entering a report) has resulted in entities not filing required reports. Collecting usability information will allow DPDB to continually enhance the system so that those required to use the NPDB can use the system with ease, which will increase overall use and, thus, the value of the NPDB to the U.S. health care system.

3. Use of Improved Information Technology

The usability test will be conducted online (e.g., WebEx) to allow for respondents to schedule the test at a time and place convenient for them. Usability testing also may be done in-person at national health-related conferences that DPDB attends. (The test would still be conducted on a computer in a pre-production environment.) The survey will likely be given by DPDB or its contractor(s) either in person or via telephone. However, DPDB also may use technology, such as a web-based survey platform, to administer the survey. Whether oral, electronic, or written, the OMB control number will be either read to the respondent or clearly visible on the survey.

4. Efforts to Avoid Duplication

This data collection is not duplicative of other information collection efforts. It will be conducted during specific instances around IQRS enhancements or website changes.

5. Involvement of Small Entities

The survey will not have a significant impact on small businesses or other small entities.

6. Consequences if Information is Collected Less Frequently

DPDB proposes to conduct usability testing for all major enhancements as necessary. Consequences of not conducting usability testing at all, or less frequently (e.g., not for each enhancement), could mean that enhancements would not be user-friendly. DPDB would have no knowledge of a user’s experience with the system and could design enhancements that are not user-friendly or that contain issues that hinder its use. In addition, DPDB would not be able to effectively minimize user burden, facilitate compliance, or determine return on the investment of its implementations.

7. Consistency With the Guidelines in 5 CFR § 1320.5(d)(2)

These surveys will be implemented in a manner fully consistent with 5 CFR § 1320.5(d)(2).

8. Consultation Outside the Agency

In accordance with 5 CFR § 1320.8(d), on April 24, 2009, a 30-day notice was published in the Federal Register for HRSA’s generic clearance, OMB Control No. 0915-0212 (74 FR 18726). No public comments were received.

9. Remuneration of Respondents

Respondents in usability testing will not be provided with payments or gifts.

10. Assurance of Confidentiality

Confidentiality of respondents cannot be assured (i.e., the test moderator and survey administrator know who participated). The survey is, therefore, not anonymous. However, the survey itself does not collect any personal information. Data will be kept private to the extent allowed by law.

11. Questions of a Sensitive Nature

There are no questions of a sensitive nature on the survey.

12. Estimates of Annualized Hour Burden

Respondents:

We estimate that up to 300 respondents will participate in usability testing during a 12-month period (200 for the IQRS and 100 for the website). Respondents will be solicited via phone or email from current or prospective NPDB users. Respondents may participate in more than one usability test during a 12-month period; however, each test is a unique response and is counted as such. Each usability test is approximately one hour in duration, which includes time for the survey. This burden information is based on past usability tests and surveys conducted by DPDB for system enhancements.²

Annual burden estimates:

Type of Collection	Number of Respondents	Responses per Respondent	Total Responses	Hours per Respondent	Total Burden Hours	Wage Rate	Total Hour Cost
Usability Testing Survey-IQRS Form	200	1	200	1	200	$39/hr3	$7,800
Usability Testing Survey-Website Form	100	1	100	1	100	$39/hr	$3,900
Total	300	1	300	1	300	$39/hr	$11,700

We estimate that up to 300 respondents will participate in usability testing, 200 for the IQRS and 100 for the website, for a cost of $11,700. Since many of the NPDB users are state or local government agencies, we obtained a median state government worker hourly rate (rounded) for management-related occupations to calculate the cost.

Planned frequency of information collection:

Usability testing generally will occur quarterly. Testing may occur more or less frequently depending on the number of enhancements made to the NPDB system; however, the total number of respondents will not exceed 300 (200 for the IQRS and 100 for the website) in a 12-month period.

13. Estimates of Annualized Cost Burden to Respondents

The only associated cost to respondents is their time to participate in testing and providing the requested information.

14. Estimates of Annualized Cost to the Government

Usability testing will be conducted by DPDB staff or its contractor(s). The total cost to the Federal Government for usability testing in a 12-month period is $121,034.50. This cost is already covered within an existing contract and is not an additional cost. DPDB plans to conduct usability testing indefinitely and will resubmit its plans for OMB clearance prior to the expiration of the current HRSA generic clearance. The Government cost estimate was calculated as follows:

Total cost to Federal Government: $121,034.50

• Contract Support Costs - $114,604

  • Human Factors Engineer ($134.04/hr) x 855 hrs/12-month period = $114,604

• DPDB Staff Costs - $6,430.50

  • GS-15 level ($61.48/hr) x 20 hrs/12-month period = $1,225.60

  • GS-14 level ($55.45/hr) x 20 hrs/12-month period = $1,109.00

  • GS-13 level ($45.51/hr) x 90 hrs/12-month period = $4,095.90

15. Change in Burden

Not applicable. This is a new activity under HRSA’s generic clearance and will be included in the total burden currently approved by OMB under OMB Control No. 0915-0212.

16. Plans for Analysis and Timetable of Key Activities

Prior to any usability test, DPDB or its contractor(s) recruits respondents to participate. Once someone agrees to participate in usability testing, a time is scheduled to conduct the test and administer the survey. Once all of the usability tests have been completed, the SUS component is scored and the open-ended questions are coded across the respondents according to themes or emerging issues. The entire process from recruitment to data analysis takes about 1-2 months per usability test.

17. Exemption for Display of Expiration Date

No exemption is being requested. The expiration date will be displayed.

18. Certifications

This information collection activity will comply with the requirements in 5 CFR § 1320.9.

1 U.S. Department of Health and Human Services. (n.d.). System usability scale (SUS). http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

2 Based on prior information: 56 participants in 2012 spent, on average, one hour in usability testing.

3 http://www.bls.gov/oes/current/naics4_999200.htm#00-0000