Usability Testing and Evaluation for Phase 1 of the QualityNet Portal (QNP) Redesign Project

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 0938-1185)

TITLE OF INFORMATION COLLECTION:

Usability Testing and Evaluation for Phase 1 of the QualityNet Portal (QNP) Redesign Project

PURPOSE:

The Centers for Medicare & Medicaid Services (CMS) Center for Clinical Standards and Quality (CCSQ) engaged the Project/Program Management and Business Requirements Contract (PMBR) to integrate QualityNet.org with the QualityNet Secure Portal (QSP) under a single, publically available entry point. The results of this consolidation effort will be referred to throughout this document as QNP.

On the QualityNet.org web site, CMS Quality Program participants access and retreive important program information such as specification manuals, measures information, benchmarks of care, and more. Users also obtain their QSP log in credentials from registration information posted on QualityNet.org.

In QSP, CMS Quality Program registered users can upload quality data, securely transfer files and emails, obtain reports, and manage facility information.

The QNP integrated project team (IPT) is currently conducting analysis and gathering data from existing QualityNet.org and QSP users in order to create a user-friendly navigation structure for the future QNP.

This fast track request to conduct usability testing is a part of the QNP analysis and requirements creation process, while also supporting the user-centered design guidelines set forth in the CMS Technical Reference Architecture (TRA) – Web-Based User Interface Supplement.

The data collected in the usability testing will be used to uncover usability issues with proposed navigation structure, compare proposed QNP navigation against the current QualityNet.org and QSP navigation, and guide the QNP IPT in decision-making around future navigation. An card sorting evaluation will also be used to help establish awareness around current QNP user mental models related to the content on the site.

The usability testing will feature a series of navigation-related tasks wherein the participant will use think-aloud protocol to explain the rationale behind the completion of each task. The usability testing will also include a questionnaire to help with System Usability Scoring (SUS) at the end of each participant’s session. It is anticipated that each usability testing session will run 45 minutes to 1 hour in duration.

Benefits of the usability testing and evaluation include:

• Benchmarking and revealing usability issues with the current QualityNet.org and QSP navigation

• Knowledge around existing mental models that users have around QualityNet.org and QSP content

• Awareness and ability to address usability issues with the proposed QNP navigation prior to finalization of design

• Understanding of QNP user preferences, satisfaction, and suggestions related to the future QNP navigation

• Comparative measurement of the proposed QNP navigation against the current QualityNet.org and QSP navigation

The usability testing will be led by a facilitator, and will have two to three additional observers for note taking purposes. The testing will be conducted in-person (where applicable) or via a remote moderated session wherein the participant will utilize web-based meeting software, Adobe Connect, to share their computer screen.

The card sorting evaluation will be conducted via an online tool, Optimal Sort, and operates very similarly to a web-based survey. Participants will be provided 1-2 introductory questions, and then provided 30 cards to sort and organize into categories.

The information collected through the usability testing and evaluation is both qualitative and quantitative in nature. The data collection will not include statistical analysis, nor will it be shared publicly. There is no potential for controversy around the data collected, and it will not set any policy.

DESCRIPTION OF RESPONDENTS:

Target respondents are the primary audience of QualityNet.org and QSP. This audience is made up of CMS Quality Program contractors, vendors, hospital and providers, and are from the private sector. It is estimated that approximately 30% of participants will be small entity, and may represent either for-profit or not-for-profit institutions.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [] Customer Satisfaction Survey

[X] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [ ] Other: ______________________

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name:

Olaoluwa Ayilegbe

IT Specialist (SYSTEMS ANALYSIS)

Centers for Medicare & Medicaid Services (CMS)

Center for Clinical Standards and Quality (CCSQ)

Information Systems Group (ISG)

Division of Quality Systems Governance, Engineering and Development (DQSGED)

Office: 410-786-8367

Email: [email protected]

7500 Security Blvd,

Baltimore, MD 21244-1850

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ ] Yes [X] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ ] Yes [X] No

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Usability Testing, Private Sector	30	1 hour	30 hours
Card Sorting, Private Sector	10	15 minutes	~2 hours
Totals	40	75 minutes	~32 hours

FEDERAL COST: The estimated annual cost to the Federal government is none. Current QNP Phase 1.0 has allocated the survey cost.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [X] Yes [ ] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

Participants will have the opportunity to opt-in to participate in the usability testing, and a request for participation may be established through existing relationships with the QNP IPT or through recruitment requests via established Quality Program list serves. Any additional participants will be CCSQ support contractors that also utilize QualityNet.org and QSP for their job functions.

Existing Quality Program listserve recipients for the following Hospital Quality Reporting (HQR) programs include:

• Inpatient Quality Reporting

• Outpatient Quality Reporting

• Ambulatory Surgical Center Quality Reporting

• Inpatient Psychiatric Facility Quality Reporting

• PPS-Exempt Cancer Hospital Quality Reporting

• End-Stage Renal Disease (EQRS) Quality Reporting System

The QNP IPT will be working closely with the CMS support contractor HSAG to assist in the distribution via listserve as they are the current owners of the distribution lists.

Participation will not be indicated as mandatory for good standing in any quality reporting program. All participation data will be logged anonymously, and no PII will be captured.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[ ] Web-based or other forms of Social Media

[ ] Telephone

[X] In-person (where possible)

[ ] Mail

[X] Other, Explain: For usability testing, Web-based testing using HCQIS-approved web meeting software, Adobe Connect, will be used. For card sorting, Optimal Sort, a web-based survey and sorting tool.

2. Will interviewers or facilitators be used? [X] Yes [] No

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time and divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

8