Supporting_Statement_A_7.06.21

Supporting Statement – Part A

Evaluation of the CMS Network of Quality Improvement and Innovation Contractors Program (NQIIC) (CMS-10769)

Background

The purpose of this Information Collection Request (ICR) is to collect data to inform the program evaluation of the Centers for Medicare & Medicaid Services (CMS) Network of Quality Improvement and Innovation Contractors (NQIIC) Program. Given the breadth and scope of NQIIC’s activities, the current ICR focuses on evaluating specific contract activities. Specifically, this package supports evaluation of the Quality Innovation Network- Quality Improvement Organization (QIN-QIO) Program’s nursing home support and Hospital Quality Improvement Contractors (HQIC) Program. This ICR is part of a larger evaluation of the overall impact of the NQIIC. Subsequent ICRs will be submitted to include evaluation of the QIN-QIO Program’s work in other settings, including community-based healthcare facilities and practices.

The purpose of NQIIC is to support quality improvement efforts across settings and programs for maximum impact to health care and value to taxpayers in a manner that aligns with CMS and the U.S. Department of Health and Human Services’ (HHS) priorities. The NQIIC quality improvement efforts involve the QIN-QIO Program, which is one of the largest federal programs dedicated to improving health quality for Medicare beneficiaries. As such, the QIN-QIO Program has been an important resource in CMS’ efforts to improve quality and efficiency of care for Medicare beneficiaries.¹ QIN-QIOs are groups of health quality experts, clinicians, and consumers who work with health care providers, community partners, beneficiaries, and caregivers on data-driven initiatives designed to improve the quality of care for people with specific health conditions.² QIN-QIOs also serve as quality improvement experts, facilitators, and change agents for healthcare transformation that will span across focus areas to improve care for rural, medically underserved, and vulnerable populations; reduce health disparities; and improve customer experience.³

The 12 QIN-QIOs for the nursing home program have recruited more than 9,000 long-term care nursing homes, hereafter referred to as nursing homes, to focus in six broad areas:

• COVID-19 Pandemic Response and Infection Control

• Reduce Opioid Utilization and Misuse

• Increase Patient Safety

• Increase Chronic Disease Management

• Improve Care Coordination

• Increase Adult Immunization

¹ Centers for Medicare & Medicaid Services. (2020). Quality Improvement Organizations. Retrieved December 15, 2020 from https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment- Instruments/QualityImprovementOrgs

² Quality Improvement Organizations, Centers for Medicare & Medicaid Services. (2020). About. Retrieved December 15, 2020 from https://www.qioprogram.org/about

³ Ibid.

Following are some of the improvements QIN-QIOs are required to make in quality and patient safety in long-term care settings by 2024: reduce all-cause harm and reduce adverse drug events (ADEs), reduce the percentage of nursing homes receiving infection control deficiencies upon survey inspection, improve compliance with infection control protocols and prevent the occurrence and spread of COVID-19 in nursing homes. QIN-QIOs are expected to provide support to the nursing homes most in need of quality improvement based on a combination of requirements related to their size, geographic location, star rating, and survey inspection results.

The NQIIC focus on acute care hospitals involves over 2600 rural hospitals, critical access hospitals, and other acute care hospitals that are low performing and serving vulnerable populations. The nine HQIC contractors provide technical assistance to achieve measurable outcomes focused on the following goals:

• Improve Behavioral Health Outcomes, focusing on decreased opioid misuse (decrease opioid related adverse drug events, including deaths, by 7%)

• Increase Patient Safety with a focus on reduction of harm (reduce all-cause harm, including ADEs, by 9% or more)

• Increase the Quality of Care Transitions with a focus on high utilizers in an effort to improve overall utilization (reduce readmissions by 5%)

CMS evaluates the quality and effectiveness of the QIN-QIO Program as authorized in Part B of Title XI of the Social Security Act,⁴ and has hired Booz Allen Hamilton (Booz Allen) as the Independent Evaluation Contractor (IEC) for the NQIIC Program. The IEC conducts formative evaluation, outcome evaluation, and impact evaluation including Return on Investment (ROI) and assesses provider satisfaction with the program.

This ICR is to conduct data collection using surveys with administrators or managers of nursing homes and hospitals. Table 1 provides an overview of the proposed data collection methods, including survey topics and respondent groups. The Nursing Home Survey instrument can be found in Appendix A and the Hospital Survey Instrument is in Appendix B.

⁴ Social Security Administration. Contracts with Quality Improvement Organizations. Retrieved December 15, 2020 from https://www.ssa.gov/OP_Home/ssact/title11/1153.htm

Table 1: Overview of Data Collection

Data Collection Method	Survey Topics	Respondents
Nursing Home Survey
Telephone survey of Nursing Home Administrators, Directors of Nursing, or staff member most responsible for quality improvement activities (hereafter referred to as Nursing Home Administrators)	Extent to which facility attributes quality improvement outcomes to QIN-QIO Program Level of facilities’ satisfaction with QIN- QIOs Reasons for not participating in QIN-QIO program, when applicable, or for having low engagement Resources used for quality improvement instead of QIN-QIOs among low- or non- participating facilities	250 Nursing Home Administrators of facilities that qualify for the NQIIC Program and either have low-level participation or were never successfully enrolled in the NQIIC 250 Nursing Home Administrators of facilities enrolled in the NQIIC Program that have average or high levels of participation
Hospital Survey
Telephone survey of Hospital Administrators, Directors of Nursing, or staff member most responsible for quality improvement activities (hereafter referred to as Hospital Administrators)	Hospital’s progress towards HQIC goals Extent to which facility attributes quality improvement outcomes to HQICs Level of facilities’ satisfaction with HQICs Reasons for not participating with HQIC, when applicable, or for having low engagement Resources used for quality improvement instead of those provided by HQICs among low- or non-participating hospitals	125 Hospital Administrators of facilities qualify for HQIC technical and were never successfully enrolled 375 Hospital Administrators of facilities enrolled by a HQIC that have average or high levels of participation

A. Justification

1. Need and Legal Basis

The QIN-QIO Program was mandated by Sections 1152-1154 of Part B of Title XI of the Social Security Act, as amended by the Peer Review Improvement Act of 1982 and by the Trade Adjustment Assistance reauthorization bill (Pub. L. 112-40) signed by the President in October 2011. The quality improvement efforts considered for the NQIIC indefinite delivery/indefinite quantity contract include statutorily required QIN-QIO work (Sections 1152-1154 of the Social Security Act) and statutorily required End-Stage Renal Disease (ESRD) Network work (under Sec. 1881 (c). [42 U.S.C. 1395rr]), as well as hospital-focused, large-scale improvement work; clinician-focused technical assistance work; and other quality improvement work.⁵

The Social Security Section 1153. [42 U.S.C. 1320c–2] (c)(2) includes language authorizing evaluation of the QIN-QIO Program: “the Secretary shall have the right to

⁵ CMS. Network of Quality Improvement and Innovation Contractors (NQIIC) Statement of Work IDIQ. Retrieved on December 18, 2020 from https://www.govconwire.com/wp- content/uploads/2019/01/Attachment_J.1_NQIIC_IDIQ_Statement_of_Work.pdf

3

evaluate the quality and effectiveness of the organization in carrying out the functions specified in the contract.”⁶

The data collection proposed in this ICR is necessary for CMS to evaluate the QIN-QIO Program and provide reports on the performance of QIOs. Sections 1152-1154 of Part B of Title XI of the Social Security Act requires CMS to “regularly furnish each quality improvement organization with a contract under this section with a report that documents the performance of the organization in relation to the performance of other such organizations.” Additionally, the Statements of Work (SOW) for QIN-QIO and HQIC contractors both specify engagement with IEC to aid in collect collection for formative and impact evaluation.

2. Information Users

The primary use of this data collection is to provide information to enhance the program evaluation of the NQIIC initiative. The IEC will use survey findings to inform the formative and impact evaluation on the level of engagement and satisfaction with the NQIIC contractors and test these factors for possible associations with process and clinical outcomes. The CMS leads for the NQIIC program may use perceived satisfaction responses to inform the contract management of QIN-QIOs and HQICs; CMS Contract Officer Representatives may share the findings with contractors as part of the contractors’ continual improvement. The findings will also be used to inform CMS’ annual reports to Congress, and its reports and briefings to the Office of Management and Budget (OMB) and other stakeholder groups. The results from this data collection may be published in annual program reports and peer-reviewed journal publications.

3. Use of Information Technology

IEC will conduct telephone surveys to effectively balance the need for program information with the costs of data collection and potential burden on program staff and stakeholders. We will conduct telephone surveys using Computer-Assisted Telephone Interviews (CATI) technology. The interviewer will be guided by the survey questionnaire displayed on the computer screen. The CATI program will reflect the survey logic and skip patterns. Responses will be entered directly into the survey database in a structured format, which will eliminate the need for additional data processing (e.g., transcription, data entry, and coding), thus reducing cost and enhancing data accuracy. Another key advantage of conducting the surveys using the CATI technology is to ensure that the data are collected from the right person. The survey questionnaire includes a series of screening questions; only facility administrators, directors of nursing, or staff member most responsible for quality improvement activities will be selected to participate in the survey. The interviewer will terminate the interview with individuals who do not meet participation eligibility criteria and will continue attempts to reach the right person at the selected facility. Participant screening by a trained interviewer - as opposed to by an electronic survey – will reduce the waste of time by facility staff who attempt to take the survey but are not the right individuals for

⁶ Social Security Administration. Contracts with Quality Improvement Organizations. Retrieved December 18, 2020 from https://www.ssa.gov/OP_Home/ssact/title11/1153.htm

providing information required for the NQIIC evaluation, and ensure the validity of the responses.

This data collection does not require a signature from participants. Consent will be obtained verbally and recorded in the system.

4. Duplication of Efforts

IEC has carefully tracked all sources of data collected by the QIN-QIOs and HQICs for nursing homes and hospitals, as well as data reported by nursing homes and hospitals, including Medicare claims and other HHS data. We are only proposing additional collection of data necessary to inform the NQIIC evaluation questions that are not currently available in existing data sets, program reports, or other sources. IEC is collaborating with other CMS contractors to exchange information and data in order to avoid any duplication of data collections from QIN-QIOs and the facilities they serve.

5. Small Businesses

An estimated 250 nursing homes and 275 hospitals to be surveyed are considered small businesses.⁷ To reduce the impact on these small businesses and entities, data collection will be streamlined and focused, limited to only the collection of data required to answer the evaluation questions. Surveys will be no longer than 15 minutes (for hospitals) or 20 minutes (for nursing homes) in duration. Surveys will be administered by telephone at times that are convenient to the participants. Pre-notification emails will be sent out to respondents prior to data collection to inform them about the purpose of the data collection, expected time required, and provide other elements of informed consent (Appendices C.1 for nursing homes and C.3 for hospitals).

6. Less Frequent Collection

If these information collection activities are not conducted, CMS will not be able to fulfill the mandates of the Social Security Act Title XI, Section 1153, to evaluate the QIN-QIO Program and provide an annual report on performance of contracted QIN-QIOs to the

U.S. Congress. If the data collection occurs less frequently, QIN-QIOs and HQICs will not receive timely feedback from an independent source in order to improve their services to meet the NQIIC goals, which represent HHS and CMS priorities and goals. CMS will not be able to evaluate provider perceptions of and satisfaction with the NQIIC contractors, assess the extent to which providers perceive the NQIIC Program to have contributed to quality improvement progress, gather important information for assessing return on investment, and make any necessary adjustments to increase the program effectiveness and efficiency. Also, evaluating the program tasks requires early and frequent inputs to make appropriate changes in time for the QIN-QIO 13th SOW.

7 References used for estimating the proportion of small businesses include: https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/CertificationandComplianc/NHs; https://www.cdc.gov/nchs/data/series/sr_03/sr03_43-508.pdf; and https://www.shepscenter.unc.edu/wp-content/ uploads/2015/02/21stCenturyRuralHospitalsChartBook.pdf

7. Special Circumstances

There are no special circumstances relating to the Guidelines of 5 CFR 1320.5.

8. Federal Register/Outside Consultation

The 60-day Federal Register notice published on March 9, 2021 (86 FR 13566). CMS received one question from an organization with interests in long-term care inquiring about other means CMS is using to assess the value of the NQIIC Program. The response is included as part of this ICR. In addition to the required public notices, CMS consulted the IEC team, including the survey partner, The Henne Group (THG), on the availability of secondary data, method and frequency of the primary data collection, clarity of instructions to the interviewer and respondent, data collection database, disclosure and confidentiality statements, and anticipated survey reporting format. THG is a market research company with over 30 years of experience with federal, state, and local agencies, as well as with private sector companies and major

U.S. academic institutions. THG is known for its ability to obtain higher than anticipated response rates, success in recruiting hard-to-reach populations, and ability to navigate gatekeepers and reach target respondents.

We also conducted pre-testing with nursing home and hospital administrators using cognitive interviews, which provided substantive input from the targeted respondents to make sure that questions are clearly stated and understood as intended. We have made the required changes to questions to optimize response validity before fielding the survey.

We minimized the extent of testing required to reduce burden and increase quality by relying heavily on previous surveys with similar groups, and those items made up the majority of the questions for both surveys.

The 30-day Federal Register notice published on July 2, 2021 (82 FR 35300).

9. Payments/Gifts to Respondents

No payments or incentives will be issued for survey participants. The burden of the response is low, so we do not think the lack of payment or incentives will impact response rates. While incentives have the potential to encourage participation and can be used to compensate participants for their time, our team and other projects have had success conducting surveys with program stakeholders without incentives. Strategies for successful responses include mailings letters of prenotification and scheduling surveys at convenient times for participants (e.g., early morning or evening phone calls).

The pre-tests involved approximately one hour of time for each participant. For this reason, nursing home and hospital administrators participating in pre-tests were compensated at market rates.

10. Confidentiality

To protect the privacy of participant data, each survey respondent will be de-identified and given a unique identification (ID) number. This ID number will be the only information that is recorded on data-collection instruments, and the data-collection instruments will be stored separately from other data collected within this project.

Contact information (names, telephone numbers, and email addresses) of participants will be stored separately from data files and will only be accessed by authorized team

members for logistical reasons (e.g., scheduling, follow-ups, avoiding recruitment for survey participation with the same individuals in subsequent years). Individuals will also not be identified in the transcripts that are used for pre-testing the survey instruments.

Facility identifiers may be used to construct relevant variables from the raw survey data and linked to outcomes data for the construction of more robust analytic data sets.

No one outside the IEC team will have access to the individual responses, nor will anyone outside the team be able to identify any individual respondent by their responses. Reports on data collected will be presented in aggregate form only. At the end of the project, the Primary Investigator will arrange for the proper storage and destruction of all data in compliance with all relevant government regulations and policies.

11. Sensitive Questions

The surveys do not include any sensitive questions related to private matters.

12. Burden Estimates (Hours & Wages)

The category of respondents for each of the data collections and the estimated annual burden (number of burden hours per year) for the specific information collection are outlined in Table 2.

Table 2: Estimated Burden Hours and Cost

Data Collection Activity	Estimated Number of Respondents (1)	Number of Responses per Respondent (2)	Hours per Response (3)	Estimated Burden Hours (4=1*2*3)	Hourly Wage Rate8 (5)	Estimated Total Respondent Cost (6=4*5)
Nursing Home Survey	500	1	0.33	165	$55.34	$9,131.10
Hospital Survey	500	1	0.33	165	$80.88	13,345.20
Total	1,000	1	--	330	--	$22,476

The estimated number of respondents reflects the planned sample of 500 per year for each survey. The burden hour estimates for the survey are based on pre-tests of the length of time each type of respondent is likely to need to complete the survey screener and questions. The pre-tests confirmed that completion of surveys averaged around 20 minutes if the facilities are participating in the NQIIC, and less time if they are not

⁸ Based on May 2017 National Industry-Specific Occupational Employment and Wage Estimates, Bureau of Labor Statistics, North American Industry Classification System (NAICS) 623100, https://www.bls.gov/oes/2017/may/naics4_623100.htm#11-0000 for Nursing Home Administrators: Top Executives in “Nursing Care Facilities (Skilled Nursing Facilities)” on average earned $55.34 in 2017. Based on NAICS 622000, https://www.bls.gov/oes/2017/may/naics3_622000.htm#11-0000 for Hospital Administrators: Top Executives in “Hospitals” on average earned $80.88 in 2017. Cognitive interview participants will also be Nursing Home and Survey Administrators.

participating. Since the majority of the facilities in the sample will be participating in the NQIIC at different levels, Table 2 uses 20 minutes as the average time.

The estimated annual hour and cost burden is based on the (most recently available) 2017 hourly wage rate of the categories of respondents for these data collections (Table 2). The total cost is calculated by multiplying the number of responses by the average time per response by the hourly wage. IEC then summed the costs to derive the total cost for all respondents.

13. Capital Costs

There are no capital costs.

14. Cost to Federal Government

The cost of this information collection effort to the federal government consists of the costs for government (CMS) activity and CMS’ contractor activity (Table 3). The costs to CMS involve labor costs for overseeing the contractor’s work and reviewing and providing guidance on data collection instruments, the OMB clearance package, and other materials. Labor costs were estimated using average salaries representative of the professional levels and steps for CMS personnel. CMS contractor costs represent labor for survey development and testing; sample recruitment, screening, and scheduling; survey administration and management; data cleaning and analysis; and developing reports. Operational expenses include overhead, survey scripting, data processing, and coding.

For purposes of OMB review and approval, we have annualized the number. As shown in Table 3, the estimated annual cost to the federal government over a standard three-year OMB approval period will be $443,253.

Table 3: Annual Cost to the Federal Government

Activity
Government Activity Review and provide guidance on instruments, OMB clearance, and data collection approach	$25,000
Contractor Activity Instrument development, testing, administration, management; sample recruitment and scheduling; Data coding/transcribing; Analysis and reporting	$418,253
Total	$443,253

15. Changes to Burden

The burden increased from an estimated total of 300 hours to a total of 330 hours after pretests found both surveys averaged 20 minutes.

16. Publication/Tabulation Dates

The Independent Evaluation Contractor’s period of performance is effective from September 25, 2020 through September 24, 2025. Our plans and timeline for reports and

publications using survey findings are outlined in Table 4 and include program management reports that provide ongoing performance data that can guide CMS’ program decisions regarding continuation or modification of contract recruitment and performance targets, measurement strategies, and recommended evidence-based interventions. Table 4 also identifies documents and reports suitable for presentation to various audiences, national stakeholders, and policymakers, including presentations at professional meetings and publications in peer-reviewed journals.

Table 4: Deliverable Schedule for Data Collection and Reporting Activities

Deliverables	Timeline
Survey Reports	Annual Reports 2021-2024
Evaluation Progress Reports (includes survey findings)	Bi-annually (2020-2025)
Ad Hoc/Occasional Reports	Up to 3 Annually
Presentations	2021-2025
Publications (including peer-reviewed manuscripts)	2021-2025
Final Technical Report	4/2025

Supporting Statement B provides an overview of the statistical techniques IEC will use to analyze survey data.

17. Expiration Date

The expiration date will be displayed on the collection instrument. The expiration date will be mentioned by the interviewer before the survey is administered.

18. Certification Statement

There are no exceptions to the certification statement.

1