2nd Passback CMS-10467 Evaluation of GNE Demo Supporting Statement A_CLEAN_TO CMS

EVALUATION OF THE GRADUATE NURSE EDUCATION DEMONSTRATION PROGRAM

OMB No. 0938-1212

Expires 9/30/2016

Supporting Statement A

Contract No. HHSM-500-2011-00013I

Task Order No. HHSM-500-T0009

Table of Contents

A. Background 3

B. Justification 3

1. Need and Legal Basis 3

2. Purpose and Use of the Information Collection 5

3. Use of Information Technology and Burden Reduction 6

4. Duplication of Efforts 6

5. Impact on Small Business and Other Small Entities 7

6. Less Frequent Collection 7

7. Special Circumstances 8

8. Federal Register/Outside Consultation 8

9. Payments/Gifts to Respondents 8

10. Confidentiality 8

11. Sensitive Questions 9

12. Burden Estimates (Hours & Wages) 9

13. Capital Costs 14

14. Cost to the Federal Government 14

15. Changes to Burden 15

16. Publication/Tabulation Dates 19

17. Expiration Date 21

18. Certification Statement 21

Supporting Statement A

Evaluation of the Graduate Nurse Education Demonstration Program (CMS-10467)

1. Background

Advanced practice registered nurses (APRNs) play a critical role in the U. S. health care delivery system, providing services in a variety of roles in acute, ambulatory, and population-based settings. APRNs are registered nurses who have completed an advanced specialized formal education to develop knowledge and skills in diagnosing and managing common acute and chronic diseases, ordering diagnostic tests, prescribing medications and performing minor procedures.¹ The demand for APRN-provided care has increased in recent years because of the shortage of primary care physicians and the rise in the demand for primary care services. It is expected to continue increasing as more Americans acquire access to health care coverage due to the Affordable Care Act (ACA).

The Graduate Nurse Education (GNE) Demonstration project aims to increase the supply of APRNs in the U.S. health care delivery system by providing Medicare payments to five selected hospitals for the reasonable cost of providing clinical training to APRN students. This project also involves the creation of partnerships between hospitals, schools of nursing (SONs), and community-based care settings (CCSs). Payments to the participating hospitals will be linked directly to the number of additional APRNs that the hospitals and their partnering entities are able to train as a result of their participation in the Demonstration. Current Medicare payment policies do not provide a source of funding for graduate clinical nurse education. Under Section 5509 of the ACA,¹ the GNE project is intended to test the feasibility of Medicare funding the clinical nurse education training for APRN students which is not otherwise possible under current Medicare policy. This legislation also authorizes an evaluation of the GNE project to determine its impact on APRN growth, costs to Medicare, and other topics of interest to the Secretary.

Five networks (each consisting of a lead hospital and affiliated SONs and CCSs) were awarded funding through the GNE Demonstration. The Demonstration began in fall 2012 and was slated to end in spring 2015. During this period, each network was enrolling, educating, and graduating APRN students using Demonstration funds. CMS elected to extend the Demonstration through spring 2018, to allow networks to use remaining funds to graduate APRN students enrolled during the original Demonstration period; no new enrollments are permitted during the Demonstration extension period.

The GNE Demonstration Evaluation began in fall 2012 and will be completed in 2019.

GNE Evaluation data collection activities were approved by the Office of Management and Budget (OMB) with an expiration date of 09/30/2016 (OMB # 0938-1212). The purpose of this Information Collection Request (ICR) is to seek approval for revised primary data collection activities, adjusted to account for the final years of the Demonstration. The revised activities include only qualitative telephone interviews, and will result in a reduction of data collection burden. Qualitative data collected during this period will supplement secondary quantitative data analyses on the impact of the Demonstration on APRN growth, costs to Medicare, and spillover effects.

Summary of Revised Data Collection Activities

The nature of GNE Demonstration networks’ activities are changing in the Demonstration extension period; rather than actively enrolling new students, networks are instead using funds to allow APRN students enrolled during the more active phase of the Demonstration to complete their education and graduate. As such, there is no longer a need to monitor implementation of the Demonstration as closely, and many of the network stakeholders important to earlier qualitative data collection are no longer actively engaged in the Demonstration. This revised ICR requests approval to continue qualitative data collection with fewer stakeholders, and only by phone once per year; no primary quantitative data collection nor in-person interviews or focus groups are needed. Exhibit A-1 documents the changes to primary data collection activities.

Exhibit A-1: Changes to Primary Data Collection Activities in Revised ICR

Current Package	Revised Package	Rationale
Qualitative Activities
In-Person Site Visits: SON Admin and GNE Oversight Team	Eliminated	No longer necessary due to changing nature of Demonstration networks’ activities
30- Minute Telephone Check-in Calls: SON Admin and GNE Oversight Team	Extension-Year 60-Minute Telephone Check-in Calls: SON Admin and GNE Oversight Team	Check-in Calls will be expanded to 60 minutes to ensure enough time to discuss project updates since in-person site visits will no longer occur.
--	Post-Demonstration 60-Minute Telephone Interviews: SON Admin and GNE Oversight Team	Telephone interviews will occur after the extension year to gather information on changes since the end of the Demonstration
APRN Student Focus Groups	Eliminated	No longer necessary as students are no longer actively enrolling into Demonstration activities
Clinical Placement Coordinator In-Person Interviews	Eliminated	No longer necessary as Demonstration activities are coming to a close
Clinical Faculty Focus Groups	Eliminated	No longer necessary as Demonstration activities are coming to a close
Preceptor Interviews	Eliminated	No longer necessary as Demonstration activities are coming to a close
Clinical Director Interviews	Eliminated	No longer necessary as Demonstration activities are coming to a close
Chief Financial Officer Interviews	Eliminated	No longer necessary as Demonstration activities are coming to a close
Quantitative Activities
Implementation monitoring through six instruments tracking information on networks’ APRN students, preceptors, alumni, hospital, SON, and CCSs.	Eliminated	No longer necessary as Demonstration activities are coming to a close, and no new enrollments are permitted.

2. Justification

1. Need and Legal Basis

The GNE project is mandated under Section 5509 of the Affordable Care Act (Pub. L. 111-148), which states,

“The Secretary shall establish a graduate nurse education Demonstration under Title XVIII of the Social Security Act (42 U.S.C. 1395 et seq.) under which an eligible hospital may receive payment for the hospital’s reasonable costs . . . for the provision of qualified clinical training to advance practice nurses”.²

The law identifies clinical nurse specialist, nurse practitioner, certified registered nurse anesthetist, and certified nurse midwife programs as the advanced practice registered nurse specialty programs to be included in this Demonstration.

In addition, ACA Section 5509 states that a final Report to Congress (RTC) of the GNE project must be completed no later than October 17, 2017. Per the ACA mandate, the final RTC will include an analysis of the following at a minimum:

1. The growth in the number of APRNs with respect to a specific base year as a result of the Demonstration.

2. The growth in the number of APRNs within specialties.

3. The costs to the Medicare program as result of the Demonstration.

4. Other items the Secretary determines appropriate and relevant.

2. Purpose and Use of the Information Collection

All information collected through the Evaluation of the GNE project will be used by the Centers for Medicare and Medicaid Services (CMS) through its contractor IMPAQ International, LLC to meet the requirements specified under the ACA Section 5509. CMS will also use the information to determine the overall effectiveness of the GNE project.

CMS will use results of the qualitative data collection to understand how the Demonstration is/was implemented overall, how that implementation has changed over time, which aspects of the Demonstration have been successful or unsuccessful, and what plans the sites have for the remainder of the implementation and after the Demonstration formally ends. Findings of the Evaluation will answer qualitative questions, such as: “Have there been any changes to the process of matching students with clinical preceptors?” and “What have been the experiences of students, preceptors, and faculty in the clinical placements?”

The qualitative data collection instrument protocols are provided in Attachment 1.

Primary quantitative data will no longer be used to inform the evaluation.

3. Use of Information Technology and Burden Reduction

Qualitative check-in calls and post-demonstration interviews will occur via telephone, and with participants’ consent, recorded via voice-over internet protocol software. Interviews will cover network stakeholders’ opinions and perceptions of the Demonstration’s successes, challenges, effects, future plans, and similar topics. Protocols for interviews are provided in Attachment 1. Recordings and associated transcripts will be stored securely on a FISMA-compliant enclave.

No primary quantitative data collection will occur for the remainder of the evaluation period. No other information aside from annual telephone calls will be collected.

4. Duplication of Efforts

Qualitative telephone check-in calls and post-demonstration interviews will collect key information about the Demonstration implementation and stakeholders’ experiences that CMS believes is not captured elsewhere. This information collection does not duplicate any other effort and the information cannot be obtained from any other source. Primary quantitative data are no longer necessary, due to the changing nature of the Demonstration activities. Secondary data purchased from the American Association of Colleges of Nursing (AACN) will be used to estimate the impact of the Demonstration on APRN growth.

5. Impact on Small Business and Other Small Entities

Demonstration network hospital and SON members are not small businesses; all respondents of primary qualitative data collection activities are employed by the network hospitals and SONs. While prior data collection included interviews with health care providers employed by network CCSs, some of which included small businesses or small entities, primary quantitative and qualitative information will no longer be collected from CCS employees as part of the evaluation.

6. Less Frequent Collection

The revised ICR reduces the frequency of qualitative data collection from annual in-person site visits and check-in telephone calls to annual check-in calls only during the Demonstration extension, and a one-time telephone interview in the post-Demonstration year. Annual qualitative data collection is necessary to understand GNE oversight team and SON administration experiences, opinions, and perspectives related to the Demonstration over the prior academic year.

7. Special Circumstances

There are no special circumstances.

8. Federal Register/Outside Consultation

The 60-day Federal Register notice was published on October 16, 2015 (80 FR 62535). Comments were received and have been addressed.

9. Payments/Gifts to Respondents

In the previous evaluation years, APRN students received $25 for participating in focus groups; no other participants received payments/gifts. However, going forward, we will no longer be conducting student focus groups because the Demonstration is ending and students are no longer actively enrolling. Therefore, there will be no further payments or gifts to respondents.

10. Confidentiality

Individual-level participant data continues to be maintained as provided by the Privacy Act of 1974 (5 U.S.C. 552a). In accordance with this act, IMPAQ publishes its Privacy Policies and Procedures on its corporate website and internal intranet and provides a compliance hotline for reporting any privacy issues anonymously. All privacy issues invoke IMPAQ’s incident response plan and IMPAQ shall report the breach/loss of privacy within the timeline required and remediate accordingly. Remediation is then tracked on IMPAQ’s Plan of Action and Milestones (POAM) until resolution.

After conducting the check-in calls and post-demonstration telephone interviews, the evaluation team will transcribe all recordings into Microsoft Word files. Any identifying information (e.g., names, titles) will be removed from the transcripts during the quality assurance review. All recording data will be stored in a FISMA-compliant enclave. De-identified transcripts and the analysis files are stored on our secure file system, along with secondary data files (reports and other documents). Individual files have password protection. Additionally, our Information Technology team runs routine compliance scans on all file-shares for the detection of PII, EPHI, etc. The scans are performed every month utilizing Tenable Nessus scanners.

Quantitative data are no longer collected from networks. However, all primary data that have been collected will remain on IMPAQ’s FISMA Enclave (IFE). The IFE is comprised of a state-of-the- art firewall, separate Layer 2 switching to ensure physical separation from other hosted VLANs, and dedicated Systems Security Plan and Privacy Impact Assessments to ensure privacy. Access to the data for validation and analysis will be limited to project personnel who have been granted specific user-credentials for the task and have signed an Assurance of Confidentiality agreement and also signed IMPAQ’s IFE Rules of Behavior (RoB) (Attachment 2). IFE access is facilitated via a virtual Private Network (VPN) requiring multi-factor authentication.

11. Sensitive Questions

No information of a sensitive nature will be collected.

12. Burden Estimates (Hours & Wages)

Qualitative Data Collection

Telephone check-in calls and post-demonstration telephone interviews will be conducted with networks’ strategic planning team members and SON administrators. These calls will last approximately 60 minutes and will focus on barriers/facilitators, sustaining the program, and continuing benefits of the program post- Demonstration. This represents a change from prior activities, which included in-person site visits and focus groups with faculty and students. As the Demonstration is coming to an end, however, it is not necessary to collect detailed implementation monitoring information; it is only necessary to collect information on network leadership’s experiences and perceptions of the Demonstration.

Quantitative Data Collection

Quantitative data will no longer be collected from the five Demonstration networks because detailed implementation monitoring is not necessary as the Demonstration comes to a close.

Estimates of annualized hour burden and annualized cost respondents are displayed in Exhibits A-2 and A-3, respectively. The estimated total number of respondents is 51 annually. The number of burden hours by year are as follows: 51 hours in year 4, 51 hours in year 5, and 51 hours in year 6. The total burden hours for all years is approximately 153 and the average annual burden is approximately 51 hours. The estimated respondent cost by year is as follows: $2,131.59 for year 4, $2,131.59 for year 5, and $2,131.59 for year 6. The total respondent cost for all years is $6,394.77 and the average annual cost burden is $2,131.59.

Estimates of the total annual respondent cost for the collection of information use the appropriate wage rate categories. For the respondents, the hourly wage rate is based on information from the Bureau of Labor Statistics 2014 National Occupational Estimates.³

³ See http://www.bls.gov/oes/current/oes_nat.htm#00-0000.

1. 

   Data Collection Activity	Type of Respondent	Estimated Number of Respondents	Year 4			Year 5			Year 6			All Years Total Hours	Average Annual Burden Hours
   			Responses per Respondent	Burden per Response	Annual Burden Hours	Responses per Respondent	Burden per Response	Annual Burden Hours	Responses per Respondent	Burden per Response	Annual Burden Hours
   Qualitative Data Collection Instruments	Project Strategic Planning Team	25	1	1	25	1	1	25	1	1	25	75	25
   	SON Administrator	26	1	1	26	1	1	26	1	1	26	78	26
   Grand Total		51			51			51			51	153	51

   
   

   Estimated Annualized Burden Hours Exhibit A-2. Estimated Annualized Burden Hours

8

2. Annualized Cost to Respondents

Exhibit A-3. Estimated Annualized Cost to Respondents

Data Collection Activity	Type of Respondent	Estimated Total Annual Burden Hours			Hourly Wage Rate*	Estimated Total Annual Respondent Cost			All Years Total Cost Burden	Average Annual Cost Burden
		Year 4	Year 5	Year 6		Year 4	Year 5	Year 6
Qualitative Data Collection Instruments	Project Strategic Planning Team	25	25	25	$40.97	$1,024.25	$1,024.25	$1,024.25	$3,072.75	$1,024.25
	SON Administrator	26	26	26	$42.59	$1,107.34	$1,107.34	$1,107.34	$3,322.02	$1,107.34
Grand Total		51	51	51		$ 2,131.59	$2,131.59 2,131.59	$2,131.59 2,131.59	$ 6,394.77	$ 1,186

* Based on the mean wages for occupations. National Compensation Survey: Occupational wages in the United States May 2014, U.S. Department of Labor, Bureau of Labor Statistics.

9

13. Capital Costs

There are no capital costs.

14. Cost to the Federal Government

Exhibit A-4 lists the estimated annualized costs of the project for each of the next three years as well as the overall cost to the Federal government.

Exhibit A-4. Estimated Annualized Cost to the Federal Government

Activity/ Partner	Specific Activities	Year 4 Cost	Year 5 Cost	Year 6 Cost	All Years Cost	Average Annual Cost	Cost Description
Government	Reviewing and providing guidance on instruments, OMB clearance, and data collection approach.	$2,366 (40 hours)	$1,479 (25 hours)	$1,479 (25 hours)	$5,324	$1,775	GS-14 staff4: 90 hours x $59.14
Contractor	Telephone check-in calls and post-demonstration telephone interviews	$15,323.20 (160 hours)	$15,323.20 (160 hours)	$15,323.20 (160 hours)	$45,969.60	$15,323.20	Contractor staff5: 480 hours x $95.77
Total	--	$17,689.20	$16,802.20	$16,802	$51,293.60	$17,098.20	--

⁴ Average GS-14 salary for 2015 for locality area of Washington-Baltimore-Northern Virginia, across all ten steps, is $59.14 per hour. https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary- tables/pdf/2015/DCB_h.pdf

⁵ According to national industry-specific occupational employment and wage estimates, social scientist and related workers in “Management, Scientific, and Technical Consulting Services” (NAICS 541600) on average earned

$41.64, which is approximately $95.77 including overhead, fringe and general and administrative indirect rate ($41.64 * 2.3). National Compensation Survey: Occupational wages in the United States May 2014, U.S. Department of Labor, Bureau of Labor Statistics. http://www.bls.gov/oes/current/naics4_541600.htm

10

15. Changes to Burden

Current data collection activities include in-person site visits and focus groups, 30-minute check-in telephone calls with Demonstration network leadership, and primary quantitative data to monitor Demonstration implementation and changes in enrollments, faculty, and preceptor characteristics and numbers. However, most of these activities are no longer needed as the Demonstration reaches an endpoint. Data collection adjustments (see Exhibit A-1) that reduce burden include:

• eliminating the primary quantitative data collection activity,

• reducing the overall number of qualitative data collection respondents, and

• eliminating in-person site visits to each Demonstration network.

Only telephone check-in calls and post-demonstration telephone interviews will be conducted with GNE strategic planning team members and SON administration from each network. Because the Demonstration is entering extension years, in which networks are not allowed to enroll new students but instead assist students enrolled during the Demonstration to graduate, detailed implementation monitoring data are no longer needed. Qualitative telephone check-in calls and interviews will continue to collect network leadership’s experiences and perceptions of the Demonstration’s successes and challenges, and future plans.

Currently, the CMS is approved for 6,740 burden hours. With this ICR, the total burden hours for years 4, 5, and 6 for the Evaluation of the Graduate Nurse Education Demonstration Program is approximately 153 hours, which is a decrease in burden of 6,587 hours. The bulk of this reduction is from eliminating the primary quantitative data collection, in-person site visits, and a number of stakeholders from qualitative data collection.

a. Revisions to the Study Design

The quantitative data will no longer be collected starting with this revision due to the approaching end of the Demonstration. Notable changes have been made to the qualitative data collection methodology: a shift in the mode of data collection from in-person site visits and telephone interviews to only telephone interviews; reduction in the number of participants involved in data collection from seven stakeholders at each Demonstration network to two (see Exhibit A-1 for more detail); and additions/deletions within interview guides to reflect interviews being conducted in the post-Demonstration period (Exhibit A-5).

Exhibit A-5: Revisions to Qualitative Instruments

QUALITATIVE INSTRUMENT REVISIONS
Instrument	Information Collected	Instrument Changes
GNE Project Strategic Planning Team Telephone Interview Guide	Changes since GNE Demonstration Project began Outcomes, barriers, and facilitators Perceived return on investment and sustainability plans	For any participants who were not interviewed in earlier data collection, six short background questions were added (e.g., title, role, responsibilities). Added a question about online enrollment. Added a probe about effects of demonstration on enrollment and graduation. Added three questions about positive and negative outcomes of the demonstration. Deleted question about advice to other organizations.
SON Administration Telephone Interview Guide	Changes since GNE Demonstration Project began Outcomes, barriers, and facilitators Perceived return on investment and sustainability plans	For any participants who were not interviewed in earlier data collection, six short background questions were added (e.g., title, role, responsibilities). Reworded a probe about clinical opportunities and effects on acceptance decisions. Added a question about online enrollment. Added a probe about effects of demonstration on enrollment and graduation. Reworded question about additional precepting hours. Added two questions about positive and negative outcomes of the demonstration.
Clinical Faculty Telephone Focus Group	Changes since GNE Demonstration Project began Outcomes, barriers, and facilitators Perceived return on investment and sustainability plans	Instrument eliminated
APRN Student Telephone Focus Group	Placement in clinical rotations Clinical training experiences	Instrument eliminated
Clinical Placement Coordinators Telephone Interviews	Changes since GNE Demonstration Project began Outcomes, barriers, and facilitators Perspectives on sustainability	Instrument eliminated
Preceptors Telephone Interview Guide	History as a preceptor Preceptor experiences	Instrument eliminated
Director of Nursing/Clinical Director Telephone Interview Guide	Changes since GNE Demonstration Project began Outcomes, barriers, and facilitators Perceived return on investment and sustainability plans	Instrument eliminated
Chief Financial Officer Telephone Interview Guide	Investments related to the GNE Demonstration project Perceived return on investment and sustainability plans	Instrument eliminated
GNE Strategic Planning Team Post-Demonstration Telephone Interview Guide	Changes since demonstration project end Outcomes, barriers, facilitators Perceived return on investment and sustainability plans	For any participants who were not interviewed in earlier data collection, six short background questions were added (e.g., title, role, responsibilities). Reworded questions to “since the demonstration ended”. Added a question about effects of demonstration on enrollment and graduation. Added two questions about positive and negative outcomes of the demonstration.
SON Administration Post-Demonstration Telephone Interview Guide	Changes since demonstration project end Outcomes, barriers, facilitators Perceived return on investment and sustainability plans	For any participants who were not interviewed in earlier data collection, six short background questions were added (e.g., title, role, responsibilities). Reworded questions to “since the demonstration ended”. Reworded a probe into a question about clinical opportunities and effects on acceptance decisions. Added a question about effects of demonstration on enrollment and graduation. Added question on increasing preceptor hours. Reworded question on effects of preceptor hours. Added two questions on positive/negative effects of demonstration.

16. Publication/Tabulation Dates

The project timeline for the remaining data collection, completion of report and other actions are included in the table below.

17. Expiration Date

The project team will display the expiration date on the front page of each interview protocol.

18. Certification Statement

No exceptions are requested.

1 http://www.hhs.gov/sites/default/files/v-healthcare-workforce.pdf

2 Public Law 111-148: Patient Protection and Affordable Care Act. 42 U.S.C. § 18001 (2010). Washington, DC:

U.S. Government Printing Office. http://www.hhs.gov/sites/default/files/v-healthcare-workforce.pdf

8