CMS-10467 Supporting Statement Part A

Supporting Statement – Part A
Evaluation of the Graduate Nurse Education Demonstration Program
A.

Background

Advanced practice registered nurses (APRNs) play a critical role in the United States (U.S.) health
care delivery system, providing services in a variety of roles in acute, ambulatory, and populationbased settings. The demand for APRN-provided care has increased in recent years because of the
shortage of primary-care physicians and the rise in the demand for primary-care services. This
demand is expected to continue increasing as more Americans acquire access to health care
coverage because of the passage of the Affordable Care Act (ACA). The Graduate Nurse
Education (GNE) Demonstration aims to increase the supply of APRNs in the U.S. health care
delivery system by providing Medicare payments to five selected hospitals for the reasonable cost
of providing clinical training to APRN students. This demonstration also involves the creation of
partnerships between hospitals, schools of nursing (SONs), and community-based care settings
(CCSs).
Optimal Solutions Group, LLC (Optimal), and its partner, American Institutes of Research (AIR),
are designing and implementing a program evaluation to inform the demonstrations’ Report to
Congress (RTC). The final report will include an analysis of the following at a minimum:
1. The growth in the number of APRNs with respect to a specific base year as a result of the
demonstration.
2. The growth for each of the following specialties: clinical nurse specialist, nurse
practitioner, certified nurse anesthetist, certified nurse-midwife.
3. The costs to the Medicare program as result of the demonstration.
Quantitative and qualitative data from primary and secondary sources will be gathered and
analyzed for this evaluation. The primary data will be collected through site visits, key stakeholder
interviews, small discussion groups and focus groups, telephone interviews, electronic templates
for quantitative data submission, and quarterly demonstration-site reports. The secondary data will
come from mandatory hospital cost reports provided to the Center for Medicare and Medicaid
Services (CMS), and several other existing secondary data sources, such as the American
Association of Colleges of Nursing (AACN).
The primary data elements to be collected are divided into six broad categories:
• Characteristics of APRN applicants, current students, and alumni;
• Characteristics of preceptors to APRN students;
• Characteristics of nursing faculty;
• Characteristics of partner hospitals that are part of the demonstration networks;
• Characteristics of schools of nursing (SONs) that are part of the demonstration networks;
and
• Characteristics of community-based care settings (CCSs) that are part of the demonstration
networks.
1

The evaluation team is conducting both a process and impact evaluation within a rapid-cycle
framework to systematically collect and report data and to ensure the timely submission of highquality deliverables to inform CMS’ Center for Medicare and Medicaid Innovation (CMMI), its
stakeholders, and write the demonstration’s RTC. The process evaluation will analyze the
implementation of the demonstration and allow for course corrections during the demonstration
period. The impact evaluation will measure changes in key data elements from baseline. In order
to conduct the process and impact evaluation, it is vital that Optimal and its partner AIR collect
information pertaining to all stages of the demonstration, including historical, baseline, transition,
implementation and post-implementation stages. For more information on the Graduate Nurse
Education Demonstration see http://innovations.cms.gov/initiatives/gne/.
There are two types of data collection for this project: quantitative and qualitative. Table 1
summarizes the purpose and the respondents for each type of data collection at each of the five
demonstration sites.
Table 1. Data Collection Types
Data Collection Type
Purpose
Quantitative
Process and Impact
Evaluation
Qualitative
Process Evaluation

B.

Respondents
Demonstration Sites
Hospital and SON
demonstration
administrators, CCS
administrators,
preceptors, APRN
students

Justification
1.

Need and Legal Basis

The Graduate Nurse Education (GNE) Demonstration is mandated under Section 5509 of the
Affordable Care Act under title XVIII of the Social Security Act (42 U.S.C. 1395 et seq.).
According to Section 5509 of the ACA, the five selected demonstration sites receive “payment for
the hospital’s reasonable costs for the provision of qualified clinical training to advance practice
registered nurses”.
In addition, ACA Section 5509 states that an evaluation of the graduate nurse education
demonstration must be completed no later than October 17, 2017. This evaluation includes
analysis of the following:
1. The growth in the number of APRNs with respect to a specific base year as a result of the
demonstration.
2. The growth for each of the following specialties: clinical nurse specialist, nurse practitioner,
certified nurse anesthetist, certified nurse-midwife.
3. The costs to the Medicare program as result of the demonstration.
2

2.

Information Users

All information collected through the Evaluation of the Graduate Nurse Education Demonstration
will be used by the Center for Medicare and Medicaid Innovation (CMMI) through its contractors
Optimal and its partner AIR to conduct the analysis of the three items specified under ACA
Section 5509.
CMS will use the results of the process evaluation to inform demonstration sites on how to
improve their operations. CMS will use the results of the impact evaluation to inform Congress on
the effectiveness of the Graduate Nurse Education Demonstration.
3.

Use of Information Technology

Quantitative Data Collection
The data collection forms will be provided to respondents (see table 1) in a common electronic file
type (i.e., Microsoft Excel) and can be submitted via a secure File Transfer Protocol (FTP) site.
Respondents can submit data outside of the provided template forms as long as all data elements
are clearly labeled and the file type is Microsoft Excel, .csv, or tab delimited text file. This
approach is most efficient and reduces burden as respondents have the data available electronically
and are compiling data electronically. Each site will be provided a secure login to the FTP site.
Signatures from respondents are not required.
Qualitative Data Collection
Opinion on and details of the perceived progress or success of the demonstration implementation
will be collected through in person and telephone interviews. Collecting the data electronically
would actually pose a higher burden on participants as it would require them to provide their
perceptions in writing.
4.

Duplication of Efforts

In order to identify and prevent duplication of efforts, Optimal has contacted the five
demonstration sites to inquire about current data collection efforts. Optimal worked to incorporate
demonstration sites’ feedback in determining which data elements would result in excessive
burden to be collected.
Additionally, Optimal conducted secondary data research, identifying data sources that can
provide comparable information when data would be too burdensome to collect directly from
demonstration sites. One example is the collection of historical data for APRN alumni, which
demonstration sites indicated would be difficult to gather for all baseline years (2006-2010).
Optimal identified the American Association of Colleges of Nursing (AACN) and the National
League of Nursing (NLN) as secondary data sources that will substitute and/or complement the
primary data gathered from demonstration sites. Optimal has taken measures to ensure that the
primary data collection does not duplicate any other effort and the information cannot be reliably
obtained from any other source.

3

5.

Small Businesses

While demonstration hospitals and SONs are not small businesses, some of the community-based
care settings (CCSs) within the de demonstration networks may be considered small businesses or
small entities. However, CCSs can be Federally Qualified Health Center, public health facilities,
or other types of CCS which do not qualify as small businesses. As part of the primary data
collection efforts, some of the demonstration sites may request additional information from partner
CCSs, thus potentially having a minor impact on a few small businesses or entities. This burden is
a requirement of participation in the demonstration network by the CCS and they are aware of this
requirement. Therefore, the burden imposed upon small businesses is likely to be negligible.
6.

Less Frequent Collection

Quantitative Data Collection
The frequency of data collection, which varies by data element, has been determined by the
frequency of the natural availability of data. For example, because APRN students enroll and
graduate each academic period, data on student enrollment are requested with a frequency of once
per academic period (semester or quarter). The frequency of the data collection as described above
is necessary to execute the process evaluation as well as the impact evaluation as required by the
ACA. Collecting data at specific intervals throughout the demonstration allows for the analysis of
the implementation of the demonstration and subsequent course corrections.
Qualitative Data Collection
Part of CMS’ objective in fulfilling the legislative requirement of GNE demonstration evaluation
is to conduct a process evaluation. Qualitative methodology is commonly used to conduct process
evaluations, given its ability to capture the complexity of interventions as well as individuals’
perspectives. The qualitative data collection is an essential component of the evaluation as it
allows for rich and detailed information about the projects, their challenges and successes, and
potential for sustainability.
Data collection for the qualitative component of the evaluation will be conducted at three time
points. Time 1 (T1) will be as close to the beginning of the grant period as feasible. Time 2 (T2)
will be roughly six months after T1. Time 3 will occur 6 months after T2. The purpose of
collecting data at multiple time points is to document the progress, challenges, and implementation
and development strategies associated with the GNE demonstration projects. To collect data less
frequently would reduce the reliability, specificity and thoroughness of the analysis due to
challenges associated with requiring respondents to recall activities over long periods of time.
7.

Special Circumstances

None of the special circumstances apply to this data collection effort.

8.

Federal Register/Outside Consultation

4

CMS and the evaluation contractors worked with the demonstration sites to determine what data
was available historically and what data can be collected moving forward. While there are slight
variations across sites, the data collection forms reflect what can be realistically collected and
what is necessary to answer key research questions.
9.

Payments/Gifts to Respondents

Quantitative Data Collection
No payments or gifts will be provided to respondents other than the reimbursement of reasonable
costs associated with the clinical training of APRNs, as part of the GNE demonstration mandated
by the ACA Section 5509.
Qualitative Data Collection
While we assume that the participation of key informants from the GNE personnel at the
demonstration sites, the clinical sites, and the nursing schools will be subsumed under the
responsibilities and duties of their employment, APRN students are under no obligation to provide
their perspectives on the activities of the GNE demonstration. Thus, we will provide an incentive
in the amount of $25 to APRN students who participate in focus groups as part of the process
evaluation. This incentive is intended as a gesture of thanks to the students for the time they will
take out of their busy schedules to participate in the qualitative data collection efforts.
10.

Confidentiality

Quantitative Data Collection
Individual-level participant data will be maintained as provided by the Privacy Act of 1974 (5
U.S.C. 552a). The requested data set does not include personally identifiable information such as
participant names, addresses, or Social Security numbers. To ensure the confidentiality of the data
during transmission from the awardees to Optimal, we will set up secure file transfer protocol
servers through which the data are to be transmitted. The data to be transmitted will be encrypted
using American Encryption Standard (AED) 256-bit. Access to the data for validation and analysis
will be limited to project personnel who have been granted specific user-credentials for the task
and have signed an Assurance of Confidentiality agreement. Server access requires a virtual
Private Network (VPN) account with a login and password assigned, as well as a login and
password for the server. Additionally, since these machines operate on a secure virtual local area
network (VLAN) that does not have access directly to the Internet, the only way to move files on
or off is through a secure FTP.
The VLAN is a managed hosted VLAN that includes a Cisco ASA hardware firewall and separate
Layer 2 switching such that our network is physically separate from other hosted VLANs and
private.
The data collection effort assures respondents that the raw data will be treated as proprietary.
Optimal has established stringent procedures and safeguards for securing and protecting data
against inappropriate disclosure or release of confidential information that will be collected. The
contractors handling the GNE data will not release any of the information collected in such a way
that it can be linked to individual demonstration participants or their partners. All results will be

5

presented in the aggregate.
Qualitative Data Collection
The confidentiality of the data collected through the qualitative instruments will be maintained as
provided by the Privacy Act of 1974 (5 U.S.C. 552a). Data collectors will upload all audiorecordings of interviews within 24 hours of completing an interview. To protect the confidentiality
of respondents, each interview will be assigned an identification number. Identifiable information
such as names and organizations of respondents will be stored separately from these data, with a
key as the only crosswalk between the data and identifiers. Data and the key will be stored on
password protected server hosted by the American Institutes for Research. Only evaluation team
members and contractors, such as transcriptionists, who have signed an Assurance of
Confidentiality Agreement, will have access to the data for purposes of transcribing and analysis.
Transfer of audio and transcript files between AIR and the transcriptionist service and/or will use a
secure FTP server encrypted using American Encryption Standard (AED) 256-bit.
11.

Sensitive Questions

No information of a sensitive nature will be collected.
12.

Burden Estimates (Hours & Wages)

Quantitative Data Collection
The quantitative data collected includes individual data on preceptors, APRN students, and APRN
alumni and aggregate data on hospitals, APRN program applicants, SONs, SON faculty, and
CCSs. Year 1 startup costs include provision of historical data (2007-2012) in addition to the
collection of contemporaneous data. Calculations are based on an initial start-up burden to provide
historical data of 62-104 hours per respondent plus an annualized burden to provide
contemporaneous data of 137-552 hours per-respondent..
Regarding the monetary costs of compiling and uploading this information, the evaluation team
estimates that administrative personnel time will account for 90% of the time spent in these tasks.
The remaining 10% of the time will be accounted for by executive level staff. Accordingly, the
resulting hourly rate is $24.3/hour, which is the weighted average of 90% of time at $21.9/hour
and 10% of the time valued at $45.74/hour. These rates are based on information from the Bureau
of Labor Statistics 1. To calculate the monetary burden to respondents, the estimated time burden
per respondent was multiplied by the time cost for each respondent, and the resulting figure was
multiplied by 5, which is the total number of respondents (demonstration sites). The resulting
estimated range of total monetary costs is $40,790-$146,830, which are divided between $24,16079,735 for the first year and $16,630-$67,095 for the second year. The difference in costs between both
years is explained by the startup costs that are incurred during the first year of the evaluation only.

1 See http://www.bls.gov/oes/current/oes_nat.htm#00-0000.

6

Table 2. Total Respondent Hour Burden by Year for Quantitative Data Collection
Year 1
Activity

Total
Respondents

Start-up
(one time)

5
5
5

Reporting
TOTAL

Hours per
response

Year 2
Annualized
Response Burden

62 - 104
137-552
199-656

310-520
685-2760
995-3280

Total
Respondents

Hours per
response
0
5
5

0
137-552
137-552

All Years
Annualized
Response Burden
0
685-2760
685-2760

Total Hours

310-520
1370-5520
1680-6040

Table 3. Total Respondent Cost Burden by Year for Quantitative Data Collection
Year 1
Activity
Start-up
(one
time)
Reportin
g
TOTAL

Total
Respondents

Cost per
response

Year 2
Annualized Cost
Burden

Total
Respondents

All Years

Cost per
response

Annualized
Cost Burden

Total Cost

5

$1506 - $ 2528

$7530-$12640

0

$0

$0

$7530-$12640

5
5

$3326-$13419
$4832-$15947

$16630-$67095
$24160-79735

5
5

$3326-$13419
$3326-$13419

$16630-$67095
$16630-$67095

$33260-$134190
$40790-$146830

7

Qualitative Data Collection
The qualitative data collection is structured to conduct interviews and focus groups with multiple key informant types. Each key
informant will contribute information based on their role and responsibility. As a result, the following will vary widely: the
interview/focus group length, the utility of multiple interviews, and the costs per interviewee associated with their hourly wages. Thus,
estimated burden calculations will be reported in a range. There will be between 50 and 77 interviews per site (GNE partnership
network). Each participant will respond 1 – 2 times per year. Each interview/focus group will last .5 – 1.5 hours. The estimated hourly
wage of expected respondents ranges from $20.91 (Administrative Assistant) to $120 (Dean of a Nursing Program). Aggregated, these
estimates equate to an annualized hourly burden of 367.5 in year 1 and 332.5 hours in year 2 and an annualized cost burden of
$16,170.95 - $19,729.53 in year 1 and $15,656.40 - $18,121.10 in year 2.
Table 4. Total Respondent Hour Burden by Year for Qualitative Data Collection
Instrument
GNE Strategic planning and
implementation team - T1
SON Administration - T1
Clinical faculty - T1
Clinical placement coordinator - T1
Director of Nursing/Clinical Director T1
APRN Student
Preceptor
Interim Check-In
GNE Strategic planning and
implementation team - T3
SON Administration - T3
Clinical faculty - T3
Clinical placement coordinator - T3
Director of Nursing/Clinical Director T3
CFO/Business Manager
Totals

# of
people

Year 1
Hours per Annualized
response
hours

# of
people

Year 2
Hours per
response

All Years
Annualized
hours

Total hours

15

1

15

15

20
60
20

1
1.5
1

20
90
20

20
90
20

15

1

15

15

100
20
75

1.5
1
0.5

150
20
37.5

100
20
0

1.5
1

150
20
0

300
40
37.5

0

0

15

1

15

15

0
0
0

0
0
0

20
60
20

1
1.5
0.75

20
90
15

20
90
15

0

0

15

1

15

15

0
367.5

15
265

0.5
.75 – 1.5

7.5
332.5

7.5
700.0

0
325

.5 – 1.5

8

Table 5. Total Respondent Cost Burden by Year for Qualitative Data Collection
Year 1
Instrument

# of
people

Hours per
response

GNE Strategic planning and
implementation team - T1

15

1

SON Administration - T1

20

1

Clinical faculty - T1
Clinical placement coordinator T1
Director of Nursing/Clinical
Director - T1
APRN Student
Preceptor

60

Hourly
wage level

Annualized
Cost Burden

# of
people

Hours
per
response

All Years
Total cost
burden

1.5

$328.65 $1203.75
$810.40 $2400
$4578.30

$328.65 $1203.75
$810.40 $2400
$4578.30

20

1

$33.91

$678.20

$678.20

15

1

$41.54

$623.10

$623.10

100
20

1.5
1

75

0.5

$31.71
$80.29
$21.91 $80.25

$7135.50
1605.8
$411 $1504.88

1.5
1

GNE Strategic planning and
implementation team - T3

15

1

SON Administration - T3

20

1

Clinical faculty - T3
Clinical placement coordinator T3
Director of Nursing/Clinical
Director - T3
CFO/Business Manager

60

Totals

Annualized
Cost
Burden

$21.91 $80.25
$40.52 120
$33.91

100
20

Interim Check-In

Year 2
Hourly
wage
level

325

8.5

9

$21.91 $120

$16170.95 $19729.53

$31.71
$80.29

$7135.50
$1605.80

1.5

$21.91 80.25
$40.52 $120
$33.91

$328.65 $1203.75
$810.40 $2400
$4578.30

$14271
$3211.60
$411 $1504.88
$328.65 $1203.75
$810.40 $2400
$4578.30

20

0.75

$33.91

$381.45

$381.45

15

1

$41.54

$623.1

$623.10

15

0.5

$51.52

$193.20

265

8.25

$21.91 $120

$15656.40 $18121.10

$193.20
$31827.35
–
$37850.63

All Data Collection
Total respondents for all data collection are 330; this number assumes 325 respondents for the qualitative data collection and 5
respondents (sites) for the quantitative data collection. The total burden estimate across both years and for both types of data collection
is 3,750 to 12,260 hours. As a result, the estimated range for the overall monetary cost burden is $72,617.40 to $184,681.
Table 6. Total Respondent Hour Burden by Year for All Data Collection
Year 1
Activity
Quantitative
Qualitative
Grand Total

Total
Respondents
5
325
330

Hours per
Response
199-656
.5 – 1.5
199.5-657.5

Year 2
Annualized
Response Burden

Total
Respondents

995-3280
367.5
1362.5-3647.5

5
265
270

Hours per
Response
137-552
.75 – 1.5
137.75-553.5

All Years
Annualized
Response Burden
685-2760
332.5
1017.5-6407.5

Total Hours
3050-11560
700
3750-12260

Table 7. Total Respondent Cost Burden by Year for All Data Collection
Year 1
Total
Respondent
Activity
s
Quantitative
5
Qualitative
Grand
Total

325
330

Cost per
Response
$4832-$15947
$21.91 - $120
$4853.91$16067

Year 2
Annualized Cost
Burden

Total
Respondents

Cost per
response

$24160-$79735
$16170.95 $19729.53

5

$3326-$13419

265

$40330.95-$99464.50

270

$21.91 - $120
$3347.91$13539

10

All Years
Annualized Cost
Burden
$16630-$67095
$15656.40 $18121.10
$182286.40$85216.10

Total Cost
Burden
$40790-$146830
$31827.35 –
$37850.63
$72617.40$184681

13.

Capital Costs

There are no capital costs. .
14.

Cost to Federal Government

Table 6. Cost to the Federal Government
Activity/
Partner

Year 1
Cost

Year 2
Cost

Cost all
Years

Specific Activities
• Reviewing and
providing guidance on
Start-up/
instruments, OMB
Government
clearance, and data
collection approach.
$4570
$1428
$5998
• Developing data set
requirements
• Setting up FTP site
for file transfer
• Providing assistance
Start-up/
to respondents
Contractor
• Testing FTP site
• Retrieving Data
• Site Visits, Calls,
Focus Groups
$158,326 $136,771 $295,097
Total
-$162,896 $138,199 $301,096

15.

Cost Description

GS-14 staff:
105 hours X $57.13

Contractor staff: 2
2895 hours x $101.94
--

Changes to Burden

Not applicable as this is a new information collection.
16.

Publication/Tabulation Dates

The publication dates for the GNE evaluation reports are provided in the project timeline table
below:

2 According to national industry-specific occupational employment and wage estimates, social scientist and related
workers in “Management, Scientific, and Technical Consulting Services” (NAICS 541600) on average earned $43.75
in 2011, which is approximately $101.94 including overhead, fringe and general and administrative indirect rate
($43.75 * 2.3). http://www.bls.gov/oes/current/naics4_541600.htm

11

Table 7. GNE Task Area #2 -- Analyze and Report on the Demonstration Site Efforts Production
Schedule
Deliverable
Number

5

6

7

Deliverable Name
Reporting System
Design Reporting System (Identify data
elements, data collection frequency, finalize
forms and develop business rules)
Interim Qualitative and Quantitative
Analyses Report
Draft and Revise Interim Qualitative and
Quantitative Analyses Report -- Draft
Interim Qualitative and Quantitative
Analyses Report -- Final
Final Qualitative and Quantitative Analyses
Report
Final Qualitative and Quantitative Analyses
Report -- Draft
Final Qualitative and Quantitative Analyses
Report -- Final
Year One Site Visits - Key Informant
Interviews, Small Group Interviews, Focus
Groups
Scheduling of Site Visits
Site Visits
Analysis of data from site visits
Year Two Telephone Interviews
Scheduling of telephone interviews
Telephone Interviews
Analysis of data from telephone interviews
Year Two Site Visits - Key Informant
Interviews, Small Group Interviews, Focus
Groups
Scheduling of Site Visits
Site Visits
Analysis of data from site visits

Official
Due Date
Ongoing

Start
Date

End Date

1/7/2013

3/5/2013

8/22/2013

9/13/2013

9/20/2013

9/25/2013

5/2/2014

7/3/2014

7/18/2014

8/1/2014

5/4/2013
7/8/2013
8/9/2013

6/3/2013
8/8/2013
8/22/2013

Revised to
12 months
of award

20 months
of award

11/4/2013
11/18/201
3
12/2/2013

2/4/2014
4/15/2014
4/22/2014

11/18/201
3
12/2/2013
12/9/2013

2/14/2014
6/17/2014
7/18/2014

The quantitative data gathered at the beginning and during the length of the project from the
demonstration networks and secondary sources will be analyzed utilizing both a simple differences
and a Difference-in-Differences (DID) approach to estimate the impact of the demonstration. In
order to meet project deadline of September 2013 for the preliminary findings report, historical
and first year demonstration quantitative data must be collected by June 2013. Qualitative data
will be collected for the final report.
12

The DID approach requires both a treatment group (the demonstration sites) and a comparison
group. Due to the unfeasibility of obtaining data from hospital networks not included in the
demonstration, secondary data on APRN training nationwide will be utilized for comparison
purposes.
In contrast to uniquely estimating the effect of the demonstration by comparing the before and
after outcomes, the DID estimator calculates the difference in outcomes between the pre- and postdemonstration-comparison differences of outcomes—thus, the difference in the differences
between both groups over time. The basic purpose of DID in this project will be to examine the
effect of the demonstration by comparing the outcomes of the treatment group after treatment to
the outcomes of the treatment group before treatment and to the comparison secondary data. DID
accounts for other confounding effects that may occur at the same time as the demonstration and
that may affect its outcome. The DID approach uses the comparison group to subtract these
confounding effects, assuming that these confounding effects are the same for the demonstration
and the comparison groups.
The data obtained from the demonstration sites for which no comparison secondary information
can be obtained will be analyzed using a time-series/before-and-after approach.
17.

Expiration Date

CMS would like an exemption from displaying the expiration date, since this is a quarterly data
collection instrument to be used on continuing basis.

13