CMS-10467 Qualitative Data Collection Protocols

EVALUATION OF THE GRADUATE NURSE EDUCATION DEMONSTRATION PROGRAM

OMB No. 0938-1212

Expires 9/30/2016

Attachment 1: Qualitative Data Collection Protocols

Contract No. HHSM-500-2011-00013I

Task Order No. HHSM-500-T0009

07/06/2015

Table of Contents

I. Annual Qualitative Check-in Telephone Call Protocols 1

Graduate Nurse Education Evaluation Interview Guide: GNE Project Oversight Team 2

Graduate Nurse Education Evaluation Interview Guide: SON Administration Team 6

II. Post-Demonstration Telephone Interview Protocol 10

Graduate Nurse Education Evaluation Interview Guide: GNE Project Oversight Team 11

Graduate Nurse Education Evaluation Interview Guide: SON administration Team 15

1. Annual Qualitative Check-in Telephone Call Protocols

Graduate Nurse Education Evaluation Interview Guide: GNE Project Oversight team

_{Outline with Timings for Interviewer Use}

Time in each section (in minutes)	Topic	Elapsed time at end of section (in minutes)
5	Introduction and Informed Consent	5
8	Changes to demonstration project	13
30	Outcomes, barriers, and facilitators	43
15	Perceived return on investment and sustainability plans	58
2	Conclusion	60

Introduction and Informed Consent

The purpose of this interview is to discuss your ongoing activities related to the GNE

Demonstration Project, including changes, challenges, successful strategies, perceived outcomes, and sustainability plans.

Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty.

The interview will last about 60 minutes and with your permission will be audio-recorded for our records. Only IMPAQ staff will have access to the recordings and they will be destroyed upon completion of the evaluation.

This interview will work best if you do most of the talking. Feel free to speak openly and candidly about your experiences and perspectives regarding this demonstration.

We will treat the information you supply in a confidential manner. Only selected staff from the evaluation team will have access to your actual interview responses. You will not be identified by name in any report to the public or the funder, but we may refer to you by title, e.g., “Dean, School of Nursing,” “Chief Nursing Officer.”

As part of the evaluation, we will share our findings with CMS. The results of the evaluation will describe strengths as well as areas for improvement. Findings will describe the progress made and difficulties encountered in achieving project goals. Although you will not be identified by name, it is likely that readers will be able to identify specific GNE projects and may also be able to identify participating individuals based on their role within the project.

After the interview is completed, we will ask you to identify any information which you would NOT want presented in any form that might be linked with your personally or with your GNE project. We will comply with your request by being especially careful to not present such information in any way that would enable it to be linked to you or your GNE demonstration project.

[Ask all participants] Do you agree to participate in this interview? [Yes No]

• [If no]: Can you share some of the reasons why you are declining? Thank you for your time.

• [If yes]: Do we have your permission to audio record this interview? [Yes No]”

Do you have any questions for me before we begin? Ok, let’s get started.

If New Interviewee: Interviewee Background

1. What is your title?

2. How long have you been with this organization?

3. What are your main responsibilities in your position?

4. What are your main responsibilities in relation to the GNE Demonstration Project (if different or more specific)?

5. Are these responsibilities similar to what they were prior to the GNE Demonstration Project? In what way?

6. What percentage of your FTE is dedicated to the GNE Demonstration Project?

Changes since the GNE Demonstration Project (8 minutes)

I’d like to begin by discussing any key changes since the last time we spoke.

1. Have there been any changes to the process for coordinating students’ clinical placements?

• If yes, tell me about those changes.

• What prompted those changes?

2. Have there been any key staffing changes?

• If yes, tell me about those changes.

• [If same interviewee as prior year] Has your role changed since the last interview?

• If yes, what is your role now?

• How does it relate to the Demonstration?

3. Are there any organizations or health care agencies that are no longer participating in since the demonstration project ended?

• If yes, which organizations are no longer participating?

• What made these organizations decide not to continue their participation?

4. Are there any new partners?

• If yes, who are the new organizations?

• How did these organizations become involved?

5. Are there any organizations that increased or decreased the number of preceptors?

• If applicable, which organizations increased the number of preceptors?

• If applicable, which organizations decreased the number of preceptors?

6. What, if any, changes to policy or procedure has your organization made to support the demonstration project?

• Does your SON accept students into an online training program?

• If yes, do you permit online-enrolled students to participate in the GNE demonstration? Describe the GNE precepting process for online-enrolled students.

Outcomes, barriers, and facilitators (30 minutes)

7. What effects do you believe the demonstration has had on enrollment? Graduation?

• Do you think changes in enrollment and graduations are primarily because of the demonstration, or are there other factors influencing these?

8. What, if any, outcomes do you feel you have achieved through the demonstration project?

• How do you think this program has benefited your organization?

• What are the benefits of this project?

• To what extent did the demonstration project result in the outcomes you desired?

• If no outcomes or benefits were achieved, why do you think this project has not benefited your organization?

9. What components of your demonstration project have allowed you to be successful in achieving the goals you set out to accomplish?

10. What are the most difficult challenges you have faced in implementing the demonstration project?

• What made those challenges so difficult?

• What components of the demonstration project have created challenges to achieving the goals you set out to accomplish?

11. What are positive and/or negative effects of the GNE demonstration on non-demonstration SONs?

11. Have any aspects of the demonstration produced undesirable or negative outcomes? Describe these aspects, the negative outcomes, and possible reasons/explanations for the negative outcomes.

13. If you had to start over from the beginning, what would you do differently knowing what you now know?

Perceived return on investment and sustainability plans (15 minutes)

14. To what extent do you think that the investments you have made to support the demonstration project have paid off?

• Has it been worth it?

• What, if at all, do you see as the business case for expanding the number of APRN clinical placement opportunities?

15. In what ways, if any, will you sustain the GNE Demonstration Project activities beyond grant funding?

• If sustaining activities, what resources or investments will be necessary to sustain them?

• Where will those resources or investments come from?

• If not sustaining activities, what makes the project unsustainable?

16. If you could redesign the GNE Demonstration Project for the future, what changes would you suggest to the Centers for Medicare and Medicaid Services?

Conclusion (2 minutes)

17. Describe the best outcome or aspect of the GNE Demonstration? The worst?

18. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today?

19. Are there any areas covered by this interview that you are particularly concerned about having your identity connected with? If so, which ones?

Graduate Nurse Education Evaluation Interview Guide: SON administration Team

_{Outline with Timings for Interviewer Use}

Time in each section (in minutes)	Topic	Elapsed time at end of section (in minutes)
5	Introduction and Informed Consent	5
8	Changes to the demonstration project	13
30	Outcomes, barriers, and facilitators	43
15	Perceived return on investment and sustainability plans	58
2	Conclusion	60

Introduction and Informed Consent

The purpose of this interview is to discuss your ongoing activities related to the GNE

Demonstration Project, including changes, challenges, successful strategies, perceived outcomes, and sustainability plans.

Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty.

The interview will last about 60 minutes and with your permission will be audio-recorded for our records. Only IMPAQ staff will have access to the recordings and they will be destroyed upon completion of the evaluation.

This interview will work best if you do most of the talking. Feel free to speak openly and candidly about your experiences and perspectives regarding this demonstration.

We will treat the information you supply in a confidential manner. Only selected staff from the evaluation team will have access to your actual interview responses. You will not be identified by name in any report to the public or the funder, but we may refer to you by title, e.g., “Dean, School of Nursing,” “Chief Nursing Officer.”

As part of the evaluation, we will share our findings with CMS. The results of the evaluation will describe strengths as well as areas for improvement. Findings will describe the progress made and difficulties encountered in achieving project goals. Although you will not be identified by name, it is likely that readers will be able to identify specific GNE projects and may also be able to identify participating individuals based on their role within the project.

After the interview is completed, we will ask you to identify any information which you would NOT want presented in any form that might be linked with your personally or with your GNE project. We will comply with your request by being especially careful to not present such information in any way that would enable it to be linked to you or your GNE demonstration project.

[Ask all participants] Do you agree to participate in this interview? [Yes No]

• [If no]: Can you share some of the reasons why you are declining? Thank you for your time.

• [If yes]: Do we have your permission to audio record this interview? [Yes No]”

Do you have any questions for me before we begin? Ok, let’s get started.

If New Interviewee: Interviewee Background

1. What is your title?

2. How long have you been with this organization?

3. What are your main responsibilities in your position?

4. What are your main responsibilities in relation to the GNE Demonstration Project (if different or more specific)?

5. Are these responsibilities similar to what they were prior to the GNE Demonstration Project? In what way?

6. What percentage of your FTE is dedicated to the GNE Demonstration Project?

Changes to the Nursing program or GNE Demonstration Project (8 minutes)

I’d like to begin by discussing any key changes to the demonstration project:

1. [If same interviewee as prior year] Has your role changed since the last interview?

• If yes, what is your role now?

• How does it relate to the Demonstration?

2. Have there been any changes to the nursing program overall?

• For example, have there been changes to student enrollment, course offerings, specific curriculum, or staffing?

3. Have there been any changes to the process for coordinating students’ clinical placements?

• If yes, tell me about those changes.

• What prompted those changes?

4. How does the number of clinical opportunities affect your acceptance decisions for qualified applicants?

5. Have there been any key staffing changes made to support the demonstration project?

• ^{If yes, tell me about those changes.}

6. What, if any, changes to policy or procedure has your organization made to support the demonstration project?

• Does your SON accept students into an online training program?

• If yes, do you permit online-enrolled students to participate in the GNE demonstration? Describe the GNE precepting process for online-enrolled students.

Outcomes, Barriers, And Facilitators (30 minutes)

7. What effects do you believe the demonstration has had on enrollment? Graduation?

• Do you think changes in enrollment and graduations are primarily because of the demonstration, or are there other factors influencing these?

8. What, if any, outcomes do you feel you have achieved through the demonstration project?

• How do you think this program has benefited your organization?

• What are the benefits of this project?

• To what extent did the demonstration project result in the outcomes you desired?

• If no outcomes or benefits were achieved, why do you think this project has not benefited your organization?

9. What components of your demonstration project have allowed you to be successful in achieving the goals you set out to accomplish?

10. What are the most difficult challenges you have faced in implementing your demonstration project?

• What made those challenges so difficult?

• What components of your demonstration project have created challenges to achieving the goals you set out to accomplish?

10. Have any aspects of the demonstration produced undesirable or negative outcomes? Describe these aspects, the negative outcomes, and possible reasons/explanations for the negative outcomes.

10. How did you increase preceptor hours to meet the needs of your students?

• Are additional precepted hours at the expense of preceptors at other schools? If yes to other schools, are these other schools within the GNE demonstration network or outside of the GNE demonstration network?

10. What are positive and/or negative effects of the GNE demonstration on non-demonstration SONs?

10. If you had to start over from the beginning, what would you do differently knowing what you now know?

10. If you had to give advice to other organizations considering similar programs, what advice would you give?

Perceived Return on Investment and Sustainability Plans (15 Minutes)

16. To what extent do you think that the investments you have made to support the demonstration project have paid off?

• Has it been worth it?

• What, if at all, do you see as the business case for expanding the number of APRN clinical placement opportunities?

17. In what ways, if any, are you sustaining the GNE Demonstration Project activities beyond grant funding?

• If sustaining activities, what resources or investments will be necessary to sustain these activities?

• Where will those resources or investments come from?

• If not sustaining activities, what makes the project unsustainable?

18. If you could redesign the GNE Demonstration Project for the future, what changes would you suggest to the Centers for Medicare and Medicaid Services?

Conclusion (2 minutes)

18. Describe the best outcome or aspect of the GNE Demonstration? The worst?

18. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today?

21. Are there any areas covered by this interview that you are particularly concerned about having your identity connected with? If so, which ones?

2. Post-Demonstration Telephone Interview Protocol

Graduate Nurse Education Evaluation Interview Guide: GNE Project Oversight team

_{Outline with Timings for Interviewer Use}

Time in each section (in minutes)	Topic	Elapsed time at end of section (in minutes)
5	Introduction and Informed Consent	5
8	Changes since demonstration project end	13
30	Outcomes, barriers, and facilitators	43
15	Perceived return on investment and sustainability plans	58
2	Conclusion	60

Introduction and Informed Consent

The purpose of this interview is to discuss your ongoing activities related to the GNE

Demonstration Project, including changes, challenges, successful strategies, perceived outcomes, and sustainability plans.

Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty.

The interview will last about 60 minutes and with your permission will be audio-recorded for our records. Only IMPAQ staff will have access to the recordings and they will be destroyed upon completion of the evaluation.

This interview will work best if you do most of the talking. Feel free to speak openly and candidly about your experiences and perspectives regarding this demonstration.

We will treat the information you supply in a confidential manner. Only selected staff from the evaluation team will have access to your actual interview responses. You will not be identified by name in any report to the public or the funder, but we may refer to you by title, e.g., “Dean, School of Nursing,” “Chief Nursing Officer.”

As part of the evaluation, we will share our findings with CMS. The results of the evaluation will describe strengths as well as areas for improvement. Findings will describe the progress made and difficulties encountered in achieving project goals. Although you will not be identified by name, it is likely that readers will be able to identify specific GNE projects and may also be able to identify participating individuals based on their role within the project.

After the interview is completed, we will ask you to identify any information which you would NOT want presented in any form that might be linked with your personally or with your GNE project. We will comply with your request by being especially careful to not present such information in any way that would enable it to be linked to you or your GNE demonstration project.

[Ask all participants] Do you agree to participate in this interview? [Yes No]

• [If no]: Can you share some of the reasons why you are declining? Thank you for your time.

• [If yes]: Do we have your permission to audio record this interview? [Yes No]”

Do you have any questions for me before we begin? Ok, let’s get started.

If new interviewee: Interviewee Background

1. What is your title?

2. How long have you been with this organization?

3. What are your main responsibilities in your position?

4. What are your main responsibilities in relation to the GNE Demonstration Project (if different or more specific)?

5. Are these responsibilities similar to what they were prior to the GNE Demonstration Project? In what way?

6. What percentage of your FTE is dedicated to the GNE Demonstration Project?

Changes since the GNE Demonstration Project (8 minutes)

I’d like to begin by discussing any key changes since the end of the demonstration.

Since the demonstration ended:

1. Have there been any changes to the process for coordinating students’ clinical placements?

• If yes, tell me about those changes.

• What prompted those changes?

2. Have there been any key staffing changes?

• If yes, tell me about those changes.

• [If same interviewee as prior year] Has your role changed since the last interview?

• If yes, what is your role now?

• How does it relate to the Demonstration?

3. Are there any organizations that are no longer participating since the demonstration project ended?

• If yes, which organizations are no longer participating?

• What made these organizations decide not to continue their participation?

4. Are there any new partners?

• If yes, who are the new organizations?

• How did these organizations become involved?

5. Are there any organizations or clinical agencies that increased or decreased the number of preceptors?

• If applicable, which organizations or clinical agencies increased the number of preceptors?

• If applicable, which organizations or clinical agencies decreased the number of preceptors?

6. Since the demonstration ended, what, if any, changes to policy or procedure has your organization made?

Outcomes, barriers, and facilitators (30 minutes)

7. What effects do you believe the demonstration has had on enrollment? Graduation?

• Do you think changes in enrollment and graduations are primarily because of the demonstration, or are there other factors influencing these?

8. What, if any, outcomes do you feel you have achieved through the demonstration project?

• How do you think this demonstration project has benefited your organization?

• What are the benefits of this demonstration project?

• To what extent did the demonstration project result in the outcomes you desired?

• If no outcomes or benefits were achieved, why do you think this project has not benefited your organization?

9. What components of the demonstration project have allowed you to be successful in achieving the goals you set out to accomplish?

10. What are the most difficult challenges you faced in implementing the demonstration project?

• What made those challenges so difficult?

• What components of the demonstration project created challenges to achieving the goals you set out to accomplish?

11. Have any aspects of the demonstration produced undesirable or negative outcomes? Describe these aspects, the negative outcomes, and possible reasons/explanations for the negative outcomes.

12. If you had to start over from the beginning, what would you do differently knowing what you now know?

13. If you had to give advice to other organizations considering similar programs, what advice would you give?

Perceived return on investment and sustainability plans (15 minutes)

14. To what extent do you think the investments you made to support the demonstration project have paid off?

• Has it been worth it?

• What, if at all, do you see as the business case for having expanded the number of APRN clinical placement opportunities?

14. In what ways, if any, are you sustaining the GNE Demonstration Project activities now that the funding has ended?

• If sustaining activities, what resources or investments are necessary to sustain them?

• Where are those resources or investments coming from?

• If not sustaining activities, what makes the project unsustainable?

14. If you could redesign the GNE Demonstration Project for the future, what changes would you suggest to the Centers for Medicare and Medicaid Services?

Conclusion (2 minutes)

14. Describe the best outcome or aspect of the GNE Demonstration? The worst?

14. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today?

19. Are there any areas covered by this interview that you are particularly concerned about having your identity connected with? If so, which ones?

Graduate Nurse Education Evaluation Interview Guide: SON administration Team

_{Outline with Timings for Interviewer Use}

Time in each section (in minutes)	Topic	Elapsed time at end of section (in minutes)
5	Introduction and Informed Consent	5
8	Changes since demonstration project end	13
30	Outcomes, barriers, and facilitators	43
15	Perceived return on investment and sustainability plans	58
2	Conclusion	60

Introduction and Informed Consent

The purpose of this interview is to discuss your ongoing activities related to the GNE

Demonstration Project, including changes, challenges, successful strategies, perceived outcomes, and sustainability plans.

Your participation in this interview is voluntary. If, at any time, you wish to discontinue participation, you may do so without penalty.

The interview will last about 60 minutes and with your permission will be audio-recorded for our records. Only IMPAQ staff will have access to the recordings and they will be destroyed upon completion of the evaluation.

This interview will work best if you do most of the talking. Feel free to speak openly and candidly about your experiences and perspectives regarding this demonstration.

We will treat the information you supply in a confidential manner. Only selected staff from the evaluation team will have access to your actual interview responses. You will not be identified by name in any report to the public or the funder, but we may refer to you by title, e.g., “Dean, School of Nursing,” “Chief Nursing Officer.”

As part of the evaluation, we will share our findings with CMS. The results of the evaluation will describe strengths as well as areas for improvement. Findings will describe the progress made and difficulties encountered in achieving project goals. Although you will not be identified by name, it is likely that readers will be able to identify specific GNE projects and may also be able to identify participating individuals based on their role within the project.

After the interview is completed, we will ask you to identify any information which you would NOT want presented in any form that might be linked with your personally or with your GNE project. We will comply with your request by being especially careful to not present such information in any way that would enable it to be linked to you or your GNE demonstration project.

[Ask all participants] Do you agree to participate in this interview? [Yes No]

• [If no]: Can you share some of the reasons why you are declining? Thank you for your time.

• [If yes]: Do we have your permission to audio record this interview? [Yes No]”

Do you have any questions for me before we begin? Ok, let’s get started.

If new interviewee: Interviewee Background

1. What is your title?

2. How long have you been with this organization?

3. What are your main responsibilities in your position?

4. What are your main responsibilities in relation to the GNE Demonstration Project (if different or more specific)?

5. Are these responsibilities similar to what they were prior to the GNE Demonstration Project? In what way?

6. What percentage of your FTE is dedicated to the GNE Demonstration Project?

Changes to the Nursing program since the GNE Demonstration Project ended (8 minutes)

I’d like to begin by discussing any key changes since the end of the demonstration project.

Since the demonstration ended:

1. [If same interviewee as prior year] Has your role changed since the last interview?

• If yes, what is your role now?

• How does it relate to the Demonstration?

2. Have there been any changes to the nursing program overall?

• For example, have there been changes to student enrollment, course offerings, specific curriculum, or staffing?

3. Have there been any changes to the process for coordinating students’ clinical placements?

• If yes, tell me about those changes.

• What prompted those changes?

4. How does the number of clinical opportunities affect your acceptance decisions for qualified applicants?

5. Have there been any key staffing changes made since the end of the demonstration project?

• ^{If yes, tell me about those changes.}

6. Since the demonstration project ended, what, if any, changes to policy or procedure has your organization made?

Outcomes, barriers, and facilitators (30 minutes)

6. What effects do you believe the demonstration has had on enrollment? Graduation?

• Do you think changes in enrollment and graduations are primarily because of the demonstration, or are there other factors influencing these?

6. What, if any, outcomes do you feel you have achieved through the demonstration project?

• How do you think this program has benefited your organization?

• What are the benefits of this project?

• To what extent did the demonstration project result in the outcomes you desired?

• If no outcomes or benefits were achieved, why do you think this project has not benefited your organization?

1. What components of your demonstration project allowed you to be successful in achieving the goals you set out to accomplish?

2. What were the most difficult challenges you faced in implementing your demonstration project?

• What made those challenges so difficult?

• What components of your demonstration project created challenges to achieving the goals you set out to accomplish?

3. Have any aspects of the demonstration produced undesirable or negative outcomes? Describe these aspects, the negative outcomes, and possible reasons/explanations for the negative outcomes.

12. How did you increase preceptor hours to meet the needs of your students?

• Are additional precepted hours at the expense of preceptors at other schools? If yes to other schools, are these other schools within the GNE demonstration network or outside of the GNE demonstration network?

• If yes to other schools, are the hours then reduced for other schools within the GNE demonstration network or for schools outside of the GNE demonstration network?

13. If you had to start over from the beginning, what would you do differently knowing what you now know?

14. If you had to give advice to other organizations considering similar programs, what advice would you give?

Perceived return on investment and sustainability plans (15 minutes)

15. To what extent do you think the investments you made to support the demonstration project have paid off?

• Has it been worth it?

• What, if at all, do you see as the business case for expanding the number of APRN clinical placement opportunities?

15. In what ways, if any, are you sustaining the GNE Demonstration Project activities beyond grant funding?

• If sustaining activities, what resources or investments are necessary to sustain these activities?

• Where are those resources or investments coming from?

• If not sustaining activities, what made the project unsustainable?

17. If you could redesign the GNE Demonstration Project for the future, what changes would you suggest to the Centers for Medicare and Medicaid Services?

Conclusion (2 minutes)

18. Describe the best outcome or aspect of the GNE Demonstration? The worst?

19. Is there anything that I did not ask about that you would like to share with me, or do you have any additional thoughts about what we have discussed today?

20. Are there any areas covered by this interview that you are particularly concerned about having your identity connected with? If so, which ones?