Interview Protocol

Form Approved

OMB No. 0990-0450

Exp. Date XX/XX/20XX

SUSTAINABILITY STUDY OF PROGRAMS FUNDED BY OAH IN 2010
DISCUSSION GUIDE FOR FORMER TPP GRANTEES

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-0450. The time required to complete this information collection is estimated to average 10 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer

Prior to the phone interview, we plan to:

1. Review the grantee’s application and the final or last report received.

2. Customize the TPP grantee protocol to confirm information collected through document review and supplement with questions about specific experiences of this grantee.

3. Tailor the introduction to each site

Identifying Information [to be completed by Interviewer]:

TPP Grantee Organization Name
Program status
Respondent Name
Respondent Phone Number
Date and Time of Call
Interviewer Name
Note-taker Name
Other Comments

Introduction (5 minutes)

Thank you for agreeing to speak with us. My name is [Lead Interviewer Name]. My colleague [Note-taker name] is also joining the call to listen in and take notes. We are from Mathematica, an independent research firm contracted by the Office of Adolescent Health (OAH) of the U.S. Department of Health and Human Services to conduct this study.

The purpose of our discussion today is to learn about your efforts to sustain your 2010-2015 Teen Pregnancy Prevention program (TPP) or services over time, the types of support or technical assistance your organization has received for that purpose, and any challenges or successes related to your efforts in sustaining your program or services after grant funding ended. For the purposes of our discussion, by “TPP program” we mean all of the program components, curricula, or services that were funded by the Office of Adolescent Health TPP grant from 2010 to 2015.

After this interview, we plan to follow-up with you once more in later this year or in early 2018 to learn about the ways in which OAH-funded programs or services may be sustained over time. Your point of view and the lessons you may have learned as part of your efforts are extremely valuable to OAH.

[For grantees who also participated in the Cost Study, add: We understand you may have shared some of this information as part of the TPP Cost Study. However, this study is a separate, independent, effort and we may ask you some questions that feel similar in order to make sure the data gathered through program documents and reports is accurate.]

There are just a few things I wanted to note before we begin:

• Our conversation today will take up to 90 minutes.

• Please keep in mind there are no right or wrong answers to our questions; we just want to hear about your experiences and opinions.

• In order to make sure our notes are accurate, would it be okay with you if we record this call? No one else besides our internal team will listen to this recording, and it will only be used to ensure that the notes accurately reflect our discussion. The recording will get erased once we have completed the notes.

• We will use the information we gather from all of our discussions to prepare case studies and summary reports for OAH that may highlight particular themes and experiences, but we will not identify you by name in our reporting.

Do you have any questions for us before we start?

A. Respondent background (5 minutes)

Could you start by describing your role at the [Name of Grantee Organization]?

1. How long have you been in this role?

1. What was your role with the program funded by the TPP grant? How long were you in that role?

B. Former OAH Funded program(s) (10 minutes)

[NOTE: When preparing for interview, lead interviewer should review relevant reference notes on: implementation setting, program model(s) and services, implementation sites, partners and partner staff, number of youth enrolled, and whether an evaluation was conducted – then tailor the questions below accordingly to probe further as needed.]

Let’s talk a little bit about the program(s) or services that were funded by OAH in 2010.

3. Can you briefly describe the way [Name of TPP program] came about?

4. Can you briefly confirm the core components and services of the program funded in 2010? [NOTE: List the components described in the document review and probe for any missing information]

5. Can you talk about how the program delivery was structured during the grant period?

We are also interested in learning about the different types of support you relied on to implement the programs or services that were funded by the OAH TPP grant in 2010.

6. In addition to your funding from OAH, did you receive support for the program from anywhere else, including financial or in-kind support (this could include support from within your organization or external to it)?

• If yes, can you describe the support and how it was used to support program implementation?

C. Sustainability planning and outcomes (40 minutes)

Our next questions are about your organization’s activities related to sustainability planning before, during, and after the 2010 - 2015 grant period.

7. In your email, you had mentioned that your program is [still operating in some form/not operating]. Is that correct?

8. When the grant period began, did you have plans to sustain [TPP program name] once the federal funding ended? If yes, could you describe what your plans were?

9. What were the critical elements or components of the program that you felt needed to be present for the program to be sustained?

• Did you feel you had those elements in place?

10. Did you receive technical assistance regarding sustainability planning?

• From what sources did you receive technical assistance or guidance on sustaining your program?

• Were specific types of technical assistance particularly useful? Why?

• Are there types of technical assistance that you did not receive but think would have been useful for your organization to have received?

11. Now let’s talk about the factors that affected whether or not your program was sustained. We’ll start with the eight key elements that OAH has identified as part of their sustainability planning toolkit, but after that we will also discuss any additional factors that may have affected your program’s sustainability.

[NOTE: Interviewer should probe on each of the bulleted elements during the discussion]

• Did you develop or utilize an action plan for sustaining your program? Probes:

• If yes, when did you begin developing the plan?

• Who was involved in this effort?

• Did you receive assistance from OAH or anyone else in developing the plan?

• Did you conduct an assessment of the needs of target communities? Probes:

  • How and when did you assess community needs?

  • Did this influence program selection initially or any changes you made to the program’s implementation?

  • How did this affect program sustainment?

• Did you take any steps to integrate the program into community infrastructures? Probes:

  • For example: through sub-grantee organizations, implementation sites, youth advocacy networks, etc.) where it was operating?

  • Did you meet with any challenges or successes in this process?

• Did you take steps to obtain support and/or buy in from the local community or key stakeholders? Probes:

  • Who were the key organizations or individuals whose support was important to sustainment?

  • Did you meet with any challenges or successes in this process?

• Did you establish or use a leadership team for sustainability planning? Probes:

  • If yes, when did you establish the team?

  • Who were the key members of the team? What roles did they each play?

  • Did you have support structures and guidance for staff in place?

  • What steps did the leadership team take to help sustain the program?

  • Did the leadership team face any challenges working together for program sustainment?

• Did you identify or access additional sources of funding for program sustainment? Probes:

  • If yes, when did you begin identifying additional funding sources? How did you identify them?

  • Did you face constraints or fluctuation in funding during or after the grant period?

  • What steps did you take to obtain additional funding to sustain the program (if at all)?

• Did you develop strategic partnerships for program sustainment?

Probes:

• Did you have partnerships or relationships in place that you depended on for program sustainment?

• Did you develop any new partnerships for the purpose of implementing and sustaining the program?

• Did you have any challenges working with partners? Did these affect the program’s sustainability?

• Did you feel the program was adaptable in order to be sustained? Probes:

  • Did you feel the program could adjust to your particular circumstances and needs?

  • In what ways did you adapt the program? How did these changes affect program sustainability?

12. Can you think of any other factors that affected program sustainability? [Note to interviewer: Use the document review to tailor the following list as needed]

Probe on:

• [If grantee had sub-grantees/partners] How did the relationship between your organization and sub-grantees (or implementing partners) affect program sustainment?

  • Did sub-grantees take any additional measures for sustaining the program?

• [If an evaluation was conducted] How did findings from the evaluation of the funded program affect program sustainability?

• Did you face any unexpected costs at start-up or during program operations? If yes, how did this affect program sustainability?

• Were there any state or local policies that influenced sustainability of the program?

• Were there any economic or socio-political changes that affected program operations and sustainability?

• Did access to other similar services in your target communities have any effect on program sustainment?

• Did training and support costs have any effect on program sustainment?

• Did buy-in among front-line staff have any effect on program sustainment?

• Did existence or support of a program “champion” or advocate affect program sustainment? [Explain, if needed: Was there a person within or outside your organization who advocated on behalf of the program with senior management or community leaders, for example? How did this person champion the program?]

13. Did you take any other steps (in addition to those described earlier) to sustain program outcomes?

14. Did your plans or goals for sustaining the program evolve or change by the end of the grant period? If yes, how and why did they change?

1. programs that are no longer operating

15. Did your organization attempt to sustain the program?

• [If yes, ask]: What types of funding sources did you try to obtain?

• [If no, ask]: Please tell us why.

16. Do you know when the program ended?

17. Are you aware of any current teen pregnancy prevention programs or services [Grantee Name] is offering for youth? If yes, what are the services, and how are they funded?

18. [If there was an evaluation, ask] Did you take steps to disseminate outcomes or findings from the evaluation?

19. For future efforts, if you could change anything, what would you do differently?

20. Do you know if the grantee organization is actively looking for funding to restart the program or services?

-------------[ SKIP to LESSONS LEARNED]-------------

2. programs that are operating in some form

[NOTE: If respondent has indicated they do not have this information, please ask if they can get this info from the organization(s) currently implementing the program, and how long they expect it to take. We can share these questions with them if that would be helpful, and then contact them again in a couple of weeks.]

21. To sustain the program, did you make any changes to the program’s goals, target population, length, content, staffing, or setting?

22. What is the current structure of the program? How is it the same as or different from what was funded?

23. How do you think the changes you made may have affected or are affecting program outcomes? Did you take any steps to ensure that the program’s outcomes are sustained?

24. What funding sources are currently supporting the program’s administration and implementation? When did you begin receiving this funding?

• Did you reapply for the TPP grant?

25. What would you highlight as strategies or support that most fostered program sustainability?

D. Lessons learned for future efforts (10 minutes)

[NOTE: Ask these questions only if not covered in previous section or to confirm overall impression]

26. What were the primary challenges and barriers you faced related to program sustainability?

27. What types of support and technical assistance were critical for sustaining your program?

• When would you say is the best time to provide this support to grantees?

• What different support is needed to promote short-term and long-term sustainability?

28. What advice or lessons learned would you give future grantees about sustaining their programs? Are there specific strategies they should pursue?

29. What advice would you give funding agencies around supporting grantees’ sustainability efforts? What could be done better?

E. Grantee organization (10 minutes)

[NOTE: Ask the questions below if time is remaining at the end of the interview. When preparing for interview, lead interviewer should review relevant reference notes on: grantee mission, number of years operating, number of staff, operating budget, funding sources, target population, partnerships, other programs/services offered – then tailor the questions below to probe further or fill in any gaps.]

Before we wrap up, we would just like to get a little background on your organization (or department/office) and it’s footprint in the community where it operates [if large university, state, or county agency, focus questions more narrowly on specific department/office responsible for funded program/services].

30. About how many people work for your organization (full-time, part-time, seasonal, etc.)? Has the staff structure changed over the last 5 years?

31. What were the main funding sources for your organization’s work before and during the grant period? How does this compare to the main funding sources for your organization’s work now?

32. Could you tell us the current operating budget of your organization (or department/office)? [Let respondent know that if they do not have this information readily available, it is fine to send it via email after the call. An approximate range or estimated amount is fine].

F. Closing/Thank you

These are all the questions I have for you today.

• Is there anything else anyone would like to say before we wrap up?

• Is it okay if we contact you to confirm or clarify any aspects of our discussion today?

Thank you for taking the time to speak with us!

8