Form Grantee Project Di Grantee Project Di Grantee Project Director Interview

August 2014

PROGRAM EVALUATION FOR PREVENTION: PARTNERSHIPS FOR SUCCESS

State Project Director Interview Protocol

Contents

Section Page

Overview and Purpose 1

Preparation for Interviews 1

Interviews 1

Postinterview Activities 2

Partnerships for Success State Project Director Interview Guide 3

1. Introduction 3

2. Interview Questions 5

2.1 Planning and Decision Making—Subrecipient Selection 5

2.2 Evidence-Based Programs, Policies, and Practices (EBPPPs) 6

2.3 Infrastructure and Capacity 6

2.4 Leveraging Funds and Resources 8

2.5 Monitoring, Performance Measurement, and Evaluation 9

2.6 Cultural Competency 9

2.7 Health Disparities 10

2.8 Challenges and Catalysts 10

3. Closing 10

Overview and Purpose

The PEP-C team will conduct initial telephone interviews with each Partnerships for Success (PFS) grantee Project Director during the winter or early spring of 2015, with follow-up interviews to be conducted using a similar protocol in Years 3 and 5 of the project in accordance with the Evaluation Plan.¹ The primary purpose of the interviews is to obtain the perspective of the implementing Project Directors on several important topics related to planning, subrecipient selection, capacity, leveraging of funds, evaluation, cultural competency, health disparities activities, and selection and implementation of interventions. These semistructured interviews will last approximately 90 minutes and will proceed using the included guide. The PEP-C team will use the information gathered during, and in preparation for, the interview to produce a profile for each interviewee program, as well as an aggregate-level summary report. This profile will include information on each grantee’s strengths as well as potential areas for improvement, and it will provide information for monitoring purposes. In addition, the data gathered during the interview will be used in qualitative and quantitative analyses to answer the PFS cross-site evaluation questions for PFS cross-site evaluation reports.

Preparation for Interviews

Before each interview:

• The interviewer will arrange and confirm the interview time and telephone number with each grantee Project Director using an email template approved by SAMHSA and will obtain written (email) consent to record the interview.

• The interviewer will review grantee-specific information, including quarterly reports, proposal responses, Grantee-Level Instrument responses, and other documents, as applicable.

Interviews

Data collection will consist of individual telephone interviews with the State-level Project Director for the PFS Program.

The interviewer will document the interviews through notes taken on the interview forms in this document. The files containing the completed interview forms will be placed in the relevant grantee’s folder under the Data Collection folder on SharePoint (see [LINK TBD]). With participant consent, the interviews may be audio-recorded to facilitate reporting after the interview.

Postinterview Activities

Within a week after each interview, the interviewer will send an email to the Project Director, thanking him or her for taking the time to be interviewed. The interviewer will follow up by telephone within 2 weeks for any outstanding information or document requests (e.g., contact information, plans, reports, local analyses).

The interviewer will complete the interview process by:

• Filing the completed interview forms (see remaining sections of this document);

• Filing any written materials obtained from the grantee;

• Providing a draft copy of a one- to two-page interview summary profile to the interviewee for review that focuses on grantee strengths and areas for improvement;

• Obtaining feedback on the summary profile from grantees and CSAP staff (including grantee Project Officers), with SAMHSA approving final version; and

• Preparing and filing the final summary profile for each grantee.

Partnerships for Success State Project Director Interview Guide

1. Introduction

Hello, my name is ______________ and I work for The Program Evaluation for Prevention Contract, or PEP-C. PEP-C is part of the team conducting an evaluation of Partnerships for Success (PFS), which is sponsored by the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Prevention (CSAP). The overall goal of the cross-site evaluation is to document and assess the effectiveness of the PFS approach to SAMHSA’s mission of reducing underage drinking and prescription drug misuse and abuse.

This interview is part of a series of interviews that will be conducted at the beginning and during Years 3 and 5 of the project (FOR PFS II GRANTEES – AT THE BEGINNING AND END OF THE FINAL YEAR OF THE PROJECT). You are one of more than 40 grantee-level Project Directors who will participate in this study. We are contacting Project Directors to get their unique perspectives on several important topics related to the implementation of PFS. All grantee-level Project Directors will be asked the same questions.

The interview will help the PEP-C team understand your approach to planning and implementation and examine the impact of your efforts on underage drinking and prescription drug misuse and abuse in your state, tribe, or jurisdiction. The PEP-C team will use this information to produce a 2- to 3-page profile for your program, as well as an aggregate-level summary report across all PFS grantees. The profile will include information on the strengths of your PFS implementation as well as potential areas for improvement, and it will provide information to your Project Officer for monitoring purposes. In addition, the data gathered during the interview will be used in qualitative and quantitative analyses to answer the PFS cross-site evaluation questions for PFS cross-site evaluation reports.

We will share results of this interview with SAMHSA via the profile. We will provide you a copy of the profile to review for accuracy, but SAMHSA will approve the final version of your profile. We hope that this will be a helpful source of insights about your project’s strengths and potential areas for improvement. CSAP will clearly see which grantees provided specific information, so data are not confidential or anonymous. Your participation in the PFS cross-site evaluation, including this interview, is required by SAMHSA.

This interview will take about 90 minutes of your time. In addition, we may contact you after this interview if necessary to clarify responses or to obtain outstanding information.

In preparation for this interview, you had responded positively to an email requesting your permission for audio recording. We are audio-recording this interview as a backup to our written/typed notes. This recording is only for the use of our evaluation team and will be destroyed after interview notes are finalized. Is it OK if I begin recording now?

Thank you for agreeing to participate in this interview. Do you have any questions for me before we begin?

2. Interview Questions

2.1 Planning and Decision Making—Subrecipient Selection

1. [BASELINE ONLY] Describe the process you use for making Partnerships for Success (PFS) subrecipient selection and funding decisions. Which criteria does your jurisdiction use for selecting PFS subrecipients?

PROBES:

• Did your jurisdiction use a competitive or noncompetitive process?

• Did your jurisdiction fund prior SPF SIG grantees with PFS funds?

2. [BASELINE ONLY] How did your jurisdiction define “high need” in selecting PFS subrecipients?

PROBE: Which of the following criteria did you consider?

• Size/magnitude of the problem

• Prevalence

• Severity of consequences

• Geographic distribution

• Sociodemographic distribution (e.g., ethnicity, race, gender, economic status)

• Benchmark comparisons

• Time trends

• How changeable that need was

• How easily you could evaluate changes in that need

3. [BASELINE ONLY] How did your jurisdiction define low readiness/capacity of subrecipients? [NOTE TO INTERVIEWER: This question should be asked even if low readiness/capacity is not a criterion for their subrecipient selection.]

PROBE: Which of the following criteria did you consider?

• knowledge of/experience with the Strategic Prevention Framework

• knowledge of/experience with substance use prevention interventions

• availability of trained prevention staff

• access to high-quality data

• ability to use data in prevention planning

• access to prevention services

• fiscal/financial resources

• ability to collaborate with other organizations on interventions

2.2 Evidence-Based Programs, Policies, and Practices (EBPPPs)

The implementation of evidence-based interventions is key to the overall success of the PFS program.

4. What criteria are used for defining EBPPPs for PFS in your State?

[IF APPLICABLE:] Do the same criteria hold for both programs and environmental strategies?

IF THE GRANTEE NEEDS EXAMPLES OF CRITERIA, YOU CAN SHARE:

• Inclusion in a Federal registry of evidence-based interventions

• Found to be effective (on the primary targeted outcome) in a peer-reviewed journal

• Based on a theory of change that is documented in a clear logic or conceptual model

• Similar in content and structure to interventions that appear in registries, peer-reviewed literature, or both

• Supported by documentation of effective implementation multiple times in the past (showing consistent pattern of positive effects)

• Reviewed by a panel of informed experts, including qualified prevention researchers, local prevention practitioners, and key community leaders (e.g., law enforcement and education representatives, elders within indigenous cultures)

5. How consistently are these criteria for defining EBPPPs used:

• Across other prevention agencies in your State? (e.g., in the departments of health or mental health)

• Across other prevention programs or initiatives within your organization?

6. Please describe any resources or technical assistance available to assist subrecipients in the selection, implementation, and adaptation of EBPPPs for the focal areas of PFS.

6a. How is information on the criteria for evidence-based programs, policies, and practices disseminated to subrecipients?

2.3 Infrastructure and Capacity

One of the goals of the Partnerships for Success program is to strengthen prevention capacity and infrastructure at the State and community levels. The PEP-C team and CSAP are interested in better understanding the many ways in which grantees and subrecipients achieve this goal. The following questions are intended to provide a comprehensive picture of processes used to strengthen your capacity and infrastructure.

You can think of infrastructure as the foundation that supports a system, community, or society in achieving desired outcomes. For Strategic Prevention Framework (SPF)-PFS grantees, key infrastructure components include groups such as an advisory body, epidemiological outcomes workgroup, and evidence-based practice workgroup. Infrastructure can also include systems such as a data collection and monitoring system and training and technical assistance system. Infrastructure also encompasses things like funding and resource allocation policy, and systematic promotion of collaboration.

7. How did the PFS contribute to the current prevention planning process in your jurisdiction? This process might involve your advisory body, epidemiological outcomes workgroup, and evidence-based practice workgroup.

7a. What prevention planning operations developed during SPF State Incentive Grant (SIG) funding were sustained? What features were added or enhanced?

8. How has the PFS contributed to the current decision-making process in your jurisdiction? How are community stakeholders incorporated?

8a. What decision-making operations developed during SPF SIG funding were sustained? What features were added or enhanced?

9. [BASELINE ONLY] How much time elapsed between the end of your SPF SIG funding and the start of your PFS funding (in months)?

[NOTES TO INTERVIEWER: If grantee had SPE funding, this is counted as part of the elapsed time. If less than 6 months have elapsed between SPF SIG and PFS, please skip Question 10 and its sub-questions.]

10. [BASELINE ONLY] I’m now going to ask you to describe how you maintained a number of infrastructure components between SPF SIG and PFS funding.

10a. What did the Advisory Council do between the end of SPF SIG funding and the start of PFS?

10b. How did you sustain and maintain the operation of your Advisory Council between SPF SIG and PFS funding?

10c. What did the epidemiological outcomes workgroup (EOW) do between the end of SPF SIG funding and the start of PFS?

10d. How did you sustain and maintain the operation of your EOW between SPF SIG and PFS funding?

10e. What did the evidence-based practice workgroup (EBPW) do between the end of SPF SIG funding and the start of PFS?

10f. How did you sustain and maintain the operation of your EBPW between SPF SIG and PFS funding?

10g. What kinds of training and technical assistance (T/TA) did your [STATE/TRIBE/JURISDICTION] provide between the end of SPF SIG funding and the start of PFS?

10h. How did you sustain and maintain the operation of your T/TA system between SPF SIG and PFS funding?

2.4 Leveraging Funds and Resources

Leveraging resources describes the process of combining PFS-funded resources with other resources to augment prevention strategy delivery (i.e., to do more together than with PFS resources alone).

Redirecting or realigning resources describes the process of shifting resources (fiscal and nonfiscal) from one focal area to a focal area that complements PFS efforts.

Neither of these definitions includes using PFS funds to “free up” previously programmed funds for other uses, as this is not permitted under the grant.

11. a. Based on your quarterly progress report responses, it appears that you may have leveraged funds from [FILL IN]. What other strategies did your jurisdiction employ for leveraging prevention funds and resources (e.g., staff) in support of your PFS efforts?

IF NEEDED, EXAMPLES YOU CAN SHARE WITH THE GRANTEE INCLUDE:

• Building off other prevention funding streams like Safe Schools/Healthy Students or Drug-Free Communities grants

• Conducting joint trainings with other agencies

• Implementing interventions jointly (or coordinating) with other agencies

• Engaging prevention providers or coordinators from block grants or Safe and Drug-Free Schools and Communities

• Collaborating with other agencies to sponsor statewide survey

11b. Based on your quarterly report responses, it appears that you may have redirected or realigned funds from [FILL IN]. What other strategies did your jurisdiction employ for redirecting or realigning prevention funds and resources in support of your PFS efforts?

IF NEEDED, EXAMPLES YOU CAN SHARE WITH THE GRANTEE INCLUDE:

• Redirecting training to high-priority areas

• Realigning focus from solely individual services to also include population-focused practices

• Restricting funded coalitions to target same priority need

12. How has your jurisdiction encouraged the use of local funds to supplement resources provided by PFS?

IF NEEDED, EXAMPLES YOU CAN SHARE WITH THE GRANTEE INCLUDE:

• Providing T/TA around obtaining local funds

• Encouraging subrecipients to apply for locally available grant funding, such as Drug-Free Communities grants

13. What funding strategies were added or enhanced for SPF-PFS compared with SPF SIG?

14. [FOLLOW-UP ONLY] In what areas did your agency collaborate with other State agencies? If none, why not?

PROBES:

• Sharing information or data

• Conducting joint strategic planning

• Implementing joint programming

• Providing training or technical assistance

• Jointly funding projects or interventions

• Jointly funding a position

• Jointly designing a program

2.5 Monitoring, Performance Measurement, and Evaluation

15. Please describe any performance management systems in place for monitoring your jurisdiction’s prevention system milestones and progress.

16a. How often are evaluation data reviewed to mark progress on set goals and objectives?

16b. Are monitoring and evaluation findings tied back to the needs assessment process (SPF Step 1)? In what ways?

IF NEEDED, EXAMPLES YOU CAN SHARE WITH THE GRANTEE INCLUDE:

• Using evaluation data to update needs assessment

• Evaluating whether identified needs have been met

17. Your jurisdiction proposed to look at the following indicators of prevention system milestones and progress:

[PREPOPULATE USING GRANT APPLICATION, EVALUATION PLAN, OR BOTH; LIST LIKELY INCLUDES (1) NUMBER OF TTA ACTIVITIES, (2) NUMBER SERVED BY TTA, (3) PERCENTAGE OF SUBRECIPIENTS THAT INCREASED NUMBER/PERCENT OF EBPPPS, (4) PERCENTAGE OF SUBRECIPIENTS REPORTING AN INCREASE IN PREVENTION ACTIVITIES SUPPORTED BY LEVERAGING, AND (5) PERCENTAGE OF SUBRECIPIENTS THAT SUBMIT DATA USING GRANTEE DATA SYSTEM]

Please describe the data your jurisdiction is using to assess these indicators.

2.6 Cultural Competency

18. Many jurisdictions have plans and policies in place to address cultural competence in their substance abuse prevention system. How has PFS contributed to cultural competence plans and policies in your jurisdiction?

19. Please describe any resources or technical assistance available to assist subrecipients in the selection, implementation, and adaptation of culturally and linguistically competent programs, policies, and practices.

PROBES:

• in-person trainings

• direct assistance/facilitated co-development of materials

• online courses

• written materials or guidance from lead agency

2.7 Health Disparities

20. Which health disparities are you addressing in your jurisdiction through the PFS?

PROBES:

• Gender?

• Race/ethnicity?

• Lesbian, gay, bisexual, transgender, queer (LGBTQ)?

• Rural/urban?

• Military families?

21. Please describe your [plan/processes] for addressing health disparities at the grantee level. How will you ensure that strategies reach populations affected by health disparities?

22. How are you [planning to support/supporting] subrecipients in addressing health disparities?

2.8 Challenges and Catalysts

23. [FOLLOW-UP ONLY] What were the most significant barriers to the success of your subrecipients’ PFS-related efforts? Why?

24. [FOLLOW-UP ONLY] What factors were most important in helping subrecipients overcome challenges or barriers to their implementation of their PFS interventions?

3. Closing

If you have any questions or concerns about this interview, please contact the PEP-C Help Desk through the MRT system, by leaving a message at (866) 558-0724, or by sending an email to [email protected]. You can request assistance at any time and someone will respond to you within 24 hours or on the next business day.

Do you have any final questions or concerns to share with me at this time?

Thank you for your time.

1 PFS 2 grantees will complete the interview twice (after OMB approval and at or near the end of their grants). PFS 2013 grantees will complete the initial interview during Year 2 of their grants, and PFS 2014 grantees will complete the initial interview during Year 1 of their grants.