Instrument 2_Followup Questions for Executive Directors

Instrument 2:

Follow-up Questions for Executive Directors

Activity 2: EXECUTIVE DIRECTOR FOLLOWUP INTERVIEW

Pre-Introduction

IF ADVANCE EMAIL WAS SENT BUT RESPONDENT DID NOT REPLY TO EMAIL/ OR THIS IS A BLIND CALL/EMAIL WAS NOT AVAILABLE: This is ________________ with Westat and the Chafee Strengthening Outcomes for Transition to Adulthood project.

[NOMINATED PROGRAM] was recently nominated as an example of an innovative service program for young people, ages 14-26, either youth transitioning out of foster care or young adults formerly in care who transitioned out of care in the recent past. We’d like to learn more about [NOMINATED PROGRAM] and explain our interest in following up. However, if at this time, you’d prefer to decline to participate in this discussion for any reason, that is completely acceptable. We can discontinue the call and we will not contact you again.

(IF ADVANCE EMAIL WAS SENT BUT RESPONDENT DID NOT REPLY TO EMAIL: I recently sent you an email to tell you I was going to call.)

Is this a good time to talk?

IF NO:

• At this time, if you’d prefer to decline to participate in this discussion for any reason, that is completely acceptable. We can discontinue the call and we will not contact you again. How would you like to proceed?

• [If they indicate they’d like to continue] When would be a good time to talk? IF NEEDED: I have a short set of questions I’d like to email you in advance of our call. [CONFIRM OR COLLECT EMAIL AND SET APPOINTMENT]

IF YES:

• CONTINUE:

Introduction

Thank you for talking with us today about your program. We are researchers from [Westat, an independent research firm located in Rockville, MD / the Kempe Center for the Prevention and Treatment of Child Abuse and Neglect, Department of Pediatrics, School of Medicine University of Colorado]. We have been contracted by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) to conduct an evaluation of programs serving youth in or transitioning out of foster care (called the Chafee Strengthening Outcomes for Transition to Adulthood [Chafee SOTA] Project).

 

As you know, [NOMINATED PROGRAM] was nominated as an innovative program for youth transitioning from foster care. During the conversation today, we are interested in collecting information about how this program works, what its goals and activities are, and how it has evolved over time. At this time, we are not conducting an evaluation that will make conclusions about whether the program is working or not working – we are just gathering enough information to determine if your program is appropriate to participate in an evaluability assessment, the next step in this process. An evaluability assessment (EA) is an assessment to help determine whether a program is ready for an evaluation. Evaluability assessments will be conducted on a subset of programs that we speak with to determine if they would be good candidates for, and are interested in participating in further evaluation activities. Sites that are selected will be provided some financial assistance to participate in the evaluation. Please note that this interview will focus on your program overall and will not ask for information on individual clients.

Your participation in this interview is voluntary and it should take about an hour to complete. We will keep your responses private. In addition, please feel free to let me know if you’d like to skip any question I ask by saying, “I’d prefer to skip this question.”

We will be recording this session for analysis purposes; it also helps us with accurate note-taking. If you are not comfortable being recorded, please let me know. Recorded interviews will be stored in a secure location and will be destroyed after the recording has been transcribed.

Before we get started, do you have any questions for me?

Program Overview/Background

We’d like to begin with a few questions about [ORGANIZATION NAME] and your role within it, and then we’d like to learn a bit more about the [NOMINATED PROGRAM] and the services it provides. We have some information from the nomination form (and the internet), but we would like to ask some more detailed questions to supplement what we already know.

1. What is your position and role in the organization

• How long have you been with this organization? In this position?

2. What services/programs does your organization provide, overall?

3. What is the organization’s (or nominated program’s) target population?

• Does it serve both youth transitioning out of foster care and young adults formerly in care who transitioned out of foster care in the recent past?

• What are the age ranges for youth served?

• About how many youth are served each year?

• How are youth typically identified or referred for the program?

• What are the program’s eligibility criteria?

• Can you describe the youth who are typically served by the program? (Ex: Ages, racial composition, employment, and education status)

3. Who would you consider to be the organization’s (or program’s) key partners in providing services to youth transitioning from foster care? These are organizations with whom you collaborate to serve these youth and who might need to be included in any evaluation of your program. We ask you about this now because if your program is selected to move to the evaluability assessment process, we will include your partner agencies in data collection.

5. Outside of receiving services, do youth provide input around the program? (IF NEEDED: For example, are youth involved in planning services or processes, giving feedback for program improvements, or on a youth board?)

IF YES:

• 5a. In what ways are youth involved?

IF NO:

• 5b. [If youth are not currently involved in any of these activities] Are you and your staff amenable to working with our team to develop and implement a plan for engaging youth in evaluation activities? Involvement of youth in evaluation is a key component of our project.

Nominated Program Outcomes

6. What short-term and long-term outcomes have you seen?

6. What short-term and long-term outcomes are you hoping for?

6. How do you track outcomes for participants in the program?

• What kind of data systems do you use?

• What outcome data do you collect on youth/young adults? In what format are the data? [Are there codebooks or data collection forms we could review?]

• Who does the data collection/data entry? How frequently?

• What data are currently available?

• Do you have any reports or summaries that you could share?

9. Administrative Data Review

• Do you collect and maintain administrative data (e.g., demographics, enrollment and attendance, and length in the program) data on the youth you serve? 

• If yes – If you were chosen to participate in an evaluability assessment, would you be willing to work with us to conduct a review of these data to determine its nature and the extent to which it can be used for evaluation purposes?  To do this effectively, we may ask that your program data administrator be available for a discussion with us.

Evaluation Capacity

10. Do you have a logic model for the program or a Theory of Change? [If so, is that something you can share with us?]

10. Has this program been evaluated previously?

    1. If so, when?

    2. Who conducted it?

    3. What was the nature of the evaluation? (Ex. Research questions, data collection methods, process or outcome)

    4. Do you have any materials you can share with us about the evaluation?

10. One of the next steps in the selection process is to undergo an evaluability assessment. An evaluability assessment (EA) is an assessment to help determine whether a program is ready for an evaluation]. This step will require an onsite visit where our team would implement the EA assessment process, which includes collecting additional information from you, your staff, and the youth that are served through [NOMINATED PROGRAM] through interviews and focus groups. If selected, would you be interested and willing to participate in this process?

10. Is there interest in/enthusiasm for an evaluation of the program? IF YES: What would you hope to learn?

10. We’d like to ask if you could send us any relevant and easily accessible documents – those that can be emailed - that will help us continue to assess your program for advancement to the evaluability assessment stage. But these should include only those that are easily accessible to you. Some examples might include program and implementation plans, logic models, monitoring reports or even grant applications, if appropriate to share. If you have any questions about what to send, please feel free to email me directly.

Also, if you are selected to participate in the evaluability assessment, we may ask for additional documentation before we conduct our site visit. But we will work with you around any additional request.

Conclusion

10. Is there any other information about your program that you think would be helpful for us to know?

4 Instrument 2:

Follow-up Questions for Executive Directors