Instrument 6B – Site Visit 3 Interview Guide for Administrators_ Licensed LifeSet Experts

LifeSet Licensed Program Expert Interview Site Visit #3

Instructions:

Thank you for us joining today. We would like to invite you to participate in an interview that will last about 60 minutes. We are conducting an evaluation of LifeSet, and today we’d like to learn more about how the program has been operating over the last year. Specifically, we will talk about challenges and barriers the program has experienced, provider performance monitoring, and aspects of the program model among other topics. We will use this information to better understand how the LifeSet program operates and serves young adults in New Jersey.

Your participation in this interview is voluntary. You can choose not to answer any question or not participate in the interview at all. There will be no consequences to you if you choose not to participate. We will keep the information you provide private and will not share it with anyone except for research staff working on the study. Additionally, federal law states that an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this data collection is 0970-0577 and the expiration date is XX/XX/XXXX.

Background and Role

Current position and professional experience& Main responsibilities and role within LifeSet program

Thank you for taking the time to talk with me today.

[If this is the first time interviewing this respondent] I’d like to start by learning a little bit about you.

1. What is your current position and how long have you been in it?

2. What are your main responsibilities?

3. [If held current position since before March 2020] Are these the same responsibilities you had pre-COVID? If not how did your responsibilities change?

4. How long have you worked at Youth Villages?

5. What other positions have you held at Youth Villages?

6. What role do you play as part of the LifeSet program?

7. Have you worked with former foster youth or other vulnerable youth populations before working for Youth Villages? If so, in what capacity?

8. Have you held other positions – client facing or management – in social services before? If so, what were they?

9. Is there a degree/credential or experience requirement for your position? What is it?

[If this is a follow-up interview with this respondent] First I’d like to ask about any changes in your role as a Program Expert.

10. Has anything has changed regarding your position and main responsibilities since we last spoke on [date]? 

    1. Please describe any changes that have occurred in your position.

    2. When did these changes occur?

    3. Why were these changes made?

Implementation Infrastructure

The first topic is about challenges and supports LifeSet has experienced in New Jersey, as well as the impact of the evaluation on implementation.

Challenges, supports, and barriers

11. What challenges have you encountered implementing LifeSet? How did you overcome these challenges?

1. [If applicable] Has Covid-19 presented any challenges in implementing LifeSet? Has it been beneficial in any way?

12. What things have helped support the implementation of LifeSet?

1. [If needed] For example, DCF policies, data systems or technology, coordination with partners, or funding.

13. What things have hindered or created barriers to the implementation of LifeSet?

14. In what ways has the New Jersey LifeSet implementation team (the regular calls with DCF and providers) affected program implementation, both positive and negative?

    1. Are there ways has it supported implementation? Are there ways it has hindered program implementation? What are some examples?

Impact of the evaluation

15. In what ways has the external evaluation by the Urban Institute impacted the way LifeSet is implemented, both positive and negative?

Performance Monitoring

I’d like to understand how Youth Villages monitors the performance of agencies that provide LifeSet Services. We are defining performance monitoring broadly to include not just processes or activities but also outcome monitoring and reporting.

16. Can you describe the general performance monitoring requirements for providers of LifeSet services?

17. Are there some LifeSet providers that are more effective than others?

    1. [If yes] How do you know whether some providers are more effective?

       1. [Probes] Is this informed by data, either qualitative or quantitative?

    2. In your view, what makes those providers more effective at delivering LifeSet?

Collection and use of feedback

18. Does Youth Villages collect feedback from LifeSet supervisors and specialists to understand program needs?

1. If so, how, and how often is this feedback collected? 

2. How, if at all, does this input change LifeSet service delivery?   

19. Does Youth Villages collect feedback from LifeSet youth? 

1. If so, how, and how often is this feedback collected? 

2. How, if at all, does this input change LifeSet service delivery? 

Program Model 

We spoke about the program model and fidelity during our second site visit in June 2022. The next few questions will help to assess changes in those areas since then.

Main components 

20. To your knowledge, has anything changed regarding the program’s model since June 2022?

    1. [If yes] Can you please describe what has changed and why?

Fidelity 

21. Do you believe LifeSet is being delivered with fidelity in New Jersey?  

1. If yes, how do you know? 

2. If no, what areas of the model are not being delivered as intended? How do you know?

22. Currently, do you know of differences in how services are delivered across the four providers? 

1. If so, please tell us about some of those differences.   

2. If not, please tell us how LifeSet is able to maintain consistency in how services are delivered. 

Staff Requirements and Responsibilities

The next topics are about your processes of staff supervision, consultation, and assessment.

Supervision and consultation

23. What areas of LifeSet do you provide support to for the New Jersey providers?

1. Are there any areas in particular that providers need extra support in?

24. Can you describe what a typical meeting with the New Jersey providers looks like?

1. How often do you meet with them?

2. What are the meeting requirements?

25. Are there resources from Youth Villages or the LifeSet model that you refer to often in your work as a program expert?

1. If so, what are those resources?

2. Are there resources from Youth Villages that you often point other LifeSet staff towards?

3. What resources that aren’t available do you think might be helpful in your work as a program expert?

Staff assessment

26. Does the LifeSet model include an assessment of supervisors’ and specialists’ performance?

    1. [If yes] Please describe the assessment process, such as your role in the assessment, the methods you use, and how often they are done.

    2. [If no] What informal ways do you use to assess supervisors’ and specialists’ performance?

27. How is feedback about performance shared with supervisors and specialists?

Service Delivery

I’d now like to ask you some questions about youth engagement and then some questions about the cultural responsiveness of LifeSet.

Youth Engagement

28. We understand that youth engagement is an important part of the LifeSet model. Is this something the model trains or instructs specialists on how to do? If so, how is this done?

29. Is a specialist’s ability to engage with youth part of their [formal/informal, as appropriate] performance assessment?

1. [If yes] Please describe how you assess a specialist’s ability to engage youth.

30. What are the characteristics of LifeSet Specialists that make them especially good at engaging with youth?

31. What challenges have LifeSet specialists in New Jersey had in engaging youth in services?

1. [If applicable] How do you help address these challenges in your role as program expert?

32. In what specific ways do you think engagement with youth has led to successful outcomes?

33. Have you noticed any changes or improvements in the New Jersey specialists’ ability to engage youth over the last couple years?

    1. [If yes] Can you describe what changes you have noticed?

Cultural responsiveness and disparities in services

34. Do you consider the LifeSet program to be culturally responsive? If so, in what ways? Could you give some examples?

35. Are changes needed to LifeSet to make it more responsive to youths' diverse characteristics?,

    1. [If yes] What changes need to be made? Are steps being taken to make those changes?

Partners

Now, I’d like to learn a bit about your relationships with DCF and the LifeSet providers.

Relationship among implementing agencies

36. How would you describe Youth Villages’ relationship with DCF?

1. Now that LifeSet is being implemented, how often does Youth Villages coordinate and collaborate with DCF?

37. How would you describe Youth Villages’ relationship with LifeSet providers?

1. How much does Youth Villages usually coordinate and collaborate with LifeSet providers?

Reflection on LifeSet Services

Finally, I’m interested in your opinion about the LifeSet program.

38. Is there anything about LifeSet that you would like to change? If so, what is it?

39. What does the LifeSet program does well?

40. What does the LifeSet program could improve upon?

Closing Questions

Thank you for taking the time to talk with me today. I have a couple closing questions.

41. Is there anything that I did not ask about that you think I should know about LifeSet or your experience?

42. Do you have any final questions for me about the study, or about the research team?

The Paperwork Reduction Act Statement: This collection of information is voluntary and will be used to evaluate the programs and services provided to young adults who are currently or were previously in foster care. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0577, Exp: XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Michael Pergamit at [email protected].