Round 2 Site Visit Interview Guide

This site visit interview guide contains a complete inventory of questions, arranged by topic area that will be explored during site visits to participating Jobs Plus developments. There are seven categories of program partners who will be interviewed during each site visit, and not every question will be posed to every program partner. At the beginning of each topic area, the interview guide notes which program partners will be asked the questions included under that topic area.

Categories of program partners include:

• Program Directors (PD)

• Case Managers (CM)

• Community Coaches (CC)

• Directors at Partnering Agencies (DPA)

• Street-level Referral Partners (SRP)

• Housing Authority Administrators (HAA)

• Property Management Staff (PMS)

Appendix C: Site Visit Interview Guide Round 2

Questions on Program Support

We’re interested in the ways that HUD and PHAs learn Jobs-Plus and make it their own, as well as the ways that the PHAs and HUD support strong performance and accountability, internally to the program and externally to the Agency. This means that we’d like your perspectives on the different program supports that were provided by HUD, and how you feel about their successes and challenges. Then we’ll turn to the ways that you support accountability to the program, and what’s been successful and challenging in that regard.

HUD supports

Unless otherwise noted, questions are for:

Program Directors (PD)

Case Managers (CM)

What we mean by HUD support includes a number of interactions between the PHA and the agency. Let me describe a number of them, and you can tell me some of the strengths and challenges of these interactions in helping [you or the PHA] learn Jobs Plus as a model – by that we mean understanding how to use it fully to benefit your population, make a strong program, and how to make it your own – meaning relevant to your housing development.

First, let’s talk about the technical assistance offered by Abt through webinars

Could you describe their value to you to learn the Jobs-Plus model since we last spoke? [If relevant, given visit timing or interviewee perspective.] What are some of the benefits? What are some limitations?

1. Any particular moment that was helpful or “ah-ha” moments?

2. Any aspect that didn’t seem to help?

3. Were there any changes or decisions made as a result of assistance offered by Abt?

1. Next, let’s talk about the TA conferences in DC since we last spoke. [If relevant, given visit timing or interviewee perspective.] What are some of their strengths and challenges?

   1. Any particular moment that was helpful or “ah-ha” moments?

   2. Any aspect of the TA conferences that didn’t seem to help?

   3. Any changes or decisions made as a result of TA conferences?

2. Now, let’s talk about the site visits HUD employs. Could you describe their value to your learning Jobs-Plus, since the time we last spoke? [If relevant, given the timing of the visit or interviewee perspective]

   1. Any particular moment that was helpful or “ah-ha” moments?

   2. Any aspect of these visits that didn’t seem to help?

   3. Any changes or decisions made as a result of HUD’s visits?

3. Finally, let’s talk about day-to-day interaction with the program office and focus on any updates since we last spoke. [If relevant, given visit timing or interviewee perspective.] How frequent is this connection, what happens, and how helpful or unhelpful has it been in building a strong program?

4. How about the financial and compliance side of administering Jobs-Plus. Did your program budget(s) adequately capture all of the costs incurred in the delivery of Jobs-Plus? Have you experienced any challenges related to permissible costs, or other areas related to HUD guidance?

5. Are there any other supports I haven’t mentioned?

HUD accountability tools

Unless otherwise noted, questions are for:

Housing Authority Administrators (HAA)

Program Directors (PD)

Now, let’s turn to ways that HUD supported accountability. We’d like to understand how effective you think these have been in helping learn Jobs Plus, in promoting strong performance, and in making Jobs Plus the PHA’s “own.”

7. Could you talk about day-to-day interaction with the program office, and how it may have changed since our first site visit? [If relevant, given visit timing or interviewee perspective.] How has that acted as an accountability vehicle, to correct or shore up performance? How successful has that been?

1. Any particular moment that was helpful?

2. Any particular moment that didn’t seem to help?

8. Next, could you talk about reporting, metric development, and your review of these with HUD? Any evolution since we past spoke? [If relevant, given visit timing or interviewee perspective.]

1. Any particular moment that was helpful?

2. Any particular moment that didn’t seem to help?

Finally, let’s talk about funding and contract requirements itself.

9. To what extent are you now finding these to be helpful or effective tools in promoting strong performance? [If relevant, given visit timing or interviewee perspective.]

3. Do you currently feel that funding levels from HUD, in your view, are adequate to carry out the Jobs-Plus program with the right level of staff and support for the JPEID? Why or why not?

4. Since we last spoke, how effective has HUD been in communicating those requirements and how responsive has HUD been in helping to resolve any issues or questions you have encountered?

10. Are there any other HUD accountability tools that I’ve missed?

Balancing accountability and learning (HUD)

Unless otherwise noted, questions are for:

Housing Authority Administrators (HAA)

Program Directors (PD)

Case Managers (CM)

Finally, let me ask a few questions about the continuing balance and/or trade-offs between learning and accountability.

11. Are there any moments when you’ve felt tensions between HUD giving the site a chance to learn and adapt the model, and the need for strong performance?

1. Around what types of issues, performance challenges, administrative issues?

12. On the whole, what do you think HUD’s emphasis has been since we last spoke?[If relevant, given visit timing or interviewee perspective.] On giving time to learn and adapt Jobs Plus, or in ensuring strong performance?

    1. How, if at all, was this emphasis an effective one, given the site and its needs? How has it shifted since we last spoke? [If relevant, given visit timing or interviewee perspective.]

13. Do you have any updates as to what extent has the HUD influencedyour decisions about how budget funds were utilized and/or expended on Jobs-Plus (e.g., rate of spending)? [If relevant, given visit timing or interviewee perspective.] Please describe.

Accountability efforts by PHAs

Unless otherwise noted, questions are for:

Housing Authority Administrators (HAA)

Program Directors (PD)

Case managers (CM)

Now, let’s talk about accountability within the PHA itself. This means efforts on the part of senior management or external agencies to ensure strong performance.

14. We recognize that implementing a new program within a PHA can take some time and effort. In your view, how successful has senior PHA leadership been in providing the basic supports required to operate the program?

1. Hiring, procurement, JPEID (will probe deeper in other sections).

15. How access to PHA’s resources evolved over the course of implementation of Jobs-Plus?

1. Were there any points at which you requested/identified additional support that you would have liked to have received but had it denied by the PHA?

   1. If so, please describe the circumstances. What did you do to address it within the PHA, and how did it respond?

16. External to the PHA, are there other local agencies who are interested in strong performance of Jobs Plus? How, if at all, are they involved?

1. Mayors’ offices, external workforce, community colleges state agencies

2. In the absence of a formal governance role, how do they express their interests in the program?

17. If any of these other actors voiced concern about Jobs Plus program implementation, how was it articulated, and how were these issues addressed?

5. Numbers served, relating to HUD’s expectations, etc.

Balancing accountability and learning (PHAs)

Unless otherwise noted, questions are for:

Housing Authority Administrators (HAA)

Program Directors (PD)

Case managers (CM)

Finally, let me ask a question about the balance and or trade-offs between learning and accountability.

18. Are there any moments when you’ve felt tensions between the PHA giving the site a chance to learn and adapt the model, and the need for strong performance?

1. Around what types of issues, performance challenges, administrative issues.

19. On the whole, what do you think the PHA’s emphasis has been? On giving time to learn and adapt Jobs-Plus, or in ensuring strong performance?

20. How, if at all, was this emphasis an effective one, given the site and its needs?

Questions on Program Sustainability

Unless otherwise noted, questions are for:

Housing Authority Administration (HA)

Program Directors (PD)

Case managers (CM)

Community Coaches (CC)

Because Jobs Plus is a four year program, we’re interested in your thoughts about direction of the program over the longer term. While you’re not obligated to keep the program running past its funding date, we are interested in providing feedback to HUD about what, if any, aspects of Jobs Plus may be “left behind” in the developments over the longer term.

21. How, if at all, do you think Jobs-Plus has informed the way the PHA thinks about self-sufficiency and workforce development activities? Exclude CC

1. Saturation model, development of new partnerships, individual staff learning,

2. Identify any staffing or service delivery efficiencies?

3. Are any of these practices likely to continue? Why or why not?

   1. Probe: Data, staffing, morale, internal partnerships (between divisions).

22. Could you talk about CSW activities? To what extent do you think these can be sustained after Jobs Plus has ended?

1. Neighbor-to-neighbor contacts, informal associations, outreach.

23. What about the way that partnering organizations are involved with the PHA? How do you see these partnerships continuing over time?

1. Which may be sustained, and which do you think will not be? Why?

24. Do you have any other reflections on sustainability that you’d like to share?

Questions on Employment Services

Evolution of employment services

Unless otherwise noted, questions are for:

Program Directors (PD)

Case Managers (CM)

Community Coaches (CC)

First, I would like to hear your thoughts about the evolution of employment-related services at [SITE].

33. Has your thinking about the goals of employment related services changed over time?

    1. What led to thinking of employment services differently?

34. I would like to get a sense of any changes in the employment-related services that [SITE] offers since we last spoke in [DATE]

    1. Probe: Messaging or service emphasis; service uptake in general and among subgroups (families with/without young children, working/nonworking at baseline)

    2. Have you added new employment related services to the program?

1. If yes: Could you speak a bit about why you added these services? What was the rationale behind adding these services? What drove this change?

2. Probe: Any related to sectoral or career pathways or advancement.

1. Have you discontinued any services you were providing before?

1. If yes: why?

Evolution of employment-related partnerships

Unless otherwise noted, questions are for:

Program Directors (PD)

Case Managers (CM)

Community Coaches (CC)

Now, I would like to talk about partnerships the program has developed as part of employment-related services. First, let’s talk about the partnerships you developed early on in the implementation of the program, and then we’ll talk about newer partnerships.

33. How have the partnerships you developed early on evolved over time? Probe as needed:

1. New services provided?

2. Different approach to the provision of services?

3. Targeting services to residents in different situations?

4. New mechanisms for referrals?

33. What new partnerships have you developed?

1. What drove the development of those newer partnerships?

33. What have these newer partnerships added to your work and the implementation of Jobs Plus?

    1. Have you seen any new funding streams for services provided by external partners?

34. Have there been any challenges developing new partnerships? If so, what have they been?

I would like to get some final thoughts about employment services within the broader context of the Jobs Plus program.

33. How if at all, would you describe any synergies between Employment Services and the other components of the program (i.e., JPEID and CSW) as they’ve been put into practice here? If you don’t see it, why not? If you do, what are they?

    1. Probe: Engagement of program participants, motivation, retention, etc.

Questions on JPEID

Unless otherwise noted, questions are for:

Housing Authority Administrators (HAA)

Property Management Staff (PMS)

Program Directors (PD)

Case managers (CM)

Community coaches (CC)

Evolution of JPEID

Another key component of Jobs-Plus is the Jobs-Plus Earned Income Disregard, or JPEID. In this portion of the interview, we want to understand any changes that each site made to the JPEID component of their programs, any challenges that came up during the implementation of the JPEID, and what kinds of outcomes have been achieved.

41. Have you made any changes in the following aspects of the JPEID component of the Jobs-Plus program since it was initially rolled out? (IF “YES” TO ANY QUESTION, ASK WHY AND HOW THAT CHANGE HAS WORKED OUT.) Any change in . . . (Exclude CC)

    1. The targeting of the program to specific sub-groups?

    2. The messages you use when marketing JPEID?

    3. The methods you use to market the JPEID component of the program (e.g., events, work with property management posters, word of mouth, etc.)? How if at all has this made a difference?

    4. How the JPEID has been integrated into the other aspects of the Jobs-Plus program?

    5. The way the JPEID works in conjunction with other self-sufficiency programs?

    6. The introduction or elimination of any related financial incentives (e.g. IDAs, financial counseling)?

    7. The data management system you use to track the JPEID?

    8. Are there any other changes that have been made in the JPEID component of the program that you can think of?

42. As you think about your total experiences with the program, how helpful was Abt’s technical assistance in developing your approach to implementing the JPEID? How about assistance from HUD program staff? (Exclude CC)

Assessment of JPEID

Unless otherwise noted, questions are for:

Housing Authority Administrators (HAA)

Property Management Staff (PMS)

Program Directors (PD)

Case managers (CM)

Community coaches (CC)

Thinking about the implementation of the JPEID so far…

41. What do you think has gone as you thought it would? Why do think it worked out that way? What kinds of surprises – either good or challenging – have you encountered? Why do you think that happened?

42. What do you think have been the most critical elements of the JPEID implementation?

43. (For sites that offer other financial incentives, excluding HAs) How important was the coordination between the JPEID and the other financial services provided to program participants—such as financial counseling, IDA or incentive accounts—in the degree of program success? (Exclude CC)

    1. Did you run into any challenges with these other financial services or how they worked with the JPEID?

44. Are there any particular subgroups of residents that have taken advantage of the JPEID more than others?

    1. What do you think explains these higher rates?

    2. What subgroups have not taken advantage of the JPEID? Why do you think that is?

45. What advice would you give to others about implementing the JPEID? Did the pace of program start-up affect enrollment in JPEID?

46. How important was the JPEID in achieving the outcomes of your program?

    1. Could the program have worked if the JPEID wasn’t part of it?

    2. Why or why not?

47. As the JPEID program nears its end, what efforts have you made to prepare residents? (probes: marketing or information campaigns, financial counseling)

48. If you were going to do it over again, what would you do differently with respect to the JPEID?

49. What advice would you give to other PHAs on developing and implementing a JPEID?

Questions on CSW

We are now going to ask you to talk about how the community support for work component of Jobs-Plus has been going since our last visit. We have four main topics to explore with you about community support for work: 1) how you understand the ideas behind the strategy, 2) how it has been implemented at your site, 3) what the outcomes have been and 4) what some of the key challenges have been.

Conceptualizing CSW

Unless otherwise noted, questions are for all respondents:

Program Directors (PD)

Case Managers (CM)

Community Coaches (CC)

Housing Authority Administrators (HAA)

Property Management Staff (PMS)

Directors at Partnering Agencies (DPA)

Street-level Referral Partners (SPA)

First, let’s discuss how you and others now think about the community support for work component.

65. At our last visit, people here told us that the key ideas behind community support for work are AAA, BBB, and CCC. How does this compare to how you understand community support for work? Probe, if needed: to what extent has your thinking around CSW changed over time?

66. In your view, what is the value of community support for work to the goals of Jobs-Plus? Has that changed for you over time? Do you think others here agree about its value?

    1. Probe if not yet mentioned: One part of community support for work has been around building shared expectations and aspirations about work in the community. To what extent do you think this has been considered as part of the strategy? Would you say it has become more or less about this over time?

Implementing CSW

Unless otherwise noted, questions are for all respondents:

Program Directors (PD)

Case Managers (CM)

Community Coaches (CC)

Housing Authority Administrators (HAA)

Property Management Staff (PMS)

Directors at Partnering Agencies (DPA)

Street-level Referral Partners (SPA)

65. Please describe the ongoing guidance and support in terms of management, TA, conferences, training and HUD program staff support that you have received around your community support for work efforts. (excl. HUD)

66. Please describe the interactions between program staff and residents.

    1. What is the frequency of these interactions? Have these changed over time?

    2. What is the tone and quality of these interactions? Has this changed over time?

67. How would you describe social relations among residents at this site?

    1. How strong are the social networks here? Have these changed over time?

68. Please describe the role or roles (if any) that Housing Authority property managers play in implementing the community support for work strategy. To what extent has this changed over the time of the program? (excl. HUD)

69. What sorts of expenses have been incurred in connection with community support for work? What funds are used to pay for these expenses?

    1. Have there been any opportunities to split cost with other programs, partner organizations?

CSW Outcomes

Unless otherwise noted, questions are for all respondents:

Program Directors (PD)

Case Managers (CM)

Community Coaches (CC)

Housing Authority Administrators (HAA)

Property Management Staff (PMS)

Directors at Partnering Agencies (DPA)

Street-level Referral Partners (SPA)

65. Please think about the implementation of Jobs-Plus so far. What influence do you think community support for work efforts have had on the engagement of residents in Jobs-Plus so far?

    1. What evidence would you point to that demonstrates how it’s had that influence?

    2. Probe as necessary for:

1. Participation in Jobs-Plus

2. Resident leadership

3. Relationships

   1. Among residents on site

   2. Between residents and the broader community

4. Resident empowerment and shift of mindset

   1. Shared expectations and aspirations around work

1. With which types of residents has community support for work been most successful and which ones have been harder to engage?

65. How do you think that residents perceive the value of community support for work activities? (excl. HUD)

66. What thing or things do you think are most important for getting residents to commit to making changes in their lives? What is most important for them taking action to get engaged in the Jobs-Plus program? (excl. HUD)

67. Do you think that community support for work has driven those participants engaged in Jobs-Plus to take greater advantage of the other core services that the program offers?

CSW Challenges and Implications

Unless otherwise noted, questions are for all respondents:

Program Directors (PD)

Case Managers (CM)

Community Coaches (CC)

Housing Authority Administrators (HAA)

Property Management Staff (PMS)

Directors at Partnering Agencies (DPA)

Street-level Referral Partners (SPA)

65. What have been the key challenges in implementing community support for work?

66. What are the implications for how community support for work could be implemented (even) more effectively?

67. Are there any final reflections on the community support for work component that you’d like to share?