Semi-structured management, staff, and supervisor interviews

OPRE Evaluation: Evaluation of Employment Coaching for TANF and Other Related Populations [Experimental impact study and an Implementation study]

Attachment D. Semi-structured management staff and supervisor interviews_rev

Semi-structured management, staff, and supervisor interviews

OMB: 0970-0506

Document [docx]
Download: docx | pdf

Attachment D

discussion guides for SEMI-structured interviews

D1. Program DIRECTOR/management Discussion Guide

D2. program frontline staff Discussion Guide

D3. Program supervisor Discussion Guide


This page has been left blank for double-sided copying.




D1. Program DIRECTOR/management Discussion Guide



PROGRAM DIRECTOR/MANAGEMENT DISCUSSION GUIDE


Note to site visitors: This guide is intended for individuals identified as directors and/or other senior/program managers in the coaching interventions selected for site visits. Respondents will be familiar at a high level with the lead agency or program administrative structure, services, mission and goals, as well as the rationale, development, design, implementation history, structure and performance of the coaching intervention. In preparation for the interview, review the notes from the previous interview, ongoing technical assistance calls, any materials collected from the program about its response to COVID-19, and public information about the state’s and county’s infection rates, responses, and guidance.

Date of Interview:

Interviewer(s):

Site:

Respondent Name(s):

Title(s):

Respondent(s) Affiliation:

E-Mail(s):

Phone(s):

Address(es):

Fax(es):

Introductory Statement and Consent

Thank you for participating in the Evaluation of Employment Coaching for TANF and Related Populations, a national study being conducted by Mathematica and its partner Abt Associates for the U.S. Department of Health and Human Services.

We very much appreciate all your work on the study. We learned a lot about your program when we talked with you last year. Now that the world has changed so much because of COVID-19, we want to know how [PROGRAM] and the services it provides have changed. This will be helpful to understand how the program has changed for our study participants. In addition, other coaching programs will want to learn from your experiences and successes.

The length of the interview is about 90 minutes. Your participation in this study is important and will help us understand how coaching programs have adapted during a disaster situation, how participants have been affected, and lessons learned.

Your participation in this study is voluntary. During our conversation, anything you say will be kept private and we won’t use your name or any other identifying information when we report the results of our study. No identified information from the interviews will be shared by the researchers with anybody from [PROGRAM NAME or AGENCY]. Your responses will be combined with the responses of other staff and no individual names will be reported. You may also choose not to answer any question you do not want to answer. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0506 and the expiration date is 03/21/2021.

I would like to record our conversation so I don’t miss anything. No one will hear the recording except for researchers. Is it okay with you if I record this conversation? If you want me to stop recording for any reason or at any time, just say so.

Shape1

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN

Through this information collection, ACF is gathering information to understand how coaching programs are serving individuals. Public reporting burden for this collection of information is estimated to average 90 minutes per respondent, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0506 and the expiration date is 3/21/2021. If you have any comments on this collection of information, please contact Sheena McConnell, Mathematica, 1100 First St. NE, Washington, DC 20002-4221.

[INTERVIEWER: IF RESPONDENT CONSENTS, PRESS THE RECORD BUTTON]








  1. Current Position and Responsibilities

I would like to begin by asking a little about your position at your [agency/organization]. [Note to interviewer: Probe the following responsibilities, as appropriate to the respondent(s)]

  1. Have your roles and responsibilities at [agency/organization] changed as a result of the COVID-19 pandemic.

  • Agency administration/planning/policy and procedure development

  • Responsible for administrative oversight and/or policies and procedures for:

Eligibility and intake

Case management

Social work

Counseling

Coaching

Employment services

Referrals to other programs and services

Performance management

Other oversight responsibilities

  • Other organizational responsibilities, such as:

Community involvement and outreach

Hiring

Budget planning

Fundraising

Other organizational responsibilities

  1. Local Economic, Policy, and Program Context

I would like to focus now on the local economic and policy environment in which [agency/organization] operates.

Socio-Economic Context

              1. How has the local economy been impacted by COVID-19? For example, have local businesses laid off or furloughed employees? Have businesses closed? Have closures been temporary or permanent?



Organizational background and administrative structure

  1. Has the administrative structure of your agency changed at all during COVID-19? What is the overall staffing pattern? Have the number of coaches changed? Has the average caseload for coaches changed? [Note: ask if relevant] Is the caseload for coaching different than it is/was for case management?

  1. Coaching Program Design and Structure

We are especially interested in how the structure and implementation of your organization’s coaching services in the [name of program] program may have changed due to COVID-19.

Coaching Program Development and Design

  1. Have any of the core elements/components of your coaching model and practice changed due to COVID-19? For example:

Participant eligibility requirements?

What coaching strategies are used by coaches? [Probe] For example do they use motivational interviewing, “mindfulness” training, cognitive therapy, other techniques? Is coaching one-on-one or in groups?

How often coaches meet with clients and for how long of a period they work with a client?

Requirements on program participants?

What tools coaches use?

Policies on using incentives

How are coaches trained? How often and by whom? Are new trainings being offered or planned?

How are coaches supervised, monitored, and rated for performance?

What associated training is used to support client change? For example, are clients trained in life skills, workplace skills, financial planning, in other skill areas?

  1. Did when and how coaching is first presented to clients change as a result of COVID-19? How are clients assigned to coaches?

For each change made ask:

  • Why was it made? Probe for federal or state regulations, program’s own decision.

  • When was it made?

  • Was the change made program-wide or in only some program locations? Describe.

  • What problem was it meant to solve?

  • Was it successful? If not, why?

  • [If change made to respond to COVID-19] Is this a temporary change? If yes, when would you change it back?

3. Do you anticipate any future program changes as a result of COVID-19? Describe.

Monitoring Performance

  1. Did the goals you have for [name of program] change as a result of COVID-19? For example, do you have goals for:



Client program participation and attendance at coaching sessions?



[If relevant]: Work participation rates?



Specific client outcomes? If so, which outcomes? How and when do you measure those outcomes?



  1. Have changes been made to the consequences the program faces if performance goals are unmet? Describe.



  1. [For Jefferson County and FaDSS:] Have program performance measures changed?



  1. Have changes been made to how you use data and data systems to monitor overall goals during COVID-19?

Have changes been made to what program data you collect as a result of COVID-19? Describe.

Are you planning to make any changes to how you collect or use data as a result of the pandemic? Describe.

Other COVID-19 Related Questions

  1. Have the costs of the program changed as a result of COVID-19? Describe.

Do you anticipate any future changes to program costs as a result of the pandemic? Describe.

  1. Has the availability of program funding changed? Has the amount of program funding changed?

If so, what caused these changes?

Were funds diverted to other program needs? Describe.

Were additional funds obtained? Describe.

Do you anticipate any future program funding changes as result of the pandemic? Describe.

  1. [For FaDSS and LIFT leadership and MDRC for MyGoals:] Has your relationship with the local providers changed? Describe.

Have the local providers faced any capacity issues that could impact their ability to serve participants? Describe.

How were these capacity issues addressed? What role did your organization play?

  1. Have existing relationships with employers changed, or new relationships formed? Describe.

  1. What staff changes has the program experienced, if any, as a result of the pandemic?

Have there been furloughs, layoffs, hours reduced or increased, or reassignments?

Have any staff quit, become unavailable, or reduced their hours?

Has the program hired new staff?

Have roles and responsibilities been reallocated among staff? How?

Are you expecting any changes in staffing related to COVID-19 in the future? Describe.

  1. How has the program implemented social distancing?

Are staff working from home?

If so, describe the timing and process for shifting to work-from-home.

What were the organizational successes?

What were the organizational challenges and how did you work to overcome them?

What worked well and what was challenging from the staff perspective?

Have staff been given flexibility in when they work or their number of work hours? Describe.

  1. What challenges have staff faced implementing coaching during the pandemic? Probe for challenges related to stress, wellness, scheduling, child care, health, technology.

  2. What additional resources or support has the program provided staff?

Are additional resources or supports planned?

Are any other resources or supports needed?

  1. Describe your response to the pandemic as a program leader.

What challenges did you face responding to the pandemic? What were some of the successes of your response?

What lessons did you learn from your response?

  1. Describe the planning process you have undertaken to respond to the pandemic. Probe for who was involved, content and frequency of meetings, timing, outcomes.

Have you engaged consultants or technical assistance providers to help manage the crisis? Describe.

  1. Has your capacity or availability changed as a result of the pandemic? Describe.

How has this impacted the program, if at all?

  1. How have you communicated with staff during the pandemic? Probe for mode and frequency of communications, content, tone.

  2. Describe how the organization as a whole has functioned during the pandemic. For example, have program operations been able to continue normally or have there been backlogs in work? Has the organization been able to respond to the pandemic smoothly or have there been major disruptions?

What about the organization has facilitated any successes? Caused challenges?


Challenges and Solutions

  1. In your view, what have been the strengths of the coaching program during COVID-19? What, if any, have been weak points during the pandemic?

  2. During COVID-19, how well do you think the support services you offer, other than coaching, met families’ needs?

    1. Are the services helpful?

    2. What do you think families need more or less of?

    3. What are the most common services that families took up?

    4. Have you heard any feedback from families regarding the services offered? If so, what was it? Do you regularly elicit feedback from individuals or families you serve?

  1. Thinking about [organization/program’s] response to the pandemic overall, what worked well?

  2. What about [organization/program’s] response to the pandemic could be improved?

  3. What are the opportunities for how [organization/program] can better respond moving forward?

  4. What were the most urgent challenges [organization/program] faced so far during the pandemic?

    1. What challenges will persist?

    2. What new challenges do you foresee?

  5. What needs does [organization/program] continue to face in responding to the pandemic?

    1. What needs do staff continue to face?

    2. What needs do participants continue to face?

  6. How prepared was [organization/program] to respond to the pandemic?

    1. Which aspects of the crisis was [organization/program] most prepared for? Least?

  7. What, if any, changes made during the pandemic do you anticipate continuing after the pandemic?

  8. What, if any, additional changes is [organization/program] planning to make in response to the pandemic?

  9. What lessons have you learned from this experience that can help inform future crisis response? Probe for lessons related to policy, process, service delivery, staff, participants.

  1. Wrap-Up – Take-Aways from this Study

Thank you so much for sharing this information with us. As we mentioned at the beginning, we expect this study to result in a report and there may be other opportunities to share selected results. We sincerely hope that this study can be useful to you. So, before we finish, we would like to hear what you would most like others to know about your experiences running a coaching program, and what you might like to learn from our study when it is completed.

  1. In your view, what would you most want stakeholders – for example, the public, federal policymakers, other [state/non-profit] administrators, or others – to know about your [agency’s/organization’s] experiences with serving low-income individuals and, in particular, coaching them for employment success? What would you most want stakeholders to know about your experiences serving individuals during the pandemic?

  2. Is there anything you can think of that you would most like to learn from our study when it is finished?

  3. Is there anything else you would like to share with us?

  4. If we have any follow-up questions as we write our report, may we contact you again?

THANK YOU FOR YOUR TIME



D2. Program frontline staff Discussion Guide



PROGRAM FRONTLINE STAFF

DISCUSSION GUIDE

[Note to site visitors: This guide is intended for individuals identified as coaches, case managers, or other frontline program staff in the offices in the programs selected for site visits. Respondents will be familiar with how coaching or case management is delivered to clients, as well as with the content and delivery of employment, family, and other services, as applicable. In preparation for the interview, review the notes from the previous interview, ongoing technical assistance calls, any materials collected from the program about its response to COVID-19, and public information about the state’s and county’s infection rates, responses, and guidance.]

Date of Interview:

Interviewer(s):

Site:

Respondent Name(s):

Title(s):

Respondent Affiliation:

E-Mail:

Phone:

Address:

Fax:

Introductory Statement and Consent

Thank you for participating in the Evaluation of Employment Coaching for TANF and Related Populations, a national study being conducted by Mathematica and its partner Abt Associates for the U.S. Department of Health and Human Services.

We very much appreciate all your work on the study. We learned a lot about your program when we talked with you last year. Now that the world has changed so much because of COVID-19, we want to know how [PROGRAM] and the services it provides have changed. This will be helpful to understand how the program has changed for our study participants. In addition, other coaching programs will want to learn from your experiences and successes.

The length of the interview is about 90 minutes. Your participation in this study is important and will help us understand how coaching programs have adapted during a disaster situation, how participants have been affected, and lessons learned. Your participation in this study is voluntary. During our conversation, anything you say will be kept private and we won’t use your name or any other identifying information when we report the results of our study. No identified information from the interviews will be shared by the researchers with anybody from [PROGRAM NAME or AGENCY]. Your responses will be combined with the responses of other staff and no individual names will be reported. You may also choose not to answer any question you do not want to answer. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0506 and the expiration date is 03/21/2021.

I would like to record our conversation so I don’t miss anything. No one will hear the recording except for researchers. Is it okay with you if I record this conversation? If you want me to stop recording for any reason or at any time, just say so.

Shape3

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN

Through this information collection, ACF is gathering information to understand how coaching programs are serving individuals. Public reporting burden for this collection of information is estimated to average 90 minutes per respondent, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0506 and the expiration date is 3/21/2021. If you have any comments on this collection of information, please contact Sheena McConnell, Mathematica, 1100 First St. NE, Washington, DC 20002-4221.

[INTERVIEWER: IF RESPONDENT CONSENTS, PRESS THE RECORD BUTTON]










[NOTE TO INTERVIEWER: If this site is a TANF agency that uses more traditional case managers to serve control group sample members, or uses both case managers and coaches to serve treatment group members, and you are conducting small group discussions, please ensure that members of each discussion group are either all coaches or all case managers. Also be aware of guidance in the discussion guide about specific questions or groups of questions to ask of all workers, just case managers, or just coaches.]

  1. Current Position and Responsibilities

[Note to interviewer: use for all frontline staff]

Respondent Background

I would like to begin by asking a little about your position and your [agency/organization]. [Note to interviewers: Probe for the following responsibilities, as appropriate to the respondents]

  1. Have your roles and responsibilities at [agency/organization] changed as a result of the COVID-19 pandemic? Probe for roles related to:

      • Program enrollment/intake

      • Client assessment

      • Coaching

      • Case management

      • Social work

      • Counseling

      • Delivering employment services

      • Arranging enrollment in other programs [Ask respondent to name other programs]

      • Making referrals/navigation of other services

      • Other responsibilities

Organizational Background and Work Structure

  1. How is your work organized on a daily/weekly basis? Summarize a typical work day for you prior to COVID-19 and a typical day during the pandemic.



  1. Are you working from home? Have you reduced your hours?



  1. Has what other staff members you work with most changed during COVID-19? In what ways?



  1. Do the other organizations you work with during the course of everyday changed during COVID-19?

[Probe: formal vs. informal relationship; referral process; new partnerships or referral processes created]

  1. Local Economic and Policy Context

[Note to interviewer: use for all frontline staff]



              1. What types of jobs are typically available for the clients you work with? Are there enough jobs for those who want to work? Has this changed as a result of COVID-19?

[Ask items 2-3 for TANF agencies or other agency or contractor that administers TANF]:

  1. Can you summarize the current work requirements for the TANF recipients you work with? (Probe: hours requirements, allowable activities, exemptions/good cause for nonparticipation, sanctioning.) Have the work requirements changed as a result of COVID-19?

  2. Do you face challenges in applying current TANF work requirement policies? If so, what are they?


  1. Client Strengths, Challenges, and Engagement


[Note to interviewer: use for all frontline staff]



  1. What personal strengths have participants relied on in dealing with the pandemic? Probe for existing support networks; skills developed while engaging in the program; abilities like resourcefulness, resilience, motivation, work ethic.



    1. During COVID-19, what personal challenges and needs did clients face?



[Probe these areas if not explicitly mentioned]

Unstable housing or homelessness

Lack of transportation or child care

Meeting basic needs (food, rent)

Mental illness

Physical health

Stress, including financial

Personal and family well-being

Substance abuse

Other

  1. How has the program supported participants in resolving these challenges and needs? Probe for additional program supports, referral resources.



    1. Have staff supported participants in applying for government benefits, such as SNAP or Unemployment Insurance? Describe.

    2. What other sources of support have participants relied on? Probe for family, church, community-based organizations, government benefits.

    3. To what extent were participants’ needs met?

  1. What employment challenges have participants faced during the pandemic? Probe for job loss, layoffs, safety concerns, working remotely, lack of job opportunities.

    1. How has the program supported participants in resolving these challenges? Probe for additional program supports, referral resources.

  2. How knowledgeable are participants about COVID-19?

    1. How did they receive information about the virus and its impact on their community?

    2. Did the program provide information to participants? If so, how and when?

    3. What are their main questions?

  3. Do participants have access to technology to support remote program engagement and access to other programs and services during the pandemic?

    1. If yes, what forms of technology and platforms do participants use (cell phones, computers, video conferencing, social media)?

    2. What challenges to accessing technology do participants face? Probe for digital literacy issues, lack of hardware or software, lack of internet, lack of phone minutes.

    3. How did the program support participants in accessing technology?

    4. What other resources did participants access to support their use of technology (for example, free Wi-Fi, computers provided by children’s school)?

  4. Has engagement among existing participants changed as a result of COVID-19?

    1. Did participants who were unengaged reengage? Did engaged participants disengage?

    2. Has the number of participants leaving or disengaging from the program changed?

    3. Were participants more or less likely to engage in other program services (workshops, training and education, peer support)?

    4. What reasons did participants share for why their engagement changed?

  1. Case Management Implementation and Practice

[Note to interviewer: use this section only for case managers.]

  1. During COVID-19, how do you approach working with clients? For example, what do you try to accomplish in the first meeting? How has this changed from how it was before COVID-19?

  2. Have your overall goals in your casework changed during COVID-19? Have how you monitor client progress towards those goals changed?

  3. Do the types of assessments you use to help tailor client services changed during COVID-19?

[Probe for responses if not mentioned]:

    1. Barriers to self-sufficiency

    2. Cognitive skills

    3. Academic skills

    4. Executive functioning skills

    5. Employment history

    6. Employability/basic skills

    7. Psychological (e.g., personality tests)

    8. Job knowledge/career aptitude and preference

    9. Others

  1. Have the specific assessment tools you use changed during COVID-19?

  2. Approximately how often do you meet or communicate with clients and by what means? About how long are typical meetings? Are meetings one-on-one or in groups or both? Has this changed during COVID-19?

  3. Briefly describe what happens in a “typical” case management session during COVID-19. How is this different from what happened before the pandemic?

  4. Has what you try to accomplish in client meetings changed during COVID-19?

  5. Do you monitor client compliance? If so, how? Has this changed during COVID-19?

  6. What tools do you use in your case management practice? Do you have a manual? Has this changed during COVID-19?

  7. What training did you receive for case management? Have you received additional training during COVID-19?

  8. What ongoing support and supervision are available? How is your performance rated? Has this changed during COVID-19?

  9. Do you have an assigned caseload? If so, how large is it? If not assigned, approximately how many clients do you typically work with on a monthly basis? Has this changed during COVID-19?

  10. What are the participation and other behavioral requirements for clients? What are the consequences for noncompliance? Are there any less formal mechanisms than sanctioning through which clients are held accountable? About what percentage of clients are noncompliant? Has this changed during COVID-19?

  11. Has the process for sanctions changed during COVID-19? About how often (percentage of cases) do you begin sanctioning procedures? Has this changed during COVID-19?

  1. Coaching Implementation and Practice

[Note to interviewer: use this section only for coaches. For any changes made related to COVID-19, ask when the changes were made and why they were needed.]

  1. During COVID-19, how do you approach working with clients? For example, what do you try to accomplish in the first meeting? What types and frequency of interactions do you typically have with clients? How has this changed from how it was before COVID-19?

  2. What are your overall goals in coaching? Have these changed during COVID-19? Has how you monitor client progress towards those goals changed?

  3. Have the types of assessments you use to help tailor client services changed during COVID-19?

[Probe for responses if not mentioned:]

    1. Barriers to self-sufficiency

    2. Cognitive skills

    3. Academic skills

    4. Executive functioning skills

    5. Employment history

    6. Employability/basic skills

    7. Psychological (e.g., personality tests)

    8. Job knowledge/career aptitude and preference

    9. Others

  1. Which specific assessment tools do you use? Have these changed during COVID-19?

  2. Has what you try to accomplish in each client meeting changed during COVID-19? Is there an agenda for each meeting? How do you organize the meetings? Has this changed during COVID-19?

  3. Briefly describe what happens in a “typical” coaching session during COVID-19. How is this different from what happened before the pandemic?

    1. Has the content of the discussions changed? If so, how? What are topics that are more likely to be discussed now? What are topics that are less likely to be discussed?

    2. Are participants still setting goals?

    3. Have the types of goals participants set changed? If so, how?

    4. Do you interact with participants differently because of COVID-19? For example, being more focused on personal well-being, less focused on compliance, being more directive.

  4. During COVID-19, approximately how often do you meet in person with clients? If not meeting in person, by what mode are coaching sessions taking place? Are they taking place by telephone or by video conference? How often do you meet using these other modes? [Ask for a rough breakdown of what percentage of sessions are taking place by each mode.]

      • How has the mode by which coaching sessions take place changed as a result of the pandemic?

      • If taking place by video conference, which platforms are they using?

      • What have been the successes of using technology for communicating with participants?

      • What have been the challenges? How have you worked to overcome these challenges?

  5. During COVID-19, about how long are typical coaching sessions? Is different from the length of sessions before the pandemic? Over what time period is coaching delivered? Is it for a prescribed period? Until specific milestones are reached, or for some other period?

  6. During COVID-19, how frequently are you in touch with clients between formally scheduled meetings? Has this changed with the pandemic?

Probe: what are some common reasons why coaches would be in touch with clients between meetings? Has this changed with the pandemic? What method of contact do they use – phone, email, text message, video? Has this changed with the pandemic? Has their approach to these interactions changed? If so, when?

[For FaDSS]: Are they visiting participants home and if so, for what reasons? Has the content of the communications changed?

  1. What are the participation and other behavioral requirements for clients in the coaching program? What are the consequences for missed coaching sessions? Has this changed during COVID-19?

About how often does someone with a coach leave the program before completing the entire coaching process or achieving a desired outcome? Has this changed during COVID-19?

[For workers in TANF agencies only] Are there any less formal mechanisms than sanctioning through which clients are held accountable? About what percentage of clients do not comply with participation requirements? Has this changed during COVID-19?

  1. Has the process for sanctions changed during COVID-19? About how often (percentage of cases) do you begin sanctioning procedures? Has this changed during the pandemic?

  2. If a particular coach-client match is not working, are clients reassigned to new coaches? Who initiates this? Has this change during COVID-19?

  3. What are common challenges or barriers to client progress? How are they addressed? Has this changed during COVID-19?

  4. What tools do you use in your coaching practice? What activities are clients asked to do between sessions? Has this changed during COVID-19?

  5. What training did you receive for coaching? Did training change during COVID-19?

  6. What ongoing support and supervision are available to you? Did supervision change during COVID-19? How is your performance rated?

    1. Did you receive additional resources or support during COVID-19? Are any other resources or supports needed?

  7. Do you have an assigned caseload for coaching? If so, how large is it? If not assigned, approximately how many clients do you typically work with on a monthly basis? Has this changed during COVID-19?

  8. During COVID-19, have there been changes to what other services are offered to participants (either by the program itself or its partners)? Note to interviewer: tailor this question based on the services we know the program provides, such as referrals to training programs or lending circles.

    1. Have any new services become available? Describe.

    2. Are any previously available services no longer offered? If so, what caused this change?

    3. Has the amount of services available changed? If so, what caused this change?

  9. What have been the successes of offering coaching and other services to participants during the pandemic? What have been the challenges? What have you done to overcome those challenges?

    1. What have been the successes and challenges of any changes made to the coaching process and procedures?

  10. What challenges have you personally faced implementing coaching during the pandemic? Probe for challenges related to stress, wellness, scheduling, child care, health, technology.

  1. Response to COVID-19 and lessons learned

  1. Describe how program leadership has responded to the pandemic. Probe for whether the response of their program leadership was proactive or reactive, inclusive, understanding.

  2. Has the capacity or availability of program leadership changed as a result of the pandemic? Describe.

    1. How has this impacted the program, if at all?

  3. How has leadership communicated with staff during the pandemic? Probe for mode and frequency of communications, content, tone.

    1. Has leadership’s communication been effective? Why or why not? How could it be improved?

  4. Describe how the organization as a whole has functioned during the pandemic. For example, have program operations been able to continue normally or have there been backlogs in work? Has the organization been able to respond to the pandemic smoothly or have there been major disruptions?

    1. What about the organization has facilitated any successes? Caused challenges?

  5. Thinking about [organization/program’s] response to the pandemic overall, what worked well?

  6. What about [organization/program’s] response to the pandemic could be improved?

  7. What are the opportunities for how [organization/program] can better respond moving forward?

  8. What were the most urgent challenges [organization/program] faced so far during the pandemic?

    1. What challenges will persist?

    2. What new challenges do you foresee?

  1. What needs does [organization/program] continue to face in responding to the pandemic?

  1. What needs do staff continue to face?

  2. What needs do participants continue to face?

  1. How prepared was [organization/program] to respond to the pandemic?

  1. Which aspects of the crisis was [organization/program] most prepared for? Least?

  1. What lessons have you learned from this experience that can help inform future crisis response? Probe for lessons related to policy, process, service delivery, staff, participants.

  1. Wrap-Up: Desired Take-Aways from this Study

[Note to interviewer: use this section for coaches, only.]

Thank you so much for sharing this information with us. As we mentioned at the beginning, we expect this study to result in a report and there may be other opportunities to share selected results. We sincerely hope that this study can be useful to you. So, before we finish, we would like to hear what you would most like others to know about your experiences running a coaching program, and what you might like to learn from our study when it is completed.

              1. In your view, what would you most want stakeholders – for example, the public, federal policymakers, [state/non-profit] administrators, front-line social service workers, or others – to know about your [agency’s/organization’s] experiences with serving low-income individuals, and, in particular, coaching them for success? What would you most want stakeholders to know about your experiences serving individuals during the pandemic?

              2. Is there anything you can think of that you would most like to learn from our study when it is finished?

              3. Is there anything else you would like to share with us?

              4. If we have any follow-up questions as we write our report, may we contact you again?

THANK YOU

[Note to interviewer: use this section for case managers, only]

Thank you so much for sharing this information with us. As we mentioned at the beginning, we expect this study to result in a report and there may be other opportunities to share selected results. We sincerely hope that this study can be useful to you. So, before we finish, we would like to hear what you would most like others to know about your experiences conducting case management, and what you might like to learn from our study when it is completed.

  1. In your view, what would you most want stakeholders – for example, the public, federal policymakers, [state/non-profit] administrators, front-line social service workers, or others – to know about your [agency’s/organization’s] experiences with serving low-income individuals and, in particular, preparing them for employment? What would you most want stakeholders to know about your experiences serving individuals during the pandemic?

  2. Is there anything you can think of that you would most like to learn from our study when it is finished?

  3. Is there anything else you would like to share with us?

  4. If we have any follow-up questions as we write our report, may we contact you again?

THANK YOU



D3. Program supervisor Discussion Guide


PROGRAM SUPERVISOR DISCUSSION GUIDE

[Note to site visitors: This guide is intended for individuals identified as supervisors of coaches, case managers, or other frontline program staff in the offices/programs selected for site visits. Respondents will be familiar with how coaching or case management is delivered to clients, as well as with the content and delivery of employment, family, and other services, as applicable. In preparation for the interview, review the notes from the previous interview, ongoing technical assistance calls, any materials collected from the program about its response to COVID-19, and public information about the state’s and county’s infection rates, responses, and guidance.]

Date of Interview:

Interviewer (s):

Site:

Respondent Name(s):

Title(s):

Respondent Affiliation:

E-Mail:

Phone:

Address:

Fax:

Introductory Statement and Consent

Thank you for is participating in the Evaluation of Employment Coaching for TANF and Related Populations, a national study being conducted by Mathematica and its partner Abt Associates for the U.S. Department of Health and Human Services.

We very much appreciate all your work on the study. We learned a lot about your program when we talked with you last year. Now that the world has changed so much because of COVID-19, we want to know how [PROGRAM] and the services it provides have changed. This will be helpful to understand how the program has changed for our study participants. In addition, other coaching programs will want to learn from your experiences and successes.

The length of the interview is about 90 minutes. Your participation in this study is important and will help us understand how coaching programs have adapted during a disaster situation, how participants have been affected, and lessons learned. Your participation in this study is voluntary. During our conversation, anything you say will be kept private and we won’t use your name or any other identifying information when we report the results of our study. No identified information from the interviews will be shared by the researchers with anybody from [PROGRAM NAME or AGENCY]. Your responses will be combined with the responses of other staff and no individual names will be reported. You may also choose not to answer any question you do not want to answer. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0506 and the expiration date is 03/21/2021.

I would like to record our conversation so I don’t miss anything. No one will hear the recording except for researchers. Is it okay with you if I record this conversation? If you want me to stop recording for any reason or at any time, just say so.

Shape6

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN

Through this information collection, ACF is gathering information to understand how coaching programs are serving individuals. Public reporting burden for this collection of information is estimated to average 90 minutes per respondent, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0506 and the expiration date is 3/21/2021. If you have any comments on this collection of information, please contact Sheena McConnell, Mathematica, 1100 First St. NE, Washington, DC 20002-4221.



[INTERVIEWER: IF RESPONDENT CONSENTS, PRESS THE RECORD BUTTON]








[NOTE TO INTERVIEWER: If this site is a TANF agency that uses more traditional case managers to serve control group sample members, or uses both case managers and coaches to serve treatment group members, please ensure that all members of each discussion group are supervisors of either only coaches or only case managers. If supervisors are normally responsible for both coaches and case managers, any group of supervisors may be interviewed as a group. Also be aware of guidance in the discussion guide about specific questions or groups of questions to ask of all supervisors, just supervisors of case managers, or just supervisors of coaches.]

  1. Current Position and Responsibilities

[Note to interviewer: use for all supervisors]

Respondent Background

I would like to begin by asking a little about your position and your [agency/organization]. [Note to interviewers: Probe for the following responsibilities, as appropriate to the respondents]

              1. Have your roles and responsibilities at [agency/organization] changed as a result of the COVID-19 pandemic.

Probe for whether the respondent supervises and/or conducts directly:

      • Recruitment/public information sessions

      • Program enrollment/intake

      • Client assessment

      • Coaching

      • Case management

      • Social work

      • Counseling

      • Delivering employment services

      • Arranging enrollment in other programs [Ask respondent to name other programs]

      • Making referrals/navigation of other services

      • Other supervisory or direct staffing responsibilities, including

Coach or case manager performance review and assessment

Other oversight responsibilities

      • Other organizational responsibilities, such as:

Community involvement and outreach

Hiring

Other organizational responsibilities

Organizational Background and Work Structure

  1. How is your work organized on a daily/weekly basis? Summarize a typical work day for you prior to COVID-19 and a typical day during the pandemic.

  2. Are you working from home? Have you reduced your hours?

  3. Did the other staff members you work with most changed during COVID-19? In what ways?

  4. How are you supervised? To whom do you turn most often with work issues or problems? Has this changed during COVID-19?

  5. Did the other organizations you work with during the course of your everyday job change during COVID-19?

[Probe: formal vs. informal relationship; referral process; new referral processes created]

  1. Have relationships with existing referral sources or other community partners changed during COVID-19? Describe.

    1. Have new relationships formed? If so, how and why?

  1. Have existing relationships with employers changed, or new relationships formed during COVID-19? Describe.

  1. Local Economic and Policy Context

[Note to interviewer: use for all supervisors]

  1. What types of jobs are typically available for the clients you and your coaches/case managers work with? Are there enough jobs for those who are looking for work? Has this changed during COVID-19?

[Note: ask next two questions only for a TANF agency or contractor that administers TANF]

  1. Can you summarize the current work requirements for the TANF recipients your agency serves? Have the work requirements changed as a result of COVID-19? (Probe: hours requirements, allowable activities, exemptions/good cause for nonparticipation, sanctioning)

  2. Do the workers you supervise face challenges in applying current TANF work requirement policies during COVID-19? If so, what are they?

  1. Client Strengths and Challenges



[Note to interviewer: use for all supervisors]

  1. What personal strengths have participants relied on in dealing with the pandemic? Probe for existing support networks; skills developed while engaging in the program; abilities like resourcefulness, resilience, motivation, work ethic

  2. During COVID-19, what challenges and needs did clients face?

[Probe these areas if not explicitly mentioned]

Unstable housing or homelessness

Lack of transportation or child care

Meeting basic needs (food, rent)

Mental illness

Physical health

Stress, including financial

Personal and family well-being

Substance abuse

Other

  1. Do you have any particularly effective strategies that you share with your coaches/case managers in dealing with the clients facing the most challenges during this time? What are they?

  1. Supervising Case Management Implementation and Practice

[Note to interviewer: use this section only for supervisors of case managers, or supervisors of both case managers and coaches. If interviewees supervise both coaches and case managers, ask them to respond to the following questions for supervising case managers only.]

  1. How many case managers do you supervise? Has this changed during COVID-19?

  1. How are clients assigned to case managers? Has this changed during COVID-19?

  2. How do case managers approach working with clients? For example, what do they try to accomplish in the first meeting? Has this changed during COVID-19? What types and frequency of interactions do they typically have with clients? What guidance do you give them? Has this changed during COVID-19?

  3. What are the overall goals for casework with your clients? How do you monitor your case managers’ clients’ progress towards those goals? Has this changed during COVID-19?

  4. During COVID-19, how often do you meet with the workers you supervise and for what purposes? What do you do in these meetings? Has this changed from before COVID-19?

  5. How do you assess the performance of case managers? How do you help case managers improve in their practice? Has this changed during COVID-19?

  6. What types of assessments do case managers use to help tailor client services? Has this changed during COVID-19?

[Probe for responses if not mentioned]:

                1. Barriers to self-sufficiency

                2. Cognitive skills

                3. Academic skills

                4. Executive functioning skills

                5. Employment history

                6. Employability/basic skills

                7. Psychological (e.g., personality tests)

                8. Job knowledge/career aptitude and preference

                9. Others

  1. Which specific assessment tools do they use? Has this changed during COVID-19?

  2. During COVID-19, approximately how often do case managers meet or communicate with clients and by what means? Did this change from before the pandemic? About how long are typical meetings? Did this change? Are meetings one-on-one or in groups or both? Do you ever monitor or sit in on client/case manager meetings?

  3. Briefly describe what happens in a “typical” case management session during COVID-19. How is this different from before the pandemic?

  4. During COVID-19, what do case managers try to accomplish through client meetings?

  5. Do case managers monitor client compliance? If so, how? How do you interact with case managers about client compliance? Do you also monitor client compliance? If so, how? Has this changed during COVID-19?

  6. In general, what tools do case managers use? Do they have a manual? Is it adequate? Has this changed during COVID-19?

  7. How are clients assigned to case managers? Has this changed during COVID-19? If a particular case manager-client match is not working, are clients reassigned to new case managers? Who initiates this? How are you involved? Does this happen very often? Has this changed during COVID-19?

  8. What training do case managers receive? What training do you receive? Has this changed during COVID-19?

  9. What ongoing support and supervision are available to you? Has this changed during COVID-19? How is your performance rated?

  10. Do case managers have an assigned caseload? If so, how large is it? If not assigned, approximately how many clients do case managers typically work with on a monthly basis? Has this changed during COVID-19?

  11. [Ask supervisors in TANF agencies] What are the participation and other behavioral requirements for clients? What are the consequences for noncompliance? Has this changed during COVID-19? Are there any less formal mechanisms than sanctioning through which clients are held accountable? About what percentage of clients are noncompliant?

  12. [Ask supervisors in TANF agencies] Has the process for sanctions changed during COVID-19? About how often (percentage of cases) do case managers begin sanctioning procedures? Are you involved in sanctioning? If so, in what ways?

  13. How would you describe “successful” case management? How do you work with case managers to help them be successful? Has this changed during COVID-19?

  1. Supervising Coaching Implementation and Practice

[Note to interviewer: use this section only for supervisors of TANF coaches, or supervisors of both case managers and coaches. If interviewees supervise both coaches and case managers, ask them to respond to the following questions for supervising coaches only. For any changes made related to COVID-19, ask when the changes were made and why they were needed.]

A. What coaches do

  1. How has outreach and recruitment changed as a result of COVID-19? For any changes made, ask when they were first implemented.

    1. How are you communicating about your services to potential participants during this time?

    2. Has the number of applicants for the program changed?

  2. How has the enrollment process for new participants changed as result of COVID-19? For any changes made, ask when they were first implemented.

    1. Has the number of new participants changed?

  3. How are clients assigned to coaches? Has this changed during COVID-19?

  1. During COVID-19, how do coaches approach working with clients? For example, what do they try to accomplish in the first meeting? What types and frequency of interactions do they typically have with clients? How has this changed from how it was before COVID-19?

  2. What are the overall goals in coaching TANF clients? Other low-income individuals? Have these changed during COVID-19? How do you monitor your coaches’ clients’ progress towards those goals?

  3. What types of assessments do coaches use? Have these changed during COVID-19?

[Probe for responses if not mentioned:]

                1. Barriers to self-sufficiency

                2. Cognitive skills

                3. Academic skills

                4. Executive functioning skills

                5. Employment history

                6. Employability/basic skills

                7. Psychological (e.g., personality tests)

                8. Job knowledge/career aptitude and preference

                9. Others

  1. Which specific assessment tools do coaches use? Have these changed during the pandemic?

  2. What do coaches generally try to accomplish in each client meeting during the pandemic? How individualized are the meeting goals? Has this changed from before the pandemic?

  3. How are the meetings organized during COVID-19? Are they usually one-on-one or group formats? Has this changed from before the pandemic?

  4. Briefly describe what happens in a “typical” coaching session during the pandemic. How is this different from what happened before the pandemic?

    1. Has the content of the discussions changed? If so, how? What are topics that are more likely to be discussed now? What are topics that are less likely to be discussed?

    2. Are participants still setting goals?

    3. Have the types of goals participants set changed? If so, how?

    4. Do coaches interact with participants differently because of COVID-19? For example, being more focused on personal well-being, less focused on compliance, being more directive.

  1. During COVID-19, approximately how often do coaches meet in person with clients? If not meeting in person, by what mode are coaching sessions taking place? Are they taking place by telephone or by video conference? How often do you meet using these other modes? Ask for a rough breakdown of what percentage of sessions are taking place by each mode.

      • How has the mode by which coaching sessions take place changed as a result of the pandemic?

      • If taking place by video conference, which platforms are they using?

      • What have been the successes of using technology for communicating with participants?

      • What have been the challenges? How have you worked to overcome these challenges?

  1. During COVID-19, about how long are typical coaching sessions? Is this different from the length of sessions before the pandemic?

  1. During COVID-19, how frequently are coaches in touch with clients between formally scheduled meetings? What are some reasons why coaches would be in touch with clients between meetings?

What method(s) of contact do they use – e.g., phone, email, text message, video? Has this changed with the pandemic? Has coaches’ approach to these interactions changed? If so, when?

[For FaDSS]: Are they visiting participants home and if so, for what reasons? Has the content of the communications changed?

  1. What are the participation and other behavioral requirements for clients who have a coach? What are the consequences for missed coaching sessions? Has this changed during COVID-19?

About how often does someone with a coach leave the program before completing the entire coaching process or achieving a desired outcome? Has this changed during COVID-19?

[For supervisors in TANF agencies only] Are there any less formal mechanisms than sanctioning through which clients are held accountable? About what percentage of clients do not comply with participation requirements? Has this changed during COVID-19?

  1. Has engagement among existing participants changed as a result of COVID-19?

    1. Did participants who were unengaged reengage? Did engaged participants disengage?

    2. Has the number of participants leaving or disengaging from the program changed?

    3. Were participants more or less likely to engage in other program services (workshops, training and education, peer support)?

    4. What reasons did participants share for why their engagement changed?

  2. What is the process for sanctions during COVID-19? About how often (percentage of cases) do coaches begin sanctioning procedures? Are you involved in sanctioning? If so, in what ways? Has this process changed from before the pandemic?

  3. If a particular coach-client match is not working, are clients reassigned to new coaches? Has this changed during COVID-19? Who initiates this? How are you involved? Does this happen very often? Has the frequency changed during COVID-19?

  4. During COVID-19, have there been changes to what other services are offered to participants (either by the program itself or its partners)? Note to interviewer: tailor this question based on the services we know the program provides, such as referrals to training programs or lending circles.

    1. Have any new services become available? Describe.

    2. Are any previously available services no longer offered? If so, what caused this change?

    3. Has the amount of services available changed? If so, what caused this change?

  5. What are common challenges or barriers to client progress? Have they changed during COVID-19? How do coaches address them?

  6. What tools do coaches use in their coaching practice? What activities are clients asked to do between sessions? Has this changed during COVID-19?

  7. What training do coaches receive? What training did you receive? Did this change during COVID-19?

    1. Are new trainings being offered or planned?

  8. Do coaches have an assigned caseload for coaching? If so, how large is it? If not assigned, approximately how many clients do coaches typically work with on a monthly basis? Has this changed during COVID-19?

B. Supervising coaches

  1. What staff changes has the program experienced, if any, as a result of the pandemic?

    1. Have there been furloughs, layoffs, hours reduced or increased, or reassignments?

    2. Have any staff quit, become unavailable, or reduced their hours?

    3. Has the program hired new staff?

    4. Have roles and responsibilities been reallocated among staff? How?

    5. Are you expecting any changes in staffing related to COVID-19 in the future? Describe.

  2. How many coaches do you supervise? Did this change from before COVID-19?

  3. During COVID-19, do you ever sit in on a coach/client meeting? Do you give coaches feedback on their coaching? Did this change from before COVID-19?

  1. During COVID-19, how often do you meet with the coaches you supervise and for what purposes? Did this change from before COVID-19?

  2. What ongoing support and supervision are available to you? Did you receive additional resources or support during COVID-19 from the program? Did coaches? Are any other resources or supports needed? How is your performance rated?

C. Opinions of coaching

  1. How would you describe “successful” coaching during COVID-19?

  1. What have been the successes of offering coaching and other services to participants during the pandemic? What have been the challenges? What have you done to overcome those challenges?

    1. What have been the successes and challenges of any changes made to the coaching process and procedures?

  2. What challenges have you personally faced implementing coaching during the pandemic? Probe for challenges related to stress, wellness, scheduling, child care, health, technology.

  1. Response to COVID-19 and lessons learned

  1. Describe how program leadership has responded to the pandemic. Probe for whether the response of program leadership was proactive or reactive, inclusive, understanding?

  2. Has the capacity or availability of program leadership changed as a result of the pandemic? Describe.

    1. How has this impacted the program, if at all?

  3. How has leadership communicated with staff during the pandemic? Probe for mode and frequency of communications, content, tone.

    1. Has leadership’s communication been effective? Why or why not? How could it be improved?

  4. Describe how the organization as a whole has functioned during the pandemic. For example, have program operations been able to continue normally or have there been backlogs in work? Has the organization been able to respond to the pandemic smoothly or have there been major disruptions?

    1. What about the organization has facilitated any successes? Caused challenges?

  5. Thinking about [organization/program’s] response to the pandemic overall, what worked well?

  6. What about [organization/program’s] response to the pandemic could be improved?

  7. What are the opportunities for how [organization/program] can better respond moving forward?

  8. What were the most urgent challenges [organization/program] faced so far during the pandemic?

  1. What challenges will persist?

  2. What new challenges do you foresee?

  1. What needs does [organization/program] continue to face in responding to the pandemic?

  1. What needs do staff continue to face?

  2. What needs do participants continue to face?

  1. How prepared was [organization/program] to respond to the pandemic?

  1. Which aspects of the crisis was [organization/program] most prepared for? Least?

  1. What, if any, changes made during the pandemic do you anticipate continuing after the pandemic?

  1. What, if any, additional changes is [organization/program] planning to make in response to the pandemic?

  2. What lessons have you learned from this experience that can help inform future crisis response? Probe for lessons related to policy, process, service delivery, staff, participants.

  1. Wrap-Up: Desired Take-Aways from this Study

Thank you so much for sharing this information with us. As we mentioned at the beginning, we expect this study to result in a report and there may be other opportunities to share selected results. We sincerely hope that this study can be useful to you. So, before we finish, we would like to hear what you would most like others to know about your experiences, and what you might like to learn from our study when it is completed.

              1. In your view, what would you most want stakeholders – for example, the public, federal policymakers, [state/non-profit] administrators, front-line social service workers, or others – to know about your [agency’s/organization’s] experiences with serving low-income individuals, and, in particular, coaching them for employment success? What would you most want stakeholders to know about your experiences serving individuals during the pandemic?

              2. Is there anything you can think of that you would most like to learn from our study when it is finished?

              3. Is there anything else you would like to share with us? If we have any follow-up questions as we write our report, may we contact you again?

THANK YOU





File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorBethany Boland
File Modified0000-00-00
File Created2022-04-22
© 2024 OMB.report | Privacy Policy