ACF BIAS-NG Project: Generic ICR for Qualitative and Descriptive Quantitative Data Collection for a Second TANF Site

OMB # 0970-0502

Expiration Date: 08/31/2020

Appendix B.1 Frontline Staff Interview/Focus Group Protocols

Note to reviewers: This appendix includes protocols for discussions with frontline staff.

The Case Study Reviews will be conducted in individual discussions with both program and control group-serving staff. Research staff will use these guides to prompt staff to recall details of Comprehensive Evaluations they recently conducted with clients in their respective study groups.

In a separate discussion, research staff will conduct semi-structured interviews or focus groups with a subset of both groups of frontline staff, separately. The research team will decide whether to conduct semi-structured interviews with individual staff or focus groups with groups of frontline staff based on the offices’ respective staffing structure and needs as we develop our site visit agendas. However, we will use the same protocols for either approach, and the maximum number of staff with whom we will speak will remain the same regardless of the method selected.

Contents:

Frontline Staff Consent Form (all activities) 2

Appendix B.1.1 Case Study Review Guide 4

Appendix B.1. 2 Program Group-Serving Frontline Staff Interviews/ Focus Groups 6

Appendix B.1.3 Control Group-Serving Frontline Staff Interviews/Focus Groups 10

Frontline Staff Consent Form (all activities)

Informed Consent for Frontline Staff

You are invited to participate in an [interview/focus group] for the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project.

What is the study?

MDRC, a nonprofit social policy research organization, and its partner MEF Associates are conducting this project on behalf of the Office of Planning, Research, and Evaluation in the Administration for Children and Families. Its goal is to use behavioral science, which incorporates ideas from psychology and economics, to improve communication with and participation of individuals who receive services from Washington State Department of Social and Health Services (DSHS).

What will I need to do?

We are talking with clients who have participated in the WorkFirst program as well as staff involved in WorkFirst services at DSHS.

[IF CASE STUDY REVIEW] This discussion will focus on 2-4 cases where you conducted a Comprehensive Evaluation in the past 1-3 weeks. This will take up to 1 hour.

[IF INTERVIEW/ FOCUS GROUP] The [interview/focus group] will focus on your thoughts and experiences related to the WorkFirst Comprehensive Evaluation and the assignment of WorkFirst activities. This [interview/focus group] will take up to 90 minutes. 

What are the risks and benefits of participating in this [case study review/interview/focus group]?

There are no significant risks to your participation. Sometimes people feel uncomfortable answering some questions. If that happens, you do not have to answer them. You may stop participating at any time. We would like to record your responses to help with our notes. However, if you do not want us to record, we will not. If we do record, you can ask to stop the recording at any time.

Participating in the [interview/focus group] will not benefit you directly but sharing your thoughts on and experiences with these interventions may help improve the experiences of individuals involved in DSHS. We have very strong security measures in place and will make every effort to protect your privacy.

[IF FOCUS GROUP] The nature of focus groups means that research staff cannot promise that everyone in the group will keep everything you say private. We will remind everyone to respect the privacy of their fellow participants.

Will you share information from the [case review/interview/focus group]?

Only the study team will see the notes or hear the recordings. Your name will not be listed in any published reports, and comments will not be attributed to you. Your answers will be kept private to the extent permissible by law. We will destroy the notes and recordings at the end of the study.

Do I have to do the [case review/interview/focus group]?

Your participation is voluntary and you do not have to answer any questions you do not want to complete. Whether or not you choose to participate will not be shared with anyone.

Questions:

If you have questions about the study, you can call the MDRC project director, Frieda Molina, at 212-340-8660.

According to the Paperwork Reduction Act of 1995 (Pub. L. 104-13), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this collection is 0970-0502 and the expiration date is 08/31/2020.

[FOR WRITTEN CONSENT] Statement

“I have read this form and agree to be in the study. I know that my participation is voluntary and that MDRC follows strict rules to protect my privacy. I know that I can refuse to answer any questions and that I can stop participating at any point.”

VERBAL CONSENT

Are you willing to participate in this interview?

• If yes, proceed.

• If no: “No problem. Thank you for your time today.”

Record response.

Do you feel comfortable with me recording our discussion?

If yes, turn on the recorder and proceed. If no, continue without recorder.

Appendix B.1.1 Case Study Review Guide

Format: Individual interview (15 minutes per case, 2-4 cases per interviewee; 60 minutes) with program and control group-serving staff.

Introduction

Thank you for agreeing to participate in this interview. Your insights are a really important part of the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The goal of the project is to explore ways that we can apply principles from behavioral science—which uses ideas from psychology and economics—to make your jobs easier and help better serve individuals and families.

We want to hear your thoughts about how Comprehensive Evaluations are completed and how IRPs are developed in your office. We would like to walk through some specific cases to learn about the Comprehensive Evaluation you conducted with those specific clients. This interview is completely voluntary and private. Your supervisor, other DSHS staff, and clients will not see these responses. The research team will never identify an individual nor link any answers back to a specific individual.

I will record the conversation in order to make sure we capture the information correctly, but you can tell us if you would like us to remove anything you have said from the interview transcript at any time. You may also choose not to answer questions, and you may choose to stop the interview at any time. 

As is stated in the consent form that I’ll be handing out, participation is voluntary, and the interview should take about 1 hour of your time. According to the Paperwork Reduction Act of 1995 (Pub. L. 104-13), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this collection is 0970-0502 and the expiration date is 08/31/2020.

Please read through and sign the consent form and we can get started.

[Read and sign consent]

Protocols

I’d like to ask you about a few specific clients you recently served. Let’s start with a new client you conducted a Comprehensive Evaluation with in the past month. I will ask a series of questions about the CE conducted with the client and their participation plan before we turn to a case that has left TANF in the past year and returned to TANF in the past month. If time allows, I will ask about one or two additional cases of new or returning clients.

[Interviewers: Use this form to collect and contrast responses to specific examples of service delivery among program- and control-serving staff. Core contrasts to draw: what goals were recorded and how, how the case manager used those goals to discuss the various WF activities, and what their eventual IRP assignment details were.]

Background/case info:

1. How many children are in this case/household? What are their approximate ages? (infant, toddler, child, early teen, late teen): ____________________________________________

2. How many times has this individual opened a TANF WorkFirst case in Washington State?

3. How many months of TANF has this individual received in total?

4. Is this case a 1- or 2- adult household?

• 1

• 2

Motivation:

5. What did you learn about this client’s goals and personal situation during the Comprehensive Evaluation (CE) conversation?

   1. What goals were recorded for this client?

   2. What interests did the client have?

   3. What challenges did the client have?

6. What activity/ies did you assign to the client? Why?

   1. What, if any, information the client provided during the CE conversation contributed to your recommendation?

7. How did you communicate the reason you recommended these activities to the client?

Planning: Notes to review in Individual Responsibility Plans

8. What were the next steps you discussed with this client?

   1. Outside of WorkFirst activity referrals, did you make any additional referrals? If so, what were they?

   2. Did you learn anything new about the client’s situations, routine, or habits when discussing their next steps? If so, what? Did these affect the way you communicated next steps to the client?

   3. Did you help the client develop a plan? If so, tell us more about that.

9. How did you communicate to the client the importance of completing their next steps?

10. Did the client identify any barriers to completing these next steps? If so, what? If so, how did you address these barriers?

11. How confident were you that the client was going to complete their next steps? Why?

12) Did the client participate in their assigned WorkFirst activity within 30 days? Do you have any more information about why/why not? If so, what?

Appendix B.1.2 Program Group-Serving Frontline Staff Interviews/Focus Groups

Format: Semi-structured interviews OR focus groups, TBD (90 minutes)

Introduction

Thank you for agreeing to participate in this [interview/focus group]. Your insights are a really important part of the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The goal of the project is to explore ways that we can apply behavioral science—which uses psychology and economics—to make your jobs easier and help better serve families.

We are interested in learning how you approach the Comprehensive Evaluation (CE) session and how you create Individual Responsibility Plans (IRPs) for clients during the TANF WorkFirst intake sessions.

I will record the conversation in order to make sure we capture the information correctly, but you can tell us if you would like us to remove anything you have said from the [interview/focus group] transcript at any time.

[If interview] You may also choose not to answer questions, and you may choose to stop the interview at any time. 

[If focus group] If you prefer to answer questions privately, you are also welcome to share any comments with any one of us separately.  

If you prefer to answer questions privately, you are also welcome to share any comments with any one of us separately. As is stated in the consent form that I’ll be handing out, your participation in this [interview/focus group] is voluntary, and the [interview/focus group] should take about 90 minutes (1.5 hours) of your time. According to the Paperwork Reduction Act of 1995 (Pub. L. 104-13), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this collection is 0970-0502 and the expiration date is 08/31/2020.

[For Focus Groups and In-Person Interviews (Written Consent)] Please read through and sign the consent form and we can get started.

[Read and sign consent]

[For Phone Interviews (Verbal Consent)]: I will now read you a form about what it means to participate in this research, describe your rights as a participant, and ask for your consent to continue with the interview.

Protocols

Introduction

1. Please state your name and position at DSHS.

   1. How long have you worked in this position? In other positions at DSHS?

2. Are you on any special teams or committees at DSHS?

3. What is the size of your current caseload? Does it change from month to month or by season, say summer versus winter? How many people do you meet with each week?

4. To what extent do you feel like you have time to connect with each of your clients and track their progress through the TANF program?

   1. Do you take any actions if a client misses an appointment?

Previously we talked through some specific cases with [some/all] of you to understand what occurred during their specific CE conversations with you. Now we are going to be asking you some questions about how you work/worked with clients during CE intake sessions during the BIAS-NG pilot.

Description of workflow and communication with clients during CE

5. Please walk me through a typical Comprehensive Evaluation, starting with when the client sits down at your desk.

   1. Probes: How do you typically describe the purpose of the CE and the questions you’re asking clients? What sections do you ask first? Next?

6. Part 2 of the CE starts by asking clients about their goals. How did you encourage clients to set goals for themselves?

   1. Probe: What sorts of phrases did you use? Do you generally ask why that goal is important to them?

Employment section/shaping the IRP

7. Do you typically explain all of the different WorkFirst activities available to clients? Why/why not?

   1. If so, how?

      1. Probes: What are common phrases you use? Do you use any tools (e.g., visual aids) to describe them? When in the CE session do you do this?

   2. If so, how do clients react to your explanation of the activities? What questions do they have?

8. How do you determine which WorkFirst activity/ies are a good fit for clients?

   1. Probe: How does information you gather when conducting your CE affect your decisions? To what extent do clients participate in determining which WorkFirst activities are a part of their IRP?

9. How do you explain to clients the benefits of participating in their WorkFirst activity?

   1. Probes: Does this explanation differ for different activities? For different clients?

10. What are common next steps for clients once their WorkFirst activity/ies are selected? How do you communicate these to clients?

    1. Probes: Aside from the IRP document, do you give clients any other documents outlining the next steps? If so, please describe. When do you advise clients to complete their next steps? (Probe: do you give them a deadline?)

11. What strategies do you find most effective to engage clients in the CE/IRP session?

12. What strategies do you find most effective to encourage clients to participate in their assigned WorkFirst activities after the CE/IRP session?

13. How long does the full CE session take, on average?

14. How often do you receive incoming calls from clients after you’ve completed their IRP?

    1. What are common questions you receive? How do you address these?

15. Do you interact with clients at any other points besides the intake sessions? If so, when and with what kinds of clients?

Delivering the Intervention Components

Now, we'd like to ask you a little more specifically about the BIAS-NG materials you used during the Comprehensive Evaluation session with clients in the BIAS-NG Program group, beginning with the WorkFirst Activity Flashcards.

Note to interviewers: You may present the intervention materials to clients as a visual reference throughout this section if needed. Some of these questions may have been discussed in previous section about staff members’ overall CE intake session flow.

16. How do you use the flashcards with clients?

    1. Probes: How do you describe the purpose of this tool? When did you typically first use the Activity Flashcards with clients? Who holds the cards, and how quickly do they review them, are the front and back reviewed? About how long did you spend using the flashcards?

17. Did you gather any information from clients as a result of the flashcards? If so, did that influence your decisions when developing their IRP? How?

18. Did you change the way you used the flashcards with clients over time? If so, how?

19. Overall, do you think the flashcards were helpful to clients? Why or why not?

    1. Probe: Did clients discuss their perceptions of the flashcards with you? If so, what types of things did they bring up? Were any specific words or images more salient than others?

20. Overall, were the flashcards helpful for you during the CE/IRP session? Why or why not?

Next, let’s talk about the Blueprint Implementation Prompt.

21. How did you use the Blueprint throughout the CE session?

    1. Probes: When did you typically first present and begin using the Blueprint with clients? How did you describe the purpose of this tool? Did you fill it out as you went through the CE or separately (e.g., at the end of the session)? Who typically recorded the client’s responses on the blueprint itself (client, caseworker, both)? About how long did you spend working through the Blueprint with clients per session (Front side? Back side?)

22. How did you encourage clients to set goals for themselves?

    1. Probes: What sorts of phrases did you use? What was most salient?

23. How did you encourage clients to reflect on the ways their activities fit into their goals?

    1. Probes: What sorts of phrases did you use? What was most salient? What were some of the common reasons clients described the activities fitting into their goals? Who came up with these reasons (staff, client, mix)?

24. Did the Blueprint allow you to obtain additional/different information from clients that informed your decision about which WF Activities to assign them? If so, can you give an example?

    1. Probe: How did you incorporate the client’s situation and stated barriers into the goals they set?

25. What were some of the common next steps discussed with clients after they completed the Blueprint tool?

    1. Probe: Did clients mention questions and concerns about completing their agreed upon next steps? If so, what are some examples, and how did you handle them?

26. Did you change the way you used the Blueprint with clients over time? If so, how?

27. Overall, do you think the Blueprint was helpful to clients? Why or why not?

    1. Probe: Did clients discuss their perceptions of the Blueprint with you? If so, what types of things did they bring up? Were any specific words or images more salient than others?

28. Was the implementation prompt helpful for you during the CE/IRP session? Why or why not?

Dosage

29. On average, out of every 10 clients who completed the CE with you, to how many did you show the flashcards?

    1. Probe: How did you decide whether to show them or not?

30. On average, out of every 10 clients who completed the CE with you, with how many did you complete the blueprint tailoring tool/implementation prompt?

    1. Probe: How did you decide whether to complete this or not? To skip sections or not?

31. Which tool did you tend to spend more time on during the CE?

General Reflections

32. In your own words, what do you see as the purpose of this intervention overall? Did that change over time? If so, how?

33. Do you have a story that you consider a particular success or failure of delivering the intervention to a client to help them participate in WorkFirst activities? What helped them succeed? What got in the way?

34. Did you experience any challenges during the BIAS-NG pilot? If so, what were they?

35. Thinking back to the time before the BIAS-NG pilot, how have your interactions with clients changed since you started using the new tools and procedures?

    1. Probe: Did clients contact you more as a result of the implementation prompt (i.e. because it included their case manager’s contact info)?

36. Is there anything you would change about the materials themselves to make them more useful for staff as you develop a client’s IRP? As you motivate clients to attend their assigned WorkFirst activity/ies?

Conclusion

37. Is there anything else you would like us to know about your experience during the BIAS-NG pilot?

Thank you for your time!

Appendix B.1.3 Control Group-Serving Frontline Staff Interviews/Focus Groups

Format: Semi-structured interviews OR focus groups, TBD (90 minutes)

Introduction

Thank you for agreeing to participate in this [interview/focus group]. Your insights are a really important part of the Behavioral Interventions to Advance Self-Sufficiency Next Generation (BIAS-NG) project. The goal of the project is to explore ways that we can apply behavioral science—which uses psychology and economics—to make your jobs easier and help better serve families.

We are interested in learning how you approach the Comprehensive Evaluation (CE) session and how you create Individual Responsibility Plans (IRPs) for clients during the TANF-WorkFirst intake sessions.

I will record the conversation in order to make sure we capture the information correctly, but you can tell us if you would like us to remove anything you have said from the [interview/focus group] transcript at any time.

[IF INTERVIEW] You may also choose not to answer questions, and you may choose to stop the interview at any time. 

[IF FOCUS GROUP] If you prefer to answer questions privately, you are also welcome to share any comments with any one of us separately.

As is stated in the consent form that I’ll be handing out, your participation in this [interview/focus group] is voluntary, and the [interview/focus group] should take about 90 minutes (1 hour and half) of your time. According to the Paperwork Reduction Act of 1995 (Pub. L. 104-13), an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this collection is 0970-0502 and the expiration date is 08/31/2020.

[For Focus Groups and In-Person Interviews (Written Consent)] Please read through and sign the consent form and we can get started.

[Read and sign consent]

[For Phone Interviews (Verbal Consent)]: I will now read you a form about what it means to participate in this research, describe your rights as a participant, and ask for your consent to continue with the interview.

Protocols

Introduction

1. Please state your name and position at DSHS.

2. How long have you worked in this position? In other positions at DSHS?

3. What is the size of your current caseload? Does it change from month to month or by season, say summer versus winter? How many people do you meet with each week?

4. To what extent do you feel like you have time to connect with each of your clients and track their progress through the TANF program?

5. Do you take any actions if a client misses an appointment?

6. Are you on any special teams or committees at DSHS?

Description of workflow and communication with clients during CE

3. Please walk me through a typical Comprehensive Evaluation, starting with when the client sits down at your desk.

   1. Probes: How do you typically describe the purpose of the CE and the questions you’re asking clients? What sections do you ask first? Next?

4. Part 2 of the CE starts by asking clients about their goals. How did you encourage clients to set goals for themselves?

   1. Probe: What sorts of phrases did you use? Do you generally ask why that goal is important to them?

Employment section/shaping the IRP

3. Do you typically explain all of the different WorkFirst activities available to clients? Why/why not?

   1. If so, how?

      1. Probes: What are common phrases you use? Do you use any tools (e.g., visual aids) to describe them? When in the CE session do you do this?

   2. If so, how do clients react to your explanation of the activities? What questions do they have?

4. How do you determine which WorkFirst activity/ies are a good fit for clients?

   1. Probe: How does information you gather when conducting your CE affect your decisions? To what extent do clients participate in determining which WorkFirst activities are a part of their IRP?

5. How do you explain to clients the benefits of participating in their WorkFirst activity?

   1. Probes: Does this explanation differ for different activities? For different clients?

6. What are common next steps for clients once their WorkFirst activity/ies are selected? How do you communicate these to clients?

   1. Probes: Aside from the IRP document, do you give clients any other documents outlining the next steps? If so, please describe. When do you advise clients to complete their next steps? (Probe: do you give them a deadline?)

7. What strategies do you find most effective to engage clients in the CE/IRP session?

8. What strategies do you find most effective to encourage clients to participate in their assigned WorkFirst activities after the CE/IRP session?

9. How long does the full CE session take, on average?

10. How often do you receive incoming calls from clients after you’ve completed their IRP?

    1. What are common questions you receive? How do you address these?

11. Do you interact with clients at any other points besides the intake sessions? If so, when and with what kinds of clients?

General Reflections

3. Overall, what would you describe as the purposes of the CE/IRP session? How does the CE tool support/hinder those goals?

   1. What strategies do you find most effective to engage clients in the CE/IRP session?

4. Do you have a story that you consider a particular success or failure of using the CE to help clients choose and participate in WorkFirst activities? What helped them succeed? What got in the way?

5. What are some of the most common challenges for clients after their CE/IRP session once they are assigned to their WorkFirst activities?

   1. What strategies do you find most effective to encourage clients to participate in their assigned WorkFirst activities after the CE/IRP session?

6. Has your unit implemented any process changes in the last six months or so? If so, how would you describe purpose of those changes?

   1. Probe: How much do they understand about the BIAS-NG intervention?

7. How would you describe your interactions with clients during the CE/IRP session? Has this changed over time? If so, how?

   1. Probe: Levels of engagement, understanding, motivation

Conclusion

3. Is there anything else you would like us to know about your experience working with new TANF WorkFirst applicants?

Thank you for your time!

6