B1. Summary of Intervention Designs June 15 2023

B1. Summary of Intervention Designs June 15 2023 Clean.docx

Rapid Cycle Evaluation of Operational Improvements in Supplemental Nutrition Assistance Program (SNAP) Employment & Training (E&T) Programs

B1. Summary of Intervention Designs June 15 2023

OMB: 0584-0680

Document [docx]
Download: docx | pdf


Appendix B1. Summary of Intervention Designs



This page has been left blank for double-sided copying.

Appendix B1. Table B2.a.1. Summary of interventions and evaluations

Site and agency

Summary of challenge and intervention

Evaluation and intervention components

Colorado



Colorado Department of Human Services

Challenge: There has been little investment in marketing messages, materials, and strategies at the State level. County staff are unsure of how to best ‘sell’ the SNAP E&T program to clients. Eligibility workers and case managers do not receive much training on why someone would want to participate in the Employment First program. There are no marketing materials other than one standard State-offered flyer that counties can customize. All other marketing happens during outbound telephone calls that case managers make although many clients do not answer these calls. As a result, case managers believe that most clients do not know what the Employment First program offers, including supportive services and E&T activities, and enrollment is low.



Intervention: The site will use a suite of marketing materials consisting of text messages, emails, and postcards to conduct outreach to referred work registrants, or SNAP participants who do not meet a federal exemption from general work requirements, in place of making outbound phone calls. Four treatment groups will receive different combinations of text messages and emails at different frequencies. Three treatment groups will get text messages and emails one week after a SNAP certification or recertification interview: one group will receive three text messages one week apart, a second group will receive two text messages two weeks apart, and a third group will receive two text messages and two emails two weeks apart. A fourth treatment group will receive the same series of messages as the third group except the first text and email will be sent two weeks after the certification or recertification interview. A fifth treatment group will receive two postcards one week apart, starting one week after the certification or recertification appointment. A control group will continue to receive telephone recruiting calls from Employment First case managers.


Target population: SNAP participants who are work registrants

Intervention locations: Four counties (Larimer, Montrose, Denver, Broomfield)

Target number of participants: 9,000



Evaluation design: Multi-arm RCT, with equal probabilities used to assign individuals across research groups.

Research questions:

  • What package of behaviorally informed marketing outreach is most effective in increasing enrollment in SNAP E&T, compared to existing marketing practices?

  • What frequency of text messaging outreach is most effective in increasing enrollment?

  • What mode of outreach is most effective in increasing enrollment?

  • Is an up-front or delayed approach to outreach more effective?

Outcomes: The intervention is designed to increase engagement in an initial phone call or in-person meeting with a case manager to enroll in SNAP E&T services.



Survey sample size: 800

Survey design: Strata will be formed (1) based on research groups (four treatment groups and the control group) and county for three of four counties and (2) based on research groups (two treatment groups and the control group) within the fourth county. (There are fewer treatment groups in this county due to reduced cell phone coverage and inability for SNAP participants to receive text messages.)

Connecticut



12 Connecticut community colleges

Challenge: Standard strategies and tools with which SNAP E&T coaches or coordinators can address students’ challenges are lacking. Coaches use ad-hoc approaches for assessing student barriers, including conversational approaches, questions on an intake form, and formal assessment tools (typically focused on assessing educational readiness). Coaches also feel unprepared to respond to students’ challenges. Additionally, students lack awareness about the program and the supports available to them. Together, these conditions contribute to inadequate mitigation of barriers, potentially hindering students’ ability to participate in and complete the SNAP E&T program.



Intervention: Coaches at the site will administer a new type of assessment to SNAP E&T participants on community college campuses. This assessment will use a strengths-based approach to identify a range of barriers students may be facing by asking students to identify both areas where they need help and areas in which they are thriving. This approach will include a goal-setting process intended to increase participants’ motivation to achieve their goals. Coaches will use the assessment to determine the participant reimbursements or support service referrals each student needs. Students who get a referral to receive support services will receive a behaviorally informed text message to remind them of the referral information and to motivate them to pursue the service. Individuals will be randomly assigned to one of three research groups: a treatment group that receives the new participant-driven assessment and, if referred to a provider, receives behaviorally informed text message reminders for the referral; a treatment group that receives the new participant-driven assessment and, if referred to a provider, does not receive text message reminders; and a control group that receives the existing assessment that coaches currently use.


Target population: SNAP-participants who are also work registrants attending community colleges

Intervention locations: 12 community colleges

Target number of participants: 808



Evaluation design: Multi-arm RCT with two treatment groups and one control group

Research questions:

  • What is the effect of a participant-driven assessment compared to a provider-driven assessment?

  • What is the effect of a behavioral nudge compared to no text message among individuals who receive a participant-driven assessment?

  • What is the effect of the “full package” of receiving a participant-driven assessment along with a behavioral nudge text compared to receiving only a provider-driven assessment?

Outcomes: The intervention is designed to increase participation in and completion of SNAP E&T services.



Survey sample size: 808

Survey design: The participant survey sample will include all SNAP participants enrolled in Connecticut’s community colleges to ensure sufficient statistical power with which to identify intervention effects

District of Columbia



District of Columbia Department of Human Services, Economic Security Administration (DHS)

Challenge: Current assessment tools used to support case management are lacking in several areas. Assessments need to be more customer and goals-focused and need to help guide conversations around goal setting, action planning around goals and potential barriers, and strategies for finding and maintaining meaningful employment. Additionally, participants may not have a strong understanding of the various dimensions that make a job a good fit. Most participants focus on wages, but understanding factors like benefits, flexibility in work schedule, career trajectory, and having a supportive work environment are important. The lack of goals-based assessments and participant awareness may contribute to low engagement and misalignment between program offerings and participants’ needs.



Intervention: The site will create a participant-driven assessment tool that guides case managers’ conversations with participants around goal setting, identifies barriers to employment, and describes steps involved in finding and maintaining meaningful employment. Staff also will receive training to provide enhanced case management to participants. The new assessment and enhanced case management will be provided to all SNAP E&T participants; individuals will not be randomly assigned to receive these components. Individuals will be randomly assigned to receive behaviorally informed text message reminders for appointments.


Target population: SNAP E&T participants ages 16 and older

Intervention locations: District wide

Target number of participants: 375



Evaluation design: Quasi-experimental pre-post design to evaluate change in assessment and case management approach.

Single-arm RCT to evaluate effectiveness of electronic message reminders.

Research questions:

  • Are the new assessment and case management approaches more effective in increasing SNAP E&T engagement than pre-intervention practices? Do they increase the likelihood of individuals identifying and obtaining participant reimbursements? Are they associated with individuals being more informed about the quality of jobs they accept and having better matches with employers?

  • Compared to participants exposed to pre-intervention practices, are participants who receive new assessment and case management approaches more likely to consider long-term career goals with short-term goals?

  • To what extent is there a difference in the pre-and post-intervention percentages of participants who, after becoming employed, stay engaged in SNAP E&T or enroll in education or training programs?

  • Are reminders and behaviorally informed text messages more effective in increasing SNAP E&T engagement and improving attendance of appointments than current practices?

  • Which type of reminder message or message content is most effective in increasing SNAP E&T engagement with a case manager?

Outcomes: The intervention is designed to strengthen case management approaches to motivate participants to remain engaged in SNAP E&T and focus on their long-term goals. Engagement will be measured using administrative records (e.g. whether an appointment was attended, number of months in which they attend at least one appointment, enrollment in education or training programs).

Kansas



Kansas Division of Children and Families (DCF)

Challenge: Kansas’s SNAP E&T program transitioned from a voluntary program that served anyone over 18 years of age with no dependents to a mandatory program that served ABAWDs only, as of October 2022. Early reports from DCF staff indicate low levels of engagement among ABAWDs. SNAP participants who are referred to E&T are notified via a letter that they must participate in the E&T program and are given an appointment date with a Career Navigator. At that point they are considered a mandatory SNAP E&T participant, and will lose their benefits if they do not meet participation requirements. Staff estimate about 30% of participants attend their first scheduled appointment, and it is difficult for staff to reach participants by telephone to reschedule meetings. As the mandatory program continues, DCF expects high good cause determination and/or sanction rates among the mandatory participants, as many are not a good fit for the program or are not meeting participation requirements.



Intervention: The site will send a series of text messages informed by behavioral science to SNAP E&T participants to increase engagement in SNAP E&T activities. After completing an assessment, reminder messages and/or behavioral nudges will be sent to E&T participants to remind them of SNAP E&T appointments and events. There will be four research groups: three treatment groups and one control group. One treatment group will receive text messages reminding them of appointments. A second treatment group will receive behavioral nudges designed to promote engagement in the program. A third treatment group will receive both appointment reminder messages and behavioral nudges related to engagement. The control group will receive neither reminder messages nor nudges.

Target population: SNAP E&T participants who are Able-Bodied Adults Without Dependents (ABAWDs)

Intervention locations: Statewide

Target number of participants: 1,200



Evaluation design: Multi-arm RCT with three treatment groups and one control group

Research questions:

  • Do text message reminders increase attendance to appointments?

  • Do behavioral messages (nudges) motivate participants to stay engaged in the SNAP E&T program?

  • Do text message reminders and behavioral messages (nudges) reduce the likelihood of being sanctioned?

  • Is sending both text message reminders and behavioral messages (nudges) more effective in improving engagement in SNAP E&T than only sending one type?

Outcomes: The intervention is designed to increase individuals’ appointment and event attendance and engagement in SNAP E&T activities.

Massachusetts



Massachusetts Department of Transitional Assistance (DTA)

Challenge: DTA eligibility staff do not have a clear assessment process or referral protocol for potential SNAP E&T participants, resulting in some participants being unaware of the services available to them. Additionally, there are no dedicated E&T staff or case managers who work with SNAP participants, and a clear process does not exist for staff to assess participants’ E&T needs and interests and match them to a provider that can meet their needs. Staff currently provide participants with a link to a DTA Path to Work website to locate providers. There also is a lack of communication with career centers, the only statewide E&T provider, including challenges related to referring participants to the centers. These factors have contributed to low E&T enrollment and inadequate referrals in the state.



Intervention: The site will use a new referral process with four components. A text message will be sent inviting potential SNAP E&T participants to learn about E&T services. Individuals who affirm they are interested in learning more will receive an online, self-administered screening form to assess their work readiness. Based on their level of work readiness, individuals will receive a one-on-one assessment by an E&T worker to assess participant fit and readiness for a referral to a local career center. Those deemed to be work ready in the one-on-one assessment will receive a referral and warm handoff to a career center. There will be two treatment groups that are differentiated only by the behaviorally informed content used in the initial text message. Individuals will be randomly assigned to one of those two groups or to a control group that has access to a website to independently learn more about E&T services available at the local career center. Within each treatment group, individuals who pass the online, self-administered screener will be randomly assigned to receive a one-on-one assessment by an E&T worker. Individuals deemed to be work-ready in the one-on-one assessment by the E&T worker will then be randomly assigned to receive a warm handoff to the career center. At each of these points of random assignment, control group members will be referred to the career center website to independently learn more about E&T services available.


Target population: Adult SNAP participants who do not receive Supplemental Security Income benefits and who have agreed to receive communication from DTA through text messages

Intervention locations: Five counties where there is a DTA office and at least one career center location were purposively selected

Target number of participants: 30,000



Evaluation design: Multi-arm RCT

Two treatment groups and a control group to evaluate the effectiveness of electronic messaging. One treatment group and one control group to evaluate effectiveness of one-on-one assessment. One treatment group and one control group to evaluate effectiveness of warm handoff and referral to career center

Research questions:

  • Which types of outreach messages yield the highest percentage of individuals who express interest in learning more about SNAP E&T services?

  • Which types of outreach messages lead to higher rates of enrollment in SNAP E&T?

  • Does completing a full assessment lead to higher rates of enrollment among individuals who express interest in learning more about E&T services and pass a pre-screener survey for the assessment?

  • Does a warm handoff referral to career center staff among work-ready participants lead to higher rates of SNAP E&T enrollment?

Outcomes: The intervention is designed to increase enrollment in SNAP E&T and improve assessment of work readiness for SNAP E&T services.



Survey sample size: 1,200

Survey design: Three strata will be formed based on research groups (treatment group 1, treatment group 2, and the control group) and the five geographic locations where the intervention will be implemented. The team will also create strata based on the assessment and referral treatment and control groups within treatment groups 1 and 2 to ensure sufficient numbers of respondents who will be eligible for an assessment or a referral to a career center.

Minnesota-Hennepin



Hennepin County Department of Human Services

Challenge: Individuals referred to SNAP E&T currently receive a system-generated letter with information about the program. SNAP E&T staff do not believe the letter is effective at encouraging potential participants to enroll in the program, particularly given low enrollment.



Intervention: The site will send a series of behaviorally informed text messages to ABAWDS to encourage enrollment in SNAP E&T to help them meet their work requirements and avoid losing SNAP benefits. Due to the end of COVID-19 ABAWD waivers, SNAP participants will need to enroll in E&T by the end of October 2023, to avoid losing benefits. (If they do not meet E&T or other work requirements, November will be the final month of benefits). SNAP participants will receive up to three text messages at three different time points: mid-September, end of September, and the first week of October 2023. The content of each of the three messages will vary across each point in time: the first message will try to increase awareness (mere-exposure effect) and make it clear how to enroll in SNAP E&T. The second message will clarify that changes to work rules are soon to be in effect and emphasize that there are not many steps to enroll in E&T (loss aversion and endowed progress effect). The third message will notify participants of the upcoming end of the waiver and consequent benefit disruption (loss aversion). The timing and behavioral message content will be the same for each of the two treatment groups; the groups will only differ according to whether they receive a short single text in each round of messages (first treatment group), or a slightly longer set of two messages that include the participant’s name (second treatment group) (there is a possibility that the second group will not have the name included). The control group will continue to receive a system-generated letter with information about the program and will not receive text messages from Hennepin County.




Target population: SNAP participants who are newly certified ABAWDs

Intervention locations: Hennepin County

Target number of participants: 4,700



Evaluation design: Multi-arm RCT

Three treatment groups and one control group

Research questions:

  • Which text messaging approach (existing versus text messaging intervention) yields the most enrollment in SNAP E&T?

  • Which approach yields the most contact with the SNAP E&T team?

  • Which of the three text messaging outreach approaches was most effective in increasing enrollment in SNAP E&T, compared to existing outreach?

Outcomes: The intervention is designed to increase the number of participants who contact the county SNAP E&T team and enroll in SNAP E&T.

Minnesota-

Rural



Minnesota Department of Human Services on behalf of E&T providers in 33 rural counties

Challenge: E&T provider staff report that staff who interact with potential SNAP E&T clients at all levels and across counties often have different or inconsistent ways of communicating the benefits of SNAP E&T. SNAP E&T providers are left to develop recruitment messages on their own, but often lack in-house expertise in marketing and confidence that their messages will be effective at reaching their target population. This can lead to inconsistent messaging about the program which may impact referrals.



Intervention: The site will send a series of behaviorally informed text messages to SNAP participants to encourage enrollment in SNAP E&T. Individuals will be randomly assigned to either a treatment group that receives up to three text messages or a control group that receives providers’ existing recruiting materials, consisting of word-of-mouth approaches, flyers, or community partner referrals that are not specifically targeted to SNAP participants. Treatment group members will receive the first text message within three weeks after SNAP certification, and the second and third message approximately one and two months later, respectively. The content of each message will vary across each point in time: the first message will try to increase awareness (mere-exposure effect), the second message will emphasize that SNAP E&T staff want to support participants without the need for them to take many additional steps to enroll (endowed progress effect ), and the third message will aim to increase the feeling of entitlement to or ownership of the E&T program and reiterate that staff want to support them in getting employment (endowment effect).


Target population: SNAP participants who are work registrants

Intervention locations: 33 counties

Target number of participants: 4,500



Evaluation design: Single-arm RCT

One treatment group and one control group

Research questions:

  • Which approach (existing recruiting and outreach model versus text messaging intervention) yields the most contact with SNAP E&T providers within 90 days?

  • Which approach (existing recruiting and outreach model versus text messaging intervention) yields the most enrollments in SNAP E&T within 90 days?

  • Of participants that have enrolled in SNAP E&T, what percentage are engaged in a component within 60 days of enrollment?

Outcomes: The intervention is designed to increase the number of participants who contact SNAP E&T provider staff, enroll in SNAP E&T, and engage in a program component.

Rhode Island



Rhode Island Division of Human Services (DHS)

Local Initiatives Support Corporation (LISC)


Challenge: Outreach messaging needs to be strengthened. Existing recruiting approaches consist of sending emails to eligible SNAP participants referring them to the SNAP E&T website, mailing letters or brochures, attending community events, discussing with SNAP participants in SNAP offices, and distributing flyers to community service providers. Improvement to the referral process also is needed. SNAP E&T staff receive few referrals which could reflect a lack of staff discussing the program and its offerings with potential participants or a lack of participants’ interest in the program.



Intervention: The site will send a series of behaviorally informed text messages and emails to eligible SNAP participants to encourage enrollment in SNAP E&T. The intervention will target two groups separately: (1) work registrants who are not also an ABAWD and (2) ABAWDs. The follow up messages for ABAWDs will focus on loss aversion, as Rhode Island anticipates the ABAWD waiver will expire in July 2022 and many of those they are targeting will have a time limit. Individuals will be randomly assigned to one of five research groups: a treatment group in which individuals receive the text message containing a link to the SNAP E&T website to learn more about the services available; a treatment group in which individuals receive an email containing a link to the SNAP E&T website to learn more about the services available; a treatment group in which individuals receive a text message requesting them to reply directly to the text for more information; a treatment group in which individuals receive an email requesting them to reply directly to the email for more information; or a control group that receives the existing, standard outreach materials. In the treatment groups that request a reply from participants to learn more about available services, individuals who affirm their interest are randomly assigned again either to receive an existing assessment or an enhanced, provider-informed assessment that determines participant skills and interests, and matches them with providers that accommodate the person’s background, skill level, and interests.


Target population: SNAP participants: both ABAWDs and work registrants who are not also ABAWDs

Intervention locations: Statewide

Target number of participants: 5,000



Evaluation design: Multi-arm RCT

Four treatment groups and one control group to evaluate effectiveness of electronic messaging

One treatment group and one control group to evaluate effectiveness of new, enhanced assessment

Research questions:

  • Are targeted, behaviorally informed emails more effective than current practices in increasing rates of contact with SNAP E&T staff and rates of enrollment in SNAP E&T?

  • Does new behaviorally informed messaging using text messages or emails lead to higher rates of initial contact with DHS staff and enrollment in E&T?

  • Do requests to reply to outreach messages yield higher rates of contact with DHS staff and higher rates of enrollment in SNAP E&T than current practices of providing website links for individuals to contact E&T staff on their own?

  • Does receiving a provider-informed assessment lead to greater percentages of individuals who finish the assessment and are referred to a provider, compared to those receiving a current assessment, among individuals who express interest in learning more about E&T services?

Outcomes: The intervention is designed to determine whether the format and content of outreach messaging, along with an optimized assessment process, increases SNAP E&T enrollment and engagement in program components.



Survey sample size: 1,200

Survey design: Five strata will be formed based on research groups (treatment groups 1 to 4 and the control group). The team will also create strata based on the assessment treatment and control groups within two of the initial-stage treatment groups to ensure sufficient numbers of respondents who will be eligible for an assessment.



File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified0000-00-00
File Created0000-00-00

© 2024 OMB.report | Privacy Policy