SSB - MOMS GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Next Generation of Enhanced Employment Strategies Project

Formative Data Collections for Program Support

0970-0531

Supporting Statement

Part B

September 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers

Marie Lawrence

Gabrielle Newell

Sarita Barton

Megan Reid

Part B

B1. Objectives

Study objectives

The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services proposes to conduct formative evaluation activities for the purpose of piloting and refining a program that is being evaluated as part of the Next Generation of Enhanced Employment Strategies (NextGen) Project (OMB #0970-0545).

The NextGen Project is intended to build on the findings and lessons learned from these past and ongoing evaluations by identifying and rigorously evaluating the “next generation” of employment strategies for highly vulnerable populations with complex barriers to obtaining and retaining employment. One program being evaluated under the NextGen Project is Mental Health Outreach to MotherS (MOMS) Partnership.

The ultimate objective of this requested formative data collection is to ensure that MOMS Partnership is well implemented when the program is being evaluated under the NextGen Project. The formative data collection will also help in refining the procedures for recruiting and enrolling study participants.

Generalizability of results

This study is intended to present an internally-valid description of the implementation of MOMS Partnership. It is not intended to promote statistical generalization to other programs or service populations.

Appropriateness of study design and methods for planned uses

These data collection efforts are intended to improve the implementation of the MOMS Partnership program and the recruitment and enrollment procedures. The study team will collect feedback on the implementation of program components and the procedures for recruiting and enrolling study participants and use the feedback to identify opportunities to strengthen and refine the program’s implementation and recruitment and enrollment efforts. Working with the program provider, Viability, we will make changes to the program implementation or recruitment and enrollment efforts, and then collect feedback on the refined approach. We will iterate until either we have conducted four rounds, or the feedback suggests the approach is working well, whichever comes earliest.

The surveys are designed to capture people’s perspectives about the program component or activity at a point of time. The surveys are short so that people will be more likely to respond to them and the burden is low.

The surveys will be administered at the end of, or directly after, the component or activity so that the respondents will not have problems with recall. One survey is for people who are potentially eligible for MOMS Partnership but not yet participating and have heard the description of the program and study. Shortly after they have heard the message, we will ask them questions about how they perceived the message. Another survey is for staff at referral agencies for them to comment on the messages about the program and study, which will be administered right after they have heard the message. The three other surveys are for MOMS Partnership participants and will be administered at the end of a class, group, or one-on-one session for their perspectives of that activity. The expected sample sizes are large enough to conclude whether there are specific problems with the program and areas that need improvement. The sample is not meant to be representative of people who are eligible for MOMS Partnership, referral agency staff, or MOMS Partnership participants.

The focus groups are designed to provide more in-depth information about any problems or challenges that arise. We will form two focus groups—one with staff from the Department of Transitional Assistance (DTA), the main referral agency, and one with staff from other referral agencies. We separate the staff from DTA and the other referral agencies because the approach to recruitment differs between DTA and the other agencies. The staff will be selected purposively—they will not be representative of staff at these agencies. The other focus groups will be of staff in MOMS Partnership. We will include all staff who are involved in the provision of that component.

This data collection is not designed to provide representative estimates of perspectives of any target population. It will be used to make refinements in the program implementation and recruitment and enrollment procedures; it will not be used to make any inferences about the impact of the program. The limitations of these data will be described when findings from the analysis are presented.

This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target population

The target population includes program participants, program staff, staff at agencies that refer potential participants to the program, and potential program participants.

• Program participants. This includes people who are participating in MOMS Partnership. People are eligible for the program if they present as female, are caregivers of children under 18, have depressive symptoms, have low income, and are not currently receiving Social Security disability benefits. We will conduct short surveys of program participants (Instruments A, C, and E).

• Program staff. This will include staff at Viability, which will implement the MOMS Partnership program for the NextGen Project. The staff will include orientation facilitators, Stress Management Course facilitators, MOMS Moving Forward coaches, intake specialists, supervisors, and managers. We plan to conduct virtual focus groups with these staff (Instruments B, D, F, and J).

• Staff at referral agencies. This includes frontline staff and their supervisors and managers at the Massachusetts DTA) and other agencies who will refer people to MOMS Partnership. We will conduct short surveys of staff at referral agencies as well as virtual focus groups (Instruments G and I).

• Potential program participants. This includes people receiving services at DTA and other referral agencies who are potentially eligible to participate in MOMS Partnership and are attending recruitment events. We will conduct short surveys of potential program participants to ask for their feedback about the messages about the program (Instrument H).

Sampling and site selection

For respondent recruitment, all the program staff in relevant roles will be included in the focus groups. For example, we will include all the orientation facilitators in the focus groups about the orientation.

We will ask all program participants who attend specific orientations, Stress Management Course classes, or MOMS Moving Forward sessions to complete a short survey. We will select particular orientations, classes, or sessions to study in a way that includes different program staff, virtual as well as in-person services, different times of the day or days of the week, and other ways in which they may differ. We will include program participants who attend a one-on-one orientation session as well as a group orientation session.

We will ask select staff at DTA and other referral agencies to conduct a short survey. These will be staff who attend meetings at which Viability staff describe MOMS Partnership and the study. We will conduct one focus group with DTA staff and another focus group with staff from other referral agencies.

We will ask potential program participants at specific recruitment events to complete a survey. Some of these recruitment events will be at DTA. Other recruitment events will be at other referral agencies.

B3. Design of Data Collection Instruments

Development of data collection instruments

The instruments were adapted from protocols and survey instruments used successfully for other projects such as the Supporting Partnerships to Advance Research and Knowledge (Project SPARK)¹. Hence, these data collection instruments will not be pretested.

The instruments are designed to address each of the research objectives (see table below). The short surveys—completed in 5 minutes or less—are designed to detect issues with the implementation of the component. The focus groups are designed to go into more detail about the nature of the issue, the root causes, and potential solutions. We streamlined the data collection instruments to ask only questions necessary to achieve the objectives of the data collection.

Research question	Instrument
	A	B	C	D	E	F	G	H	I	J
1. Are there ways to make the orientation to MOMS more informative, engaging, and easy to attend?	X	X
2. Are there ways to make the Stress Management Course more informative and increase the likelihood of MOMS participants engaging in the Course?			X	X
3. Are there ways to make the MOMS Moving Forward sessions more informative and increase the likelihood of MOMS study participants engaging in the sessions?					X	X
4. Are there ways to increase the number of eligible study participants to sign up for the study enrollment session and increase the number of those who sign up to the session who agree to participate in the study?							X	X	X	X

B4. Collection of Data and Quality Control

ACF has contracted with Mathematica to conduct this information collection and the related NextGen Project. Viability, the organization that will implement MOMS Partnership, will administer the surveys and Mathematica staff will facilitate the focus groups.

The surveys will be administered via the web. Mathematica staff will train Viability staff on how and when to administer the surveys. They are simple surveys with mainly closed-ended answers. Mathematica staff will code the open-ended responses. They will also check for missing data. If data are missing, we will not try to reinterview the respondent. We do not need responses from everyone interviewed because the data collected are not meant to be representative and we are interviewing enough respondents that we can collect sufficient information with some item nonresponse.

A Mathematica staff person will facilitate the focus groups. This staff person will be trained on facilitation procedures as well as the objectives of the group. With the participants permission, the focus groups will be recorded. Another member of the Mathematica research team will sit in on the focus group to monitor whether everyone has a chance to speak, one or two focus group participants do not dominate the conversation, and the facilitator covers all the questions. If problems arise, the facilitator will receive additional training.

B5. Response Rates and Potential Nonresponse Bias

Response rates

The surveys and other qualitative data collection activities are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be reported.

Nonresponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be investigated.

B6. Production of Estimates and Projections

Data collected will support program improvement. The data will not be used to generate population estimates, neither for internal use nor dissemination.

B7. Data Handling and Analysis

Data handling

No personally-identifiable information will be given to anyone outside of the study team. Typed notes and audio recordings will be stored on Mathematica’s network, which is accessible only to the study team, and destroyed upon completion of the study.

Most of the questions in the survey will have closed-ended responses. The surveys will be administered by a program (such as Menti) that automatically calculate frequencies, reducing the opportunity for errors.

Data analysis

This project will not employ complex analytical techniques on the data collected. We will present the results of the surveys using descriptive statistics such as frequencies and means.

To analyze qualitative data, such as notes from focus groups, we will use standard qualitative analysis techniques such as thematic identification.

A second Mathematica staff member will conduct a quality control check on any programming conducted on survey data and the conclusions from the focus groups.

Data use

The study team will present the results of the data collection to the Viability staff and use the results for decisions about how implementation of the program should be refined. The descriptive report on MOMS Partnership, to be shared publicly, will include information on the refinements made and the lessons learned. In the report, we will describe the limits to the generalizability of the findings.

B8. Contact Persons

• Sheena McConnell, Mathematica, project director, [email protected]

• Marie Lawrence, OPRE, federal project officer, [email protected]

• Gabrielle Newell, OPRE, federal project officer, [email protected]

Attachments

Instrument A: Survey of orientation attendees

Instrument B: Protocol for focus group of orientation facilitators

Instrument C: Survey of Stress Management Course attendees

Instrument D: Protocol for focus Groups of Stress Management Course Facilitators

Instrument E: Survey of MOMS Moving Forward attendees

Instrument F: Protocol for focus groups of MOMS Moving Forward coaches

Instrument G: Survey of staff at referral agencies

Instrument H: Survey of attendees at recruitment events

Instrument I: Protocol for focus groups of staff at referral agencies

Instrument J: Protocol for focus groups of intake specialists

1 Data collection approved in December 2020 under the umbrella generic for Formative Data Collections for ACF Program Support (OMB #0970-0531)

7