Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Next Generation of Enhanced Employment Strategies Project
Formative Data Collections for Program Support
0970-0531
Supporting Statement
Part A
September 2021
Submitted by:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C St., SW
Washington, D.C. 20201
Project Officers:
Marie Lawrence
Gabrielle Newell
Sarita Barton
Megan Reid
Part A
Executive Summary
Type of request: This information collection request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).
Description of request: This request includes formative evaluation activities to refine the implementation of a program being evaluated as part of the Next Generation of Enhanced Employment Strategies (0970-0545; approved April 2020). The program is the Mental Health Outreach to MotherS (MOMS) Partnership. Proposed data collection efforts include surveys of program participants and potential participants who attend information sessions about the program and focus groups of program staff and staff at referral agencies. We do not intend for this information to be used as the principal basis for public policy decisions.
Time sensitivity: To begin the formative evaluation process when MOMS Partnership is first piloted in November 2021, we request approval by October 31, 2021.
A1. Necessity for the data collection
The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services proposes to conduct formative evaluation activities for the purpose of refining the implementation of a program that is being evaluated as part of the Next Generation of Enhanced Employment Strategies (NextGen) Project (OMB #0970-0545). ACF has contracted with Mathematica to conduct the NextGen Project.
ACF has spent decades studying strategies to help low-income people find and advance in jobs. Findings from these studies have been mixed, revealing variation in what works for whom and the duration and magnitude of impacts. The NextGen Project is intended to build on the findings and lessons learned from these past and ongoing evaluations by identifying and rigorously evaluating the “next generation” of employment strategies for highly vulnerable populations with complex barriers to obtaining and retaining employment.
One program being evaluated under the NextGen Project is Mental Health Outreach to MotherS (MOMS) Partnership. MOMS was designed to serve female caregivers with depressive symptoms and low income. It has been implemented in Connecticut, Vermont, Kentucky, and Washington, DC. For the NextGen Project, it will be implemented and evaluated in Springfield and Holyoke, Massachusetts. A nonprofit organization, Viability, will administer the program. The core of the MOMS Partnership model is a stress management course delivering cognitive behavioral therapy (CBT). The course includes eight weekly 90-minute group classes teaching practical skills for managing life stress and improving mood. These classes will be co-facilitated by a clinician and a staff person with lived experiences similar to the program participants. In addition, the MOMS Partnership as implemented in Massachusetts will offer participants the opportunity to meet with an employment coach, individually and in groups, in sessions called MOMS Moving Forward. Participants for the programs will be recruited from the Massachusetts Department of Transitional Assistance (DTA) and other agencies that serve mothers with low-income (such as Head Start, American Job Centers, and nonprofit organizations serving low-income populations).
The current data collection request is necessary to ensure high-quality implementation of the program while it is being evaluated. While the main features of the MOMS Partnership will not change, we expect that the details of its implementation will be refined in response to the data collection. We will encourage Viability, the program administrator, to continue these types of data collection independently after the formative evaluation is complete as part of continuous quality improvement of the program.
There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose
Purpose and use
This proposed information collection meets the following goal of ACF’s generic clearance for formative data collections for program support (0970-0531):
Use of rapid-cycle testing activities to strengthen programs in preparation for summative evaluation.
The purpose of this data collection is to ensure that the evaluation of MOMS Partnership conducted as part of the NextGen Project produces useful information for ACF, Viability, other organizations planning to implement the MOMS Partnership program, and other stakeholders. This information collection will be used to refine implementation of the MOMS Partnership to ensure high-quality implementation of the program while it is being evaluated. ACF will use the information collected through surveys of participants and potential participants and focus groups of program staff and staff at referral agencies to work with MOMs Partnership staff to improve the implementation of program components. The evaluation will be a fairer test of whether MOMS Partnership is effective if the program is well implemented.
This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
Study objectives
The ultimate objective of this requested formative data collection is to ensure that MOMS Partnership is well implemented when the program is being evaluated under the NextGen Project. The formative data collection will also help in refining the procedures for recruiting and enrolling study participants. Four research questions will be addressed:
Are there ways to make the orientation to MOMS more informative, engaging, and easy for the participants to attend?
Are there ways to make the Stress Management Course more informative and increase the likelihood of MOMS participants engaging in the Course?
Are there ways to make the MOMS Moving Forward sessions more informative and increase the likelihood of MOMS participants engaging in the sessions?
Are there ways to increase the number of eligible study participants who are referred to MOMS Partnership, who sign up for the study enrollment session, and who agree to participate in the study?
Study design
Starting in November 2021, Viability will implement a pilot of the MOMS Partnership program. They will provide an orientation, the MOMS Stress Management Course, and MOMS Moving Forward sessions to about 20 mothers who are likely eligible for MOMS Partnership. These participants will receive the MOMS Partnership services but will not be part of the summative evaluation (0970-0545; approved April 2020). In late December 2021, Viability will begin recruiting participants for the summative evaluation. Services will be provided to the study participants assigned to the research group that is offered a slot in MOMS Partnership beginning in January 2022.
The study design includes rapid learning cycles. Each cycle involves: collecting data on a program component, analyzing the data, and working with the program staff to refine implementation of the component. The cycles are repeated so that data is collected on the refined implementation of the component. We will examine four components: (1) the MOMS Partnership orientation; (2) Stress Management Course classes; (3) MOMS Moving Forward sessions; and (4) outreach, recruitment, and enrollment into the study.
The first data collection for the orientation, Stress Management Course, and MOMS Moving Forward sessions will occur within the first four weeks of the MOMS Partnership pilot in November 2021. A second cycle will occur in December 2021. Further cycles of these components will occur once the summative evaluation has begun in late December 2021. The first cycle to collect data on the outreach, recruitment, and enrollment into the study will begin in January 2022. We will conduct up to four cycles of each component. Hence, there will be several months in which we are both collecting data for the formative evaluation (this information collection) and the summative evaluation (0970-0545).
We will collect data on the program implementation from both program staff and participants and on recruitment from staff at referral agencies and potential study participants. The data collection is designed to be low burden.
After a cycle is concluded, the study team will analyze data collected and work with program staff to identify lessons learned and opportunities for additional improvement. Together, the study team and program staff members will refine the program implementation or recruitment efforts and plan for the next iterative learning cycle if needed.
Universe of data collection efforts
The current request includes ten data collection efforts (summarized in the table below). The efforts fall into four main categories:
Short surveys of MOMS Partnership participants (Instruments A, C, and E). For each of the four learning cycles, we will conduct short surveys of about 20 participants at the end of selected orientations, Stress Management Course classes, and MOMS Moving Forward sessions. For each learning cycle, we expect to administer the surveys at four or five orientations, two Stress Management Course classes (such as the first and the fourth in the series), and four or five MOMS Moving Forward sessions (including both virtual and in-person sessions). These surveys will be short and ask for assessments of the facilitation and logistics (such as place, time, day, and childcare) of the sessions and classes. The surveys will ask about the content of the orientation and MOMS Moving Forward sessions; the surveys will not ask about the content of the Stress Management Course classes as the curriculum cannot be changed before this evaluation.
Focus groups of MOMS Partnership staff (Instruments B, D, F, and J). For each of the four learning cycles, we will conduct focus groups with the staff facilitating the orientation, staff facilitating the Stress Management Course classes, MOMS Moving Forward coaches (who facilitate the MOMS Moving Forward sessions), and intake specialists who conduct recruitment, outreach, and the study enrollment interviews. We expect about eight participants to attend a focus group during each of the four cycles. At these groups, we will gather feedback on challenges staff see with the implementation of a specific program component and how it could be improved.
Surveys and focus groups of staff at referral agencies (Instruments G and I). For each of the four learning cycles, we will conduct short surveys of staff at referral agencies. For example, we expect to conduct periodic meetings of DTA staff and staff at other referral agencies in which we describe the study and the program. At the end of selected meetings, we will ask staff to respond to a quick survey about the content of the meeting. We expect about 60 survey respondents who will respond during each cycle. To understand more fully how we can improve our messaging, we will also conduct focus groups with about eight staff at DTA and eight staff at other referral agencies. These staff will attend a focus group during each of the four cycles.
Surveys of potential study participants (Instrument H). At regular intervals, MOMS Partnership intake specialists will present to groups of potential program participants who may be eligible for the program and study. For each of the four learning cycles, we expect to administer a short survey to these people to assess the clarity and efficacy of the messages. We expect to administer surveys to about 30 people in each cycle.
Data collection activity |
Instruments |
Respondent, content, purpose of collection |
Mode and duration |
|||
Orientation |
||||||
Survey of orientation attendees |
Instrument A |
Respondents: Attendees at orientation
Content: Attendees perceptions of the orientation
Purpose: To learn how to improve the orientation |
Mode: Web
Duration: 5 minutes
|
|||
Focus group of orientation facilitators |
Instrument B |
Respondents: Orientation facilitators
Content: Challenges faced and opinions of how it could be improved
Purpose: To learn how to improve the orientation |
Mode: Video conference
Duration: 60 minutes
|
|||
MOMS Partnership Stress Management Course |
||||||
Survey of Stress Management Course attendees |
Instrument C |
Respondents: People who attended a Stress Management class
Content: Satisfaction with facilitation and logistics of the classes
Purpose: To learn how to improve the facilitation and logistics of the classes |
Mode: Web
Duration: 5 minutes
|
|||
Focus Groups of Stress Management Course Facilitators |
Instrument D |
Respondents: Facilitators of the Stress Management Course classes
Content: Satisfaction with facilitation and logistics of the classes and how they could be improved
Purpose: To learn how to improve the facilitation and logistics of the classes |
Mode: Video conference
Duration: 60 minutes |
|||
MOMS Moving Forward Sessions |
||||||
Survey of MOMS Moving Forward attendees |
Instrument E |
Respondents: Attendees at MOMS Moving Forward sessions
Content: Satisfaction with content, facilitation, and logistics of the sessions
Purpose: To learn how to improve the content, facilitation, and logistics of the sessions |
Mode: Web
Duration: 5 minutes
|
|||
Focus Groups of MOMS Moving Forward coaches |
Instrument F |
Respondents: MOMS Moving Forward coaches and community mental health ambassadors
Content: Satisfaction with content, facilitation, and logistics of the sessions and how they could be improved
Purpose: To learn how to improve the content, facilitation, and logistics of the sessions |
Mode: Video conference
Duration: 60 minutes |
|||
Outreach, recruitment, and enrollment |
||||||
Survey of staff at referral agencies |
Instrument G |
Respondents: Staff at DTA and other referral agencies
Content: Response to messages and information provided about MOMS Partnership and the study
Purpose: To improve the messages given to referral staff at presentations about MOMS Partnership and the study |
Mode: Web
Duration: 5 minutes |
|||
Survey of attendees at recruitment events |
Instrument H |
Respondents: Potential study participants
Content: Potential study participants response to messages about MOMS Partnership and the study
Purpose: To improve the messages given to potential study participants at presentations about MOMS Partnership and the study |
Mode: Web
Duration: 5 minutes |
|||
Focus groups of staff at referral agencies |
Instrument I |
Respondents: Staff at DTA and other referral sources
Content: Perceptions of messages about MOMS Partnership and the study, experiences recruiting people, challenges faced when recruiting people and referring people to the study
Purpose: To learn how to improve the process of recruiting study participants |
Mode: Video conference
Duration: 60 minutes
|
|||
Focus groups of intake specialists |
Instrument J |
Respondents: Intake specialists
Content: Process of recruiting, messages, reasons people do not show up to the enrollment interview, why people who show up to the enrollment interview do not enroll in the study
Purpose: To learn how to improve the process of recruiting study participants and conducting enrollment interviews |
Mode: Video conference
Duration: 60 minutes
|
Other data sources and uses of information
The information collected by the surveys and focus groups will be supplemented with information we collect from the Random Assignment, Participant Tracking, Enrollment and Reporting system (RAPTER®). As part of the NextGen study (0970-0545; approved April 2020), this system will collect information on how many study participants are enrolled in the study group that are able to participate in MOMS Partnership and whether participants attend an orientation, each Stress Management Course class, or a MOMS Moving Forward session as well as whether they meet one-on-one with a MOMS staff person.
A3. Use of information technology to reduce burden
The surveys will be available online. We will administer them in a way that is easy for the respondent. They will typically be administered via Menti or other presentation software at the end of meetings. This will just require that participants use their smartphone or be given access to a tablet. We may also email invitations to online surveys that can be completed using a smartphone, tablet, laptop, or desktop computer.
The focus groups will be conducted virtually using videoconferencing software. Conducting the focus groups virtually will reduce the need for participants to travel to participate in the groups.
A4. Use of existing data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
None of the data collection instruments ask for information that can be reliably obtained through other sources. As MOMS Partnership is a new program in Massachusetts, no other agency will collect this information. Other sources also could not provide the data in the time frame required to make refinements in the program and test again in preparation for a summative evaluation.
A5. Impact on small businesses
No small businesses will be involved in this data collection.
A6. Consequences of less frequent data collection
A key goal of the formative data collection is iterative testing of the implementation of MOMS Partnership. Under the rapid cycle testing design, we will collect data, refine the program’s implementation, and then test again. If we collected the data less frequently, the number of iterations and hence changes we could make to the program based on the data collected would be fewer. We will conduct no more than four iterations and we will stop iterating and collecting data if the data collected suggests that the component is working well.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register notice and comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on
October 13, 2020, Volume 85, Number 198, page 64480, and provided a sixty-day period for public comment. ACF did not receive any substantive comments.
Consultation with experts outside of the study
The project team did not consult with experts outside of the study for this information collection.
A9. Tokens of appreciation
There are no tokens of appreciation associated with this information collection.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally identifiable information
Little personally identifiable information will be collected on survey respondents. If the survey is completed using a presentation software such as Menti, the responses will not require that we obtain any contact information. If we need to email the online survey, we will ask for email addresses.
We will request names and email addresses of staff at Viability who administer the program as well as staff at referral agencies in order to organize the focus groups. Recordings of the focus groups will be stored on secure servers.
Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.
Assurances of privacy
Prior to the start of any data collection, respondents will be informed about the purpose of the data collection, that their participation is voluntary, and that their information will be kept private. Mathematica will comply with all Federal and Departmental regulations for private information. Assurances will be made to survey and focus group participants before any data collection begins. We will obtain permission from all focus group participants before we record the session.
Data security and monitoring
Only staff from Mathematica will handle individual survey responses collected under this clearance. All Mathematica staff involved in the project receive annual training on (1) limitations of disclosure; (2) safeguarding the physical work environment; and (3) storing, transmitting, and destroying data securely. All Mathematica staff sign the Mathematica Confidentiality Agreement, complete online security awareness training when they are hired, and receive annual refresher training thereafter. Training addresses security policies and procedures found in the Mathematica Corporate Security Manual.
Only staff at Viability (the organization that administers MOMS Partnership) who are involved in the implementation of MOMS Partnership and staff at Mathematica involved in the study will view the focus group recordings.
Survey responses and recordings of focus groups will be downloaded to an encrypted project folder on Mathematica’s server. Mathematica uses access control lists to restrict access to the encrypted project folders where sensitive and confidential project data are stored. Access to the project folder is explicitly authorized by the Project Director on need-to-know and least privilege bases. Mathematica staff are required to change their password for computer and network access every thirty days, and passwords must adhere to strict composition standards. Staff access rights to the project folder are revoked when they leave the project. If a staff member leaves Mathematica, his or her access to computing assets, including network access, is terminated.
A11. Sensitive Information
There are no sensitive questions as part of these data collections.
A12. Burden
Estimated hours
The total number of burden hours is 255. The estimated hours for each instrument are provided in the table below. They are calculated as follows.
Survey of orientation attendees. Twenty different orientation attendees in each of four cycles will respond to a 5-minute survey (20 x 5/60 x 4 = 7 hours).
Focus group of orientation facilitators. Eight orientation facilitators will attend a one-hour focus group in each of four cycles (8 x 1 x 4 = 32 hours).
Survey of Stress Management Course attendees. Twenty participants will respond twice to a 5-minute survey after different classes, repeated with different participants in each of four cycles (20 x 2 x 5/60 x 4 = 13 hours).
Focus group of Stress Management Course facilitators. Eight Stress Management Course facilitators will attend a one-hour focus group in each of four cycle (8 x 1 x 4 = 32 hours).
Survey of MOMS Moving Forward attendees. Twenty participants will respond twice to a 5-minute survey after different sessions, repeated with different participants in each of four cycles (20 x 2 x 5/60 x 4 = 13 hours).
Focus group of MOMS Moving Forward coaches. Eight MOMS Moving Forward coaches will attend a one-hour focus group in each of four cycles (8 x 1 x 4 = 32 hours).
Survey of staff at referral agencies. Sixty staff at referral agencies who attend meetings about MOMS Partnerships will respond to a 5-minute survey in each of four cycles (60 x 5/60 x 4 = 20 hours).
Survey of attendees at recruitment events at referral agencies. Thirty different attendees at study recruitment events will respond to a 5-minute survey in each of four cycles (30 x 5/60 x 4 = 10 hours).
Focus groups of staff at referral agencies. Eight DTA staff and eight staff from other referral agencies will attend a one-hour focus group in each of four cycles ( (8 + 8) x 1 x 4 = 64 hours).
Focus groups of intake specialists. Eight intake specialists will attend a one-hour focus group in each of four cycles ( 8 x 1 x 4 = 32 hours).
Estimated annualized cost to respondents
To compute the total estimated annual cost, the total burden hours were multiplied by the estimated average hourly wage for program staff and participants. According to the Bureau of Labor Statistics’ Current Population Survey 2020, the median hourly wage for full-time community and social service specialists is $26.30. The hourly wage ($7.25) for participants and potential study participants is the federal minimum wage. The total annual burden cost is $5,887.35.
Instrument |
Respondent |
Total/ annual number of respondents |
Number of responses per respondent |
Average burden hours per response |
Total/Annual burden hours |
Average hourly wage |
Total annual cost |
Survey (A) |
Orientation attendees |
80 |
1 |
0.083 |
7 |
$7.25 |
$50.75 |
Focus group (B) |
Orientation facilitators |
8 |
4 |
1.000 |
32 |
$26.30 |
$841.60 |
Survey (C) |
Stress Management Course attendees |
80 |
2 |
0.083 |
13 |
$7.25 |
$94.25 |
Focus group (D) |
Stress Management Course facilitators |
8 |
4 |
1.000 |
32 |
$26.30 |
$841.60 |
Survey (E) |
MOMS Moving Forward attendees |
80 |
2 |
0.083 |
13 |
$7.25 |
$94.25 |
Focus group (F) |
MOMS Moving Forward coaches |
8 |
4 |
1.000 |
32 |
$26.30 |
$841.60 |
Survey (G) |
Staff at referral agencies |
60 |
4 |
0.083 |
20 |
$26.30 |
$526.00 |
Survey (H) |
Attendees at recruitment events |
120 |
1 |
0.083 |
10 |
$7.25 |
$72.50 |
Focus group (I) |
Staff at referral agencies |
16 |
4 |
1.000 |
64 |
$26.30 |
$1,683.20 |
Focus group (J) |
Intake specialists |
8 |
4 |
1.000 |
32 |
$26.30 |
$841.60 |
Estimated annual burden total |
255 |
|
$5,887.35 |
A13. Costs
There are no additional costs to respondents.
A14. Estimated annualized costs to the federal government
The total estimated cost for the federal government for the data collection activities under this current request will be $25,400. This includes personnel effort plus other direct and indirect costs.
Cost category |
Estimated costs |
Instrument development and OMB clearance |
$5,000.00 |
Data Collection |
$15,400.00 |
Analysis and PowerPoint describing findings |
$5,000.00 |
Total costs over the request period |
$25,400.00 |
A15. Reasons for changes in burden
This is an individual information collection request under generic clearance 0970-0531.
A16. Timeline
Beginning November 1, 2021, pending OMB approval of this request, Viability will pilot the MOMS Partnership program with about 20 participants. These participants will not be part of the summative evaluation as part of the NexGen project (0970-0545). The first two learning cycles of the orientation, Stress Management Course, and MOMS Moving Forward will occur in November and December 2021, respectively. Subsequent cycles will occur in early 2022. The four learnings cycles for the outreach, recruitment, and enrollment will begin in January 2022.
A17. Exceptions
All instruments will display the expiration date for OMB approval. No exceptions are necessary for this information collection.
Attachments
Instrument A: Survey of orientation attendees
Instrument B: Protocol for focus group of orientation facilitators
Instrument C: Survey of Stress Management Course attendees
Instrument D: Protocol for focus Groups of Stress Management Course Facilitators
Instrument E: Survey of MOMS Moving Forward attendees
Instrument F: Protocol for focus groups of MOMS Moving Forward coaches
Instrument G: Survey of staff at referral agencies
Instrument H: Survey of attendees at recruitment events
Instrument I: Protocol for focus groups of staff at referral agencies
Instrument J: Protocol for focus groups of intake specialists
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | DHHS |
File Modified | 0000-00-00 |
File Created | 2024-07-25 |