FAMLE Cross-Site Part b
U.S. Department of Health and Human Services Administration for Children and Families Office of Planning, Research, and Evaluation 330 C Street, SW Washington DC 20201
Federal Project Officer: Seth Chamberlain
|
OMB Supporting Statement for the Fatherhood and Marriage Local Evaluation and Cross-Site Data Collection Part B: Collection of Information Involving Statistical Methods June 2018
|
1. Respondent Universe and Sampling Methods 3
2. Procedures for collecting information 4
A. Statistical Methodology, Estimation, and Degree of Accuracy 4
B. Unusual Problems Requiring Specialized Sampling Procedures 5
C. Periodic Cycles of Data Collection 5
3. Methods to Maximize Response Rates and Data Reliability 6
4. Tests of procedures or methods 7
5. Individuals consulted on statistical methods 8
In anticipation of the healthy marriage and responsible fatherhood (HMRF) grant funding awarded in September 2015, the Administration for Children and Families (ACF) contracted with Mathematica Policy Research to conduct the Fatherhood and Marriage Local Evaluation (FaMLE) and Cross-Site project, with the dual goals of supporting quality and consistent performance measures data collection and fostering strong local evaluations by the HMRF grantees. For further background, please see Supporting Statement A.
The project relies on two complementary sources of data collection: (1) data collection by the contractor for cross-site analysis of program design and implementation, hereafter referred to as DCI (Data collected by the Contractor Itself); and (2) data collection by the grantees themselves for performance reporting and cross-site analysis, hereafter referred to as DCS (Data collected for Cross-Site analyses).
1. DCI
Respondents for Instrument DC-1, Grantee Staff Topic Guide on Program Design will be selected purposively, using information from grant applications including organizational charts and information on the role of employees at the grantee and its partner organizations. A topic guide will be used to conduct one semi-structured telephone interview with the program director or program manager at each of the approximately 60 grantees selected for stage two DCI data collection. Purposeful selection is needed for identifying staff respondents because insights and information can only come from individuals with particular roles or knowledge. In identifying whether to select the program director or manager, we will take into account each staff member’s (1) position and responsibilities, and (b) tenure with the organization.
Respondents for Instrument DCI-2, Grantee Staff Topic Guide on Implementation will be selected purposively, using organizational charts and information on each employee’s role at the grantee and its partner organizations. Semi-structured interviews, in person or by phone, will be conducted with a range of program staff from the approximately 20 grantees selected in stage three to learn about program implementation. Staff to be selected for the interviews may include: organizational leadership, project director/project manager, supervisors, staff involved with outreach and recruitment, case management/employment specialist staff, workshop facilitators, and staff at partner organizations. We estimate that on average, 15 staff at each grantee will participate in these interviews. Purposeful selection is appropriate for identifying staff respondents because insights and information can only come from individuals with particular roles or knowledge. In selecting staff, we will take into account factors such as each staff member’s (a) position and responsibilities, and (b) amount of daily interaction with participants or prospective participants.
With regard to Instrument DCI-3, Program Participant Focus Group Topic Guide, focus groups are intended to provide a sample of clients’ perspectives to provide context and anecdotes. The convenience sample approach that will be used to identify focus group participants precludes the project from using any findings from the focus group discussions to generalize to the entire FaMLE Cross-site Project grantee service population, or to any subgroups. That is, these groups will not contain, and are not intended to contain, a representative sample of any larger population. Grantees participating in the project will identify and recruit willing focus group respondents who have engaged in at least two program activities or attended a single activity two times beyond the intake interview. Identifying these clients will be purposive, based on their availability for the group and willingness to provide feedback. In HM programs, couples will be invited to the focus groups; in RF programs, fathers will be invited to focus groups.
2. DCS
A key objective of the DCS task is to facilitate the collection and reporting of quality and consistent performance measures data for all HMRF grantees funded in FY 2015. Through the nFORM system, DCS data are collected from the grantee and participant population of the HMRF organizations receiving grants. As discussed, the system provides data at the grantee, program and individual levels. Thus, data are collected on grantees’ program design and implementation and on the participants they serve. No sampling techniques are employed for the DCS performance reporting system.
ACF recognizes that the small number of participants and small number of groups, along with the sampling approach, means that the information gathered from the client focus groups will be purely anecdotal. Similarly, grantee selection for the design and implementation portion of the study will be purposive, and the project will interview all relevant staff in these purposively selected grantees. The findings from program design and implementation analyses, therefore, will not be used to extrapolate to the HMRF grantee population. Therefore, no statistical estimation or methodology is required for DCI data collection.
Power calculations tell us the likelihood of detecting changes in mean outcomes across treatment and comparison groups.1 Before the current cohort of grants was awarded, we calculated minimum detectable impacts (MDIs), as shown in Table 1, drawing on past work by Mathematica to determine population means and standard deviations.2 The power analysis assumed 150 control and 150 treatment observations from each of 35 grantees, for a total of 5,250 treatment and 5,250 control observations (or 10,500 total observations). We used about half of the sample for estimates of each of the RF and HM outcomes. Based on these assumptions, the sample would be large enough to detect the effect size of 0.09 found in Supporting Healthy Marriage (SHM) evaluation for the impact on couple’s warmth and support (Hsueh and Knox 2011)3. It is also sufficient to detect impacts on fathers’ outcomes. The evaluation of Parents’ Fair Share, which included services for low-income non-custodial parents, showed an impact of 6.8 percentage points in any payments towards a formal child support order (Knox and Redcross 2000)4. Actual effect sizes may be different. For example, the number of grantees conducting impact evaluations is currently lower than estimated, but the treatment and control observations may be higher per grantee.
Table B.1. Minimum Detectable Effects for Cross-Site Impact Analysis
|
Healthy Marriage Outcomes |
|
Responsible Fatherhood Outcomes |
||
Variable |
Quality of Parents’ Support and Affection |
Mother Reports Severe Physical Assault by a Romantic Partner (SD=1) |
Quality of the Co-Parenting Relationship |
Father Lives with Focal Child (SD=1) |
Father Provides Focal Child with Formal Financial Support |
Control Group Mean |
0 |
10 |
0 |
20 |
42 |
Assumed |
0.30 |
0.10 |
0.40 |
0.20 |
0.20 |
Minimum Detectable Impact |
|||||
|
SD |
Percentage points |
SD |
Percentage points |
Percentage points |
MDI |
0.07 |
2.4 |
0.06 |
3.0 |
3.7 |
There are no unusual problems requiring specialized sampling procedures.
This will include one round of telephone interviews with 60 grantees to gather information about program design. A subsequent round of interviews (some in person and some by telephone) will engage a wider range of staff at a subset of 20 of these grantees and will gather information on program implementation. Focus groups with HMRF clients will occur four times at each of these 20 grantees.
DCS data collection takes place with varying periodicity, depending on the content.
Program Applicant Characteristics (Instrument DCS-1) are collected from each program applicant when they enroll in the program.
Information on Program Operations (Instrument DCS-2), including marketing/outreach/recruitment, quality monitoring, staff characteristics, and implementation challenges, is entered into nFORM upon program implementation and updated quarterly.
Information on Service Receipt (Instrument DCS-3) is entered into nFORM at least once a week.
Outcomes (Instruments DCS-4HM and DCS-4RF) are collected at program entry (at the first workshop attended) and again at the last core program activity (or one month post-program exit for programs shorter than one month in duration).
Semi-structured interviews with program staff. The contractor will conduct interviews by telephone and on-site. We anticipate that all grantees selected to participate in the FaMLE Cross-Site project will agree to participate in these calls and site visits. Our past experience indicates that staff participation rates in site visits are typically higher than 90 percent among selected grantees. To ensure that response rates are maximized, the contractor will:
Identify convenient dates/times for calls and site visits. To help ensure high participation among staff for interviews, the contractor will coordinate with the selected grantees to determine convenient dates for these discussions and work with grantees to develop a schedule that accounts for the availability of program staff.
Use experienced and trained staff. All contractor staff conducting semi-structured staff interviews will have prior experience conducting semi-structured interviews and will participate in training to maximize data reliability.
Focus groups with participants. The contractor will:
Use familiar staff to recruit participants and offer reminders. Grantee staff will be asked to identify and recruit willing fathers or couples to participate in the focus groups. In addition, a letter will be mailed to each selected participant and a reminder call made prior to the focus group (included as Attachment I). To maximize response rates, we will offer a $25 gift card in appreciation of each participant’s time.
Conduct focus groups on site at a time convenient to participants. All focus groups will be held at the program location during a scheduled site visit. We will coordinate the schedule for each focus group so that it is convenient for participants to attend, for example just before or after a program group session, during the evening or weekend.
Use experienced focus group moderators. All contractor staff moderating focus groups will have prior experience with focus group moderation and participate in training to increase data reliability.
As described above, client data are continuously collected in nFORM across the current HMRF grantees. Given the range of grantee programming, surveys are administered at different points in time relative to client enrollment. As of June 1, 2018, grantees had completed enrollment for 102,660 clients; this is the count of clients who had completed Applicant Characteristics Surveys as of June 1st. Of these, 93,798 (91.4 percent) had completed a pre-test survey and 55,340 (53.9 percent) had completed a post-test survey as of June 1st.
To maximize response rates and data reliability for the current HMRF grantees, we have taken and will continue to take these steps:
Develop a user-friendly, flexible MIS. nFORM is specifically designed for use by grantee staff. As such, it is extremely user-friendly and flexible to meet each site’s needs. By providing sites with this system, we standardize the information being collected from each site and improve the reliability of our implementation and impact components.
Include data quality checks in the MIS. nFORM also improves data reliability by instituting automatic data quality checks. For example, if grantee staff enter odd or unlikely values in a particular field, the system will prompt users to check the value. For some fields, the response values are restricted; for others, grantee staff are able to override the check.
Provide training to grantee staff. To increase data quality, we provided training to nFORM system users prior to their initial use. Training was conducted in person as well as via a series of virtual trainings in which grantee staff viewed the trainer’s computer screen to get a virtual tour and illustration of how to use the system.
Monitor data quality. We also monitor the data entered by grantees and provide feedback to grantees on the accuracy and completeness of their entered data. Initially, we monitored data quality quarterly (coinciding with grantees’ quarterly reporting) and responded to specific questions from grantees. We have since monitored data quality more often in addition to continuing to respond to ad hoc requests from grantees.
Provide ongoing training and technical assistance to grantees. We provide numerous webinars and in person trainings to all HMRF grantees on protocols for collecting performance/cross-site data and entering these data into nFORM. We also develop guidance and technical assistance resources on various topics, and address specific questions from grantees via phone calls and email.
1. DCI
The semi-structured interview guides and the respondent focus group guide build on existing questions and previous experience from similar studies completed by the study team. Consequently, pretesting of previously used instruments or measures has not been planned.
In developing the self-administered pre- and post-program questionnaires, we drew heavily from existing measures, focusing as much as possible on measures with established validity and reliability. Nevertheless, some of the items have not been used with populations expected to be served by HM and RF programs—especially those for whom Spanish is their primary language. Consequently, we conducted cognitive interviews with five respondents who were participating in a RF or HM program; we included both men and women, and those for whom English and for whom Spanish was their primary language. We identified areas of misinterpretation or confusion (including translational issues) and revised the instruments as necessary. We then pilot-tested the revised instruments with four additional respondents who were participating in a RF or HM program, timing how long it took them to self-administer each instrument.
The development of nFORM was built on previous experience—most recently, in developing the MIS for the PACT Evaluation (PACTIS). The nFORM interface and the data elements being entered into nFORM have been tested previously and operated successfully in PACTIS, yielding the necessary information.
The data collection tools for this project were developed in partnership with staff in ACF’s Office of Planning, Research, and Evaluation and in consultation with staff in ACF’s Office of Family Assistance. Staff at Mathematica Policy Research were consulted on statistical methods, including:
Dr. Sarah Avellar
Senior Researcher
Mathematica Policy Research
1100 1st Street, NE
Washington, DC 20024
Inquiries regarding statistical aspects of the study should be directed to:
Mr. Seth Chamberlain
330 C Street, SW
Washington, DC 20201
Phone: (202) 260 2242
Further consultations will be made with statistical experts in the upcoming design phase of the cross-site evaluation.
1 We assume a two-tailed test with alpha=.05 and 80 percent power, and an average of 150 control and 150 treatment observations per 35 grantees (10,500 observations evenly split between treatment and control conditions), based on Mathematica’s past analysis of grantee applications for the PACT evaluation. We also assume a response rate of 87 percent, based on the 15-month evaluation of Building Strong Families (see Wood et al. 2010).
2 Quality of support and co-parenting are standard normal measures. Mothers’ reports of abuse are from the 15-month evaluation of Building Strong Families (see Wood et al. 2010). Responsible fatherhood outcomes are from a preliminary analysis of data from Mathematica’s PACT evaluation, confirmed in McLanahan and Beck (2010).
3 Hsueh, J. and V. Knox. The Supporting Healthy Marriage Evaluation: Early Impacts on Low-Income Families. OPRE Report 2011-45. February 2012.
4 Knox, V., and C. Redcross. Parenting and Providing: The Impact of Parents’ Fair Share on Paternal Involvement. MDRC. October 2000.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Emily Sama-Miller |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |