BSF - OMB Package Part B - 3-18-08

CONTENTS (continued)

Page

Contract No.: 233-02-0056

144

Building Strong Families: OMB Supporting Statement for the Second Follow-Up Data Collection - Part B (Collection of Information Employing Statistical Methods)

March 19, 2008

Submitted to: U.S. Department of Health and Human Services Administration for Children and Families Office of Planning, Research, and Evaluation 7th floor, West, Aerospace Center Building 370 L’Enfant Promenade, SW Washington, DC 20447 Project Officers: Nancye Campbell Seth Chamberlain

CONTENTS

Page

1. Respondent Universe and Sampling 1

2. Procedures for the Collection of Information 2

3. Methods to Maximize the Response Rate and to Deal with Nonresponse 5

4. Tests of Procedures and Methods to Be Undertaken 7

5. Individuals Consulted on Statistical Aspects of the Design 8

References 9

TABLES

Tables Page

1 NUMBER OF COUPLES EXPECTED TO ENROLL IN STUDY, BY SITE 1

2 MINIMUM DETECTABLE IMPACTS AT THE SECOND FOLLOW-UP

BY SAMPLE SIZE, FOR KEY OUTCOMES 3

1. Respondent Universe and Sampling

The BSF programs target low-income, adult, unwed couples who are expecting a baby (or had a baby within the past three months), who are romantically involved, and volunteer for the program. The sample frame for the evaluation is all couples in the seven BSF sites both of whose members: (1) are interested in participating in a BSF program during the sample intake period, (2) pass the eligibility criteria, (3) complete a baseline information form, and (4) give consent to participate in the study. The sample intake period varies for each program site. However, it will last until the desired sample size has been randomly assigned in each program site or until February 2008, whichever is shorter. Table 1 presents the number of couples enrolled in the overall study by site.

TABLE 1

NUMBER OF COUPLES EXPECTED TO ENROLL IN STUDY, BY SITE

Site	Total Enrolled
Atlanta, Georgia	930
Baltimore, Maryland	603
Baton Rouge, Louisiana	646
Florida (Orange and Broward counties)	696
Indiana (Allen, Marion, Miami, and Lake counties)	463
Oklahoma	1,010
Texas (San Angelo and Houston)	747
Total	5,095

^aThe number of couples is shown, rather than individual respondents, because sites must enroll both members of the couple.

While the telephone surveys are targeted at the entire evaluation sample, the target sample size for the in-home direct assessment is 3,000 families. We will choose five or six intervention sites for the in-home data collection based on an assessment of the strength of BSF program implementation at the sites. In those sites, we will randomly select families among those couples who enrolled after the first three months of intake to determine the 3,000 families sampled to participate in the in-home direct assessment.

2. Procedures for the Collection of Information

a. Statistical Methodology, Estimation, and Degree of Accuracy

The method for estimating impacts is presented in Part A Section 14. The minimum detectable impacts were estimated for three illustrative outcomes that will be measured at 36 months: (1) the percentage of BSF couples who are married; (2) the percentage of BSF couples who are still together as a couple; and (3) the supportiveness of the parent toward the child during the Three Bag Task—a structured play situation that will be videotaped. Table 2 shows the minimum detectable impacts measured using the second follow-up survey for the percentage of BSF couples who are married and the percentage of BSF couples who are still together at 36 months after random assignment (measured in percentage points). It also shows the minimum detectable impacts (measured in effect sizes) for the impact on a measure of supportiveness that would be coded from the videotape of the parent-child interaction during the Three Bags Task during the in-home assessment. We present the minimum detectable effects for the supportiveness measure separately for the mother-child and father-child interactions as we expect to be able to observe more mother-child interactions than father-child interactions.

It is important that the sample is large enough to be able to detect impacts of a size that we would expect given the intervention. Because BSF is a new and unique intervention, it is difficult to predict the size of its expected impacts. However, we do know the following:

TABLE 2

MINIMUM DETECTABLE IMPACTS AT THE SECOND FOLLOW-UP BY SAMPLE SIZE, FOR KEY OUTCOMES

	Second Follow-Up Telephone Survey of Mothers
Sample Size (Program/Control)	Percent Married (Expected Control Group: 19.5%)a	Percent Still Together as a Couple (Expected Control Group: 64.6%)a
Full Sample
5,277 (2,638/2,638)	2.6%	3.2%
50% Subgroup
2,638 (1,319/1,319)	3.7%	4.5%
30% Subgroup
1,583 (792/792)	4.8%	5.8%
One Site
750 (375/375)	7.0%	8.4%
	In-Home Direct Assessment and Observation
	Supportiveness of Parent During Parent-Child Play in Three Bags Test
Sample Size (Program/Control)	Mother (Effect Size)	Father (Effect Size)
Full Sample
3,000 (1,500/1,500)	0.09	0.10
50% Subgroup
1,500 (750/750)	0.12	0.14
30% Subgroup
900 (450/450)	0.16	0.18
One Site
600 (300/300)	0.20	0.22

Note: Calculations assume: (1) an equal number of program and control members; (2) a 95 percent confidence level with an 80 percent level of power; (3) a one-tail test; (4) a reduction in the variance of 20 percent from the use of regression models; (5) 85 percent of fathers will be in regular contact with the child and hence eligible for the in-home assessment; and (6) response rates of 85 percent for the mother survey, 85 percent for the in-home assessment with the mother, and 80 percent for the in-home assessment of the father.

^a Based on findings from the Fragile Families 36-month follow-up survey.

• The Minnesota Family Investment Program (MFIP)—a welfare reform program not specifically designed to increase marriage rates—increased the marriage rate 36 months after random assignment among unmarried, long-term welfare recipients by 4 percentage points (Miller et al. 2000). We would expect BSF to have a larger impact on marriage rates than MFIP.

• PREP, a relationship-skills program designed to improve couple relationships, increased by 24 percentage points the likelihood that couples were still married three years after the program (Markman et al. 1988). Unlike BSF, the couples were married before the intervention and typically not low-income. This study does suggest, however, that large impacts on marriage are possible.

• Early Head Start—a program designed specifically to improve outcomes for children—had an effect of 0.15 (statistically significant at the 1 percent level) on the mother’s supportiveness as observed during the Three Bags Task and an effect of 0.18 (not statistically significant) on the father’s supportiveness as observed during the Three Bags Task (Love et al. 2002). An intervention such as BSF that affects the stability of the child’s family structure over a period of three years could have impacts at least as large as Early Head Start.

When pooled across sites, the expected telephone survey sample with seven sites and an 85 percent response rate is sufficient to detect statistically significant impacts within the expected size range. With 85 percent of the full sample, we will be able to detect an impact of 2.6 points or more in the percentage of couples who are married and 3.2 points or more in the percentage of couples still together as a couple. The total sample for the in-home assessment is sufficient to detect statistically significant effect sizes of 0.09 and 0.10 for the supportiveness of mothers’ and fathers’ interactions.

The pooled telephone survey sample will also be sufficient to detect expected impacts for important subgroups. We can detect even quite small impacts with a 50 percent subsample. Subgroups that are likely to represent approximately 50 percent of the sample include African Americans, couples who entered BSF prior to the birth of their baby, and couples in which the father earned less than $15,000 in the year prior to baseline. If the impacts are large, we can also detect impacts for subgroups of 30 percent of the sample. Subgroups that are likely to represent approximately 30 percent of the full sample include those with less than a high school education, mothers who were more than 25 years old at baseline, and those who had a child by another partner prior to entering the sample.

Given the telephone survey sample size, impacts for individual sites will only be detected if they are large (such as a 7 percentage point increase in marriage). However, because BSF adheres to a set of detailed, specific guidelines in each site (Hershey et al. 2004), it is meaningful to estimate an impact of the program using pooled data from all seven sites.

Questions about the effects of the implementation of the programs across the sites can be addressed by examining the impacts of BSF on groups of sites. The telephone survey sample size is large enough to examine impacts for groups of two to four sites. These groups of sites could be chosen for the similarities in how BSF is implemented in those sites or for the similarities in the communities in which the program is implemented. A subgroup of couples in sites operating BSF within Healthy Families programs, for example, would comprise about 50 percent of the sample.

b. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

c. Periodic Cycles to Reduce Burden

Respondents will be asked to complete the 36-month telephone survey follow-up and in-home direct assessment only once.

3. Methods to Maximize the Response Rate and to Deal with Nonresponse

We anticipate a total of 5,095 couples will have enrolled in the study when enrollment ends. For the second follow-up telephone survey, our anticipated response rate is 85 percent for mothers (a total of 4,329 mothers from 2008 to 2011) and 80 percent for fathers (a total of 4.074 fathers from 2008 to 2011).

For the in-home direct assessment, our target sample size is 3,000 families. We anticipate the response rate will be 85 percent for children and mothers (an estimated 2,550 children and 2,550 mothers between 2008 and 2011) and 80 percent for fathers who are in regular contact with their child (an estimated 2,040 fathers between 2008 and 2011).

We are updating location information during the first follow-up and will send a mailing between data collections to maintain contact and to keep location information up to date (included in the mailing is MPR’s 800 telephone number and a postage paid postcard that respondents can return if they have changed address). Interviewing and scheduling will take place from MPR’s centralized telephone interviewing facility. For sample members who are hard to reach by telephone, priority letters may be sent. When necessary, field locators will do on-site searching for sample members who could not be reached or located by telephone. Field locators will also attempt refusal conversion of sample members who refused to complete the survey over the telephone. This method is considered the most cost-efficient and will give us the greatest flexibility to follow-up with sample members who are hard to reach. For example, once the sample member is located or agrees to complete the telephone interview, the field locator will contact the telephone center on an available land-line using a toll-free number or on an MPR-provided cellular telephone. The sample member will complete the interview with a telephone interviewer. If warranted, there will also be the opportunity for parents to complete the telephone interview by cellular phone during the in-home direct assessment when not engaged in the parent-child interaction task.

The following approaches will be used to maximize the response rate:

• Telephone interviewers and schedulers will be selected based on past experience and performance in comparable studies with demonstrated skills in communication and refusal conversion

• The telephone interviewers will be supplemented by telephone locators with expertise in locating sample members by telephone

• Experienced on-site staff will be hired from a pool of field data collectors with expertise in locating sample members and promoting participation in the study

• Given that some sample members will require interviews in Spanish, we will hire qualified bilingual interviewers and data collectors to complete interviews in Spanish

Some nonresponse is inevitable. We will conduct an analysis of nonresponse to assess whether the survey sample is representative of the full sample of mothers and fathers and whether the in-home direct assessment sample is representative of the subsample of families targeted. Using the data on characteristics of the couples collected by the baseline information form, we will conduct statistical tests (chi-squared and t-tests) to gauge whether the program group members who participated in data collection are representative of all the program group members, whether the control group members who participated in data collection are representative of all the control group members, and whether there are differences in the program and control group members who responded to the survey and participated in the in-home direct assessment.

We will use two approaches to correct for potential nonresponse bias in the estimation of program impacts. First, the regression models described in Part A Section 16 will adjust for any observed differences between the characteristics of program and control group respondents. Second, because this regression procedure will not correct for differences between respondents and nonrespondents in each research group, we will construct sample weights so that the weighted baseline characteristics of respondents in the program and control group in each site are similar to the full sample (respondents and nonrespondents). These weights will be constructed using data from the baseline information form.

4. Tests of Procedures and Methods to Be Undertaken

A pretest of the survey will be used: (1) to identify typical instrumentation problems such as question wording and incomplete or inappropriate response categories, and (2) to measure the response burden. Our goal is to develop a questionnaire that can be administered to respondents within 55 minutes, on average.

The instrument will be tested in two rounds in early 2008 with couples who received BSF services and couples who did not receive these services. The telephone interviews will be audio-taped and/or monitored to identify: (1) questions the respondents have difficulty understanding, (2) additional response categories that might be appropriate, and (3) wording changes that might improve the clarity of the question intent. As a result of the pretest, we expect to make minor changes to correct errors and improve the wording of the questions and their sequencing. Based on the pretest of the first follow-up survey, we anticipate that respondents generally will not have difficulty answering the questions.

The in-home assessment procedures and instruments will be pretested with a convenience sample of mothers, fathers, and their three year old children local to an MPR facility. The pretest in-home direct assessments will be observed by senior project staff. As a result of the pretest, we expect minor changes to procedures to improve the wording of instructions.

5. Individuals Consulted on Statistical Aspects of the Design

The following persons were consulted on statistical aspects of the study design:

• Irv Garfinkel, Columbia University School of Social Work, 212-854-8489

• John Gottman, Relationship Research Institute, University of Washington, 206-832-0305

• Barbara Devaney, Mathematica Policy Research, 609-275-2389

• Sheena McConnell, Mathematica Policy Research, 202-484-4518

• Robert Wood, Mathematica Policy Research, 609-936-2776

• Peter Schochet, Mathematica Policy Research, 609-936-2783

References

Hershey, Alan, Barbara Devaney, M. Robin Dion, and Sheena McConnell. “Building Strong Families: Guidelines for Developing Programs.” Washington, DC: Mathematica Policy Research, Inc., 2004.

Love, J.M., E.E. Kisker, C.M. Ross, J. Brooks-Gunn, P. Schochet, K. Boller, J. Constantine, C. Vogel, A. Fuligni, and C. Brady-Smith. “Making a Difference in the Lives of Infants and Toddlers and their Families: The Impacts of Early Head Start.” Washington, DC: Administration for Children and Families, June 2002.

Markman, Howard, F. Floyd, S. Stanley, and R. Storaasli. “Prevention of Marital Distress: A Longitudinal Investigation.” Journal of Consulting and Clinical Psychology, vol. 56, no. 2, 1988.

Miller, Cynthia, V. Knox, L. Gennetian, M. Dodoo, J. Hunter, and C. Redcross. “Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family Investment Program. Volume I: Effects on Adults.” New York: MDRC, 2000.

2