PACT follow up supporting statement B - 11.22.13 - clean

U.S. Department of Health and Human Services Administration for Children and Families Office of Planning, Research, and Evaluation Aerospace 7th Floor West 901 D Street, SW Washington DC 20447 Project Officers: Nancye Campbell and Seth Chamberlain	Parents and Children Together (PACT) Evaluation (0970-0403): OMB Supporting Statement for the Follow-up Surveys Part B: Collection of Information Involving Statistical Methods September 2013

CONTENTS

1. Respondent Universe and Sampling Methods 1

2. Procedures for Collecting Information 2

a. Statistical Methodology, Estimation, and Degree of Accuracy 2

b. Unusual Problems Requiring Specialized Sampling Procedures 4

c. Periodic Cycles to Reduce Burden 4

3. Methods to Maximize Response Rates and Data Reliability 4

4. Tests of Procedures or Methods 6

5. Individuals Consulted on Statistical Methods 6

REFERENCES 8

(Please see item A1 for a short description of the impact and implementation/qualitative only evaluations, as well as the data collection instruments already approved and currently requested, which are numbered (1) through (20); numbers in parentheses refer to the number of the instrument.)

1. Respondent Universe and Sampling Methods

Grantee Programs. The PACT Evaluation will focus on grantee programs purposively selected for the study. Up to 15 responsible fatherhood (RF) grantees and 15 healthy marriage (HM) grantees are expected to be selected across the impact and implementation/qualitative only sites.

Sample Frame for the Follow-up Surveys. The follow-up surveys will be used only for fathers and couples in the grantee programs selected for the impact study (RF programs and HM programs).

To participate in the impact study, fathers or couples must:

• Apply to a grantee program in the impact study and satisfy the grantee program’s eligibility criteria

• Satisfy additional criteria for participating in the study. Fathers are eligible for the RF impact study if they are 18 or older and have biological or adopted children or are expecting a baby. Couples are eligible for the HM impact study if they consist of a man and a woman who apply to the program together, the man and the woman are both 18 or older, and either (or both) the man or the woman has a biological or adoptive child living with him or her or they must be expecting a baby together.

• Consent to participate in the study

• Complete a baseline survey

The intake period for the study will be about 18 months for the RF grantees and 12 months for the HM grantees.

We estimate that about 421 fathers and 421 couples will apply for services in each grantee program during the intake period. Of these 421 fathers or couples, we estimate that about 400 will be eligible to participate in the program and the study, consent to participate in the study, and complete the baseline survey. These fathers or couples will be randomly assigned to either a program or control group. Follow-up survey data collection will be attempted on all 400 fathers in each grantee program’s research sample (both program and control groups) and 400 men and 400 women from each HM grantee sample’s research sample (both program and control groups).We expect that follow-up data will be successfully collected on 320 participant fathers in the RF sample per grantee program, 320 men in the HM sample per grantee program, and 320 women in the HM sample per grantee program (that is, the response rate will be about 80 percent). Our response rate estimate is based on the experience in prior studies funded by OPRE with similar populations.  Only the Baseline Survey has been fielded to date.  The Baseline interview completion rates are: 98% for the Responsible Fatherhood Survey and 99% for the Healthy Marriage Survey..

2. Procedures for Collecting Information

a. Statistical Methodology, Estimation, and Degree of Accuracy

A sample of 400, which we expect to be the grantee-level sample size at baseline, is large enough to detect impacts on key outcomes for both the RF and HM impact evaluations. As Table B.1 shows, with a single-site sample of 400 fathers (200 in the program group and 200 in the control group) with baseline data and 320 fathers with follow-up data, we are confident of detecting impacts on continuous outcomes that have an effect size of 0.20 or larger. This is sufficient to be able to detect impacts on fathers’ attitudes toward fatherhood. Cowan et al. (2009) found an effect size of 0.31 of a fatherhood program on a measure indicating the extent to which fathers viewed fatherhood as one of the main roles in their lives. A sample of 400 is also large enough to detect an impact on employment of 6 percentage points, an impact smaller than the one found in a pilot employment program for parents behind in their child support in four communities in New York (Lippold and Sorensen 2011).

Table B.1. Minimum Detectable Impacts for Key Outcomes in the RF Impact Studies

Sample Size (Fathers) (Baseline/Follow-up)	Continuous Outcome (effect size)	Fathers’ Likelihood of Employment (percentage points) Control = 0.11a	Fathers’ Annual Earnings Control Group Std Dev = $14,717 b
400/320	0.20	0.06	$2,893
600/480	0.16	0.05	$2,362
800/640	0.14	0.04	$2,046
1,800/1,440	0.09	0.03	$1,364

Note: We assume an effective response rate of 80 percent, and a 50-50 split of sample members into program and control groups. All calculations assume a 95-percent confidence level, 80-percent power, and a one-tailed test. We assume an R-squared in the impact regression of 0.50.

^a. Lippold and Sorensen (2011).

^b. Building Strong Families Study.

For the HM impact evaluation, a sample of 400 couples at baseline, which we expect to be the grantee-level sample size, is also large enough to detect impacts on key outcomes. As Table B.2 shows, with a single-site sample of 400 couples (200 in the program group and 200 in the control group) with baseline data and 320 couples with anticipated 12-month follow-up data (80 percent response rate), we are confident of detecting impacts on continuous outcomes, such as relationship quality scales or parenting scales, of effect sizes of 0.20 or larger. This sample size is sufficient to detect the impact found on relationship quality (0.31 standard deviations) in the Oklahoma site of Building Strong Families at 15 months (Wood et al. 2012).

Table B.2. Minimum Detectable Impacts for Key Outcomes

Sample Size (Couples) (Baseline/Follow-up)	Likelihood of Couple Being Still Romantically Involved (Percentage Points) Control = 0.76a	Continuous Outcome (Effect size)
400/320	8.4	0.20
600/480	6.9	0.16
800/640	5.9	0.14
1,800/1,440	4.0	0.09

Note: We assume an effective response rate of 80 percent, and a 50-50 split of couples into program and control groups. All calculations assume a 95-percent confidence level, 80-percent power, and a one-tailed test. We assume an R-squared in the impact regression of 0.50.

^a. Wood et al. 2012

However, a sample size of 400 fathers or couples per site may not be sufficient for subgroup analysis at the site level, as fewer than 400 fathers or couples per site will belong to any particular subgroup. To conduct subgroup analysis, we will need to be able to pool samples from two or more sites (depending on the size of the subgroup). Pooling sites will also allow us to measure impacts on outcomes that are more variable, such as earnings, and will allow us to measure smaller impacts. Past evaluations have demonstrated: (1) effect sizes of 0.10 or greater on relationship outcomes (Wood et al. 2012; Hsueh et al. 2012) and (2) impacts of $1,308 in increased earnings (Schochet et al. 2006). A sample of 1,440 completed follow-up surveys will position the evaluation to detect impacts of about this size. Furthermore, a sample of 1,440 will permit subgroup analyses of 320 or about 22 percent.

Based on previous experience, we are confident that an 80 percent response rate for the 12-month follow-up data collection can be achieved. The response rate for the 15-month follow-up survey for the Building Strong Families Study was 72 percent for fathers and 83 percent for mothers; at least one member of the couple responded in 87 percent of couples (Wood et al. 2012). We expect to achieve a higher response rate for fathers and mothers in PACT than in Building Strong Families for four reasons: (1) we are conducting the follow-up interview at 12 months after random assignment rather than 15 months after random assignment; (2) the baseline survey will be conducted by telephone by a trained interviewer who can collect more detailed and accurate contact information than the grantee staff members who administered the Building Strong Families baseline survey; (3) the PACT baseline survey will collect both email and social media addresses, which were not collected in the Building Strong Families Study; and (4) a reminder about the study and a request for updated contact information will be texted or emailed to respondents at about 6 months after random assignment (these are included in Appendix K and have already been approved).

b. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

c. Periodic Cycles of Data Collection

There will only be one cycle of follow-up surveys.

3. Methods to Maximize Response Rates and Data Reliability

To maximize response rates and data reliability for the follow-up survey, we will take the following steps:

• Use a pretested, straightforward, undemanding questionnaire. While the follow-up questionnaire is unique to the current evaluation, many of the questions included in the survey have been used successfully in prior studies. The questions use clear and straightforward language. The questionnaire has been extensively reviewed and reflects information obtained through cognitive interviews or pretests with nine individuals for the RF Follow-up questionnaire and nine individuals for the HM Follow-up questionnaire who have backgrounds similar to anticipated PACT Evaluation study participants. The same follow-up survey will be used across all telephone interviewers and PACT program sites, ensuring consistency in data collection. The average time required for the respondent to complete the survey is estimated at 45 minutes.

• Administer the survey using computer-assisted telephone interviewing (CATI). Administering the follow-up survey via CATI will maximize the reliability of the data entered by telephone interviewers through skip-pattern logic and checks for consistency and validity.

• Use trained experienced interviewers. Respondents will be interviewed by trained members of Mathematica’s survey operations staff. Most of these staff are familiar with similar questionnaire content. All survey staff assigned to the study will participate in both general training (if they are not already trained) and an extensive project-specific training. Interviewers will not work on the study until they have been certified as prepared. The project-specific training will include role playing with scenarios and other techniques to ensure that interviewers are ready to respond effectively to sample members’ questions. They will also focus on developing skills for securing respondents’ cooperation and averting and converting refusals.

• Reminder messages to sample members. Mathematica staff will send reminder text messages about the follow-up survey to sample members who have provided permission for Mathematica to contact them via text messages. The reminder text messages will include the toll-free number for sample members to call-in to complete the survey. We will also send similar reminder messages via email. We will send up to three reminder text or email messages to sample members. (These reminder messages were included in the ICR package approved on August 27, 2013).

• Send an advance letter. Prior to attempting to conduct the survey, Mathematica will send a letter to the sample member’s mailing address to remind them about the survey. The advance letters for the RF and HM sample members are included in Appendix L. The letter will include the toll-free number so sample members can call-in to complete the survey.

• Use specialized expert locating staff. Specially-trained staff at Mathematica will utilize multiple database searchers to find additional, updated contact information for sample members (e.g., address, telephone number) for whom attempts at contacting are unsuccessful. Attempts will then be made to contact sample members using the new contact information.

• Use trained field locators. Skilled field locators will be used to locate sample members who we are unable to be reached via telephone. The field interviewers will start their locating effort in the area of sample members’ last known residence. Once found, the field interviewer will use a cell phone to connect the sample member to the telephone interviewing staff to complete the survey at that time. Field locators will also attempt refusal conversion of sample members who refused to complete the survey over the telephone. Once the sample member is located or agrees to complete the telephone interview, the field locator will contact the telephone center on an available land-line using a toll-free number or on a Mathematica-provided cellular telephone. The sample member will complete the interview with a telephone interviewer.

• Be able to administer the survey in multiple languages. At the beginning of the call, interviewers will identify Spanish-speaking respondents and connect them to a bilingual interviewer. When necessary, translators for languages other than Spanish will be used. Mathematica employs staff who speak a wide range of languages and have experience conducting interviews in a number of languages.

• Provide appreciation for survey participants. We suggest offering a modest $25 in appreciation to follow-up survey respondents to increase response rates. (This is discussed in greater detail in Question A9.)

We anticipate high response rates to the follow-up survey. We anticipate that 80 percent of sample members will agree to complete the follow-up survey. Likewise, we do not anticipate significant item nonresponse on the baseline survey based on prior experience asking similar questions with similar populations.

Some nonresponse is inevitable. Mathematica will conduct an analysis of nonresponse to assess whether the survey sample is representative of the full sample of fathers in the RF research sample and men and women in the HM research sample. Using the data on characteristics of the couples collected by the baseline surveys, Mathematica will conduct statistical tests (chi-squared and t-tests) to gauge whether the program group members who participated in data collection are representative of all the program group members, whether the control group members who participated in data collection are representative of all the control group members, and whether there are differences in the program and control group members who responded to the survey.

We will use two approaches to correct for potential nonresponse bias in the estimation of program impacts. First, the regression models described in Part A Section 16 will adjust for any observed differences between the characteristics of program and control group respondents. Second, because this regression procedure will not correct for differences between respondents and nonrespondents in each research group, we will construct sample weights so that the weighted baseline characteristics of respondents in the program and control group in each site are similar to the full sample (respondents and nonrespondents). These weights will be constructed using data from the baseline surveys.

4. Tests of Procedures or Methods

A pretest of the survey was used: (1) to identify typical instrumentation problems such as question wording and incomplete or inappropriate response categories and (2) to measure the response burden. Each instrument was tested in two rounds in summer 2013 with fathers who had participated in a responsible fatherhood program and men and women who had participated in a healthy marriage programs. The first four interviews were conducted as cognitive interviews. The interviewed explored whether respondents understood the questions or had difficulty answering the questions. As a result of the four cognitive interviews for both the RF and HM surveys, the survey instruments were revised. Some questions were revised to be easier to understand, some questions were removed, and response categories were added as needed. The revised surveys were then pretested on another five fathers and another five participants of HM programs. As a result of the pretest, we made minor changes to correct errors and improve the wording of the questions and their sequencing.

5. Individuals Consulted on Statistical Methods

Input on statistical methods on statistical methods was received from staff in the ACF Office of Planning, Research, and Evaluation as well as staff at Mathematica Policy Research and project and a limited number of staff external to Mathematica.

Ms. Nancye Campbell

Administration for Children and Families

U.S. Department of Health and Human Services

7th Floor West

901 D Street, SW

Washington, DC 20447

Mr. Seth Chamberlain

Administration for Children and Families

U.S. Department of Health and Human Services

7th Floor West

901 D Street, SW

Washington, DC 20447

Ms. Kathleen McCoy

Administration for Children and Families

U.S. Department of Health and Human Services

7th Floor West

901 D Street, SW

Washington, DC 20447

Dr. Sheena McConnell

Mathematica Policy Research

1100 1^st Street, NE, 12^th floor

Washington, DC 20002-4221

Dr. Robert Wood

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543

Dr. Jane Fortson

Mathematica Policy Research

505 14th Street

Suite 800

Oakland, CA 94612

Ms. M. Robin Dion

Mathematica Policy Research

1100 1^st Street, NE, 12^th floor

Washington, DC 20002-4221

Ms. Heather Zaveri

Mathematica Policy Research

1100 1^st Street, NE, 12^th floor

Washington, DC 20002-4221

Mr. Shawn Marsh

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543

Dr. Cleo Jacobs-Johnson

Mathematica Policy Research

505 14th Street

Suite 800

Oakland, CA 94612

REFERENCES

Cowan, Philip A., Carolyn Pape Cowan, Marsha Kline Pruett, Kyle Pruett, and Jessie Wong. “Promoting Fathers’ Engagement with Children: Preventive Interventions for Low-Income Families.” Journal of Marriage and Family, vol. 71, August 2009, pp. 663–679.

Hsueh, JoAnn, Desiree Principe Alderson, Erika Lundquist, Charles Michalopoulos, Daniel Gubits, David Fein, and Virginia Knox (2012). “The Supporting Healthy Marriage Evaluation: Early Impacts on Low-Income Families.” OPRE Report 2012-11. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Lippold, Kye, and Elaine Sorensen. “Strengthening Families Through Stronger Fathers: Final Impact Report for the Pilot Employment Programs.” The Urban Institute, October 2011.

Schochet, Peter Z., John Burghardt, and Sheena McConnell. “National Job Corps Study and Longer-Term Follow-Up Study: Impact and Benefit-Cost Findings Using Survey and Summary Earnings Records Data.” Princeton, NJ: Mathematica Policy Research, August 2006.

Wood, Robert G., Quinn Moore, Andrew Clarkwest, Alexandra Killewald, and Shannon Monahan (2012). The Long-Term Effects of Building Strong Families: A Relationship Skills Education Program for Unmarried Parents, OPRE Report # 2012-28A, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services (OPRE).