Part B Statistical Methods_final

CSPED – Part B

U.S. Department of Health and Human Services Administration for Children and Families Office of Child Support Enforcement (OCSE) Aerospace 4th Floor 901 D Street, SW Washington DC 20447 Project Officer: Elaine Sorensen

Child Support Noncustodial Parent Employment Demonstration (CSPED) New Collection OMB Supporting Statement for Implementation, Cost, and Impact Studies Part B: Statistical Methods AUGUST 2013

1. Respondent Universe and Sampling Methods

In October 2012, the Office of Child Support Enforcement (OCSE) within the Administration for Children and Families (ACF) issued grants to eight state child support agencies to provide employment, parenting, and child support services to non-custodial parents who are having difficulty meeting their child support obligation. The overall objective of the CSPED evaluation is to document and evaluate the effectiveness of the approaches taken by these eight CSPED grantees. All eight grantees that received funding will be included in the evaluation’s two key components: (1) the implementation and cost study; and (2) the impact study.

Implementation and Cost Study

The implementation and cost study will be conducted in all eight CSPED grantees. In this ICR, clearance is sought for four instruments associated with the CSPED implementation and cost study. The sampling approaches for these four instruments are:

1. Staff interview topic guide. Interviews will be conducted with child support staff at the state and local level, as well as staff at their partner agencies that will be providing core CSPED services. Respondents will be selected purposively using organizational charts and information on each employee’s role at the host organization and its partner organizations. At the grantee level, we will interview the grantee director as well as key managers and coordinators. Some grantees are implementing CSPED in multiple counties or communities. For those grantees, we will also interview the grantee’s lead staff member in each community. All grantees are required to partner with other organizations to provide employment and fatherhood services. We will interview the lead staff person responsible for grant activities at each partner agency. In addition, we will interview a sample of frontline case support, employment, and fatherhood staff. If an agency has dedicated more than one frontline staff person to the demonstration, we will randomly pick one of them to interview. Purposeful selection is appropriate for staff selection because insights and information can only come from individuals with particular roles or knowledge.

2. Focus group guide. The evaluation team will conduct two focus groups in each of the eight evaluation sites. The team will use the study MIS to randomly select 15 participants to invite to each focus group. Individuals who have engaged in at least two program activities will be eligible to participate in a focus group.

3. Program staff survey. All staff that work directly with CSPED participants at child support and partner agencies will be asked to complete the web-based survey. It is assumed that 25 staff members will fit this criterion in each of the eight evaluation sites and be asked to complete the survey.

4. Study MIS to track program participation. The study MIS will be used by program staff to document services received by all CSPED program participants during the grant period.

Impact Study

All eight CSPED grantees will be included in the impact study. Non-custodial parents (NCPs) will be eligible for CSPED if they are not stably employed and are behind in their child support payments or appear likely to become so because of inadequate earnings. Grantees are expected to recruit 500 NCPs who are eligible for CSPED each year for three years into the research sample, yielding a total of 1,500 sample members within each site and 12,000 sample members across all eight sites. Half of these NCPs will be randomly assigned to the treatment group and will be offered CSPED services; the other half will be randomly assigned to the control group and will not be offered these services.

Because there is likely to be substantial variation in the CSPED programs offered by the eight grantees, each site will be analyzed separately, although some pooled analysis will also be conducted if such analysis is deemed appropriate once more is known about the similarity of the eight CSPED programs. To support site-level analysis, relatively large samples are required within each site to detect policy-relevant program impacts.

2. Procedures for Collecting Information

a. Statistical Methodology, Estimation, and Degree of Accuracy

Implementation and Cost Study

In this ICR, clearance is sought for four information collection activities associated with the CSPED implementation and cost study. The data collection procedures are described below:

1. Staff interview topic guide (IC #1). Interviews will be conducted with state and local child support staff and local service provider staff during site visits conducted in the first and third years of CSPED implementation. Interviews will be one-on-one or in small groups, depending on the staffing structure, roles, and number of staff in each role. Topics for the first round of interviews will include documenting the service model, implementation system, and inputs to implementation; sample recruitment strategies, challenges, and successes; assessment of early program operations and participant responsiveness; experiences of staff, participants, and community partners; operational challenges and solutions. Topics for the second round of interviews, once service delivery had reached a steady state, will include documentation of program operations, assessment of participant experiences and responsiveness, community partnerships, staff experiences, and adaptations to the service model, and cost data collection.

2. Focus group guide (IC #2). Focus groups will be conducted with program participants during site visits. The objective of the focus groups will be to explore participants’ perspectives on the availability, quality, and value of program services. The focus groups will be used to learn about participants’ motivations for enrolling in the program, their participation and response to incentives offered, their experiences with each of the core services offered, their perceptions of the benefits of participation, and their overall satisfaction with program services.

3. Program staff survey (IC #3). The web-based program staff survey will collect systematic information from a larger number of staff than would be possible during site visit interviews. The survey will be conducted twice, once in the first year and again in the third year of CSPED implementation. Topics in the staff survey include prior experience providing core CSPED services, understanding of the goals of the demonstration, collaboration with community partners, co-location of services, staff training, supervision, and the importance of different core services for the target population.

4. Study MIS to track program participation (IC #4). The MIS will be used in each site to document service use by program participants. The web-based system will allow program staff to document all individual service contacts with participants, referrals to other community services, incentives provided, and participation in group activities such as peer support groups or job readiness workshops.

Information for the implementation and cost study will be descriptive. In general, it will not involve formal hypothesis testing.

Impact Study

The data collection procedures for the CSPED impact study instruments are described below:

1. Introductory Script, Baseline Survey, and Study MIS to Conduct Random Assignment (IC #5, 6, 7, 8). In the eight evaluation sites, program staff will identify noncustodial parents (NCPs) who are eligible for CSPED services. Intake workers at each program will meet with eligible and interested NCPs in person to enroll them in the study sample. They will use the introductory script to describe the study to potential enrollees. When intake workers are ready to enroll an NCP into the CSPED study, program staff will call the University of Wisconsin Survey Center to connect with a trained interviewer who will administer the consent form and conduct the baseline survey. Once the baseline survey is complete, the interviewer will instruct the NCP to hand the phone back to the intake worker, who then will use the Study MIS to confirm that the NCP is eligible for random assignment and to conduct the random assignment process.

2. Protocol for Collecting Administrative Records (IC #9). The evaluation team will collect administrative records pertaining to the activities of study participants. These administrative records include: (1) child support records collected from state child support enforcement agencies, (2) wage and unemployment insurance benefit records from state labor agencies, (3) public assistance benefit records from state human services agencies, and (4) criminal justice records from state and county criminal justice agencies. To acquire these data, project staff will coordinate with staff at the relevant agency regarding data availability and transfer protocols. Data will be transferred to the CSPED evaluation team following all appropriate confidentiality procedures for handling sensitive data.

In addition, a follow-up survey will be conducted approximately one year after random assignment. This follow-up survey will be included in a later ICR, once that instrument is finalized. The sample for the one-year follow-up survey will include the approximately 1,500 sample members within each site who enroll in the study. A response rate of 80 percent is anticipated for the follow-up survey, or approximately 1,200 survey respondents within each site. To ensure a high response rate for the follow-up survey, study participants will be contacted approximately 6 months after random assignment via a text message or email to remind them about the study and to request updated contact information. The language for the text message/email is presented in Attachment D. All analysis of follow-up survey data will account for survey nonresponse using nonresponse weights calculated using standard techniques to estimate the probability of nonresponse as a function of baseline characteristics. Administrative records data on child support outcomes, earnings and employment, TANF, SNAP, and Medicaid benefit receipt, and criminal justice outcomes will be gathered on all 1,500 sample members within each site.

Table B.1 reports site-level minimum detectable impacts on two illustrative binary outcomes—one with 50 percent prevalence (such as employment within a quarter) and another with 20 percent prevalence (such as whether a child support order was modified over the follow-up period). The table also includes minimum detectable impacts on annual earnings, child support payments, and a continuous father engagement scale. Separate estimates are presented for measures based on follow-up survey data and administrative records data, as well as for site-level and pooled analysis.

Both the survey and administrative records samples yield adequate statistical power for detecting impacts on key outcomes within each evaluation site. In addition, the administrative records sample will be large enough to support some limited subgroup analysis at the site level, with adequate statistical power to examine 50 percent subgroups. If CSPED programs are similar enough across evaluation sites that pooled analysis is appropriate, both the survey and administrative records samples can support subgroup analysis.

Table B.1. Minimum Detectable Impacts for Key Outcomes

Sample	Employed in the Quarter (%) (Mean 50%)	Had Child Support Order Modified (%) (Mean 20%)	Annual Earnings ($) (SD = $14,717)	Monthly Child Support Payments Made ($) (SD =$217)	Father Engagement Scale (Effect Size)
Follow-up Survey (80% response rate)
Site-level, full sample (N=800)	7.9	6.3	$2,315	$34	0.16
Pooled, full sample (N=6,400)	2.8	2.2	$818	$12	0.06
Pooled, 25% subgroup (N=1,600)	5.6	4.4	$1,637	$24	0.11
Administrative Records
Site-level, full sample (N=1,500)	5.7	4.6	$1,690	$24	N/A
Site-level, 50% subgroup (N=750)	8.1	6.5	$2,390	$35	N/A
Pooled, full sample (N=12,000)	2.0	1.6	$590	$9	N/A
Pooled, 25% subgroup (N=3,000)	4.1	3.2	$1,195	$18	N/A

Note: Figures assume a 50-50 split of sample members into program and control groups, a two-tailed test with a 95-percent confidence level and 80-percent power, and an R-squared in the impact regression of 0.20. N/A = not applicable (measure not available from administrative records). Standard deviations for annual earnings and the amount of monthly child support payments made are drawn from the Building Strong Families evaluation (Wood et al. 2012).

b. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

c. Periodic Cycles to Reduce Burden

There will be only one cycle of baseline data and one cycle of follow-up data collection.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Implementation and Cost Study

Study MIS to track program participation. To maximize response rates and data reliability for the study MIS, we will take these steps:

• Develop a user-friendly, flexible MIS. The MIS will be specifically designed for use by grantee site staff. As such, it will be extremely user-friendly and flexible, so as to meet the needs of each site. Making the system simple and easy to use will improve the quality of the data collected. In addition, by providing sites with this system, we standardize the information being collected from each site and improve the reliability of our implementation and impact components.

• Include data quality checks in the MIS. The MIS will also ensure data reliability by instituting automatic data quality checks. For example, if grantee staff enter odd or unlikely values in a particular field, the system will prompt users to check the value. For some fields, the response values will be restricted; for others, grantee staff site will be able to override the check.

• Provide extensive training to grantee site staff. To increase data quality, we will provide extensive training to system users prior to initial use. Initial training will be on site; follow-up training will be conducted using web and telephone conferences. Following training, CSPED team members will conduct follow-up site visits to ensure compliance with procedures and be available by phone and email to assist users.

• Monitor data quality. We will also monitor the data entered by grantee sites and provide feedback to grantees on their data quality. Initially, we will monitor data quality on a weekly basis, tapering that gradually to monthly monitoring, as agencies demonstrate their ability to use the system correctly.

Impact Study

Baseline Survey. In evaluation sites, to maximize response rates and data reliability for the baseline survey effort, we will take the following steps:

• Use a tested questionnaire common to all sites. While the CSPED baseline survey has been tailored to the specific circumstances of the CSPED evaluation, it is based closely on the Parents and Children Together (PACT) baseline survey, an ACF initiative which has received OMB approval, was extensively tested, and is currently being fielded. The alignment of the CSPED baseline survey with the PACT baseline is described in Attachment B.

• Use a straightforward, undemanding survey. The CSPED baseline survey is designed to be easy to complete. The questions use clear and straightforward language. The average time required for the respondent to complete the survey is estimated at 30 minutes.¹

• Administer the survey using computer-assisted technical interviewing (CATI). Administering the baseline survey via CATI will maximize the reliability of the data entered by telephone interviewers through skip-pattern logic and checks for consistency and validity.

• Use trained interviewers. Respondents will be interviewed by trained members of the University of Wisconsin Survey Center’s survey operations staff, many of whom have significant experience working on similar studies. All survey staff assigned to the study will participate in both general training (if they are not already trained) and an extensive project-specific training. Interviewers will not work on the study until they have been certified as prepared. The project-specific training will include role playing with scenarios and other techniques to ensure that interviewers are ready to respond effectively to sample members’ questions. They will also focus on developing skills for securing respondents’ cooperation and averting and converting refusals.

• Be able to administer the survey in multiple languages. During telephone contact, interviewers will identify Spanish-speaking respondents and connect them to speak with a certified Spanish language interviewer. The University of Wisconsin Survey Center (UWSC) employs staff who have experience conducting interviews in Spanish. If once sample enrollment is underway, the evaluation team determines that interviews will need to be conducted in languages other than English or Spanish, UWSC will hire interviewers with the necessary language skills.

• Provide payments for survey participants. We plan to offer a modest $10 payment to baseline survey respondents to increase program applicants’ agreement to participate in the study and to reduce attrition for follow-up data collection. (This is discussed in greater detail in Section A9.)

We anticipate high response rates to the baseline survey, since only those who complete the baseline survey will be randomly assigned. Based on experience with PACT, Building Strong Families, and other prior research studies, the evaluation team anticipates that 95 percent of those offered the opportunity to enroll in the CSPED study will agree to participate in the evaluation (consent) and that 100 percent of those who do consent will complete the baseline survey as part of the intake process. Also, the evaluation team does not anticipate significant item nonresponse on the baseline survey based on prior experience asking similar questions with similar populations.

4. Tests of Procedures or Methods to be Undertaken

Study MIS. All functions of the automated version of the MIS system will be rigorously tested and evaluated by the development team to ensure proper functionality. Additionally, we will consult with practitioners on the usability of the system and engage these practitioners in the testing phase.

Program Staff Survey. The evaluation team will carefully test the web-based version of the instrument to ensure that skip patterns have been correctly programmed and the flow through the instrument is working properly.

Baseline Survey. As described in Attachment B, most questions on the CSPED baseline survey were drawn from the baseline survey used in the PACT evaluation. In-person cognitive interviews and telephone pretests of the PACT baseline survey were conducted to ensure that questions were understood and were consistent with the concepts they aim to measure; to identify typical instrumentation problems such as question wording and incomplete or inappropriate response categories; to measure the response burden; and to check that there are no unforeseen difficulties in administering the instrument via telephone.

5. Individuals Consulted on Statistical Methods

Preliminary input on statistical methods was received from staff in OCSE as well as staff at the University of Wisconsin and Mathematica Policy Research, including the following individuals:

Dr. Maria Cancian

University of Wisconsin-Madison

Institute for Research on Poverty
1180 Observatory Drive
Madison, WI 53706

Dr. Dan Meyer

University of Wisconsin-Madison

School of Social Work

1350 University Ave.

Madison, WI 53706

Dr. Robert Wood

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543

In the future, further input on analytic approaches may be sought from additional staff at these organizations, and from outside consultants.

1 As noted in burden tables, we assume that NCPs will be on the telephone with survey interviewers for a total of 35 minutes to account for the approximately 5 minutes it will take for them to complete the consent process before beginning the baseline survey.

6