SHM Control Services OMB Part B

Contract No.: 223-03-0034

Contract Amount: $74,938,364

SUPPORTING STATEMENT

FOR OMB CLEARANCE: PART B

DHHS/ACF

SUPPORTING HEALTHY MARRIAGE (SHM)

PROJECT EVALUATION

LOW-INCOME MARRIED COUPLES DATA COLLECTION ACTIVITIES –

CONTROL SERVICES SURVEY

August 22, 2007

Prepared for:	Prepared by:
Department of Health and Human Services Administration for Children and Families	MDRC
370 L’Enfant Promenade, SW	16 East 34th Street, 19th Floor
Washington, DC 20447	New York, NY 10016
Phone: 202-401-5372	Phone: 212-340-8678
Project Officer:	Principal Investigator:
Mark Fucello	Virginia Knox

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B1. Sampling and Analysis

The control services survey sample will contain between 86 and 174 respondents in each site. The size of the sample will be determined by the size of the pilot in each site. We will need somewhat larger samples in sites that are recruiting and serving sample members in more than one location in order to examine service receipt by sample members separately in each location of the study.  Thus, sites with one location will have 86 respondents; sites with two locations will have 114 respondents, and the site with 3 locations will have 174 respondents. MDRC plans to randomly select couples from our SHM pilot sample for this survey. We plan to analyze this survey data primarily for each individual site rather than pooling survey responses. Because the purpose of this analysis is to identify any location-specific problems with control group members receiving extensive services that are similar to SHM, the universe of respondents for the survey will include individuals of who differ by age, race, income, level of marital distress, and a host of other factors.

The analysis for the control services survey will be primarily descriptive. We will analyze the data to determine how many couples received marriage education, marriage counseling or therapy, or other related services. For respondents who are in the program group, we will also examine their attendance at SHM services and their reasons for absences, if applicable. The data will be used to produce descriptive tables, percents, means, and distributions for the entire sample in each site. We will also compare the program group responses to the control group responses using appropriate statistical tests. If we determine that the control group is receiving significant levels of marriage education or marriage counseling services, we may investigate these services further using qualitative methods.

B2. Procedures for Collection of Information

The following approaches will be used to collect the control services survey data:

1. MDRC will compile a list of potential survey respondents and contact information using information entered by site staff in the SHM MIS. MDRC will provide this list to Abt Associates, the survey administrators, about three to six months after the start of the site’s SHM pilot.

2. Interviewers from Abt Associates’ CATI Center will track, locate, and contact respondents to complete the control services survey. Prior to contacting potential respondents, Abt will send them a letter about the control services survey (see Attachment C).

3. Interviewers will then contact respondents to complete the brief, approximately 10-minute control services survey using the CATI system. All surveys will be completed in Abt’s CATI Center. Abt Associates will then mail respondents a $10 incentive for completing the survey.

4. A preliminary data file will be created in the first few months of data collection and provided, with documentation, to MDRC.

B2.1 Procedures for the Control Services Data Collection

Interviewers. In all SHM sites, the control services survey will be administered by professional interviewers employed by Abt Associates.

Training Site Staff. Abt Associates will train interviewers in how to administer the control services survey and utilize the CATI software. To ensure quality assurance, the FMS selects each and every interviewer’s first interview for re-contact; subsequent selection is at 10-15 percent of every interviewers completed work

Conducting Interviews. Abt Associates’ professional interviewers will handle all survey interviews. All interviews will be conducted by phone from Abt’s CATI Center. Individuals identified as potential sample members will be contacted by the interviewers, first by letter and then by phone. At that time, they will initiate the survey session and answer any questions about the survey that sample members might have. The interviewer will read a script that provides sample members with assurances of confidentiality and discusses their rights as study participants. The interviewer will then administer the brief control services survey.

B3. Maximizing Response Rates

Our goal is to administer the control services survey to all pilot participants in each site. We aim to achieve a response rate of 80 percent. We are confident that the sample achieved with an 80 percent response rate will be adequate for the analyses that we plan to conduct with these data.  Given the small sample sizes for each location, the data will not be used to make statistical comparisons but rather to examine trends in receipt of services for each group in each location.  This will serve our purposes of understanding whether a high proportion of the control group members in these locations are accessing services that are similar to SHM soon after entering the control group.  Procedures for obtaining the maximum degree of cooperation include:

• Conveying the purposes of the survey to respondents so they will thoroughly understand the purposes of the data collection and perceive that cooperating is worthwhile;

• Sending a letter to respondents prior to contacting them by phone to explain the purpose of the survey;

• Providing a toll-free number for respondents to use to ask questions about the survey;

• Training interviewers to be encouraging and supportive, and to provide assistance to respondents as needed;

• Training interviewers to maintain any pre-existing one-on-one personal rapport with respondent; and

• Offering appropriate payments to respondents.

In addition to the above procedures, use of the CATI technology can positively affect response rates by enabling people with limited literacy skills (particularly important given that the low-income married population is disproportionately Latino) to respond to survey questions.

B4. Pre-testing

Many of the questions proposed for this instrument are either identical to questions used in prior MDRC evaluations or are similar, if not identical, to questions used in previous national evaluations. Consequently, many of the items have been thoroughly tested on larger samples.

The proposed SHM control services survey instrument has undergone a number of revisions, following critiques by internal staff and by staff at HHS. Revisions were also made on the basis of a pre-test that assessed the comprehensibility of the draft survey instruments. Survey administrators at Abt Associates analyzed pre-test results and recommended appropriate revisions to the survey instrument.

A total of nine respondents from the Oklahoma City, OK, SHM demonstration completed pretest interviews:  four in the program group and five in the control group.  Respondents were a mix of men and women, none of whom were part of the same married couple. The pretests provided information about the length of the control service survey. The averages (number of minutes) from the pretest were used in computing estimates of respondent burden, as described in Section A12. Actual pretest averages were close to the targeted (and budgeted) times for instrument administration, and so it was not necessary to delete any questions.

The pretests were also undertaken with the goal of improving the quality of the data the instruments would yield, and thus great care was taken in gleaning information about question wording and the order of questions. Following each pretest, interviewers were debriefed about problems they encountered and about their recommendations for improving the instruments. Based on Abt Associates’ pretest results, a number of revisions were made to the instrument. Some of these revisions concerned the logic of the instrument’s skip patterns, and others involved improving/clarifying (and often simplifying) the wording of questions.

B5. Consultants on Statistical Aspects of the Design

There are no consultants on the statistical aspects of the design. We have drawn on the considerable expertise of the SHM team members including Dr. Charles Michalopoulos, Dr. Virginia Knox, and Dr. JoAnn Hsueh of MDRC.

2