OMB HtE PA 36M_Supporting Statement Part B REV_11-05-07

Contract No.: HHS 233-01-0012

Contract Amount: $23.78 million

SUPPORTING STATEMENT

FOR OMB CLEARANCE: PART B

DHHS/ACF/ASPE/DOL

ENHANCED SERVICES FOR THE HARD-TO-EMPLOY (HtE)

DEMONSTRATION AND EVALUATION PROJECT

PHILADELPHIA 36-MONTH DATA COLLECTION INSTRUMENT

September 26, 2007

Prepared for:
U.S. Department of Health and Human Services
	Office of Assistant Secretary for Planning
Administration for Children and Families	and Evaluation
370 L’Enfant Promenade, SW	200 Independence Avenue, SW
Washington, DC 20447	Washington, DC 20201
Phone: 202-401-5070	Phone: 202-260-0384
Project Officer: Girley A. Wright	Project Officer: Flavio Menasce
U.S. Department of Labor	Prepared by:
Employment and Training Administration	MDRC
200 Constitution Avenue, NW	16 East 34th Street, 19th Floor
Washington, DC 20210	New York, NY 10016
Phone: 202-693-3654	Phone: 212-532-3200
Project Officer: Roxie Nicholson	Project Directors: David Butler/Barbara Goldman

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B1. Sampling

The follow-up sample will consist of 1,555 participants. The follow-up sample estimate is based on the assumption that 80 percent of the research sample will be successfully interviewed.

The evaluation literature often discusses the appropriateness of the sample size for a study by focusing on the smallest program impacts that are likely to be detected with a specified level of confidence, assuming a sample of a given size and characteristics. These are usually called the program’s “minimum detectable effects” (MDEs). Analysis of MDEs is also referred to as “power analysis,” as it estimates the study's power to measure the effects it was designed to find.

As a guide to determining appropriate sample sizes, Exhibit B1-1 reports the estimated minimum detectable effects for five sample sizes. In this case, the minimum detectable effect is the smallest true effect that would generate statistically significant impacts in 80 percent of evaluations with a given sample size. The table assumes a two-group impact estimate, as the research design calls for comparing each of the program groups with the control group but does not call for comparing the two program groups with each other or for comparing all three groups. The calculations remain the same whether the program group is the TWC group or the STEP group because the groups have similar characteristics (assured by the random assignment design) and sample sizes. Also note that, as the sample size column indicates, the maximum control group sample is 487 because this is the entire control group (the calculations assume an 80% response rate, or 390 control group respondents).

Each number in the exhibit represents the smallest true program effect that is likely to generate statistically significant impacts with the given sample sizes. For example, with a sample size of 500 TWC or STEP group members and 487 control group members, the expected minimum detectable effect on the percentage employed at the 36-month follow-up survey is 8.8 percentage points. In other words, program effects for this outcome can be detected reliably only if they exceed 8.8 percentage points.¹

Exhibit B1-1
Minimum Detectable Effects
Philadelphia HTE Evaluation
		Outcomes
Sample Size of Program and Control Group		Effect size number	Employed at the survey interview (%)	Weekly earnings at the survey interview ($)
300/300		0.23	11.3	14.72
400/400		0.20	9.8	12.75
500/487		0.18	8.8	11.48
600/487		0.17	8.5	11.00
700/487		0.16	8.2	10.64

B2. Procedures for Collection of Information

The 36-month follow-up survey data will be collected through a mixture of telephone and in-person outreach and interviewing strategies to maximize response rates. MDRC will work with HumRRO to develop strategies that ensure an 80 percent response rate.

All completed interviews will be reviewed to ensure all applicable fields are correctly completed and that all relevant interviewer notes are included in the data set. Any open ended and “other, please specify” items will be coded based on codes developed by HumRRO and approved by MDRC. Preliminary data files will be created and shared – with documentation – with MDRC on an agreed-upon schedule.

B2.1 Procedures for the Administration of the Survey

Interviewer selection. MDRC will work with HumRRO to ensure that the interviewers administering the follow-up surveys are professional interviewers, many of whom have worked on social science research projects. Prefer­ence will be given to those who are multilingual, depending on the languages spoken by the research sample. Familiarity with the special requirements of interviewing low-income populations will be desirable. New personnel will be trained along with the seasoned interviewers.

Interviewer training. MDRC will work with HumRRO to ensure sufficient interviewer training. Staff administering the HTE Philadelphia 36-month survey will be required to attend a three-day training sessions. Personnel who are new to interviewing are trained in general interviewing techniques and approaches, and all interviewers will be trained on the administration of the survey. Some pre-training exercises are likely to be required, and the actual training will include an item-by-item or task-by-task review of the survey instrument, practice interviews, and critiques of those interviews.

Training will take place close to the time when the first cohorts of research subjects reach the 36-month anniversary of their random assignment date.

All interviewers will sign a confidentiality pledge during training. They will be instructed on the importance of maintaining confidentiality and told that breaches of confidentiality will lead to dismissal.

MDRC will also work with HumRRO to monitor early interviews for each interviewer and periodically monitor interviews over the course of fielding the data collection instruments (e.g., listening in on telephone interviews). Feedback will be provided to the interviewers based on these monitoring efforts.

Conducting interviews. In all cases, the interviewers will explain the purpose of the interview, and inform respondents that they will receive a small incentive for completing the survey. Each interviewer will be prepared to answer any questions about the study that sample members might have.

Interviewer Supervision. Interviewing staff will be supervised directly by staff from HumRRO.

B3. Maximizing Response Rates

The goal will be to achieve an 80 percent response rate. Procedures for obtaining the maximum degree of cooperation include:

• Conveying the purposes of the survey to respondents so they will thoroughly understand the purposes of the survey and perceive that cooperating is worthwhile;

• Providing a toll-free number for respondents to use to ask questions about the survey and the survey firm’s staff;

• Training site staff to be encouraging and supportive, and to provide assistance to participants as needed;

• Hiring interviewers who have necessary skills for encouraging cooperation;

• Training interviewers to maintain one-on-one personal rapport with participants; and

• Offering appropriate payments to participants for participating in the survey effort.

The follow-up survey is designed to be administered in the home or by telephone. Once contacted, they will be asked to complete the interview using their own phone or a cell phone provided by the representative of the survey firm.

Interviewers will also be trained to distinguish "soft" refusals from "hard" ones. Soft refusals often occur when the sample member has been reached at an inopportune time. In these cases, it is important to back off gracefully and to establish a convenient time to call or come back rather than to persist at the moment. Hard refusals do occur and must also be accepted gracefully by the interviewer.

These strategies were used at the 15-month follow-up as well. Thus far our response rate is fairly high, which bodes well for the data collection strategies being proposed for the 36-month follow-up.

B4. Pilot Testing

Most of the questions proposed for this survey are either identical to questions used in prior MDRC evaluations or are similar, if not identical, to questions used in previous national surveys or major evaluations. Consequently, many of the items and measures have been thoroughly tested on larger samples.

The modules of the ERA 42-month survey, upon which the Philadelphia 36-month survey is based, underwent a number of revisions, one following a critique by internal staff and two following the survey instrument pilot test. Linda Lepping of HumRRO worked closely with MDRC staff to analyze pilot test results and make appropriate revisions to the survey. Nine sample members with ample contact information selected from the Chicago and Corpus Christi sites completed a draft survey over the telephone for this pilot test. After the pilot test interviews were completed, interviewers were debriefed concerning the problems they encountered and their recommendations for ways to improve the interview. An analysis of the data also showed substantial variation in responses across items. The survey instrument was then revised to incorporate Ms. Lepping’s recommendations for improving the readability of questions that respondents had difficulty understanding.

MDRC will also work closely with HumRRO’s senior staff to conduct formal pilot tests of the follow-up survey, most likely with a convenience sample recruited from those Philadelphia HTE participants who are not included in the survey sample (because the sample for the pilot test will only include nine study participants, our understanding is that this effort does not require a separate OMB review and approval process, and these hours are not included in our burden estimates). These pilot tests will provide more definitive estimates about the length of the survey – and its various components – as well as lead to improvements in questions, introduction scripts, wording and document formatting. Following each of the pilot tests, respondents will be debriefed about the clarity of the questions and any potential problems with the instruments. Interviewers will also be debriefed concerning any problems they encountered in the survey – and they will recommend improvements. A translated version of the survey will be developed once English versions are finalized.

B5. Consultants on Statistical Aspects of the Design

There were no consultants on the statistical aspects of the design.

1 This calculation assumes that 50% of the people in the control group were employed at the 36 month follow-up.