MDRC_FSS - SUPPORTING STATEMENT PART B final 7-18-2013

THE FAMILY SELF-SUFFICIENCY PROGRAM

EVALUATION

SUPPORTING STATEMENT – PART B

OMB CLEARANCE PACKAGE

Revised

6/7/2013

Submitted to:

U .S. Department of Housing and Urban Development

Contract No: GS-10F-0245N

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS

B1. Respondent Universe, Sampling Selection, and Expected Response Rates

MDRC will attempt to build a sample of at least 2,000 Housing Choice Voucher households.¹ These will be assigned 1000 each to a treatment and control group via random assignment (see below). All HCV households eligible for FSS will be allowed to participate in the study. MDRC, however will strive to build a sample that allows for analysis of relevant sub-groups (see Supporting Statement A for a description of the research design and questions). Therefore, MDRC will:

• Strive for a sample with a balance of non-working vs. already-working participants. In de Silva et al. (2011), researchers estimated that 61 percent of participants were already employed when entering the program, compared with 59 percent of non-enrolled voucher-holders in the same PHAs. Yet, as previously noted, early impact findings from Work Rewards (with the HPD sample) suggest that FSS (and the FSS+incentives) programs may have little effect on participants who enter the program already employed, but a positive effect for those who are not employed at the time of program entry. A sample in the HUD FSS evaluation that includes a fairly equal balance between these subgroups would allow the study to test whether this patterns holds more broadly for FSS programs, which, if it does, would have important implications for the design or operation of FSS.

• Strive to include a substantial proportion of Supplemental Nutrition Assistance Program (SNAP) recipients. As discussed above, another key finding from Work Rewards (from both the HPD and NYCHA samples) is the large and notable increase in earnings for voucher-holders receiving SNAP (food stamps) at study entry, and no positive labor market effects on non-recipients. This pattern has also been found in several other “make work pay” programs that MDRC has evaluated, suggesting that food stamp recipients may be especially responsive to an FSS or escrow-focused program. Thus, including a sizable proportion of SNAP recipients would allow the HUD FSS evaluation to confirm whether this pattern applies beyond New York City to other FSS participants.

Note that efforts to balance the sample will focus on targeted recruiting. MDRC does not plan to stratify sample collection to achieve desirable sub-group sizes.

The research team expects sample build-up to take at least 12 months, but based on our understanding of the small numbers of individuals enrolled in a year (section below), it is possible we may need to extend the enrollment period. However, our sample build-up projection balances the need for time to build a sample that can provide acceptable Minimal Detectable Effect Sizes (MDES) (see below) with the desire to allow a follow-up period long enough to observe program impacts.

Number of PHAs

In order to achieve a total sample size of at least 2,000 households within a one year sample recruitment period, the evaluation will need to include up to 20 Public Housing Authorities (PHAs) as sites for sample recruitment and random assignment. This assumption is based on two types of information: (1) estimates of average annual FSS vacancies from de Silva et al. (2011), and (2) estimates of the likely number of openings for new enrollees in FSS recently provided to MDRC from several PHAs. 

Table 1 presents the estimated number of months necessary to enroll a sample of at least 2,000 households given variations in the average number of FSS slots that become available each month at each site and in the number of participating sites. Recent discussions with PHAs have provided estimates of the typical range of available FSS slots on a monthly basis. At the low end, some programs report an average of 3-5 opening per month. At the high end, PHAs report an average of up to 15 open slots per month. Most PHAs appear to average between 5 and 10 slots per month, which would make recruitment of the desired sample feasible within the time allotted. However, these estimates are derived from a very small and still growing set of discussions and may change as MDRC collects more information.

The Family Self-Sufficiency Evaluation Table 1 Months Needed to Enroll Full Sample in 2-Group Design (N=2,000)
Total FSS slots per month per PHA	Research groups		Months needed to enroll full sample
	FSS group	Control group	15 PHAs	20 PHAs
5	5	5	14 months	10 months
10	10	10	7 months	5 months
15	15	15	5 months	4 months

If the average number of FSS slots per month across sites is in fact lower than 5, then 15 to 20 PHAs may not yield the necessary sample size within the desired one-year timeframe. This problem could be addressed by: 1) extending the period of random assignment– e.g., to 18 months or two years, although this would shorten the follow-up period for the full sample; 2) reducing the sample size goals, although this would reduce statistical power for detecting impacts and constrain subgroup analyses; or 3) combining these strategies. MDRC is prepared to modify the work plan and budget, in consultation with HUD, during the design and site selection process as more information about the candidate PHAs is obtained.

Criteria for PHA Selection

For a number of reasons, MDRC will not draw a probability sample of PHAs:

• PHAs must volunteer to participate. If MDRC drew a random sample, there is no guarantee that the selected sites would participate. In fact, many would likely refuse given the participation activities and associated burdens.

• The sites must together be of sufficient size to generate an adequate sample size for analysis within the time allotted for intake (see below).

• The PHAs must actually be operating an FSS program as laid out in the program logic model described in supporting statement A (i.e. the program must be voluntary, offer escrow, feature a self-sufficiency contract, and provide case management and service referral) . Without engaging in some direct contact with PHAs, it is impossible to know which sites are running FSS in more than name only. MDRC must ensure that program group members across all participating sites are receiving approximately the same treatment for conclusions about the impacts of FSS to be valid.

MDRC expects to target its site selection activities on PHAs with the following characteristics:

• Size: Larger programs will be prioritized as they are more likely to have slots for which new FSS applicants can be recruited. Although it may be desirable to represent smaller programs in the evaluation – which are represented in MDRC’s initial site reconnaissance – it will not be affordable to do so and still achieve the recommended sample size within the sample intake period.

• Ability to cluster: In order to contain costs related to acquisition of administrative data and the survey, and the overall cost of the evaluation, selecting multiple eligible PHAs within a single state, in up to 5 states, will be optimal (however, if necessary, MDRC will also consider single PHAs in states which don’t offer a clustering option).

• Local diversity: The combination of states and localities in the study should vary in terms of tenants’ demographic characteristics and local conditions that could affect work and housing subsidy use, such as the structure and tightness of local labor markets and the availability of affordable housing.

• Program quality: Where there is information and scope to do so, MDRC will also attempt to identify and include better-operated FSS programs. By examining typical programs, the evaluation can speak to the value of FSS as ordinarily enacted nationally. Examining exemplary programs instead allows the evaluation to speak to the potential of the FSS model to be effective when operated well. Because both questions are of interest to HUD, MDRC will attempt to identify a balance of typical and exemplary PHAs during site-selection.

Beyond the above characteristics, MDRC will attempt to produce a sample with a balance of employed and unemployed (at time of random assignment [RA]) study participants to facilitate sub-group comparison. MDRC will do the same with regard to SNAP receipt status, a sub-group comparison that produced important early impacts in the Work Rewards evaluation. Housing agencies, as part of their ongoing programs, may give preference to the employed/unemployed or may serve a population that naturally skews in a particular direction. MDRC will consider these types of issues when making final site decisions since it may necessitate efforts during random assignment to reach out to members of subgroups underrepresented at any given site.

Strategy to engage PHAs to join the study

The recruitment process is expected to be challenging because the study will impose new burdens on PHA staff who are already straining under tight operating budgets for administering their housing programs as well as FSS, and because participation in the study is not expected to bring them substantial new revenues. However, MDRC has extensive experience attracting on-going programs to participate in RCTs. In discussions with sites, MDRC expects to use the following strategies: 1) highlight the importance of the evidence that the PHA would help to build by participating in the study; one can presume that the PHAs operating FSS believe they are helping to improve the lives of the individuals and families in their programs and are interested in sharing this knowledge; 2) highlight that often programs in research studies are given special consideration when scarce funds are being allocated [MDRC and the GTR will discuss this option]²; and 3) note that MDRC would also organize participating PHAs into an initiative-wide Learning Network to facilitate cross-site dialogue on study-related challenges (e.g., recruitment). Learning Network activities could include periodic cross-site webinars and teleconferences. MDRC would also consider convening a Learning Network event at an annual public housing conference, such as the CLPHA annual conference.

Even if PHAs are interested in being part of the study, they may feel that they do not have the staff or resources to take on the extra costs and burden of being part of an evaluation. For that reason, PHAs will be provided a modest level of funding to defray costs related to study participation. See section below for fuller discussion of this matter.

Site payments to participating PHAs

Participation in the evaluation will require PHAs to undertake activities not normally required for their regular operations of the FSS program. These activities will span the duration of the study, although they will be more intensive in the earlier years. Required tasks will fall across four areas: planning, sample build-up, data extraction and transfer, and ongoing management and review.

Planning: PHA staff will be directly involved in the design and planning prior to random assignment, which will entail mapping out current client flow to identify optimal integration of the random assignment process into service delivery. Staff will be trained on random assignment protocols, including explaining informed consent, entering data into the random assignment module, and role playing notification of random assignment results. PHA staff will also need to modify marketing materials to incorporate approved language to reference the study.

Sample build-up: PHA staff will need to continuously modify marketing and recruitment strategies to generate the needed volume of participants to meet the target sample goal, which will be at least double the number of participants they typically screen into the program. PHA staff will obtain consent for each eligible participant, complete the baseline survey, and enter all relevant data into the random assignment module. PHA staff will also confer with MDRC to return all required research documentation, and to troubleshoot any questions or issues that may arise when accessing the online system. Following sample build-up, PHA staff will be expected to implement and maintain participant embargo procedures to ensure control group members do not receive services until the embargo period has ended.

Data: Each participating PHA will be required to identify staff to serve as a data liaison to participate in preliminary data discussions with MDRC, and to oversee the extraction of data as specified in the Memorandum of Understanding executed between MDRC and each PHA. PHAs will be required to provide an at least semi-annual file transfer to MDRC, to make any systems updates or quality checks to ensure integrity of the data, and provide progress reports as requested. As part of ongoing management and review, MDRC will establish a schedule of regular check-ins with site, which will be more intensive during the sample build-up phase, but will continue until the end of data collection. PHA staff will be expected to coordinate data collection site visits, which may involve convening participants or other external providers. PHA staff will be expected to participate in cross-site learning network meetings and in any recommended technical assistance activities.

The proposed compensation would offset some of the costs incurred by the PHA from devoting staff time and resources to these types of activities. Given the additional work that study participation entails, it is important that the PHA leadership and HCV program managers incorporate the study requirements into their staff’s workload rather than allowing the study to become a source of conflict, which would substantially hurt the effort.

The FSS evaluation budget assumes about $40,000 in site payments to each participating PHA. MDRC recognizes that this amount is insufficient to fully compensate PHAs for the additional burden study participation entails; however, it is meant to help defray some of the costs associated with the activities described above. Based on experience, study burden will be higher during the first two years of participation because of concentrated recruitment and enrollment activities during this period. As a result, we expect at least 50 percent of the $40,000 sum will be distributed to PHAs during this period. Subsequent payments would help defray some costs associated with the PHAs’ ongoing reporting needs for the study, which are less intense in years 3-5. However, the final distribution of payments will be finalized with the PHAs during contract negotiations. MDRC would like to offer maximum flexibility to participating agencies in how the site payments are used, and the Memorandum of Understanding executed between MDRC and each PHA will clearly indicate the agency’s responsibilities associated with study participation.

MDRC’s experience with evaluations and demonstrations also suggests that the additional workload associated with the sample enrollment phase of research has the potential to disrupt services to a point where having at least one more staff person is deemed necessary to support site operations and participation in the research. In the case of the FSS evaluation, and the anticipated one-year sample build-up period, a part-time hire might be necessary to cover the intake process and to ensure recruitment protocols are followed in a way that doesn’t undermine the validity of the study. PHAs may use the initially larger site payments to partially fund this position (and it is our understanding that PHAs are unable to draw on their current FSS grants to support a hire for study purposes only). The ongoing site visits to PHAs that are potential candidates for the evaluation will help MDRC understand how the PHAs plan to allocate site payments to support research needs. While we may not be in a position to offer higher site payments, this will be important feedback for HUD and MDRC in the early planning phase.

Expected Response Rates

This submission does not include details on the administration and collection of data from the single follow-up survey. This information will appear in a later submission. As all other data on the sample will be derived from the Baseline Information Form (BIF) and Informed Consent Form (mandatory for study participation) and administrative records, there are no issues regarding response rate.

B2. Procedures for Data Collection and Statistical Analysis

The study will involve randomly assigning a total sample of at least 2,000 households³ to one of two groups (1,000 per group):

• FSS group. These individuals have access to the core elements of the FSS program – case management as well as rent escrow provisions.

• Control group. These individuals will not be enrolled in FSS and will not have access to FSS case management or escrow for the 3-year embargo period.

Random assignment will occur after participants sign the informed consent and complete the BIF. Random assignment will occur in real-time on site at PHA offices. Participants randomly selected for the FSS program group will move directly into scheduling a follow-up with program staff. Control group members will be given a resource list of other services in the community that they could access on their own. MDRC staff will work with each PHA to determine how best to incorporate random assignment at their offices to minimize disruptions to their normal work routines. Moreover, options for conducting random assignment may include building random assignment into their own automated systems, a process MDRC has used successfully in several prior evaluations, or developing a standalone system.

Once a family⁴ has been assigned to the program group, participants will be provided with information about their respective programs. For FSS, MDRC will work with the programs to develop an orientation packet, and an invitation to meet with a counselor or schedule a follow-up for such purposes, so as to maintain continuity of services and momentum of engagement. Families in the control group will be provided with an information packet describing their rights and terms of participation in the study.

Statistical Impact Analysis

Impact analysis (see Supporting Statement A for an overview of study components) will assess the overall and independent effects of the FSS program by comparing the key outcomes of the treatment group to the outcomes of the control group. The study will track both the program and the control groups for a number of years, using administrative and survey data to measure outcomes.

The impact analysis will examine the program’s effects on a wide range of outcomes. Key clusters of outcomes under consideration for this study are included below.

Household Composition and Structure: FSS is a voluntary program and participation in it is open to other adults in the enrolled household. To explore effects on household composition and structure, MDRC will obtain basic information about all household members, including names, ages, employment status (if appropriate), and relationship to the head of household through the baseline information form, survey, and the 50058 form.

Education and Work: MDRC will use both Unemployment Insurance wage records and the survey to collect data on employment, earnings, job characteristics, and work search behaviors. Discussions with PHAs have revealed that some programs take a human capital development approach to self-sufficiency and thus emphasize degree, diploma and certification achievement. Though potentially regarded as an intermediate outcome, MDRC will track educational attainment among study participants through survey data.

Income, assets, finances, and rent burden: If FSS affects participants’ disposable income, it may help them accumulate assets. With survey data, MDRC will assess the effects of the program on household finances and financial behaviors (such as savings, access to credit, and debt reduction). Data on income, combined with housing authority and survey data on tenant rent and utilities payments, would be used to construct measures of rent burden.

Health, material hardship, and family well-being: As in Work Rewards, Jobs-Plus, and other housing studies, the FSS evaluation will estimate the effects of FSS on residents’ overall health and specific health conditions, and their access to preventive health care. All of these indicators may be affected, indirectly, by changes in residents’ income, and by potential changes in their housing and neighborhood contexts. These factors may also affect mental health outcomes, such as depression. Increases in disposable income may also reduce material hardships, including housing-related hardships such as disconnection of phone and utilities. Food-related hardships may also be reduced. (MDRC observed such effects on poverty and hardship in its study of New York City’s conditional cash transfer program.)

The power of the experimental design for the FSS evaluation comes from the fact that random assignment ensures that the treatment and control groups are alike in all aspects of the distribution of observed and unobserved baseline and pre-baseline characteristics. As a result, any post-baseline differences between the two groups can be interpreted as effects of the intervention.

Therefore, the basic estimation strategy is to compare average outcomes for the program and control groups. We will use regression adjustment to increase the power of statistical tests that are performed, in which the outcome, such as “employment during Year 1” is regressed on an indicator for program group status and a range of other background characteristics.

The general form of the regression models which will be used to estimate program impacts is as follows:

Yi = α + βPi + δXi + εi

where

Y_i is the outcome measure for sample member i;

P_i equals one for program group members and zero for control group members;

X_i is a set of background characteristics for sample member i; and

ε_i is a random error term for sample member i.

The coefficient β is interpreted as the impact of the program on the outcome. The regression coefficients, δ, reflect the influence of background characteristics. The functional form and estimation method will depend on the scale of measurement of the outcome for which impacts are estimates; for example, continuous outcomes will be estimated using ordinary least squares (OLS) regression. We can use a more complex set of methods depending on the nature of the dependent variable and the type of issues being addressed, such as: logistic regressions for binary outcomes (e.g., employed or not); Poisson regressions for outcomes that take on only a few values (e.g., months of employment); and quantile regressions to examine the distribution of outcomes for continuous outcomes.

The evaluation will examine outcomes across a number of domains. When multiple outcomes are examined, the probability of finding statistically significant effects increases, even when the intervention has no effect. For example, if 10 outcomes are examined in a study of an ineffective treatment, it is likely that one of them will be statistically significant at the ten percent level by chance. While the statistical community has not reached consensus on the appropriate method of correcting for this problem, we propose to address it by being parsimonious in our selection of outcome variables. In particular, we plan to identify a set of “primary” outcomes and subgroups before beginning the impact analysis. All other outcomes and subgroups will be considered “secondary” and will be used to provide context for the primary impact findings or to generate hypotheses about impacts. Schochet (2008) suggests that this strategy is flexible enough to credibly test the key hypotheses about the program, while at the same time allowing the analyst to examine a range of outcomes in a more exploratory manner in order to uncover policy-relevant information.

The main analysis will be conducted over the pooled sample across all PHAs. While sample sizes will not permit the analysis of PHA-specific impacts, MDRC will examine impacts for certain clusters of programs, such as smaller versus larger programs, programs that have a strong focus on employment, or programs that are operating in strong versus weak local labor markets. The ability to conduct this type of analysis depends on the variation in program features and contexts across participating PHAs, which will be critical to capture in the implementation analysis.

Subgroups

The impact analysis will also investigate whether the intervention worked especially well for particular subgroups of families. Subgroup impacts can be calculated in several ways,⁵ and prior to the impact analysis, the evaluation team will finalize the method and prioritize the subgroups that are “confirmatory” and the remainder that are “exploratory.” The confirmatory subgroups will be specified in advance, in order to avoid the potential for data mining and the problem of multiple comparisons.⁶ Subgroups can be chosen as confirmatory because prior theory suggests program differences by a subgroup dimension, because differences in impacts by a given dimension have been found in prior evaluations, or because a given subgroup is of great policy interest.

MDRC is currently considering several subgroups of interest. Informed by the findings from Work Rewards, MDRC plans to examine impacts by work status and SNAP receipt status at program entry. As discussed above, the early impact findings for Work Rewards show that the interventions positive effects for certain subgroups that were specified before the impact estimates were calculated. Based on prior research on employment programs, it was expected that the effects of Work Rewards might vary for different types of individuals, defined, for example, in terms of their prior attachment to the labor market and prior receipt of public transfer benefits. The study found that FSS+Incentives produced large and statistically significant increases in average quarterly employment rates and average earnings for voucher holders who were not already working at the time they entered the study. Impact estimates for FSS-Only were also positive for that subgroup, but they were smaller and not statistically significant (meaning that the effects are less certain). Neither of the two interventions improved outcomes for participants who were already working when they enrolled in the study. This pattern is consistent with the study’s qualitative research finding indicating that the community organizations operating FSS had much more concrete assistance to offer to participants who needed to find jobs than they could offer to participants who were already working and hoping to increase their earnings.

The Incentives-Only intervention produced sizable and statistically significant increases in earnings for participants who were food stamp recipients, but had no effects for participants who were not food stamp recipients. Interestingly, FSS+Incentives and FSS-Only produced a similar pattern of results (and were statistically significant in the latter case). One interpretation of this general finding is that perhaps the special workforce incentives and/or the extra prodding and support offered by the FSS program helped counteract the worries that some food stamp recipients may have had about potential reductions in their food stamp benefits if they earned more money.

Finally, earlier reports on FSS, although non-experimental, suggest that education level and TANF status may be associated with success in the program.

Minimum Detectable Effect Size

MDRC will enroll at least 2,000 individuals, across all PHAs in the evaluation, with 1,000 assigned to the FSS group and 1,000 assigned to the control group. A sample size of 1,000 per research group is large enough to detect policy relevant impacts for the full sample as well as for key subgroups.

It is useful to consider the concept of Minimum Detectable Effects (MDEs) to explore the size of program impacts that are likely to be observed or detected for a set of outcomes and a given sample size. Since these are estimates, the actual MDEs may be smaller or larger than what is shown here. The estimates shown are likely to be conservative, since they assume that baseline variables are not used in the impact model to improve precision. Pre-random assignment values of key outcomes, such as employment and earnings, are likely to be highly predictive of post-random assignment values of the same outcome. In this case, the increased precision brought about by including these variables in the impact model can reduce the MDEs considerably.

Table 2 presents MDEs for the proposed sample size. The first column presents data for the full sample and the second column presents data for a subgroup within the full sample, assuming that subgroup makes up half of the larger sample. The final two columns present similar numbers for a potential survey sample and subgroups within that sample. The first several rows present MDEs for work, earnings, and poverty. For the full sample, the evaluation could detect effects (increases or decreases) as small as 5.5 percentage points on employment rates in a given year. Because sample sizes are more likely to be smaller for subgroups, the MDE for a subgroup is somewhat larger, at 7.8 percentage points. MDEs for earnings are shown in the table but are harder to predict, given the difficulty of predicting the variance of earnings. The next row presents MDEs for family income above the poverty line, which would only be available from a survey and MDEs range from 5.7 percentage points to 8.7 percentage points. The final row presents MDEs in terms of effect sizes (or the impact on a given outcome divided by the standard deviation of that outcome). Effect sizes are a useful way to present and compare impacts on outcomes that are measured in different units, such as family well-being scales. For each of the proposed sample sizes, the effects sizes (ranging from .11 to .18) are typically considered small to moderate in the evaluation literature.

In sum, the proposed sample size is adequate for detecting effects on a range of outcomes that are relatively modest but still but meaningful from a policy standpoint. This pattern holds for the full sample and for key subgroups.

The Family Self-Sufficiency Evaluation Table 2
Minimum Detectable Effects
Research Group i versus Research Group j
	Full Sample	Subgroup	Survey Sample	Subgroup
	(1000 per group)	(500 per group)	(800 per group)	(400 per group)
Percentage point effects
Employed in Year 1	5.5	7.8	6.2	8.7
Employed for all 4
quarters of Year 1	5.1	7.3	5.7	8.1
Dollar effects
Earnings	1,002	1,417	1,120	1,585
Effect size	0.11	0.16	0.12	0.18
Notes: MDEs are calculated based on a two-tailed significance test and assuming an R-squared in the impact model of 0. Average values for employment and earnings outcomes are taken from the Opportunity NYC Work Rewards sample. Average values for income below poverty are taken from the Family Rewards sample. Average values are 55% for work in Year 1; 31 percent for working all 4 quarters; 30 percent for income above the poverty line, and $6,874 (standard deviation of $9,500) for earnings. Effect sizes are measured as the impact on a given outcome divided by its standard deviation.

B3. Maximizing Response Rates and Issues of Nonresponse

As noted above, this OMB submission includes discussion of mandatory forms (BIF and informed consent) and administrative records only. Therefore this section is not applicable. A future OMB submission will detail survey data collection and efforts to maximize survey response rates.

B4. Pre-Testing

The baseline information form will be tested internally to ensure question clarity and relevance and to ensure proper implementation. MDRC will also conduct pre-testing of the survey instrument before fielding. This approach will be described in a future OMB submission that deals with the survey directly.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for the FSS studies is being collected by MDRC and its subcontractors, Branch Associates and M.Davis on behalf of the Department of Housing and Urban Development. With HUD oversight, MDRC and its subcontractors, including Ingrid Gould-Ellen and John Goering, both national experts, were responsible for developing the study documents included in this submission. The statistical aspects of the study were developed in consultation with MDRC senior economist and impact analyst, Cynthia Miller.

1 Described further in a following section, this number is derived from calculations that reveal the Minimum Detectable Effect Size (MDES) for a given sample size. This section also explores the MDES for larger samples.

2 HUD has been having internal discussions about making additional funds available to PHAs to defray the costs of participation and possibly reimburse PHAs for pro-ration of the FSS Coordinator grant.

3 See section below for the determinants of the sample size.

4 FSS is a household-level intervention; all eligible adults in a household may participate. In cases where multiple adults from the same household express interest in participation in this evaluation, MDRC will randomize such individuals as a single unit. That is, all sample members from the same household will be placed in the same group: control or FSS.

5 In “split-sample” subgroup analyses, the full sample is divided into two or more mutually exclusive and exhaustive groups, such as single-parent families at the point of random assignment versus two-parent families. Impacts are estimated for each group separately. A related type of subgroup analysis uses regression methods to see if the effects of the intervention vary significantly with a continuous baseline measure (or one that takes on many values), such as initial attendance levels or test scores. Finally, “conditional” subgroup analyses take this idea one step further by controlling for the effect of other baseline characteristics when estimating the relationship between a particular subgroup and program effects.

6 Restricting the analysis to a few confirmatory subgroups does not rule out the possibility of a more exploratory analysis of additional subgroups later in the evaluation. Findings from this analysis would necessarily be more speculative and given less weight in the discussion of program impacts.