ICR 2528 HC Part B (8.14.08)

Part B
Statistical Methods

HUD has contracted with Abt Associates to conduct an outcome evaluation of pre-purchase and foreclosure mitigation counseling. The study will focus on clients of HUD-funded counseling agencies that serve a minimum of 270 pre-purchase or foreclosure mitigation clients a year. Thirty agencies from this universe will be selected at random to provide samples of 1,000 pre-purchase clients and 1,000 foreclosure mitigation clients. We expect to achieve an 80 percent response rate or higher, so initial sample sizes will be set at 1,250 clients per group to reach the goal of enrolling 1,000 clients of each type.

B1 Potential Respondent Universe

The objective of the outcome panel is to provide HUD with a national probability sample of housing counseling clients. Because no national list of clients exists, we will use a two-stage cluster sample. Cluster sampling takes advantage of the hierarchical nesting that exists for most populations. For clients at intake, the hierarchical nesting consists of HUD-approved counseling agencies and clients within agencies. The study will focus on two client groups: 1) pre-purchase and 2) foreclosure mitigation. Both types may receive either individual counseling or group counseling (workshops).

The outcome panel will consist of a sample of clients from each of the two groups, selected from a sample of HUD-approved agencies. Clients will be sampled over an intake period that will be customized to each sample agency and will last no longer than 8 weeks. An intake period longer than 8 weeks would place too much burden on the agencies in the sample. The goal is to sample enough clients to have a sample size of about 2,000 clients who complete the baseline intake survey, equally divided between the two groups. The baseline response rate is expected to be 80 percent or higher, because the only component of unit nonresponse is failing to obtain consent. The initial sample size is therefore 2,500 clients divided between the types of clients. Thus, agencies will need to average at least 41.7 clients of either type over an 8-week period. Based on client volumes from FY 2007 reported to HUD, there are 526 agencies that have sufficient weekly volume of clients to meet the 8-week minimum client volumes. Fortunately, these agencies account for 89 percent of the total count of clients.

B2 Statistical Methods

B2.1 Sampling Plan

Stage One: Sampling Agencies

The universe of counseling agencies from which we draw the sample will be HUD-approved agencies. Because we will want to design the sampling process to take into account information such as the distribution of clients served and the size of the agency, we will use HUD 9902 data to help define strata for sampling. To be eligible a HUD-approved agency must have received funding from HUD for housing counseling. There are two reasons why it is appropriate to focus on HUD-funded agencies. First, given the OMB PART review requirements to demonstrate the effectiveness of government programs, there are benefits to designing the outcome panel to provide feedback on HUD-funded efforts. Secondly, the agreement that agencies enter into with HUD as a condition of receiving funding requires that they cooperate with efforts to evaluate this program. Thus, HUD-funded agencies have a requirement to participate while those without HUD funding may not be so obligated.

We have used FY 2007 9902 data to develop the sample design, although the delinquent client counts reported in these data have been inflated by 50 percent to reflect the expectation of substantial increases in delinquent client volumes due both to increased demand and increased supply resulting from a large increase in funding channeled through NeighborWorks America. In all, there are 1,183 agencies that reported at least one pre-purchase or foreclosure mitigation client in FY 2007. These agencies account for 15,536 clients, on average, per week, and range from 0.2 to 661 clients, on average, per week. In this situation a two-stage cluster sample design is the most appropriate design for selecting a national probability sample of clients. In two-stage cluster sampling, a probability-proportional-to-size (PPS) sample of agencies is drawn at the first stage of sampling. The measure of size for the PPS sample will be the average weekly count of clients. As noted above, taking into account the burden on the agencies, the maximum sample intake period has been set to 8 weeks. There are 526 agencies that have sufficient weekly volume of clients to meet the 8-week minimum client volumes of 41.7 clients. Fortunately, these agencies account for 89 percent of the total count of clients.

The target sample size of agencies for the first-stage sample is 30. Because a few agencies may choose not to participate, we plan to draw a somewhat larger sample of agencies and randomly designate a portion as a “reserve” sample. Before selecting the sample of agencies we will stratify all eligible agencies in the 50 states and the District of Columbia on the basis of the nine Census Divisions and the number of delinquent clients in the agency. Stratification on the basis of the nine Census Divisions will ensure a good geographic spread of the sample, while stratification on the basis of the number of delinquent clients is important because this is the smallest of the groups. In a PPS sample the largest agencies will be included in the sample with certainty (i.e., with a probability of one). Certainty agencies are those with a measure of size greater than or equal to the total measure of size for all 526 eligible agencies (13,778 clients per week) divided by the sample size of agencies (30). Based on this rule (13,778/30 = 459), there are only two agencies large enough to be included in the sample with certainty.

Sampling Clients within Agencies

Based on FY 2007 9902 data, we have divided the client population into the two groups as shown in Exhibit B-1 and have developed estimates of the size of each group and the mean number of clients per agency. The pre-purchase group is considerably larger than the delinquent group.

Exhibit B-1. Client Population Distribution Based on FY 2007 9902 Data

Group	Approximate Weekly Population of Client Intakes	Approximate Weekly Mean Number Per Agency	Approximate Mean Number Per Agency Over an 8-Week Intake Period
Pre-purchase	8,251	15.7	125.5
Foreclosure Mitigation	5,527	10.5	84.1

Our proposed sample design will yield national estimates for both of these groups. Our goal is to sample 1,250 individuals from each group so that we may have an approximately equal level of precision at the follow-up interview for each group. This will be accomplished by setting the second-stage cluster size for each sample agency at about 41.7 clients for each group.

The selection of a random sample of clients from each group at the agency level must be practical to implement in the field. Our experience indicates that attempting to select a subsample of individuals at intake is very difficult for agencies to carry out correctly. It is much easier for an agency to include all individuals in a given group in the sample, but because the agencies have different client volumes the sampling intake period will not be the same of all sample agencies. If strong seasonal or time effects existed, this selection method could introduce a selection bias. For our specific population and survey topic, having a sampling intake period on unequal length but no longer than 8 weeks should not introduce a selection bias.

We will implement the within agency sampling as follows. After the selection of the agencies, we will ask each agency to provide an estimate of the current intake flow for each group. The estimate will be compared with the measure of size used to select the agency to check that the intake volume is consistent with the size of the agency. Based on a maximum of an eight-week intake period, we will calculate the expected intake period required to obtain around 41.7 clients for each group in each sample agency. The larger agencies will have a shorter expected intake period for the pre-purchase group compared to the smaller agencies. Because of its small size, the delinquent group will in general have the longest intake period, and it is possible that it could be as long as eight weeks in the smaller sample agencies. For this approach to work, we will need each agency to provide us with the actual intake count for each group for the entire eight-week intake period. The availability of the actual size of the two groups in each sample agency for the same fixed time period provides the information needed to develop unbiased within-agency client sampling weights.

Weighting Procedures

We will calculate sampling weights for use in estimating the characteristics and outcomes of clients nationally. The sampling weights will have several components. The base sampling weight will reflect the probability of selection of each sample client. (For example, if the probability of a case’s selection is 0.25, its sampling weight is equal to 4.) For each of the groups the base sampling weights will then be adjusted for baseline survey and follow-up survey unit nonresponse within agency (assuming this follow-up survey is implemented). The nonresponse bias assessment for the delinquent group may identify some factors related to nonresponse and it may be necessary to incorporate them into the weighting procedures.

In addition to developing sampling weights it will be necessary to use software such as SUDAAN to obtain valid standard errors for the national estimates.

B2.2 Justification of Level of Accuracy

Expected Level of Precision for the Follow-Up Interview

The use of a two-stage cluster sample design will result in some reduction in the effective sample size due to homogeneity of clients in a given group within a given agency. Stated otherwise, the clients selected from within a given agency and group cannot be regarded as statistically independent of each other (in contrast to a simple random sample drawn from among all clients nationally within a given group). The degree of homogeneity is measured by the intra-cluster correlation, which we expect to average around 0.01 based on what we know about the nature of these agencies. Based on this assumption, we show in Exhibit B-2 the sample size of clients, the effective sample size of clients, and the margin of error. The margin of error represents the 95-percent confidence interval half-width for a population proportion equal to 50 percent. In other words, if the population characteristic being measured has a true value of 50 percent, the margin of error indicates the range of values within which there is a 95 percent chance that the sample mean will fall, given the indicated sample size. Both of the margins of error are within acceptable bounds of precision.

The overall sample size for follow-up from the baseline is 2,000 clients. In our sample size calculations we assumed a response rate to the six-month follow-up survey of 80 percent for the pre-purchase and 60 percent for the delinquent homeowners. For this latter group we plan to conduct an analysis of potential nonresponse bias, using information available for all cases in the initial sample from the Baseline Questionnaires and agency-captured data. This analysis will examine the extent to which the follow-up survey respondents may differ significantly in their initial characteristics from the survey nonrespondents.

Exhibit B-2. Expected Sample Sizes and Margin of Errors for the Follow-Up Survey

Group	Expected Initial Sample Size	Expected Sample Size at Follow-Up	Expected Effective Sample Size at Follow-Up	Expected Margin of Error
Pre-purchase	1,000	800	637	3.9 percentage points
Foreclosure Mitigation	1,000	600	504	.4.4 percentage points

B3 Maximizing Response Rates

During the eight-week period when the study sample is being enrolled, each new pre-purchase or foreclosure mitigation counseling client will be informed by staff of the participating counseling agency about the nature and purpose of the study being conducted for HUD, what would be required of them if they participate, and will be asked to provide their consent to participate.¹ Clients who agree to participate in the study will be asked to complete a baseline questionnaire and to provide their consent for up to three credit reports and credit scores to be obtained by study staff over a two-year period to identify outcomes from their counseling. As compensation for their cooperation, clients will be given free copies of their credit reports and credit scores each time a report is obtained for the study.² Participation in the study will be voluntary. Counseling agency staff involved in informing clients about the study will be instructed to describe the benefits of the study for the counseling industry to encourage their participation. Since the clients will have voluntarily sought out the services of the counseling agency it is anticipated that clients will have a high degree of confidence in the trustworthiness of the agency. A response rate of at least 80 percent is anticipated both because of this high level of trust and because it will require relatively little effort for clients to participate—essentially completing the 15 to 20 minute baseline questionnaire during their initial visit to the counseling agency. Completion of the service tracking surveys and educator/counselor information surveys by the counseling agencies is expected to be close to 100 percent given that the agencies have agreed to participate in the study.

B4 Tests of Procedures or Methods

Early drafts of the instrument have been reviewed by housing counseling practitioners in order to ensure that the instrument is clear, flows well, and is as concise as possible. In addition, pre-tests of the instrument and the Internet site will be conducted between May and July 2008 with up to nine respondents.

B5 Statistical Consultation and Information Collection Agents

HUD has contracted with Abt Associates Inc. and its subcontractor IMPAQ International Consulting to conduct the data collection. Christopher E. Herbert, Ph.D., of Abt Associates is the person responsible for the statistical aspects of the survey design. He can be contacted at (617) 349-2383.

1 A copy of the consent form is presented in Appendix B. The form will be translated into Spanish or other languages as needed based on the clientele of the agencies sampled for inclusion in the study.

2 While everyone is eligible to receive a free copy of their credit report from each credit reporting agency each year, credit scores can only be obtained for a fee (generally $10 to $15). In addition, clients will also be spared the effort required to make requests for their freely annual credit reports from three separate agencies.