V2_Counseling Outcome OMB Part B 11.23.10

Part B.
Statistical Methods

HUD has contracted with Abt Associates to conduct an outcome evaluation of pre-purchase and foreclosure mitigation counseling. The study focuses on a probability sample of 1,398 housing counseling clients from 25 HUD-funded counseling agencies across the country. The first phase of the study focused on recruiting sample agencies to participate in the study, enrolling a sample of clients in the study, gathering baseline information on enrolled clients, tracking the counseling services they receive over a six-month period, and identifying outcomes at the end of six months. We received OMB clearance for the baseline study in August 2008 (OMB Control #2528-0255, expiration date 09/30/2011). This second phase of the study, for which OMB clearance is being sought, involves a follow-up survey with enrolled clients just over a year after receiving counseling services to measure longer term housing outcomes. At this time, we are requesting OMB clearance only for the follow-up survey with foreclosure clients. A separate request will be submitted for the follow-up survey with pre-purchase clients.

B1 Potential Respondent Universe

The objective in designing the outcome sample was to provide HUD with a national probability sample of housing counseling clients. Because no national list of clients exists, we used a two-stage cluster sample. Cluster sampling takes advantage of the hierarchical nesting that exists for most populations. For clients at intake, the hierarchical nesting consists of HUD-approved counseling agencies and clients within agencies. The study focuses on two client groups: 1) pre-purchase and 2) foreclosure mitigation. Both types may receive either individual counseling or group counseling (workshops).

The outcome sample consists of a sample of clients from each of the two groups, selected from a sample of HUD-approved agencies. Clients were enrolled over an intake period customized to each sample agency with the goal of obtaining a sample size of about 2,000 clients who complete the baseline intake survey, equally divided between the two groups. Anticipating a response rate of 80 percent or higher to the baseline intake survey and that 30 sample agencies would enroll clients into the study, the initial sample size was 2,500 clients divided between the two client types.

Several factors made it impossible to achieve the target of enrolling 2,000 clients. As discussed in Section B2 below, only 25 of the 30 agencies recruited to participate in the study were able to recruit clients. In addition, some agencies struggled to meet their enrollment targets. Among the agencies participating in the study, about 1,660 pre-purchase clients were assisted during the sample intake time period. Similarly, these agencies assisted approximately 11,050 foreclosure mitigation clients during the sample intake time period.¹ At the conclusion of the enrollment period in December 2009, a total of 1,398 of these clients were enrolled in the study–including 573 pre-purchase clients and 825 foreclosure mitigation clients—resulting in estimated response rates of 35% and 8%, respectively, for these two types of clients.

B2 Statistical Methods

B2.1 Sampling Plan

The sample of counseling clients who will be asked to complete the follow-up survey was determined in the first phase of the study, for which OMB approval was obtained in August 2008 (OMB Control #2528-0255, expiration date 09/30/2011). The sample was developed using the two-stage cluster sampling procedure summarized below.

Stage One: Sampling Agencies

The universe of counseling agencies from which we drew the sample was HUD-approved and HUD-funded agencies. Because we wanted to design the sampling process to take into account information such as the distribution of clients served and the size of the agency, we will used HUD 9902 data to help define strata for sampling. To be eligible, a HUD-approved agency had to have received funding from HUD for housing counseling.

HUD determined that a two-stage cluster sample design was most appropriate for selecting a national probability sample of clients. A probability-proportional-to-size (PPS) sample of agencies was drawn at the first stage of sampling. The measure of size for the PPS sample was the average weekly count of clients.

The target sample size of agencies for the first-stage sample was 30. Before selecting the sample of agencies we stratified all eligible agencies in the 50 states and the District of Columbia on the basis of the 9 Census Divisions and the number of foreclosure mitigation clients in the agency. Stratification on the basis of the nine Census Divisions was designed to ensure a good geographic spread of the sample, while stratification on the basis of the number of foreclosure mitigation clients was important because this was the smallest of the two client groups.

The recruitment of agencies to the study took place in May and June 2009 and client enrollment began in August 2009. One of the agencies in the sample refused to participate and given the sampling design there was no replacement agency available. Another four agencies initially agreed to participate but had to be dropped in the study during the client enrollment phase, at a time when it was too late to find replacements. Three of these agencies were branches of ACORN Housing Corporation, which was barred from receiving federal funding shortly after client recruitment began. In addition, one other agency agreed to participate in the study, signed a memorandum of understanding, and was trained in the study protocols, but failed to recruit any clients into the study. As a result, the total number of agencies participating in the study is 25.

Sampling Clients within Agencies

Based on FY 2008 9902 data, we divided the client population into the two groups. Enrollment of clients into the study began in August 2009, after the agencies had been recruited to participate and had been trained in the study protocols. Each agency was given an enrollment target for each client group (in some cases agencies only served one of the two groups and therefore were only asked to enroll one type of client), with the expectation that they would meet or exceed that enrollment target within eight weeks. A client was considered to be enrolled when he/she signed the written consent form and completed the baseline questionnaire. Clients served by telephone provided oral consent over the phone and were mailed consent forms and questionnaires to return by mail.

Although many agencies were able to meet their recruitment targets within the 8-week period, the overall recruitment period lasted for 16 weeks, as some agencies needed more time to recruit clients and it took some time for clients served by telephone to return the written consent forms and baseline questionnaires. At the end of the enrollment period, a total of 1,398 clients were enrolled, 573 pre-purchase clients and 825 foreclosure mitigation clients. The overall sample fell short of the anticipated 2,000 clients because of the smaller number of agencies participating in the study (25 rather than the expected 30) and because several agencies were unable to achieve their client enrollment targets, even with the extended enrollment period. This was particularly true for the pre-purchase group, as the overall volume of clients seeking help buying a house has declined sharply with the foreclosure crisis leading to tighter credit and the national economic recession.

Given the low response rate for the first phase of the study, we carried out a nonresponse bias analysis to assess the extent to which the characteristics of clients enrolled into the study sample differed from the characteristics of all clients served by the agencies during the study enrollment period. The analysis is complete for the sample of foreclosure counseling clients enrolled in the study and is ongoing for the sample of pre-purchase clients. (We are not requesting clearance for the follow-up survey with pre-purchase clients at this time.) The steps taken for analyzing and correcting for non-response bias in the foreclosure counseling client sample are described below.

Nonresponse Bias Analysis for Foreclosure Counseling Sample

The foreclosure sample consists of 24 agencies (one agency in the sample of 25 only enrolled pre-purchase clients). Within each agency, a sample of foreclosure clients was drawn. For the sample clients that completed the baseline interview we have calculated a base sampling weight equal to the product of the reciprocal of the probability of selection of the agency and the reciprocal of the within agency probability of selection of the foreclosure client. We also have an agency weight equal to the reciprocal of the probability of selection of the agency.

For each of the 24 agencies we obtained administrative data for the sampling intake period for Hispanic origin, race, gross annual income, and age. In most cases, these administrative data were provided by the housing agencies themselves and represent all clients who received foreclosure counseling services during the study enrollment period (mid-August 2009 through early December 2009). National Foreclosure Mitigation Counseling (NFMC) Program administrative data were used when administrative data could not be obtained directly from the housing counseling agency. The NFMC database includes a large subset of the total universe of foreclosure prevention counseling clients and is representative of the total foreclosure prevention client population. NFMC data were used in place of all administrative data for 4 of the 24 housing counseling agencies that collected data on foreclosure clients. NFMC data were used to supplement administrative data with information on client income for an additional six agencies.

Using the agency weights, we estimated the aggregate (combined 24 agency) distribution of foreclosure clients for each of the four variables. Using the baseline client sampling weights we estimated the aggregate distribution for Hispanic origin, race, income and age. Exhibit B-1 compares the weighted distribution of the baseline sample data across each of the four variables to the weighted distribution of the administrative data across these same variables. The exhibit shows that moderate differences exist between the baseline sample and the overall client population, particularly on two of the income categories: under $20,000, where there is a difference of 5.3 percentage points, and $40,000-$59,000, where there is a difference of 3.2 percentage points. There is also a difference of 4.21 percentage points in the 45-54 age category.

Exhibit B-1. Nonresponse Bias Analysis for Foreclosure Mitigation Clients

Variable	Baseline Survey Data (Weighted Percents)	Administrative Data (Weighted Percents)	Percentage Point Difference (Baseline Survey-Admin Data)
Ethnicity:
Hispanic	13.01%	13.69%	-0.68
Race:
Asian	2.06%	2.10%	-0.03
Black	30.18%	29.47%	0.71
White	58.61%	60.45%	-1.84
Other Race	9.14%	7.99%	1.16
Gross Annual Income:
Under $20,000	22.49%	17.19%	5.30
$20,000-$39,000	36.64%	36.76%	-0.12
$40,000-$59,000	23.99%	27.19%	-3.20
$60,000-$99,000	14.44%	15.71%	-1.27
$100,000-$199,999	2.43%	2.85%	-0.42
$200,000 and above	0.02%	0.31%	-0.28
Age:
Under 35	12.00%	13.76%	-1.76
35-44	25.27%	27.55%	-2.29
45-54	36.75%	32.54%	4.21
55-64	19.61%	17.90%	1.71
65 and over	6.37%	8.25%	-1.88

We want to adjust the baseline client sampling weights so that the weighted distribution for each of the four variables agrees with the administrative data distribution. If we were adjusting on one just variable, we could simply ratio adjust the weights within each category of that variable so that agreement was achieved for each category of that variable. Given that we want to adjust on four variables, we will need to use ranking ratio estimation, which is an iterative technique that treats each variable as a margin. The iterative procedure yields a set of client weights that brings the distribution of the client sample into very close agreement on each of the four variables.

Response Rates for the Follow-Up Survey with Foreclosure Mitigation Clients

Each client enrolled in the study will be asked to complete the follow-up survey. We anticipate a 60 percent response rate among foreclosure mitigation clients. This reflects the expectation that some of these clients may be under considerable financial strain or may have experienced a foreclosure, making them less willing to want to participate in a survey. Foreclosure mitigation clients may also be difficult to locate if they have experienced a foreclosure or been forced to move into a less stable housing situation. Exhibit B-2 shows the expected sample size for the follow up survey with foreclosure mitigation clients.

Exhibit B-2. Expected Sample Size for Follow-Up Survey

Group	Initial Sample Size	Expected Survey Response Rate	Expected Sample Size at Follow-Up
Foreclosure Mitigation	825	60%	495

Weighting Procedures

We will calculate sampling weights for use in estimating the characteristics and outcomes of foreclosure counseling clients nationally. The sampling weights will have several components. The base sampling weight will reflect the probability of selection of each sample client. The base sampling weights of the baseline survey respondents will be adjusted for nonresponse as described above. We will then compare the characteristics of the follow-up survey respondents with the baseline survey respondents. This nonresponse analysis will be used to identify one or more characteristics that can be used to form nonresponse adjustment cells. We will then sort the follow-up survey respondents into the nonresponse adjustment cells and ratio-adjust the baseline survey weights to account for follow-up survey nonresponse.

In addition to developing sampling weights it will be necessary to use software such as SUDAAN to obtain valid standard errors for the national estimates.

B2.2 Justification of Level of Accuracy

Expected Level of Precision for the Follow-Up Survey

The use of a two-stage cluster sample design will result in some reduction in the effective sample size due to homogeneity of clients in a given group within a given agency. Stated otherwise, the clients selected from within a given agency and group cannot be regarded as statistically independent of each other (in contrast to a simple random sample drawn from among all clients nationally within a given group). The degree of homogeneity is measured by the intra-cluster correlation, which we expect to average around 0.01 based on what we know about the nature of these agencies. Also, sampling variances are likely to be increased by 10 to 20 percent as the result unequal weights arising primarily from agency level sample size shortfalls and unit non-response. Based on these assumptions, we show in Exhibit B-3 the sample size of clients, the effective sample size of clients, and the margin of error. The margin of error represents the 95-percent confidence interval half-width for a population proportion equal to 50 percent. In other words, if the population characteristic being measured has a true value of 50 percent, the margin of error indicates the range of values within which there is a 95 percent chance that the sample mean will fall, given the indicated sample size. The margin of error shown in Exhibit B-3 is within acceptable bounds of precision.

The overall sample size for the follow-up survey with foreclosure mitigation clients is 825 clients. In our sample size calculations we assumed a response rate to the follow-up survey of 60 percent for foreclosure mitigation clients. Following the completion of the follow-up survey, we plan to conduct an analysis of potential nonresponse bias, using information available for all cases in the initial sample from the Baseline Questionnaires and agency-captured data. This analysis will examine the extent to which the follow-up survey respondents may differ significantly in their initial characteristics from the survey nonrespondents.

Exhibit B-3. Margins of Error for the Follow-Up Survey

Group	Initial Sample Size	Expected Sample Size at Follow-Up	Expected Effective Sample Size at Follow-Up	Expected Margin of Error
Foreclosure Mitigation	825	495	360	.5.2 percentage points

B3 Maximizing Response Rates

As shown in Exhibit B-2 above, we expect a 60 percent response rate to the follow-up survey with foreclosure mitigation clients. The estimated response rate for foreclosure mitigation clients reflects the expectation that some of these clients may be under considerable financial strain or may have experienced a foreclosure, making them less willing to want to participate in a survey. To maximize response rates, an advance letter will be sent to clients enrolled in the study notifying them of the upcoming survey and reminding them of their prior agreement to participate in the study. In addition, respondents will be offered a $20 incentive for completing the survey.

B4 Tests of Procedures or Methods

Drafts of the follow-up survey instrument have been reviewed by housing counseling practitioners in order to ensure that the instrument is clear, flows well, and is as concise as possible. In August 2010, we pre-tested the survey instrument with seven respondents.

B5 Statistical Consultation and Information Collection Agents

Office of Policy Development and Research will work with Abt Associates Inc. and its subcontractor IMPAQ International Consulting to conduct the data collection. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Policy Development and Research, Program Evaluation Division, serves as Government Technical Representative (GTR). Her supervisor is Ms. Carol Star. Dr. Myhre and Ms. Star are together responsible for the statistical aspects of the study and can be contacted at (202) 402-5705 and (202) 402-6139, respectively. The Abt Associates Project Director is Ms. Jennifer Turnham, who can be reached at (410) 382-4837 and the Abt Associates survey statistician is Mr. Michael Battaglia, available at (617) 349-2425.

1 These tallies of the number of new clients assisted by the agencies take into account large agencies where a subsample of workers participated in the sampling activity.

Housing Counseling Outcomes Study Follow-Up Survey – Foreclosure OMB Part B 6