DHAP IR Part B (12 31 08)

Part B
Statistical Methods

The U.S. Department of the Housing and Urban Development (HUD) is beginning an important national study of Disaster Housing Assistance Program (DHAP) families transitioning from stepped-up rents (i.e., Phase I) and $0 rent (i.e., Phase II/Phase III) to market rate or assisted housing. The study is currently being conducted by a combination of HUD staff and Abt Associates Inc. HUD has engaged Abt Associates Inc to conduct the sampling of the Disaster Housing Assistance Program (DHAP) Phase I/II/III families and to assist the sample of 12 DHAP PHAs with consent form and baseline survey administration. The sampling, consent form, and baseline questionnaire administration is the first part of a larger DHAP Incremental Rent Transition (IRT) Study, which will be an outcome evaluation of the incremental rent aspect of DHAP. HUD expects to engage a contractor to conduct the rest of the DHAP IRT Study, including a 12-month follow-up telephone survey with the sample of DHAP Phase I/II/III families, by September 30, 2009. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Police Development and Research, is the current project director of this initial data collection. She will serve as the Government Technical Representative (GTR) of the study once a contract is awarded to a contractor. Although it is unusual for HUD to seek study participants’ consent to participate in a research study that has not yet completely been awarded to a contractor, but HUD is in an interesting position in that that study participants are not HUD clients and HUD will no longer have any access to the study participants (DHAP Phase 1, Phase II, and Phase III families) after DHAP ends on February 28, 2009.

The DHAP program presents a unique opportunity to track families transitioning from stepped-up rents (i.e., Phase I) and $0 rent (i.e., Phase II/Phase III) to market rate or assisted housing and to measure outcomes over time.  Both Phase I and Phase II/Phase III of DHAP are time-limited, ending in March 2009.  We suggest using FEMA data and Disaster Information System (DIS) data to measure initial differences between Phase I and Phase II/Phase III families and using hits on National Change of Address (NCOA) database and Public and Indian Housing Information Clearinghouse (PIC) (for families that become HUD-assisted families after DHAP ends) every six months to track outcomes for Phase I and Phase II/III families.  We will do the data tracking internally at first.  We then propose having contractors conduct a telephone survey of a sample of the Phase I and Phase II/III families 12 months after the February 28, 2009 program end date, to identify if there are different outcomes for these two groups of families. This study would enable the Department to answer the question of how does one transition people most effectively from rental assistance to market rate housing after a disaster. Additionally, since DHAP differs from the regular Housing Choice Voucher (HCV) program, HUD wants to observe if DHAP vouchers and their differential rents are having a different effect on housing, employment, education, health, and other outcomes after DHAP ends February 28, 2009.

The study will focus on DHAP families in Phase I and Phase II/III of DHAP. Public housing agencies (PHAs) administering DHAP will be selected to administer the consent form and baseline data collection form, from which a random sample of 1,500 Phase I families and 1,500 Phase II/III families will be selected. We expect to achieve an 80 percent response rate for the Phase I families and Phase II/III families, so initial sample sizes will be set at 1,500 Phase I families and 1,500 Phase II/III families to reach the goal of enrolling 1,200 clients of each type. We expect the response rate for the Phase I/II/III families to be relatively high because the contractor will work with the DHAP PHAs’ case managers to administer the consent form and baseline questionnaire.

B1 Potential Respondent Universe

The target population for this survey is families in Phase I and Phase II/III of DHAP served by PHAs with at least 600 DHAP clients or more. The cutoff of 600 clients was derived as part of the sampling plan to assure that the sampled agencies would have a sufficient volume of clients to merit including them among the agencies selected for enrolling the outcome panel and to ensure that the PHAs were large enough so that administering the consent form and baseline questionnaires to a sample of approximately DHAP clients would not be an overwhelming burden. Twelve PHAs serve at least 600 DHAP clients annually in at least one of the DHAP phases and account for 75.26% of all DHAP clients served or 23,985 out of a total of 31,870 clients. The number of DHAP clients ranges from 615 to 6,661 clients, with an average of approximately 2,665 DHAP clients per PHA.

The objective is to select a representative random sample of families from this population. The sample must be enrolled through participating DHAP PHAs. The study will focus on two groups of DHAP clients: DHAP Phase I, and DHAP Phase II/III. Each type of client is receiving case management, but the type of housing assistance varies by type of client. DHAP Phase I clients will be transitioning to market rate rents through a stepped-down rent system where the DHAP Phase I clients have seen their housing assistance payments decrease (and consequently their rents increase) by $50 a month since their DHAP start date. Phase II/III clients have an immediate transition to market-rate rents once DHAP ends on February 28, 2009 because DHAP Phase II/III clients have been paying $0 rent up to the fair market rent (FMR) since their DHAP start date.

The follow-up panel will consist of the same random ample of clients from each of the two groups, selected for the baseline study. Clients will be randomly sampled in January 2009 and consent form and baseline questionnaire administration will last approximately 4 weeks, with a possible extension up to no more than 8 weeks. An administration/data collection period longer than 8 weeks would place too much burden on the agencies in the sample and is not programmatically feasible since DHAP ends on February 28, 2009. The Department does not expect to begin sampling until January 2, 2009. The Department expects to begin consent form and baseline questionnaire administration the week of January 5, 2008 and continue through the end of January 2009 because final DHAP assistance will be issued on February 1, 2009 so we expect DHAP clients will not have much further contact with the PHA or their case management contractor after that date.

Selecting a sample of 1,500 families in each of the two lists gives a total sample of 3,000 families. The baseline response rate is expected to be approximately 80 percent for DHAP Phase I and Phase II/III families or higher, because the only component of unit nonresponse is failing to obtain consent. We expect the response rate for the Phase I/II/III families to be relatively high because the contractor will go through the DHAP PHAs’ case management contractors, who already have a relationship with the families.

B2 Statistical Methods

B2.1 Sampling Plan

As stated earlier, the universe of PHAs from which we will select the sample of families will be restricted to DHAP PHAs that serve 600 DHAP clients or more. We have used the December 1, 2008 update of the FY 2008 Disaster Information System (DIS) data to develop the sampling frame. We have restricted the selection to 12 DHAP PHAs to obtain a random sample of DHAP clients, to minimize burden on the number and size of DHAP PHAs participating in the study, and to increase our client enrollment rate.

We propose the following sampling design for selecting a probability sample of families.

The families in each of the 12 PHAs in the two groups will be listed by PHA. For example, for the Phase I list, all families in PHA 1 will appear first on the list, followed by all families in PHA 2, and so on. The families in PHA 12 will appear last in the list. A similar list of families will be developed for Phase II/III.

We plan to select an equal probability systematic sample of 1,500 families from each list, thus giving an equal probability of selection for each of the families in the first list irrespective of the PHA in the first list and similarly in the second list. The selected families will be identified by PHA. This method avoids the allocation of the total sample to each PHA in each list which may result in unequal sampling weights and design effects.

Based on FY 2008 DIS data, we have divided the client population into the two groups as shown in Exhibit B-1. Phase I group is larger than the Phase II/III group.

Exhibit B-1. Client Population Distribution Based on FY 2008 DIS Data¹

Group	Approximate Annual Population of Clients	Approximate Annual Population of Clients in 12 Sampled DHAP PHAs
Phase I	19,478	13,748
Phase II/Phase III only	12,392	10,237

Our proposed sampling design will yield estimates for the PHAs sampled for both of these groups. Our goal is to sample 1,500 clients from Phase I and 1,500 clients from Phase II/III so that we may have an approximately equal level of precision at the follow-up interview for each group.

The selection of a random sample of clients from each group must be practical to implement in the field. Our experience indicates that attempting to select a subsample of individuals near program termination is very difficult for PHAs to carry out correctly, but we will provide each PHA with a list of the sampled clients so the PHA will not have to do any sampling of their own. If strong seasonal or time effects existed, this selection method could introduce a selection bias. For our specific population and survey topic, having a sampling intake period of unequal length but no longer than 8 weeks should not introduce a selection bias.

Weighting Procedures

We will calculate sampling weights for use in estimating the characteristics and outcomes of clients in the sampled DHAP PHAs. The overall sampling weight will combine a base sampling weight and an adjustment for nonresponse. The base sampling weight is the inverse of the probability of selection. (For example, if the probability of selection of a family is 0.25 then its sampling weight is equal to 4.) For each of the groups the base sampling weights will then be adjusted for baseline survey and follow-up survey unit nonresponse within agency (assuming this follow-up survey is implemented). The nonresponse bias assessment for the nonresponse group may identify some factors related to nonresponse and it may be necessary to incorporate them into the weighting procedures.

In addition to developing sampling weights it will be necessary to use software such as SUDAAN to obtain valid standard errors for the national estimates.

B2.2 Justification of Level of Accuracy

Expected Level of Precision for the Follow-Up Interview

The overall sample size for follow-up from the baseline is 3,000 clients. In our sample size calculations we assumed a response rate of 80 percent for the Phase I and 80 percent for the Phase II/III clients in terms of study participation, and a response rate of 70 percent to the 12-month follow-up survey for the Phase I and 70 percent for the Phase II/III clients. We typically achieve an 80 percent response rate for our study participation and slightly less for follow up, but since we expect this population to be harder to engage and track due to their experiences after Hurricanes Katrina and Rita, we are anticipating a lower response rate of 70 percent. For both groups we plan to conduct an analysis of potential nonresponse bias, using information available for all cases in the initial sample from the Baseline Questionnaires and agency-captured data. This analysis will examine the extent to which the follow-up survey respondents may differ significantly in their initial characteristics from the survey nonrespondents. If the population characteristic that is being measured has a true value of 50%, then the margin of error of sample percentages based on a sample of 1,200 families will be plus or minus 2.83 percentage points at 95% confidence level. The expected margins of error for samples at follow-up are shown in Exhibit B-2.

Exhibit B-2. Expected Sample Sizes and Margin of Errors for the Follow-Up Survey

Group	Expected Initial Sample Size	Expected Sample Size at Enrollment	Expected Sample Size at Follow-Up	Expected Margin of Error
Phase 1	1,500	1,200	840	3.4 percentage points
Phase II/III	1,500	1,200	735	.3.6 percentage points

B3 Maximizing Response Rates

During the four to eight-week period when the study sample is being enrolled, each sampled DHAP client will be informed by staff of the participating DHAP PHA or their case management contractor about the nature and purpose of the study being conducted for HUD, what would be required of them if they participate, and will be asked to provide their consent to participate.² Clients who agree to participate in the study will be asked to complete a baseline questionnaire and to provide their consent to be tracked over a two-year period to identify outcomes from their counseling and participate in a follow-up telephone survey conducted 12 months after program end date by study staff. As compensation for their cooperation, clients will be given a $15 gift card at the time of completing the consent form and baseline questionnaire.³ Participation in the study will be voluntary. DHAP PHA and case management contractor staff involved in informing clients about the study will be instructed to describe the benefits of the study for the disaster housing industry to encourage their participation. Since the DHAP clients will have voluntarily sought out the services of the DHAP PHA and their case management contractor it is anticipated that clients will have a high degree of confidence in the trustworthiness of the agency. A response rate of at least 80 percent is anticipated for the Phase I and Phase II/III clients both because of this high level of trust and because it will require relatively little effort for clients to participate—essentially completing the 15 to 20 minute baseline questionnaire during a visit to their case manager.

B4 Tests of Procedures or Methods

Early drafts of the instrument have been reviewed by HUD staff involved in administering and studying DHAP in order to ensure that the instrument is clear, flows well, and is as concise as possible. In addition, pre-tests of the instrument will be conducted in December 2008 with up to nine respondents.

B5 Statistical Consultation and Information Collection Agents

HUD’s Office of Policy Development and Research will evaluate the hypothesized differential impact of the transition from stepped down rents to market-rate rents as compared to the transition from $0 rent to market-rate rent for the DHAP Phase I and DHAP Phase II/Phase III families in terms of housing, income, employment, and other outcomes. The study is currently being conducted by Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Police Development and Research, Program Evaluation Division, who will serve as the Government Technical Representative (GTR) of the study once a contract is awarded to a private research organization, and Abt Associates Inc., which has a contract to assist in the consent form and baseline questionnaire administration for the DHAP IRT Study. It is unusual for HUD to seek study participants’ consent to participate in a research study that has not yet been awarded to a contractor, but HUD faces very tight time constraints in that study participants are not HUD clients and HUD will no longer have any access to the study participants (DHAP Phase I/II/III families) after DHAP ends on February 28, 2009. The Department also expects that since the last DHAP assistance will be issued on February 1, 2009, DHAP clients will not have much further contact with the PHA or their case management contractor after that date so we do not have time to have a private research organization administer the consent form and baseline questionnaire. Marina L. Myhre, Ph.D. has primary responsibility for this study. Her supervisor is Mark D. Shroder, Ph.D. They are the people responsible for the statistical aspects of the survey design and can be contacted at (202) 402-5705 and (202) 402-5922, respectively.

1 The client population distribution is current as of December 1, 2008.

2 A copy of the consent form is presented in Appendix B. The form will be translated into Spanish or other languages as needed based on the clientele of the agencies sampled for inclusion in the study.

3 While everyone is eligible to receive a free copy of their credit report from each credit reporting agency each year, credit scores can only be obtained for a fee (generally $10 to $15). In addition, clients will also be spared the effort required to make requests for their freely annual credit reports from three separate agencies.

7