AHPP OMB Submission_Part B revised 8-19-08

Part B
Statistical Methods

HUD has contracted with Abt Associates to conduct the household outcomes survey of AHPP participants and, in Louisiana, a comparison group of households who were found eligible for AHPP but did not participate in the program.

B1 Potential Respondent Universe

The AHPP demonstration implementation is still underway, so the final number of units to be produced – and thus households to be served – is unknown. We expect the total number of units by state will be approximately as follows:

• Alabama – 100 units

• Louisiana – 500 units

• Mississippi – 3,500 units

• Texas – 100 units

This would result in a total potential respondent universe for people receiving AHPP units of 4,200 households. In Louisiana, where more applicants are expected to apply than there are available units, we will also survey people who applied for the program but were not randomly selected to receive a unit (control group members). The size of this group is expected to be at least 1,000 people, but this will not be known until the application process is complete sometime in 2008.

B2 Statistical Methods

B2.1 Sampling Plan

Household Outcomes Survey Sample Allocation. We plan on selecting a sample of 1,400 households for the survey sample. In allocating the sample, we tried to balance the goals of measuring impacts for each state and taking advantage of the more definitive estimates of impact with the treatment-control group comparison in Louisiana. We also took into account that Mississippi has the largest program and was awarded more than 70 percent of the AHPP funding and will be the first site on which outcomes are available. Based on these considerations, Exhibit 3 shows the planned allocation of the sample for the first household outcomes survey. A discussion of the sample and precision of the estimates for each state follows in Section B2.2.

Exhibit 3. Planned Samples Sizes for the Household Outcomes Survey

State	Sample Size for First Household Outcomes Survey	Number of Completed Surveys (assuming 80% response rate)
Mississippi	400	320
Louisiana	800	640
Alabama	100	80
Texas	100	80
Total	1400	1120

Sample Selection. Our plan is to select all AHPP recipients in Alabama and Texas and samples in Mississippi and Louisiana. For each applicant selected, we will target the adult in the household who completed the application or the baseline survey.

In Mississippi, our plan is to select a simple random sample of AHPP recipients who moved into their AHPP unit at least six months prior to the start of the first survey period. A simple random sample means that every person eligible to be selected for the survey has the same chance of being selected for the survey. Even if the person has moved out by the time of the survey, they will remain in the sample and we will survey them about their experiences. We do not plan to survey people in Mississippi who applied for the program, but were determined to be ineligible, turned down an offered unit, or accepted a unit but never moved into the unit. For the first follow-up survey, this means that the sample will represent applicants in Mississippi who lived in an AHPP unit for at least one day at least 6 months prior to the start of the survey. (If the survey starts in July 2008, this means everyone who moved into their unit by the end of 2007.)

The first follow-up survey in Mississippi is not being scheduled for a later time period (such as the same time as the first follow-up survey in the other sites) for several reasons. First, a few of the temporary AHPP units have already been demobilized as some of the recipients have moved into permanent housing and all the temporary units are suppose to be demobilized by the end of March 2009. We want to interview as many people as possible while living in their AHPP unit (as long as they have lived there at least six months). Second, Mississippi’s first units were occupied in June 2007 and they are the only site with occupied units as of April 2008. We want to capture the affect of AHPP on their quality of life at approximately the same time in the housing cycle as the other sites. Third, both FEMA and Congressional staffers need to know how well the program is working as soon as possible so it can factor into decisions about disaster planning.

In planning for the second household outcomes survey in 2010, we plan to include households that received their AHPP unit in the survey sample because it is important to capture any differences in either the program or households receiving units in January 2008 or later. The Mississippi program is evolving as it matures. It is expanding into more counties, is negotiating with non-profits and housing agencies to develop group sites, and is also moving from a program that provides temporary housing for up to two years to a program that provides permanent housing. (Some of the earlier households that moved into what was called temporary housing will have their housing reclassified as permanent housing if the household needs it and the local city and county governments approve the change.) Thus, for the second follow-up survey in Mississippi, we plan on splitting the sample based on whether or not the AHPP unit was occupied before or after January 2008.¹ The split will be approximately 50-50 as that is the expect distribution of units occupied before and after January 2008. The pre-January 2008 sample will be a subsample of the households selected for the first follow-up survey.

In Louisiana, we plan to select controls and treatments in exactly the same way to ensure that the only difference between the groups is their receipt of an AHPP unit. The Louisiana AHPP staff have discussed up to six distinct sites where recipients may move in Louisiana: (1) Jackson Barracks, (2) a senior village in Baton Rouge, (3) a mixed-age group site in Baton Rouge, (4) a group site in New Orleans, (5) infill housing in New Orleans, and (6) a group site in Lake Charles. It is not likely that all these sites will work out, so there will be likely be fewer sites in the end. Our plan for the July 2009 survey is to select the Louisiana sample in proportion to the number of occupied units at each of the sites. However, in consultation with HUD and FEMA, we may exclude some sites from the sample selection so that we can have larger samples at the other sites. The decision on excluding Louisiana sites would be based on the need to have more precise estimates from the sites most important for setting future disaster response policies.

B2.2 Justification of Level of Accuracy

The justification of the sampling approach and level of accuracy the sample allocation presented in Exhibit 3 (above) is as follows:

Mississippi. A relatively large part of the sample was allocated to Mississippi because we wanted to obtain reasonable levels of precision for the largest program. With 320 completes, the 95 percent confidence interval for a yes/no question in which 50 percent of the people answer “yes” is 44.5 to 55.5 percent (i.e., +/- 5.5 percentage points).

Louisiana. In Louisiana, the sample will be evenly split between treatments and controls. This means that we plan to select 400 treatments and 400 controls and expect approximately 320 respondents from each group. The main analysis for Louisiana will be based on treatment-control comparisons. With 320 completes in each group, we will be able to detect true differences between the treatment and control group of 9.7 percentage points or larger.²

Alabama and Texas. In Alabama and Texas, we are proposing to target all the AHPP recipients— expected to be approximately 100 people per state—for the survey. By including every AHPP recipient in the sample, the estimates are extremely precise even though the size of the sample is relatively small. If we achieve a 100-percent response rate, there is no sampling error and thus estimate is the true value for the population. Some people will likely decline to participate in the survey or be unavailable during the survey field period, so the response rate will be less than 100 percent, but the sampling error will be as small as is possible with this size AHPP project.

B3 Maximizing Response Rates

The Household Outcomes Survey will be administered by phone with in-person follow-up for non-responses or respondents that prefer a face-to-face interview. We will maximize participation by sending an advance letter notifying potential respondents about the study and encouraging their participation (see Appendix E). We will offer respondents the flexibility to respond either by phone or with an in-person interview. We will also use local interviewers who may have a better rapport with respondents because they have been through similar experiences during and since the hurricanes. We will offer respondents a $25 incentive payment for the first household outcomes survey and a $25 incentive for the second household outcomes survey.

We will also undertake a number of steps to minimize study subject attrition in order to maximize response rates. A tracking database is being developed to allow for the systematic collection and management of participant location data. The AHPP participant tracking system will contain information on the household and its members such as key dates (e.g., date baseline survey was received, date of random assignment), family membership, telephone and address information, and secondary contacts.

The participant tracking database will be updated on a quarterly basis. Updates will come from a mix of active and passive tracking sources to maintain the location of AHPP participants over time. Active tracking requires direct contact with a program participant and allows us to confirm that our location data is accurate. Passive tracking relies on electronic comparisons of data from the tracking database and data from outside vendors. Passive tracking methods can provide three data elements critical to sample tracking: address updates, phone number updates, and deceased status.

Exhibit 4 presents a summary of the tracking sources we will use over the life of the project and their respective timing.

Exhibit 4. Summary of Proposed Tracking Activities for AHPP Participants

Source	Who will be Covered?	Timing
Passive Tracking Activities by Source
National Change Of Address Database	All heads of household	Quarterly
Telephone Matches	All heads of household	Quarterly
Deceased Database Searches	All heads of household	Annually
Other Passive Sources	All heads of household	Semi-annually following the first NCOA update
Active Tracking Activities by Source
Participant Tracking Letter	All heads of household	8 months after first participants enrolled; again at 22 months
Field Locating	All household heads sampled for the follow-up survey data collection efforts	July 2008 in MS; July 2009 in AL, TX, and LA; July 2010 in all sites
Data System Updates
	All passive and active tracking data	Quarterly, after passive results are processed.

Beginning in May 2008, quarterly National Change of Address (NCOA) updates will be requested. NCOA updates, along with telephone updates will be obtained from a vendor. Annually, we will expand the NCOA search to include a search against the Social Security Death Index. The death index searches will be done just prior to any active tracking or the survey data collection. Identifying sample members who passed away before survey data collection begins allows us to flag them in the databases so we do not attempt to interview them. To do so could place undue burden on the deceased member’s family.

We will also employ active tracking efforts, such as customized participant tracking mailings. All sample members will receive a tracking letter asking them to update or confirm their contact information. All updates obtained through the mailing are sent out to the NCOA vendor for address standardization. This process ensures that all new address data is posted to the data system in proper US Post Office format.

We will implement field locating efforts during the data collection field period. As interviewers work the sample to complete an interview, they utilize several locating tools such as directory assistance calls, online searches for updated information, and door-to-door locating. Updated information is stored in the respondent information booklets to aid interviewers in subsequent data collection efforts. The most recent update, or one that yields the correct address, is updated into the field management system.

B4 Tests of Procedures or Methods

Early drafts of the instrument have been reviewed by HUD personnel, Abt Associates staff, and their subcontractors in order to ensure that the instrument is clear, flows well, and is as concise as possible. In addition, the instruments submitted in this package were pre-tested in April 2008.

Abt interviewers pre-tested the non-experimental version of the survey with 5 people who had lived in an AHPP unit in Mississippi for at least 6 months. The interviews were done by telephone on paper. The paper survey added a bit of time to the pretest instruments because the interviewer had to manually follow skip patterns. In addition, questions that rely on existing data had to use drop sheets with key information listed on them in the pretest. However, once programmed in CAPI, such key information will be included in the program and displayed automatically, reducing the burden on the interviewer.

The pretest findings indicated that the questionnaire was well designed. Interviewers indicated that they had little to no problems in administering the interviews. Respondents did not indicate that the questions were difficult to answer or that the recall was troublesome.

The interviews varied in length from 38 to 70 minutes. As noted above, the length was likely affected a bit by the need to conduct the pretest on paper. Therefore, it is likely that our 45 minute estimate is within the right range. However, there are several factors that could cause this to change. First, and most important, is the respondent population itself. The study population is one that has undergone extreme trauma. In such situations it is very likely that respondents will digress, and want to share their story with the interviewers. We will train our interviewers to tactfully steer the respondents back to the questions at hand, but some respondents will increase the length of the interview by sharing their stories. Other factors for consideration are household size and the presence of children as there are questions in the instrument about other household members. If a respondent lives alone, such questions would obviously be skipped.

B5 Statistical Consultation and Information Collection Agents

HUD has contracted with Abt Associates, Inc. to conduct the data collection. The data collection procedures will be similar to those used in other surveys conducted by Abt Associates. HUD also contracted with Abt Associates to design the survey and establish the sample design. The HUD Government Technical Representative (GTR) reviewed all the proposed procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call either the HUD GTR, Harold Holzman (202-402-5709) or the Abt Associates Principal Investigator, Larry Buron (301-634-1735).

1 This should not be an issue in the other three states. Although these programs do not have occupied units as of April 2008, they are smaller programs and our expected to be completed by the end of 2008.

2 This 9.7 percentage point minimum detectable effect (MDE) is based on a two-sided test at the 90 percent significance level with 80 percent power. This is the worst-case scenario for the MDE because it is based on 50 percent of the control group providing the same answer to a binary question such as a Yes/No question. If 90 percent of the control group responded “yes” to the binary question (or 10 percent said “yes” and 90 percent said “no,”) then the MDE decreases to 6.7 percentage points.

OMB Paperwork Reduction Act Submission for AHPP QOL Evaluation Appendices