RAD Phase II Part B_11_14_2017

Evaluation of the Rental Assistance Demonstration (RAD) Program, Phase 2

Supporting Statement for Paperwork Reduction Act Submission, Part B

U.S. Department of Housing & Urban Development (HUD)

Office of Policy Development & Research (PD&R)

February 28, 2017

1. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

This Information Collection Request (ICR) includes two data collection instruments: (1) a RAD Post-Conversion telephone interview to be administered to 23 PHAs currently managing the 24 RAD projects in our sample (form included in Appendix A) to help measure PHA’s experience with RAD after completion of conversion and any rehabilitation or new construction; and (2) a Resident Survey, which will be administered to 400 tenants of RAD projects to measure the impact of RAD on tenants residing in projects at the time of conversion (form included as Appendix B). See Table 10 below.

For the Post-Conversion telephone interview, the respondent universe is composed of 23 PHAs (24 RAD projects). The sample of 24 projects was stratified by PHA size and property performance (REAC score) and randomly selected from the 132 RAD public housing projects that had received a Commitment to Enter Into a Housing Assistance Payment (CHAP) between September 2012 and December 2013, and had either closed or received a RAD Conversion Commitment (RCC) by December 31, 2014. These 132 projects represent 74 PHAs. The 24 projects were selected during phase one of the study. (OMB Approval No. 2528-0304) The 23 participating PHAs have already completed the Web survey and RAD implementation interviews administered during phase one of the study.

Because a successful analysis of resident outcomes requires inclusion of former residents who do not return to converted units, enrollment and tracking of tenants to be surveyed had to begin before properties closed. This approach ensured that residents’ contact information could be obtained before they left the RAD property and were not reachable, and therefore that these residents could be contacted to determine their voluntary enrollment in the study.

The sampling for enrollment in the resident study has occurred in two stages. In the first stage, 19 RAD sites were chosen from a sampling frame of properties that have received an RCC but not yet completed closing. In the second stage, a random stratified sample of residents living in each of the RAD sites at the time of closing is drawn from the total number of units in the RAD sites. Residents are selected using HUD’s PIC records. The universe of RAD sites for the resident survey includes 3,011 units. Selection of RAD sites and a sample of residents in those sites was completed during phase one (OMB Approval No. 2528-0304).

Table 10. Number of Entities in Universe and Sample

Interview or Survey	Universe of Respondents	Sample Respondents
Post-Conversion Interview	74 PHAs owning 132 RAD projects in our universe	23 PHAs owning 24 RAD projects in our sample
Resident Impact Survey	3,011 units	400 units

Expected Response Rate

It is anticipated that almost 100 percent of the 23 PHAs will complete the post-conversion interviews. To boost response rates, we will contact non-respondents by email and telephone every two weeks during the period allotted for the data collection. In extreme circumstances, we will request HUD to reach out to non-respondents. This method improved responses during phase one of the study. All 23 PHAs completed the phase one survey and only one did not complete the phase one interview.

Based on our prior experience with similar low-income housing resident survey efforts, we expected that 20 percent of the residents contacted for the RAD survey would agree to enroll in the study, yielding a sample size of 400 for the resident survey. Enrollment across 19 properties has resulted in 512 enrollments in 1,669 units attempted. See Table 11 below. Analysis by specific PHA is not anticipated, but subgroups for analyses would include region, size of the PHA, and household-head characteristics such as race and age (senior vs. non-senior).

Table 11. Resident Impact Survey

Estimated number of units in property sample	3,011
Number of units sampled for enrollment	1,669
Number of residents enrolled	512
Number of residents expected to complete survey	400

We anticipate a response rate of nearly 80 percent for the resident survey by virtue of planned tracking efforts and having already obtained consent to participate at the time of enrollment. Enrolled residents have filled out forms providing complete contact information—including phone numbers—and granting consent for inclusion in the study. They have received a round of reminder postcards since enrolling.

Previously Conducted Collection

A survey and interview has already been collected from the RAD PHAs during phase one of the study. Due to the timing of that data collection, PHAs were not able to provide answers to questions regarding the post-conversion impacts of RAD on their operations or projects. The questions in the proposed interview will fill in this gap in the data and will not duplicate information that HUD has already collected.

Residents have been contacted to enroll in this study but no survey data has been collected. The purpose of the proposed survey is to measure the impact of RAD on their living conditions.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

1. Sample Selection Method

To select the RAD properties for the study, we stratified the RAD universe on two variables: PHA size and latest available inspection score issued by the Real Estate Assessment Center (REAC) under the physical condition indicator of the Public Housing Assessment System (PHAS). PHAs can be classified as either large, medium, or small. The PHAS/Uniform Physical Condition Standards (UPCS) scores range from 0 to 100. Based on the inspection scores, we grouped the PHAs into three mutually exclusive categories: 90 to 100, high performers; 70 to 89, standard performers (medium); and below 70, substandard or troubled (low).

We calculated the sample size in each group as the percentage of the size of the group relative to the population as a whole multiplied by the total sample size of 24 projects. The number of projects in each group reflected this proportion after allowing for a minimum of one property in each group if one were available. As with the sampling universe, the last stratum of our sample (substandard or troubled performers in small PHAs) contained zero projects. The result is a sample that is broadly representative of RAD projects that have closed or that were expected to close during phase one of the evaluation.

Interviews will be conducted by two Econometrica, Inc team members. The interviews will be conducted over the phone. At the onset, the PHA will be asked for permission to record the interview for accuracy purposes. It is at the discretion of each PHA as to whether or not the interview is recorded.

Table 12 depicts the sample of 24 RAD properties based on the stratification criteria. These 24 properties are utilized for the PHA Interview Sample.

Table 12. Percentage of Properties in the Subgroup and Number to Be Selected From Each Subgroup for RAD Sample

PHA Size	Strata	Project Performance	Number of Projects in Universe	Percentage of Projects in Universe	Proportional Distribution of Sample	Number of Projects in Sample
Large	1	High	9	6.8%	1.64	1
	2	Standard	13	9.8%	2.36	2
	3	Substandard	2	1.5%	0.36	1
Medium	4	High	34	25.8%	6.18	6
	5	Standard	45	34.1%	8.18	8
	6	Substandard	1	0.8%	0.18	1
Small	7	High	18	13.6%	3.27	3
	8	Standard	10	7.6%	1.82	2
	9	Substandard	0	0.0%	0.00	0
Total			132	100.00%	24	24

For the enrollment stage of the resident survey, properties were sampled using the same stratification criteria applied to the full universe of RAD properties. The original design expected the sample of properties to be drawn all at once from the universe of approved RAD properties. We found, however, that the episodic nature of projects made this impossible. The possibility of a property being withdrawn or revoked from RAD has also complicated selection; it is difficult to avoid mass mailings to and enrollments of residents in properties that are subsequently withdrawn. We currently consider a property eligible for sampling only after it has reached the RCC stage, since those properties are most likely to make it to closing. We have also limited the sample to no more than one property from a selected PHA, to avoid the possibility of including a single PHA with multiple properties moving through the pipeline at the same time, creating an unbalanced sample.

Table 13. Percentage of Properties in the Subgroup and Number to Be Selected From Each Subgroup for RAD Resident Survey Sample

PHA Size	Strata	Project Performance	Number of Projects in Universe	Percentage of Projects in Universe	Number of Projects in Sample Design	Number of Projects in Sample
Large	1	High	27	10.40%	3	2
	2	Standard	36	13.80%	3	2
	3	Substandard	12	4.60%	1	0
Medium	4	High	57	21.90%	5	5
	5	Standard	72	27.70%	6	4
	6	Substandard	8	3.10%	1	1
Small	7	High	23	8.80%	2	2
	8	Standard	21	8.10%	2	2
	9	Substandard	4	1.50%	1	1
Total			260	100.00%	24	19

For selected properties with 100 or fewer units, we attempted to enroll all residents. For larger properties residents at the selected properties were selected to create a sample of 100 residents, representative of the property, on the basis of four characteristics from HUD administrative data: race/ethnicity, gender, elderly/non-elderly, and disabled/non-disabled.

All enrolled residents will be asked to complete the survey

2. Procedure for Data Collection

Data for the Post-Conversion interviews of 23 PHA represented by the sample of 24 RAD properties will be collected through a telephone interview. In phase one of this study, Econometrica identified the primary respondent who will be contacted for arranging the telephone interviews in phase two. After OMB approval, we will send these respondents an e-mail in advance notifying them that we will be calling them to set up the interview and providing a copy of the questions in advance. A copy of this notification is included in Appendix C. If necessary, we will make repeated phone calls to encourage response and will enlist the support of HUD staff if we fail to receive a response.

For the resident enrollment portion of the data collection effort, targeted residents were sent a letter by a HUD Official requesting the household’s participation in the study and including the Resident Intake Study Correspondence and Intake Form. The letter and Intake Form included a phone number to call with questions. The households were offered incentives for enrollment and maintenance of continued contact. Multiple mail contacts were made to improve response rates. Approximately 500 residents enrolled during phase one of the study.

We will attempt to survey all enrolled residents, with the goal of 400 completed surveys and a minimum of 300 completed surveys for the analysis. The survey phase will begin after residents have had the opportunity to return to the rehabilitated property or have permanently relocated to a new property if the original property was replaced. Properties in the sample will reach this stage at different dates, so fielding will occur over an extended timeline. Some properties in our sample—particularly those that did not require relocation—may already have completed all work. Prior to fielding, the survey firm will conduct final address confirmation and tracking tasks. Tracking and address confirmation activities covered under OMB clearance for the base tasks (OMB Approval No. 2528-0304) can take place before OMB approval for this phase.

The survey will be pretested on no more than nine residents of properties where work has been completed. Three projects will be selected: one new construction, one rehab, and one neither rehab nor new construction. The pretest will identify problems with the instruments’ length and clarity.

We have concluded that the initial stage should rely upon call-in and phone survey administration rather than asking respondents to return completed surveys through the mail. With a planned length of 30 minutes, the survey may discourage self-administration and may prove to be a particular challenge to respondents with low literacy.

Survey fieldwork could begin shortly after OMB clearance for the survey. Fielding the survey will begin with a mailing to the enrollees, with a small ($5) pre-incentive and the offer of an additional incentive ($25) for completing the survey. The letter will request residents to call a toll-free number to complete the survey over the phone. Following the initial letter, the survey firm will make up to four calls to the telephone number collected through the enrollment form. A second mailing will be sent if there is no response, with a $2 pre-incentive. Following the second mailing, the survey firm will attempt telephone contact, with up to four additional attempts.

Because using in-person field staff to either administer the survey or facilitate completion over the phone is prohibitively expensive for 19 sites, we are relying on the dial-in and direct contact phone modes, with the cash incentive, to achieve acceptable response rates at most selected properties. For a limited number of sites, if these efforts fall short, field staff will be sent to convert non-responses through in-person contact, equipped with cell phones to allow residents to call to complete the survey over the phone.

3. Estimation

Despite the measures described above to reduce non-response rates, it is likely that some sample units will remain unmeasured. We will measure response bias and use post-survey adjustment to reduce non-response error.

The resident survey is designed to representative of the universe of RAD projects. Responses to the survey will be weighted by property to correct for differences between the sampled properties and the universe. For example, weighs will correct for the undersample of substandard performing properties, or if necessary to adjust for a low response-rate from a particular property.

Weights representing respondent characteristics will not be constructed. Respondents were selected for enrollment to be representative of their RAD properties. Through several rounds of email and telephone contact during the enrolment process, we worked to convert non-responses and ensure a representative sample. Because survey responses will not be linked to administrative data, we cannot weight survey responses to adjust for differences between the respondents and the other residents of the property.

3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield “reliable” data that can be generalized to the universe studied.

To achieve a high response rate, we will take four steps that are consistent with contemporary strategies in the field of survey research.

First, enrollment efforts for the resident survey began with a notification letter from HUD’s Deputy Assistant Secretary for the Office of Research, Evaluation and Monitoring, and an enrollment/tracking form to the sampled population. A $5 gift card incentive was offered for participation. A reminder postcard was mailed two weeks after the first mailing. A second notification letter and enrollment form is subsequently mailed to those that have not enrolled. A final recruitment package included a $2 incentive. We made up to 5 attempts by telephone to reach residents who do not enroll after receiving three mailings. Finally, we will conduct a non-response bias analysis on the results of the enrollment process and the survey.

Second, the resident survey is designed to be easily understandable and reasonably quick to complete, no more than half an hour. Attempts to gain resident cooperation will follow several stages:

• Mail an advance notification letter from HUD to the sampled population, including a $5 cash/gift card incentive for participation and request to call an 800 number to complete the survey

• Attempt four outbound calls to participants to complete survey

• Pause telephone outreach and mail 8-page questionnaire (with $2 pre-incentive).

• Carry out a follow-up survey completion effort by telephone (4 call attempts)

• Conduct in-person follow up at no more than 15 sites

• Provide $25 post-incentives for completion of survey by telephone, in person, or by mail

Third, respondents are treated as real people with distinct identities. The distribution mailings are addressed to specific respondents rather than to a generic “householder.”

Fourth, the cover letter accompanying the survey provides a clear explanation of the reasons for the data collection and urges participation. The cover letter also encourages response by explaining that responses will be kept confidential, meaning that data will be made available in a form that would not reveal the identity of the person making the comment.

This design includes several strategies to overcome non-response, of particular importance given the extended period since enrollment for some respondents. First, the pre-incentives will foster increased confidence in the full incentive for completing the survey. Second, the incentive itself should boost response rates. Finally, the option of in-person surveys will allow us to address properties with low response rates. Because we will not know the response rates through the phone survey for all properties at the same time, we will make the decision to field the survey in person at a particular property based on being able to meet the minimum number of completed surveys. We will wait to begin in-person fielding until response rates can be compared across several properties.

For the PHA interviews, we will be contacting PHAs that have already participated in the RAD evaluation during phase one. This second round of interviews will be treated as a follow-up to the initial interviews. If necessary, we will enlist the support of HUD staff to address any failures of a PHA to respond after several requests to set up an interview.

See Appendix C for copies of notifications.

Strategies for Mitigating Non-Response Bias

We do not anticipate significant non-responses for the PHA post-conversion interviews, as explained above, primarily because most of the PHAs in our sample have already participated in phase one of the study. If responses are shown to be non-representative of the sample, they will be weighted to reflect the population more accurately.

During the resident enrollment phase, the goal has been to build a sample that is representative of RAD properties and the residents in those properties, subject to the constraint of having sufficient numbers of properties that are close to closing but have not closed. We selected properties for the sample using PHA size and project performance although only properties that have received an RAD Conversion Commitment (RCC) and in which work has not yet begun are eligible. We are monitoring responses for two primary problems that could require action:

1. Enrollment rates from a particular property are below our target threshold (20%).

2. Enrollment rates from a particular group defined by available demographic characteristics are below our target threshold (to be determined by analysis of the universe of RAD property residents).

While our plan has been, if enrollment response rates for a particular property are low, to make extra effort through additional mailings and phone contact to non-respondents to increase enrollment at that property, our enrollment rates have exceeded expectations and this has not been necessary. The property sample, of 19 properties, contains one fewer property than would be ideal for a representative sample from the substandard performing property group. Responses from the properties in that group will be weighted up to correct for the undersample. Other deviations from a representative sample will be handled by weighting responses to the survey based on property characteristics. Because responses will not be linked to administrative data, weighting by respondent characteristics is not being considered.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The PHA interview instrument will be tested with HUD program staff, outside consultants and PHAs who are not in the sample. The size of the pretest population will not exceed OMB guidelines.

The resident survey will be tested by attempting completion with 9 randomly selected residents of RAD properties at which all RAD-related improvements have been completed.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

HUD has contracted with Econometrica (prime contractor) to conduct phase two of the RAD evaluation. Econometrica designed and will administer the PHA interview. The Urban Institute is Econometrica’s subcontractor and is responsible for the design and execution of the resident survey. The following table lists those who developed the collection strategy, designed the instruments, and will participate in the data collection effort, analyze the data, and prepare reports. Tables 14 and 15 below show the names, affiliations and contact information for those involved in the survey research.

Table 14. Names, Affiliations and Contact Information of Contractors Involved in Data Collection and Analysis

Contractor Personnel	Phone Number
Prime – Econometrica, Inc.
Principal Investigator
Chuck Hanson	(301) 657-9883
Project Manager
Dennis Stout	(301) 657-9883
Senior Analysts
Fred Bellemore	(301) 657-9883
Paul Watkins	(301) 657-9883
Margaret McGilvray	(301) 657-9883
David Ruiz	(301) 657-9883
Analyst
Bethany Hase	(301) 657-9883
Tim Beggs	(301) 657-9883
Research Assistant
Linden Li	(301) 657-9883
Briana Alterman	(301) 657-9883
Subcontractor – Urban Institute
Sue Popkin, Project Manager	(202) 833-7200
Chris Hayes, Principal Investigator	(202) 833-7200
Elaine Morley, Sr. Analyst	(202) 833-7200
Alex Derian, Analyst	(202) 833-7200
Subcontractor – Other
Jaime Bordenave, SME, The Communities Group	(202) 667-3002
Dr. John Weicher, SME, Hudson Institute	(202) 974-2425

Table 15. HUD Staff Who Have Advised on the Survey and Interview Instruments

Name	HUD Staff Position	Phone Number
Nathan Bossie	HUD GTR	(202) 402-2046
Paul Joice	HUD GTM	(312) 913-8597

Page 13 – Part B Supporting Statement