RAD OMB_Part B - 01-11-2021 - clean

Supporting Statement for Paperwork Reduction Act Submission, Part B

Rental Assistance Demonstration (RAD) Choice Mobility and Long-term Affordability Evaluation

(OMB Number 2528-New)

2. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, provide the actual response rate achieved during the last collection.

The information collected through this study will be used by HUD to evaluate: 1) the implementation and impact of the choice mobility option; 2) the impact of RAD on long-term preservation and the financial viability of converted properties; 3) the adequacy of asset management for RAD conversions under Project-Based Vouchers (PBVs) and Project-Based Rental Assistance (PBRA); and 4) effect of RAD on PHA’s organization, functions, structure, staffing, and resources. Below, we present a description of each information collection that will discuss: 1) sampling or respondent selection method, 2) the number of entities in the universe, 3) the sample we plan to achieve, and 4) expected response rates for each collection.

Identification of Respondents for the Census of RAD PHAs

The research team plans to survey the entire universe of RAD PHAs. There are currently 347 PHAs with 1,118 RAD conversions, which will increase over the course of this study. The research team expects the universe of RAD PHAs to be approximately 500 at the time of data collection. The research team expects a response rate of 80 percent (400 PHAs). Based on high interest in the RAD program and other surveys of PHAs, for instance HUD’s PHA Homelessness Preferences web survey, we anticipate an 80 percent response rate.

Identification of Respondents for the Survey of non-PHA Property Owners

The research team plans to survey the entire universe of non-PHA property owners. The research team plans to use the census of RAD PHAs (above) and HUD administrative data from the RAD Resource Desk to identify all the property owners of RAD conversions other than those still owned and operated by a PHA. The PHA census will be used to fill in gaps for contact information of RAD property owners not available from HUD. The research team estimates that there are approximately 280 non-PHA owners (about 25 percent of current 1,118 RAD conversions are owned by entities other than the PHA). The number of RAD conversions will increase over the course of this study and the research team expects the universe of non-PHA owners to be 350 at the time of data collection. The research team expects a response rate of 65 percent (228 non-PHA property owners).

Identification of Respondents for the Survey of Choice Mobility Residents

The research team plans to survey a large sample of former RAD residents who used the choice mobility option. The sample will be a high proportion of the universe to allow for detailed analysis. All eligible residents who moved from PBRA developments will be included in the sample; approximately 90 percent of residents who moved from PBV developments will be included. The estimate of the universe of choice mobility users will come from the final analysis of administrative data using certain assumptions as well as verification from PHAs’ response to the survey, which will be administered before the resident surveys. The research team expects the universe of RAD residents who used the choice mobility option to be approximately 1,300 and expects to sample 1,180 residents. We will stratify the universe of residents who have used the choice mobility option by key characteristics, including program, geography, PHA characteristics, elderly and disabled status, race and gender of household head, and household size. The research team expects a response rate of 60 percent (708 choice mobility residents).

Identification of Respondents for the Survey of Non-Choice Mobility Residents

The research team plans to survey a representative sample of RAD property residents who are eligible but have not exercised the choice mobility option. The research team will conduct a short survey with these residents to collect some of the same information collected from the RAD choice mobility residents on their experiences with the administration of and communication about the choice mobility option as well as to ask additional questions on why the residents have not chosen to move with a voucher. This additional sample is necessary for a complete analysis of residents’ use of the choice mobility option. The universe of RAD property residents who are eligible but have not exercised the choice mobility option is currently unknown. It will be estimated from the final analysis of administrative data using certain assumptions as well as verification from PHAs’ response to the survey, which will be administered before the resident surveys. The survey will use sampling methods similar to those for the choice mobility resident survey. The research team has a sampling goal of 420 residents and expects a response rate of 55 percent (231 non-choice mobility residents).

Identification of Respondents for the Qualitative Interviews with PHAs

The research team plans to interview PHA staff to learn about the effect of RAD on PHA’s organization. There are currently 347 PHAs with RAD conversions, which will increase over the course of this study. The research team expects the universe of RAD PHAs to be 500 at the time of data collection. The sample of PHAs will be a purposive sample developed to reflect the average characteristics of PHAs with RAD conversions; specifically, PHA size, census region, and type of housing assistance (PBV/PBRA conversion). The research team has a sampling goal of 25 primary sites (plus back-up sample of two replacements per primary site, if needed). In each site, the team expects to interview about 10 PHA staff. The team expects a response rate of 100% percent (25 PHAs and 250 individuals).

Table 1. Number of Entities in Universe and Sample Sizes

Information Collection	Universe of Respondents	Sample Respondents	Expected Response Rates (Number of Respondents)
Census of RAD PHAs	~500	500	80% (400)
Survey of RAD non-PHA property owner	~350	350	65% (228)
Survey of choice mobility residents	~1,300	1,180	60% (708)
Survey of non-choice mobility residents	~100,000	420	55% (231)
Qualitative interviews with PHAs	~500	25	100% (25 PHAs and 250 individuals)

Previously Conducted Collection

The previous RAD evaluation collected information from some of the same entities but there was a different set of questions. Because the previous data collection was a much smaller sample, the response rates cannot be directly compared to the current effort.

2. Describe the procedures for the collection, including:

1. the statistical methodology for stratification and sample selection;

2. the estimation procedure;

3. the degree of accuracy needed for the purpose described in the justification;

4. any unusual problems requiring specialized sampling procedures; and

5. any use of periodic (less frequent than annual) data collection cycles to reduce burden.

• Statistical methodology for stratification and sample selection

Table 2. Summary of Methods for Stratification and Sample Selection

Information Collection	Stratification and Sampling Methods
Census of RAD PHAs	Sample the entire population of RAD PHAs. We will use stratification to assess non-response bias. Participating PHAs will be stratified by size; region; whether the conversions are family, senior, or both; Public Housing Assessment System (PHAS) scores; and other characteristics to confirm that completed surveys represent PHAs with converted RAD projects. If necessary, the stratification system will be used to target late-stage fielding efforts and in the design of weights. Other than Questions 1 and 2 in Section 1, property-level questions will be asked for a maximum of 5 properties per PHA. Where PHAs have more than 5 properties, a sample of properties will be drawn at random.
Survey of RAD non-PHA property owners	Sample entire population of RAD property owners that are unaffiliated to the PHAs. Property-level questions will be asked for a maximum of 5 properties per property owner. Where property owners represent more than 5 properties, a sample of properties will be drawn at random.
Survey of choice mobility residents	We will stratify the universe of residents by key characteristics, including program, geography, PHA characteristics, elderly and disabled status, race and gender of household head, and household size. All former residents of PBRA properties will be included in the choice mobility option resident sample. Approximately 90 percent of former residents of PBV properties will be included.
Survey of non-choice mobility residents	Residents will be selected to be representative of the universe of RAD residents who are eligible but have not exercised the choice-mobility option.
Qualitative Interviews with PHAs	Purposive sample with selection of sites stratified based on census region, PHA size, and type of housing assistance (PBRA/PBV).

• Estimation procedure

The study team will clean data from the census and the surveys and combine it with the cleaned administrative data collected to produce final working datasets. The PHA organization change study is solely qualitative and will not be included as a dataset. For each of the survey datasets, we will determine whether the final data require weights based on the response rates by key respondent characteristics identified in the stratification systems created for the different surveys or to adjust for non-response bias. The survey team will analyze survey data using STATA or other professional statistical software packages and produce tables categorizing survey respondents overall, by the stratification criteria, and by other characteristics, with a focus on addressing the relevant research questions.

We will code responses to open-ended questions into categories in all three surveys for quantitative analysis; use open-ended responses in qualitative analysis; and may also quote significant or revealing responses, stripped of identifying information, in our reports. We may also use weights to correct for nonresponse bias, which will be determined by comparing responses across fielding stages and by stratification measures. We will analyze the comparison group of residents who have not taken up the choice mobility option for differences from choice mobility users.

We will include descriptive statistics for the quantitative variables from the surveys. We will conduct frequency (counts and percentages) and central tendency (median and average) analyses as well as t-tests or ANOVA to highlight significant differences for both one-way tabulations and crosstabulations. We will emphasize significant differences among the stratification measures and other available indicators identified in consultation with HUD and during data collection.

• The degree of accuracy needed for the purpose described in the justification.

Census of RAD PHAs

By taking a census of all possible respondents, we will have enough sample for all of the statistical procedures or inferences we propose. If we achieve the response rates in Table 1, the data will be more than sufficiently accurate for our purposes. We will conduct tests for nonresponse bias in all cases and make any necessary adjustments.

Questions 1 and 2 in Section 1 will be asked of all properties. Other property-level questions will be asked for a maximum of 5 properties. Using current administrative data, we estimate that 87 percent of PHAs have five or fewer RAD properties, and 57 percent of all properties are in that group. Overall, 67 percent of all properties will be included in the sample. For property-level analysis we will calculate weights using the inverse probability of selection.

Based on information collected from interviews with PHA staff and property owners, we anticipate few property-level differences in survey responses.

Survey of RAD non-PHA property owners

By taking a census of all possible respondents, we will have enough sample for all of the statistical procedures or inferences we propose. If we achieve the response rates in Table 1, the data will be more than sufficiently accurate for our purposes. We will conduct tests for nonresponse bias in all cases and make any necessary adjustments.

For a description of property sampling procedures, see the Census of RAD PHAs.

Survey of choice mobility residents

By taking a census of all possible respondents moving from PBRA developments and 90 percent of respondents moving from PBV developments, we will have enough sample for all of the statistical procedures or inferences we propose. If we achieve the response rates in Table 1, the data will be more than sufficiently accurate for our purposes. We will conduct tests for nonresponse bias in all cases and make any necessary adjustments.

Survey of non-choice mobility residents

The sample will be sufficient to detect differences of 20 percent or more between residents of PBV and PBRA developments, assuming a 55 percent response rate and the more difficult scenario of a higher standard deviation.

Based on data from December 2019, we estimate that 67 percent of choice mobility-option-eligible residents who have not moved live in PBV properties and the remaining 33 percent live in PBRA properties. Based on responses to selected questions in a prior evaluation, such as awareness of the RAD program and satisfaction with PHA communication about RAD, we anticipate that in this survey, proportions (e.g., responses to yes/no questions) will have a typical standard deviation of 0.4 to 0.45. Aiming for an analysis power of 80 percent for comparisons between PBV and PBRA residents produces the following sample size calculations:

Target sample size (completes) to identify significant differences of 20 percent or greater between PBV and PBRA developments converted under RAD:

• 225, assuming PBV is 2/3 of sample, standard deviation is .5, power of 0.8.

• 144, assuming PBV is 2/3 of sample, standard deviation is .4, power of 0.8.

• Any unusual problems requiring specialized sampling procedures

There are no unusual problems that require specialized sampling procedures.

• Use of periodic (less frequent than annual) data collection cycles to reduce burden.

Each of the collections proposed is a one-time occurrence, thus a period cycle is unnecessary.

3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield “reliable” data that can be generalized to the universe studied.

Census of RAD PHAs

• Survey invitation will be sent by U.S. Priority Mail on HUD letterhead, will include a URL for the online survey portal, and will explain the evaluation and stress the importance of the information being collected.

• Two email reminders will be sent at regular intervals:

  • HUD will send a reminder email to non-respondents.

  • The research team will send a reminder on HUD letterhead via First Class mail.

• No incentive will be offered.

• Reminder emails will leverage behavioral science to encourage participation. For example, we will leverage social norms by highlighting the response rate, once it is sufficiently high, and loss aversion through messaging about losing the opportunity to participate in the study.

• After the final contact, a sample of 100 non-respondents will be selected for follow-up by phone, using the sampling frame of all eligible PHAs to identify strata with low responses, bolster the sample, and test for nonresponse bias.

• Participating PHAs will be stratified by size; region; whether the conversions are family, senior, or both; Public Housing Assessment System (PHAS) scores; and other characteristics to confirm that completed surveys represent PHAs with converted RAD projects. If necessary, the stratification system will be used to target late-stage fielding efforts and in the design of weights.

Survey of RAD non-PHA property owners

• Survey invitation will be sent by U.S. Priority Mail on HUD letterhead, will include a URL for the online survey portal, and will explain the evaluation and stress the importance of the information being collected.

• The rest of the survey process will be the same as described above for the census of PHAs.

Survey of choice mobility residents

• For both choice and non-choice mobility populations, we will send residents a survey invitation with the URL for the online survey portal using U.S. Priority Mail.

• The invitation will be in English and in Spanish, as needed, and will include a small pre-incentive of $5. Those who complete the survey will then receive a $45 incentive.

• After two rounds of reminder postcards and a reminder packet including a copy of the survey and a postage-paid return envelope, we will select a sample of non-responders from the choice and non-choice samples combined to receive up to eight phone call attempts for each case.

• These non-responders will receive an additional $10 incentive. The mixed mode options for completion, plus the call-out component, will help maximize response.

Survey of non-choice mobility residents

• The process for outreach and maximizing response rates is the same as for the survey of choice mobility residents.

For both resident surveys, we will also include a call-in number to complete the survey. In the first mailing, we will give recipients the option of taking the web-based survey or calling into a toll-free number to have the survey administered by a live interviewer. If an interviewer is not available when they call in, they would be asked to leave a number for a call-back. The estimated increase to the overall response rate is 15 percent, for a total response rate of 45 percent. This would be consistent with the research team experience with the call-in component of the first RAD evaluation, which achieved a response rate of around 50 percent. The resulting number of completed surveys are shown in Table 3 This number of completions would allow almost all planned analysis.

Following all reminders, we will analyze the population who have completed surveys and draw a sample of 200 households from under-represented strata as compared to the full universe of residents who have taken the choice mobility option. This sample will receive up to three follow-up phone calls and the offer of an additional $10 incentive to complete the survey either online or by returning a hardcopy through regular mail. The survey firm SSRS will use a national telephone number look-up service to supplement phone contact information in administrative data. The total response rate following the call out effort is estimated to be 55 percent.

Table 3. Resident Survey Fielding

Proposed Survey	Universe	Sample	Target Completes	Response Rate (Minimum-Maximum)
Choice mobility residents who moved from RAD PBRA properties	250	250	125–175	50–70%
Choice mobility residents who moved from RAD PBV properties	1,050	930	465–651	50–70%
Non-choice mobility residents remaining in RAD properties	~100,000	420	210-300	50-70%

We will address non-response bias through comparison of known characteristics of respondents to non-respondents and an analysis of the non-response sample at the end of fielding. We calculate that our non-response sample of 200, resulting in at least 100 completes, will identify statistically significant bias at a small effect size (i.e., 10 to 20 percent).

3. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

PHA Census, RAD non-PHA Owners Survey, and both Resident Surveys

We will analyze the results through descriptive statistics, using frequency counts and percentage distributions. We will also test results on key questions using t-tests or ANOVA to highlight significant differences, both for one-way tabulations and cross tabulations for comparisons of interest. For implementation, property outcomes, and impacts on the voucher program, the primary unit of analysis will be PHA and PBV/PBRA projects. For tenant outcomes, the unit of analysis will be residents, although some analysis will be reported at the PHA level.

Surveys from respondents who completed the survey in the non-response effort at the end of fielding will be analyzed in comparison to all other completed surveys to test for non-response bias. Non-response sample sizes will be sufficient to detect significant differences of 20 percent or more.

Weights may be used to correct samples that are not representative according to the sampling frame constructed prior to surveying. Unscaled weights will be capped at 2.5. We do not intend to construct weights to correct for non-response bias.

We will use the census of converted RAD projects to define the non-RAD multifamily housing and public housing project samples. The non-RAD samples will be equal in size and matched to the RAD census groups using genetic matching. This technique provides a quasi-experimental approach that mitigates selection bias generated because RAD participation is voluntary. We expect the sample sizes to be large enough to conduct appropriate statistical tests for the calculated financial performance indicators between the comparison groups, such as comparisons of means tests or ANOVA. To be effective, genetic matching should incorporate all covariates of RAD participation and project selection; otherwise, the control group could differ from the treatment group in an unspecified way. In addition to analyzing matched samples, therefore, we will also use linear regression analysis to explore statistically significant relationships between project and neighborhood characteristics and financial performance, using dummy variables to indicate program (e.g., RAD vs. public housing). In addition, we will perform sensitivity testing by analyzing trends in the selected outcome measures before RAD participation with another year of data and will consider incorporating the Heckman correction into the regression model.

As shown in Table 4, the population of public housing projects is large enough to match against the combined populations of RAD PBRA and PBV projects, the population of non-RAD PBRA projects is large enough to match against the population of RAD PBRA projects, and the population of non-RAD PBV projects is large enough to match against the population of RAD PBV projects.

Table 4. Projects and Units in Select HUD Affordable Rental Housing Programs

Affordable Housing Program	Approx. No. of Projects	Approx. No. of Assisted Units
Public housing*	6,835	1,015,482
RAD Total**	1,103	195,183
Non-RAD PBRA*	17,842	1,236,408
RAD PBRA**	413	125,003
Non-RAD PBV***	5,177	137,883
RAD PBV**	690	70,180

RAD = Rental Assistance Demonstration. PBRA = Project-Based Rental Assistance. PBV = Project-Based Voucher.

* Through 2018; Public housing likely includes some RAD projects that had not closed. Non-RAD PBRA calculated by subtracting RAD PBRA from Project-based Section 8 project and unit counts. Retrieved from https://www.huduser.gov/portal/datasets/ assthsg.html.

** Closed projects through November 22, 2019. From RAD Resource Desk.

*** HUD PD&R Housing Demographic Analysis Division, dated December 23, 2020. Non-RAD PBV calculated by subtracting RAD PBV from all PBV project and unit counts.

Some of the closed RAD projects in our study may be eliminated if the conversion is too recent. For instance, a new construction project will not achieve stabilized occupancy until after construction has been completed, which could occur one or more years after conversion.¹ Also, when a project converts, it may not have a financial statement for the full year. In both cases, the first year’s financial statements would provide an unreliable measure of the building’s financial performance. To mitigate this potential bias, we will analyze RAD conversions in years 1 and 2 to assess whether we need to adjust the number of RAD projects in the study.

We will use two types analysis for this part of the study: comparison of means hypothesis testing and linear regression modeling. These populations are large enough to support tests of statistical significance between RAD and non-RAD properties, so long as the financial data can be obtained.

We will analyze and present responses to the two web-based surveys in frequency tables to describe a portrait of RAD asset management, with a focus on differences between PBV and PBRA projects and gaps between RAD asset management and the standard model of asset management. In describing differences between PBV and PBRA properties, the analysis will test for statistical significance using standard means tests.

3. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

HUD has contracted with Econometrica (prime contractor) to conduct the evaluation; the Urban Institute is Econometrica’s subcontractor. The following table lists those who were consulted or will participate in the data collection effort, analyze the data, or prepare reports. The actual collection of Web survey data will be performed through a Web service that specializes in conducting Internet surveys. Table 5 and Table 6 below show the names, affiliations, and contact information for those involved in the statistical design and the survey research.

Table 5. Names, Affiliations, and Contact Information

Firm	Role	Personnel	Phone Number/Email
Econometrica, Inc. – Prime	Principal Investigator	Fred Bellemore	(301) 657-9883/ [email protected]
	Project Manager	Dennis Stout	(301) 657-9883/[email protected]
	Long-term Preservation Study Lead	Brad Anthony	(301) 657-9883/[email protected]
	PHA Organizational Study Lead	Jennifer Stoloff	(301) 657-9883/[email protected]
	Asset Management Study Lead	Alex Thackeray	(301) 657-9883/ [email protected]
	Analyst	William Le-Hoang	(301) 657-9883/ [email protected]
	Analyst	Gabe Schneider	(301) 657-9883/ [email protected]
Subcontractor – UI	Principal Investigator	Sue Popkin	(202) 833-7200/ [email protected]
	Choice Mobility Study Lead	Chris Hayes	(202) 833-7200/ [email protected]
	Research Associate	Martena Reed	(202) 261-5304 [email protected]
	Research Analyst	Alyse Oneto	(202) 261-5219 [email protected]
	Research Analyst	Matt Gerken	(202) 833-7200/ [email protected]
	Research Assistant	Eleanor Noble	(202) 833-7200/ [email protected]
	Research Assistant	Peace Gwam	(202) 261-5549 [email protected]

Table 6. HUD Staff Who Advised on the Survey and Interview Instruments

Name	Role	Phone Number/Email
Teresa Souza	GTR	(202) 402-5540/ [email protected]

Appendix. Draft Communications for RAD Evaluation

This Appendix includes all survey documents and communications from HUD and Econometrica/UI to all potential respondents

1 Stabilized occupancy is the long-term average occupancy rate that an income-producing property is expected to achieve after leasing in the open market for a reasonable period at terms and conditions comparable to competitive offerings. In the first year of operation, as a new rental building goes through the lease-up process, it usually has an occupancy rate below stabilized occupancy, which distorts the financial picture.

Page 10 – Part B Supporting Statement