RAD OMB_Part B_FINAL_12_3

Evaluation of the Rental Assistance Demonstration (RAD) Program, Phase 1

Supporting Statement for Paperwork Reduction Act Submission, Part B

U.S. Department of Housing & Urban Development (HUD)

Office of Policy Development & Research (PD&R)

September 22, 2014

1. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g. establishments, State and local governmental units, households, or persons) in the universe and the corresponding sample are to be provided in tabular form. The tabulation must also include expected response rates for the collection as a whole. If the collection has been conducted before, provide the actual response rate achieved.

This Information Collection Request (ICR) includes three data collection instruments: (1) a Physical and Financial Conditions Web Survey to be administered to 24 PHAs currently administering RAD, and a control group of 48 PHAs not implementing RAD (instrument included as Appendix A); (2) a RAD Implementation telephone survey to be administered to 100 entities engaged in RAD implementation or in the control group, including PHAs, stakeholders, implementation consultants, and/or project managers (instrument included as Appendix B); and (3) a Resident Intake Study Correspondence and Intake Form, which will be administered to 400 tenants of RAD projects to enable tracking of residents for a future tenant-level impact evaluation (form included as Appendix C). See Table 11 below.

For the Physical and Financial Conditions Web Survey, the respondent universe is composed of 24 RAD projects (stratified randomly selected) that received a Commitment to Enter Into a Housing Assistance Payment (CHAP) between September 2012 and December 2013—approximately a 14-month period—and a comparison group that consists of 48 properties (genetically matched) that did not apply for RAD.¹

For the RAD Implementation telephone survey, the respondent universe is composed of the 24 RAD properties, 48 matching non-RAD properties, and between 10 and 25 other respondents, including non-participating PHAs, stakeholders, and subject matter experts.

A second phase of this evaluation (to be submitted to OMB as a subsequent ICR at a later date) will include a tenant impact evaluation. Because a successful analysis of resident outcomes requires inclusion of former residents who do not return to converted units, enrollment and tracking need to begin as early as possible after properties begin the RAD process to ensure residents’ contact information is obtained before they leave the RAD property. The Resident Intake Study Correspondence and Intake Form will be administered to a subset of residents in RAD properties to enable the tracking of residents for a future tenant-level impact survey. The goal will be to receive completed intake forms from 400 households.

The sampling for the resident study will occur in two stages. In the first stage, the 24 RAD sites will be chosen. In the second stage, a random stratified sample of residents living in each of the 24 sites at the time of closing will be drawn from a total of 2,000 units. Residents will be selected using HUD’s PIC records. The universe of 24 RAD sites is expected to include approximately 3,300 units.

Table 11. Number of Entities in Universe and Sample

Survey	Universe of Respondents	Sample Respondents
Physical and Financial Condition Survey	278 RAD projects and 6,739 non-RAD projects	24 RAD projects and 48 non-RAD projects
Implementation Survey	3,123 PHAs and 194 other stakeholders2	80 PHAs and 20 other stakeholders
Resident Survey	3,300 units	400 units

Power Curve for RAD Sample

Figure 1 gives three power curves for RAD projects. We have drawn these power curves to show the relationship between the difference in means and the power of the statistical test ( ).³ As the difference in the mean capital requirement between the RAD group and the non-RAD group increases, the power of the test increases.⁴ To draw these curves we have analyzed data on 360 awarded RAD projects to estimate a mean capital requirement (estimated construction cost) for the RAD group to be $63,312 per housing unit with standard deviation of $55,109.⁵ A power curve was drawn for n=24 (the sample size of the study) and for two comparison sample sizes, n=16 and n=32. Following the blue curve (n=24): if the actual difference in capital investment per unit between the RAD and the comparison group samples is $20,000, the test has a 39% chance of detecting an effect. If the actual difference is $40,000, the test has a 93% chance of detecting an effect.

Figure 1. Power Curve for the Difference in Capital Requirement: RAD vs. Non-RAD

Expected Response Rate

It is anticipated that 100 percent of the RAD properties will complete the survey and at least 80 percent of the non-RAD respondents will complete the surveys for the Physical and Financial Study component. To boost response rates, we will contact the non-respondents by email every two weeks during the period allotted for the data collection. In extreme circumstances, we will call those who do not respond to the email requests.

It is expected that 100 percent of the RAD respondents and at least 90 percent of the non-RAD respondents will complete the interviews for the Implementation Study component. This higher anticipated response rate is predicated on the effectiveness of direct personal contact in the interview process.

Based on our prior experience with similar survey efforts, it is expected that 20 percent of residents contacted will agree to enroll in the study, yielding a sample size of 400 for the resident survey. See Table 12 below. Analysis by specific housing authority is not anticipated, but subgroups for analyses would include region and size of the housing authority and household characteristics such as race and age (senior vs. non-senior).

Table 12. Resident Enrollment Survey

Estimated number of units in property sample	3,282
Number of units sampled for enrollment	2,000
Number of residents enrolled	400

Strategies for Mitigating Non-Response Bias

During the resident enrollment phase the goal will be to build a sample that is representative of RAD properties and of the residents in those properties. We will select properties for the sample using the size/performance frame already described, although only properties that have gone to closing and in which work has not yet begun will be eligible. We will monitor responses for two primary problems that could require action:

1. Response rates from a particular property are below our target threshold (20%).

2. Response rates from a particular group defined by available demographic characteristics are below our target threshold (to be determined by analysis of the universe of RAD property residents).

If response rates for a particular property are low, we will make extra effort through additional mailings and phone contact to non-respondents to increase enrollment at that property. For some properties the demographic characteristics of residents may be relatively similar to each other, but for others we may need to narrow the focus to specific demographic groups to ensure a representative sample.

For demographic break downs we have several characteristics available from HUD administrative sources for testing:

• Household size

• Number of children

• Number of seniors

• Presence of disabled

• Income

• Employment

• Race/ethnicity of householder

While it is possible to stratify the universe to compare against the sample using all these characteristics, doing so would likely result in strata with small populations; however, several variables are likely to be highly correlated with each other, and with properties. Income and employment, for instance are likely to be correlated, and some properties may be housing for seniors. Therefore, we will run correlations among the variables and use the results to pick the most appropriate set of indicators to include in the tests for bias. Testing will determine whether our sample is significantly different from the population, outside of sampling error. Where we discover statistically significant non-response bias that would be difficult to correct for through weighting, we will use targeted recruitment through additional mailings and phone calls. Priority will be given to groups or properties where the bias is greatest.

Because properties cannot be included in the sample until they have gone to closing, we will not have a clear picture of the demographic composition of our total sample compared to the universe until late in the process. We will need to continuously update testing for bias by properties and by demographic characteristics as enrollment proceeds. Early properties should give us an indication of whether the overall approach is succeeding and how to best apply our resources.

Minor deviations from a representative sample will be handled by weighting responses to the survey in the option task.

Previously Conducted Collection

There has been no previous data collection for the non-RAD properties, non-participating PHAs, stakeholders, and consultants. Some data have been collected for the RAD program participants though the application; however, none of the questions in the proposed survey will duplicate information that HUD has already collected. All data collected by HUD via RAD applications will be incorporated into the evaluation.

2. Describe the procedures for the collection, including: the statistical methodology for stratification and sample selection; the estimation procedure; the degree of accuracy needed for the purpose described in the justification; any unusual problems requiring specialized sampling procedures; and any use of periodic (less frequent than annual) data collection cycles to reduce burden.

1. Sample Selection Method

To select the RAD properties for the study, we stratified the RAD universe on two variables: PHA size and latest available inspection score issued by the Real Estate Assessment Center (REAC) under the physical condition indicator of the Public Housing Assessment System (PHAS). PHAs can be classified as either large, medium, or small. The PHAS/Uniform Physical Condition Standards (UPCS) scores range from 0 to 100. Based on the inspection scores, we grouped the PHAs into three mutually exclusive categories: 90 to 100, high performers; 70 to 89, standard performers (medium); and below 70, substandard or troubled (low).

Table 13 depicts the preliminary sample of 24 RAD properties based on the stratification criteria.

Table 13. Percentage of Properties in the Subgroup and Number to Be Selected From Each Subgroup

Stratum	Percentage in Subgroup	Number of RAD Properties in Sample From the Subgroup
Large PHA
High Performer	9.71%	2
Standard Performer	12.95%	3
Substandard Performer or Troubled	5.04%	1
Medium PHA
High Performer	21.94%	5
Standard Performer	27.34%	6
Substandard Performer or Troubled	4.32%	2
Small PHA
High Performer	8.99%	2
Standard Performer	8.27%	2
Substandard Performer or Troubled	1.44%	1

To select the 48 matching non-RAD properties, we intend to get data from HUD administrative systems: HUD’s REAC Physical Inspection Scores, the PIH Inventory Management System (IMS/PIC), the Picture of Subsidized Households, and the American Community Survey (ACS). These data include variables in common with the RAD data set.

We intend to match RAD properties to properties that are not participating in RAD using genetic matching, since the sample of RAD properties is small. This method sets up a counterfactual scenario created by comparing RAD to non-RAD properties that are as similar as possible. The strength of genetic matching lies in its utilization of equal percent bias reduction, a criterion that yields the highest possible reduction in selection bias for any linear combination of covariates matched on, even in finite samples. In addition, genetic matching performs its bias reduction algorithm on the same mathematical weighting framework as potential outcome matching and can potentially yield computationally quicker estimates if used in conjunction with propensity score methods. Effectively, genetic matching can be viewed as a broad algorithm that can incorporate propensity score and covariate matching. If either of these methods, or some combination of them, reduces biases the most, genetic matching will impose them as limiting cases of its algorithm. Potential outcome matching can be incorporated into this framework without sacrificing the effectiveness/efficiency of the genetic matching algorithm.⁶

Genetic matching is a multivariate matching method that uses an evolving search algorithm developed by Mebane and Sekhon to maximize the balance of covariates across matched treated and control units.⁷ The strength of genetic matching lies in its utilization of equal percent bias reduction, a criterion that yields the highest possible reduction in selection bias for any linear combination of covariates matched on, even in finite samples. In addition, genetic matching performs its bias reduction algorithm on the same mathematical weighting framework as potential outcome matching and can potentially yield computationally quicker estimates if used in conjunction with propensity score methods. Effectively, genetic matching can be viewed as a broad algorithm that can incorporate propensity score and covariate matching. If either of these methods, or some combination of them, reduces biases the most, genetic matching will impose them as limiting cases of its algorithm. Potential outcome matching can be incorporated into this framework without sacrificing the effectiveness/efficiency of the genetic matching algorithm.

We will use the R software environment and programming language with add-on software packages for matching. R is a free statistical software package used for data manipulation, calculation, and graphical display.⁸ R provides a selection of statistical techniques (e.g., linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering) and graphing techniques. R Match is a statistical package that can implement a variety of matching-method functions for multivariate and propensity score matching and for finding optimal covariate balance based on a genetic search algorithm.

We will prepare integrated analysis files in Excel or CSV containing the key variables for the RAD and the non-RAD properties merged from various HUD administrative and Census data sets, including HUD’s REAC Physical Inspection Scores, IMS/PIC, the Picture of Subsidized Households, and ACS. Data can be imported into R using text files, Excel spreadsheets, SPSS, or SAS. Results can be exported as text files. Once the data have been imported into R, we will install the matching package for R and utilize its GenMatch function with selected key variables. We will run the genetic match to minimize the mean squared error between RAD (treatment group) and non-RAD (control group) properties.

Table 14 below shows the key variables and the rationale for using the variables.

Table 14. Key Variables for Selecting Non-RAD Properties Similar to RAD Properties

Key Variable	Rationale
PHA Size	Large PHAs differ from smaller PHAs. PHAs have a planning process that is unique to the PHA but related to the size of the PHA. The PHA plan includes policies, programs, operations, and strategies for meeting local housing needs and goals. Factors need to be consistent with the housing and community development plans of the jurisdiction (as described in the Consolidated Plan); thus, PHA size matters.
Inspection Score	REAC conducts approximately 20,000 physical inspections on housing properties annually to ensure that families living in public housing have decent, safe, and sanitary housing that is in good repair. Scores range from 0 to 100. Properties that receive a PHAS score greater than 90 are considered high performers; properties that score between 70 and 89 are standard; properties that score lower than 70 are substandard or troubled. High-scoring properties are inspected every 3 years, standard performers are inspected every 2 years, and troubled properties are inspected every year. We are using the inspection score as a proxy for estimating the capital needs; properties with high scores are likely to have fewer capital needs than those with lower scores. The REAC file includes property inspection scores for almost all public housing projects.
Housing Conditions (not elsewhere addressed)	Properties located in the same, similar, or identical housing markets are perceived as comparable. If properties located in comparable housing markets have higher costs or needs than comparable properties, we can examine why outcomes are different. For example, the difference may be attributed to management of resources or other variables within the control of the entity, such as inefficient operation/upkeep of physical assets. HUD’s Economic and Market Analysis Division prepares Comprehensive Housing Market Analyses that can be of use for the analysis, depending on the location of the study property.
Building Type (e.g., characteristics such as elevators, multiple systems)	Property maintenance and replacement costs are commensurate with building type, such that the cost of maintaining or replacing a physical asset such as an elevator will impact the level of capital needs. The PIC database includes data on building type.
Number of Units	Property maintenance and replacement costs are commensurate with the number of units in a property. The PIC database has the number of units.
Type of Housing/ Number of Bedrooms	Costs associated with unit types/size of individual units are not equally distributed, viz., according to an Abt study (Capital Needs in the Public Housing Program), average capital needs vary by type of housing. The average amount of capital needs for an elderly unit is lower than that of a family unit. The PIC database indicates the housing type.
Age of Building/ Date of Full Availability (DOFA)/ Construction Date/ Date of Last Modernization	The age of the building is important for determining replacement needs. The DOFA establishes when a development can access the operating subsidy from a PHA’s Operating Fund. This date is available on all projects (not AMPs) and in most cases is the same as construction date. We would have a problem if the property was acquired after it was built and then converted to public housing; however, this situation is not very common. We will also take into account the last modernization date, if available. The PIC database has this information.
Vacancy Rate/ Percent of Units Occupied	This informs demand and supply and indicates how the housing market is performing vis-à-vis available housing units. The vacancy rate serves to determine the choices open to consumers in any market. This variable will be more important for the resident portion of the study. As housing supply expands, housing vacancies rise, and demand will either remain the same or decrease as more residents find available units; as vacancies decrease, the housing supply either remains the same or contracts while demand grows. Vacancy rates can be found in PIC, and the percentage of occupied units are in the Picture of Subsidized Households. We can match property identification to either or both data sets.
Census Tract Location/ ZIP Code	Although using the 5-year Census tract-level ACS data may not be completely accurate, we believe this may be useful to extract several neighborhood indicators. We have the option of matching on Census tract or ZIP Code (which are both in ACS). We would be tempted to match on ZIP Code, primarily because this would be the easiest method. The next lowest level would be Census tract, which would require us to match addresses to Census tracts. The ZIP Code data (also available on the REAC data file) would be more accurate—based on larger samples—but less representative of the near-in neighborhood. At this time we are unsure of which environment is the more important “environment”—Census tract or ZIP Code. We will defer to HUD on this issue and will also consider whether there are enough ACS sample observations available at the Census tract level to compute an estimate for key variables. Should HUD decide on matching at the Census tract level, we would need HUD’s assistance to verify addresses. There is a large amount of data relating to neighborhood housing that inform local housing and neighborhood environmental conditions. Of interest at the Census tract level from ACS are: financial characteristics (housing value, utilities, real estate taxes, rent, and mortgage); homeownership data (housing inventory estimates, occupancy status, tenure, and characteristics of occupied housing units); housing vacancy data (housing inventory estimates, rental and homeowner vacancy rates, and characteristics of vacant housing units); multifamily housing (selected characteristics by the number of units in the structure); physical characteristics (year built, units in structure, rooms, bedrooms, kitchen facilities, plumbing facilities, telephone service availability, house heating fuel, and year householder moved into unit); and rental housing data (housing inventory estimates and characteristics of renter occupied housing units). We could use ownership rates, poverty rates, minority composition, housing structure (attached and detached vs. multiunit (2+) structures), and/or proportion of subsidized housing to understand characteristics of those neighborhoods and evaluate whether RAD address the housing needs. The aim is to see what factors shape the housing quality in the RAD and non-RAD communities and understand the degree to which RAD impacts communities.

We will ask HUD to provide a file of public housing developments for Econometrica to use for matching non-RAD properties to RAD properties on key variables. The key variables include those noted in Table 13 above. We will ask HUD to remove non-viable properties, such as those that have been demolished or are no longer owned by the PHA, from the file of public housing developments that HUD provides for us to use for matching.

A sample of residents will be drawn for the resident study, with the size of the sample at each site proportionate to the total number of units within each site as compared to the total number of units within all 24 RAD sites. To ensure that a representative population is enrolled in the study, we propose stratifying the sample by age, family status/household size, and race.

The number of residents enrolled in the study will depend upon the diversity of the 24 RAD sites sampled and the final number of completed surveys needed. The more diverse the sites in terms of geography, size, or financing, the more residents will have to be enrolled. We anticipate that 400 residents enrolled will be sufficient to support analyses by household and PHA characteristics. Residents selected from the sample will be contacted by mail and phone and asked to enroll in the study. Participation in tracking and survey efforts will be strictly voluntary. Residents will be enrolled in the study, and tracking will begin after a RAD site has gone to closing.

2. Procedure for Data Collection

Data for the Physical and Financial Study component for the 24 RAD properties and 48 non-RAD comparison properties will be collected through a Web survey. An Internet service will be used to send the pre-survey email invitation, the Web link to the survey, and a copy of the survey instrument to be used locally should the respondent need to get information from other sources. The Internet service will track response rates and provide statistics on responses while the survey is underway. The Internet service will provide a file that can be used for analysis at the conclusion of the survey. Econometrica will identify a primary respondent who will receive the survey and be responsible for completing it. We identify primary respondents through HUD administrative files and the RAD applications. We will send the list of respondents to HUD for the HUD RAD Program Director or another HUD official to send an email invitation to each of these contacts to ask them to participate in the survey. 

After OMB approval and once the survey can be launched, we will work with the RAD Program Director or other HUD official to send pre-survey invitations to respondents asking them to participate. Our notification to HUD will include the suggested text for the invitation. Copies of our notification will be sent to HUD’s Government Technical Representative (GTR).  

As noted above, for the Web survey, additional key steps in the process include email reminders as needed to prompt a survey response and, if necessary, follow-up phone calls from the Econometrica team to encourage response. We will directly notify HUD that it is time to ask the RAD Program Director or other HUD official to send pre-interview invitations to participate in the survey to the contacts. We will ask for the email invitations to be sent 5 working days before we launch the survey.

For the Implementation Study interviews, the Econometrica Project Manager will contact HUD’s GTR and let him know when we will be starting the interview process. We will directly notify HUD that it is time to ask the RAD Program Director or other HUD official to prepare pre-interview invitations to participate in the study; we will ask for confirmation of the invitation prior to conducting the interviews.

For the resident enrollment portion of the data collection effort, targeted residents will be sent a letter, preferably by a HUD Official, requesting the household’s participation in the study and including the Resident Intake Study Correspondence and Intake Form. The letter and Intake Form will include a phone number to call with questions. The households will be offered incentives for enrollment and maintenance of continued contact. Multiple mail contacts will be made to improve response rates. A sample of residents will be selected for extra effort to encourage enrollment, and the characteristics of those households will be compared with other enrollees to test for response bias. We anticipate that 400 residents will initially enroll.

3. Estimation

Despite the measures described below to reduce non-response rates, it is likely that some sample units will remain unmeasured. We will measure response bias and use post-survey adjustment to reduce non-response error.

3. Describe the methods used to maximize response rates and to deal with non-response. The accuracy and reliability of the information collected must be shown to be adequate for the intended uses. For collections based on sampling, a special justification must be provided if they will not yield “reliable” data that can be generalized to the universe studied.

The accuracy and reliability of the collected data depend upon receiving a high rate of return from the respondents. To achieve a high response rate, four steps will be taken that are consistent with contemporary strategies in the field of survey research.

First, the survey distribution package is designed to be respondent friendly.

• The survey is divided into sections in order to provide visual relief from a continuous listing of questions.

• The questions are laid out in a format that is easy to read and that promotes answering.

• The content of the questions is designed to ask for types of information that program recipients are required to maintain and should have readily available as required by HUD regulations.

• The terms used in the survey are widely known among program recipients and well defined in technical materials issued by HUD’s Office of Public and Indian Housing.

Second, respondents are treated as real people with distinct identities. The distribution mailings are addressed to specific respondents rather than to a generic “housing authority or project director.” In the same way, the mailings are signed by HUD’s RAD Program Director or another HUD official. Additionally, the Econometrica team will ensure that the invitations to participate in the survey or the interviews take the form of a personalized email sent to the contact person.

Third, the cover letter accompanying the survey and/or request for interviews provides a clear and direct explanation of the reasons for the data collection and urges participation. The cover letter also encourages response by explaining that the sampling process is random and responses will be kept confidential, meaning that data will be made available in a form that would not reveal the identity of the person making the comment. 

Fourth, the initial distribution of the survey will be followed by a series of follow-up distributions to non-respondents. The first two follow-up actions are reminder email messages, which will be emailed at 2-week intervals. In extreme cases of non-response, we will follow up with a telephone call from an Econometrica staff member to the respondent to encourage the respondent’s participation in the survey. Following a response to the Web survey, we will send a thank-you email message to each respondent.

Enrollment efforts will begin with a notification letter from HUD’s RAD Program Director or another HUD official and an enrollment/tracking form to the sampled population. A $5 gift card incentive will be offered for participation. A reminder postcard will be mailed two weeks after the first mailing. A second notification letter and enrollment form will be subsequently mailed to those that have not enrolled. A final recruitment package will include a $2 incentive. We will make up to 5 attempts by telephone to reach residents who did not enroll after receiving three mailings. Finally, we will conduct a non-response bias analysis on the sample.

See the Appendix for copies of notification letters.

4. Describe any tests of procedures or methods to be undertaken. Tests are encouraged as effective means to refine collections, but if ten or more test respondents are involved OMB must give prior approval.

The data collection instrument and data collection procedures will be tested initially with HUD program staff as well as PHAs and/or property managers who are not in the sample. The pretest will not exceed the OMB guidelines.

5. Provide the name and telephone number of individuals consulted on the statistical aspects of the design, and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

HUD has contracted with Econometrica (prime contractor) to conduct the Evaluation; the Urban Institute is Econometrica’s subcontractor. The following table lists those who were consulted or will participate in the data collection effort, analyze the data, or prepare reports. The actual collection of Web survey data will be performed through a Web service that specializes in conducting Internet surveys. Tables 15 and 16 below show the names, affiliations, and contact information for those involved in the statistical design and the survey research.

Table 15. Names, Affiliations, and Contact Information

Personnel	Phone Number or Email
Econometrica, Inc. – Prime
Principal Investigator
Chuck Hanson	(301) 657-9883
Project Manager
Dennis Stout	(301) 657-9883
Senior Analysts
Paul Watkins	(301) 657-9883
Dr. Fred Eggers	(301) 657-9883
William Thorson	(301) 657-9883
David Ruiz	(301) 657-9883
Analyst
James Hedrick	(301) 657-9883
Alex Thackeray	[email protected]
Subcontractor – UI
Sue Popkin, Project Manager	(202) 833-7200
Chris Hayes, Principal Investigator	(202) 833-7200
Elaine Morley, Sr. Analyst	(202) 833-7200
Alex Derian, Analyst	(202) 833-7200
Subcontractor – Other
Jaime Bordenave, SME, The Communities Group	[email protected]
Dr. John Weicher, SME, Hudson Institute	(202) 974-2425

Table 16. HUD Staff Who Advised on the Survey and Interview Instruments

Name	HUD Staff	Contact Information
Dr. Judson L. James	HUD GTR	(202) 402-5707

1 See discussion of “genetic matching” below.

2 Other stakeholders estimated based on assumption of two lenders per project (72*2 = 144) plus one stakeholder per state (50): 144 + 50 = 194.

3 = the probability of a Type II Error, i.e., the probability of wrongly accepting a false null hypothesis. On the other hand, the power of a statistical test ( ) is the probability of correctly accepting a true alternative hypothesis. Power measures how well a test can detect an effect, if an effect actually exists. See (cite some Statistics textbook) for more on Type I and Type II errors in Hypothesis Testing.

4 Slight differences between the RAD group and the non-RAD group are difficult to detect, and consequently the power of the test is low. But as the difference grows, the likelihood that the test will be able to detect a difference grows. Eventually, for very large differences in the mean, the power of the statistical test reaches 100%.

5 Based on our analysis of data on the estimated cost of construction for 360 RAD properties with approved CHAPs. We trimmed this data to eliminate 166 unrepresentative outliers consisting of the smallest observations (below $10,000) and the largest observations (above $250,000), leaving 194 observations in the trimmed dataset.

6 A more in-depth explanation is provided in the Research Design.

7Thoemmes, FJ, & Kim, ES. (2011). A Systematic Review of Propensity Score Methods in the Social Sciences. Multivariate Behavioral Research. 46(1):90–118; Hahn J. (1998). On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects. Econometrica. 66:315–331; Diamond, A, & Sekhon, JS. (2012). Genetic Matching for Estimating Causal Effects; Diamond, A, & Sekhon, J. S. (2005). Genetic matching for estimating causal effects: a general multivariate matching method for achieving balance in observational studies. Review of Economics and Statistics. (0); Tsai, KT, & Peace, KE. Genetic Matching: An Efficient Algorithm to Adjust Covariate Imbalance for Data Analysis and Modeling.

8 R is available under the terms of the Free Software Foundation’s GNU General Public License in source-code form. It compiles and runs on a wide variety of UNIX platforms and similar systems (including FreeBSD and Linux), Windows, and MacOS. Manuals can be found at: http://www.r-project.org/.

Page 13 – Part B Supporting Statement