Final -HCV Mobility Part B

Supporting Statement Part B for Paperwork Reduction Act Submission

Housing Choice Voucher Mobility Demonstration

OMB Control Number 2528-New

Part B. Justification

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The Housing Choice Voucher (HCV) Mobility Demonstration (Demonstration) design is a large, nine-site randomized experiment. The Demonstration will enroll approximately 17,000 families over the five-year enrollment period—of which approximately 16,000 will be existing voucher families and approximately 1,000 will be new admission families. During the first three years of enrollment into the Demonstration – the time period covered by this Information Collection Request (ICR) – approximately 10,100 households are expected to enroll in the demonstration, each with a minimum of one child aged 17 or under.¹ In accordance with the Federal Register Notice Docket No. FR–6191–N–01 published July 15, 2020, two types of households with children will be enrolled in the Demonstration: existing voucher families and waitlist families. Existing voucher families are households that already have an HCV; to be eligible for the Demonstration they must have at least one child aged 17 or under living in the household and have expressed an interest in moving. Waitlist families are applicants for an HCV; among waitlist families, a preference will be given to applicants that have at least one child aged 13 or under living in the household and currently live in census tracts with a family poverty rate of 30 percent or higher. In there are insufficient applicants that qualify for this admissions preference, the public housing agency (PHA) must select the next available applicants from the waiting list that have at least one child aged 17 or under living in the household.

As described in Supporting Statement Part A, the Demonstration will have two phases of enrollment. During the first two years of enrollment (Phase 1), the Demonstration will randomly assign households to a single treatment group or a control group. The treatment condition will offer a comprehensive set of mobility-related services (CMRS) to households. During the following three years of enrollment (Phase 2), the Demonstration will randomly assign households to one of two treatment groups or a control group. The two treatment conditions will be (1) the same comprehensive set of mobility-related services (CMRS) offered during Phase 1 and (2) a narrower set of selected mobility-related services (SMRS). This information collection request covers the first three years of enrollment into the Demonstration—all of Phase 1 and up to the first year of Phase 2. Exhibit B-1 shows the anticipated enrollment into the Demonstration, by random assignment group and year, across nine sites in the first three years of enrollment.

Exhibit B-1: Anticipated Enrollment Schedule for HCVMD, by Random Assignment Group and Demonstration Year (DY)

Year	Treatment Group—CMRS	Treatment Group—SMRS	Total Treatment Group (Families Receiving CMRS or SMRS)	Control Group	Total Enrollment (Treatment & Control)
2022	1,684	N/A	1,684	1,657	3,341
2023	1,684	N/A	1,684	1,657	3,341
2024	1,144	1,145	2,289	1,129	3,418
Total	4,512	1,145	5,657	4,443	10,100

Notes: CMRS=Families randomly assigned to an offer of Comprehensive Mobility Related Services.

SMRS=Families randomly assigned to an offer of Selected Mobility Related Services.
The years are approximate, as data collection may not start exactly on January 1, 2022.

As noted in Supporting Statement A, there are six types of data collection planned for this demonstration during the first phase of the evaluation. The first five of which are the subject of this information collection request:

1. Enrollment/Baseline Data: which includes the Household Roster and Baseline Information Form (Instrument 1) and the Baseline Survey (Instrument 2), along with random assignment data—all collected through a web-based enrollment tool.

2. Service Provision Data: including information on each contact between a service provider in the Demonstration and a participating household and between a service provider and a landlord, as well as information on the services provided collected through a web-based tracking tool (Attachment G).

3. Qualitative Data Collection: in-depth interviews with PHA and service provider staff, households participating in the Demonstration, and landlords (Instruments 3–7).

4. Cost Data Collection: including interviews with PHA and service provider staff to collect data on costs to operate the program (Instrument 8) as well as requests for documents from PHA and service provider staff;

5. Administrative data from PHAs: Data on Demonstration participants not available through HUD’s existing administrative data, including information on search time and preference category for waitlist families; and

6. Administrative data from HUD. This final data source, which is not included in this ICR, will provide data on families’ income, addresses, household composition and other factors included within the administrative data that HUD collects directly from PHAs pursuant to an alternative authority.

This section documents plans for selecting households for enrollment into the program—including completing the Household Roster and Baseline Information Form and the Baseline Survey data collection. It then describes the plans for selecting households, landlords, PHA and Service Provider staff for participation in the qualitative data collection. Finally, the section provides details on the selection of PHA and Service Provider staff for completion of the cost data collection.

Sampling Plans

A. Sampling for the Enrollment/Baseline Data Collection

Approximately 94 percent of all households enrolled in the study will be households that already have Housing Choice Vouchers (referred to throughout as existing voucher families). Households on the Housing Choice Voucher waitlist are expected to account for the remaining six percent of all households enrolled (referred to throughout as waitlist families). The waitlist families will be randomly assigned to either a treatment or the control group several weeks prior to the point where they are deemed eligible for the voucher program. It is likely that existing voucher families will also be randomized into the study prior to the issuance of a voucher to enable them to search for a new unit. These procedures will allow treatment families to receive services prior to the issuance of a voucher. Since not all waitlist families will necessarily be eligible to receive a voucher, the PHAs will prescreen families so that only those families expected to be eligible for a voucher will be enrolled in the study.² All households that enroll in the study are expected to provide the necessary baseline information and complete the Baseline Survey, so the sample for the enrollment and baseline data collection will consist of 100% of the households that enroll in the study.

During the first two years of enrollment into the HCV Mobility Demonstration, households will be assigned at random to one of two groups:

1. a treatment group, which will receive a comprehensive set of mobility related services (CMRS),

2. a control group, which will receive the standard services available to all voucher holders.

The group into which a household is randomized will not affect the household’s eligibility for a voucher.

During the first two years of enrollment into the Demonstration (Phase 1 of enrollment), households will be assigned to one of these two groups at nearly equal rates.³ During the next three years of enrollment into the Demonstration (Phase 2) – which includes one year covered by this ICR and two additional years beyond this ICR – households will be assigned at random to a control group, or to one of two treatment groups. The first treatment group will receive the same CMRS provided under Phase 1; the second treatment group will receive a more limited set of selected mobility-related services (SMRS).

Exhibit B-2: Total Expected Samples and Response Rates for Baseline Data Collection in the First Three Years of Study Enrollment

Demo Year	Total New Enrollment (Treatment & Control)	Expected Responses to Baseline Data Collection	Response Rate
2022	3,341	3,341	100%
2023	3,341	3,341	100%
2024	3,418	3,418	100%
Total	10,100	10,100	100%

Note: The years are approximate, as data collection may not start exactly on January 1, 2022.

B. Sampling for the Service Provision Data from PHAs and Service Providers

Data on the provision of mobility-related services to treatment group households will be a key input for the Process Study analysis. The evaluation contractor will collect data on service provision from PHAs and mobility service providers for 100 percent of the participants in the two treatment groups. Thus, there is no sampling required for this data collection component.

The service provision data will be contained in administrative records maintained by the mobility service providers. At some sites, the mobility related services will be provided directly by PHA staff. In other sites, contracted organizations will provide mobility services.

Exhibit B-3: Expected Samples for Service Provision Data Collection in the First Three Years of Study Enrollment

Year	Total Households Assigned to Treatment Groups (CMRS and SMRS)	Number of Households with Service Provision Data Collection	Percent of Treatment Sample with Service Provision Data Collection
2022	1,689	1,689	100%
2023	1,689	1,689	100%
2024	2,292	2,292	100%
Total	5,670	5,670	100%

Notes: CMRS=Families randomly assigned to an offer of Comprehensive Mobility Related Services.

SMRS=Families randomly assigned to an offer of Selected Mobility Related Services. The years are approximate, as data collection may not start exactly on January 1, 2022.

C. Sampling for Qualitative Data Collection (PHA and Service Provider Staff, Landlords, and Households)

The evaluation contractor will conduct one round of qualitative data collection pursuant to this Information Collection Request. The number of interviews with PHA staff and mobility services providers will vary across sites depending on whether a site has one or two PHAs, and whether mobility services are provided by the PHA or an external provider. For purposes of this estimate, four sites (comprising 6 PHAs total) are assumed to use in-house service provision and five sites (comprising 7 PHAs) are assumed to partner with external service providers.

As shown in Exhibit B-4, sample selection assumed the following per visit in each of the nine sites: 3 to 6 PHA staff interviews per site and 3 to 6 mobility service provider staff interviews depending on whether they have two or three PHAs per site, four landlord interviews, and 20 family head of household interviews.

Exhibit B-4: Sample Sizes for Qualitative Data Collection in First 3 Years of Enrollment

Informant Group	2023
PHA Staff	45
Mobility services staff (External provider or PHA provider)	45
Landlords	36
Households:	180
Treatment group (waitlist families)	45
Treatment group (existing voucher families)	90
Control Group families	45

Notes: A second round of expected qualitative data collection outside the three-year window will be described in a future ICR.

^a Numbers of PHA staff respondents are exclusive of staff that provide mobility services.

^b All landlord respondents will be involved in the demonstration

Selection of Families. The evaluation contractor will select the heads of household for interviewing from among Baseline Survey respondents based on the following criteria:

• Status at enrollment: existing voucher family or waitlist family;

• Random assignment status: treatment or control group; and

• Race: families that, as a group, reflect the racial composition of the households enrolled in the Demonstration.

The selection process will work as follows: The evaluation contractor will group Baseline Survey respondents into three groups: (a) waitlist families in the treatment group; (b) existing voucher families in the treatment group; and (c) families in the control group. The contractor will then apply an eligibility screen to those households that received a voucher after enrollment in the study, eliminating from the sample any such families that have been searching for less than 60 days or have been leased up for more than 6 months. After the sample has been restricted in this manner, within each group the evaluation contractor will randomly assign numbers to households and reduce the list to twice as many households as the number of respondents expected to be interviewed within that group (see table above). Within each of the three groups, the evaluation contractor will review the sample drawn for racial composition, replacing as many families (starting from the bottom of the selected households and replacing the last household with the next household on the list that would balance the composition) as necessary until the list reflects the racial proportions of the families enrolled in the demonstration at that site. The lists will be randomized again for outreach. The evaluation contractor will make 3 attempts at contacting a sampled family before moving to the next family on the list.

Approximately half of the family interviews with heads of household at each site will be with treatment group members who are existing voucher families, and the remaining interviews will be evenly distributed between waitlist families in the treatment group and families in the control group (including both waitlist families and existing voucher families). The evaluation contractor will oversample new admission households to ensure enough sample to report on their perspective.

Selection of PHA and Service Provider staff. There are no formal sample selection criteria for these interviews since the evaluation contractor will need to speak with staff at each of the 9 sites (which are represented by 13 participating PHAs) as well as to representatives of any external service delivery organizations that are partnering with a PHA on the administration of the program. The evaluation contractor will work with the PHAs and Service Providers to review the types of information to be discussed in each interview. This will allow for the determination of the person(s) most appropriate to answer the questions covered in the interview. Since all PHAs and their service delivery partners are required to cooperate with the data collection efforts as part of their grant awards, 100% participation is expected.

Selection of landlords. The evaluation contractor will work with PHAs and Service Providers to identify a set of landlords to interview. The evaluation contractor will work with PHA and mobility services provider staff to identify landlords with properties in opportunity neighborhoods who have had some engagement, either via outreach or recruitment efforts, with mobility services staff. From a list of all such landlords, the evaluation contractor will select a sample that is twice as large as the number of interviews needed, randomize the list, and then contact landlords in the order they appear on the list to participate, stopping when enough landlords have been recruited. While not part of the sampling criteria, it is likely that some of the landlords interviewed may be property owners while others may be property managers and that some of the landlords interviewed will have rented to a study participant or another HCV family, while others will not have. The evaluation contractor will make 3 attempts at contacting a landlord before moving to the next landlord on the list.

D. Sampling for the Cost Data Collection

PHAs and service providers are required to participate in the cost data collection as it is a requirement of the evaluation. There is no complex plan required for sampling respondents for the cost data collection. Respondents will be those who are most familiar with the financial aspects of the program. The evaluation contractor will interview approximately 34 individuals. This estimate is based on 1 to-2 staff per site that are in the best position to supply the mobility-related services cost data (controller, finance director, CFO or some other position for this data collection) and is expected to be a mix of PHA staff and mobility service provider staff. Each PHA and external mobility service provider will be asked to specify which individual within that organization is in the best position to supply the necessary data on the costs of mobility-related services. In some cases, it might be the finance director, while in others it might be a controller, a CFO or other position.

5. Sampling for the Administrative Data Collection from PHAs

The evaluation contractor will collect Housing Choice Voucher program administrative records from the PHAs for all enrolled households, so no sampling is required. Exhibit B-5 shows the numbers of households for which administrative data is to be requested from HUD and the PHAs.

Exhibit B-5. Expected Samples for Housing Choice Voucher Program Administrative Data Collection from the PHAs in the First Three Years of Study Enrollment

Demo Year	Total New Enrollment (Treatment & Control)	Number of Households with Administrative Data Collection	Percent of Sample with Administrative Data Collection
2022	3,341	3,341	100%
2023	3,341	3,341	100%
2024	3,418	3,418	100%
Total	10,100	10,100	100%

Note: The years are approximate, as data collection may not start exactly on January 1, 2022.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This section describes the procedures for the collection of information, followed by the analysis and estimation procedures. The discussions are provided first for the enrollment/baseline data collection, then the qualitative data collection, and finally the cost data and administrative data collection. The section concludes with statements regarding unusual problems requiring specialized sampling and use of periodic data collection cycles to reduce burden.

Procedures for Collecting Data Procedures for Enrollment and Baseline Data Collection

The evaluation contractor, in consultation with HUD, and the contractor’s Institutional Review Board, developed a Baseline Survey, a Household Roster and Baseline Information Form, and a set of informed consent documents for use in the Demonstration, including informed consent from the adult head of household, other adult household members, and parental permission for children to participate in the HCV Mobility Demonstration. (Please see Instruments 1 and 2 for the Household Roster and Baseline Information Form and Baseline Survey; please see attachments B-E and N-P for the consent forms.)

PHA staff will conduct outreach efforts with households deemed likely to be eligible based on information in the PHA files—on a schedule to be determined in consultation with the evaluation contractor.

The schedule for inviting existing vouchers families will be designed to ensure that invited families have enough time remaining before they must renew their current lease to allow mobility-related services to be delivered. The initial working assumption is that 39 percent of existing voucher families who receive an invitation will attend an enrollment session and that 11 percent ultimately will enroll in the study. These estimates are adapted from experience in the CMTO Demonstration in Seattle and King County on which the HCV Mobility Demonstration was modeled. The number of households invited in any given month will be adjusted based on prior experience with enrollment to ensure the study’s recruitment goals are met. At most of the study PHAs, it is likely that all eligible existing voucher families will receive at least one invitation to participate over the course of the Demonstration. At the largest PHAs, however, it may be necessary to randomly select a sample of families to invite to ensure that the PHA has the resources to provide services to those who choose to enroll.

Waitlist families will be invited in the order in which they appear on the waiting list as of the time each new set of outreach letters are prepared. Given the relatively small number of waitlist families that will be enrolled during the course of the Demonstration, not all waiting list families that qualify for the special admissions preference for the Demonstration will necessarily be invited to participate. The families who qualify for the special admissions preference and are not invited to participate in the Demonstration will fall into one of two categories: (a) they are offered a regular voucher and thus no longer qualify to participate in the Demonstration or (b) they are further down on the special admissions preference waiting list than families who are invited to participate.

Outreach efforts will begin with a letter that briefly introduces the Demonstration, the eligibility and study participation requirements, random assignment process, and an invitation for likely eligible households to learn more about the Demonstration. Interested heads of household will be invited to attend an information session to learn more about the study. Information sessions may be held in person or via teleconference/webinars and will include a video overview of the study. Interested families will either schedule or proceed directly to an enrollment session with PHA staff (or an agent assigned to this task by or on behalf of the PHA).⁴ Interested heads of household also have the option to proceed directly to scheduling an enrollment session.

During the enrollment session, PHA staff will confirm eligibility to participate in the demonstration. Demonstration eligibility for existing voucher families is confirmed by ensuring the family has at least one child aged 17 or under living in the household.

To be eligible for the special admissions preference adopted for this Demonstration, waitlist families must have at least one child aged 13 or under living in the household and be living in a census tract with a family poverty rate of 30 percent or higher. If there are insufficient new admission families that qualify for this preference, the PHA must select the next available families from the waiting list that have a child 17 years of age or younger. (Demonstration eligibility is not the same as HCV program eligibility—all interested families still need to meet the HCV program eligibility requirements to receive a voucher.) The initial working assumption is that 57 percent of waitlist families who receive an invitation letter will respond and 30 percent will ultimately enroll; these assumptions, which are again adapted from the CMTO Demonstration, will be refined over time based on experience.

Once eligibility for the Demonstration is confirmed, the PHA staff member (or agent) will explain the Demonstration and the study in more detail, show a video that reviews the informed consent documents, address any questions and obtain consent from all heads of household who wish to participate in the study. Once informed consent has been obtained from the household head, PHA staff (or agent) will request and record information from the household head to complete the Household Roster and Baseline Information Form in the enrollment tool. The PHA staff (or agent) will then provide the head of household with a tablet computer (or a link if the household is using its own computer or phone) so that the head of household can complete the Baseline Survey online. Once the Baseline Survey is complete, PHA staff (or agent) will randomly assign the household and notify the head of household of the outcome and provide the head of household with their $25 incentive for completing the Baseline Survey.

Individuals with disabilities will be provided reasonable accommodations to participate in the study and the evaluation contractor, PHAs, and HUD will ensure effective communication with individuals with disabilities throughout this study.

The evaluation contractor will work with HUD and the participating PHAs to ensure meaningful access to the study by persons with limited English proficiency. The evaluation contractor will make the survey available in English and Spanish and will work with HUD and the participating PHAs to accommodate interested households whose primary language is neither English nor Spanish.

The head of household will have the option of completing the Baseline Survey in written form or having the questions and response options read aloud through (their own or provided) headphones.

Procedures for Qualitative Data Collection

Two members of the evaluation contractor research team will travel to each of the nine sites to conduct qualitative interviews; each visit is expected to last approximately four to five days, depending on the size of each site. During these site visits, the research team members will conduct interviews with: PHA and service provider staff; households participating in the Demonstration study and Landlords. The cost data collection will take place during these visits as well, which will include both interviews and requests for information from PHA or service provider records.

The evaluation contractor, in consultation with HUD, has developed semi-structured interview protocols for the collection of PHA and service provider site visits, the cost data collection, the family interviews, and landlord interviews. Families asked to participate in the qualitative interviews will go through an informed consent process prior to the interview. This process will clearly describe the purpose of the qualitative interviews, risks, benefits, and voluntary nature of participation in the process study—as opposed to participation in the overall study. All other protocols begin with a brief introductory consent script that summarizes the overall evaluation, the focus of each interview, how respondent privacy will be protected, and how data will be aggregated. A member of the evaluation contractor research team will then ask the respondent if it is okay to proceed with the interview, a yes response implies consent to proceed. (See Instruments 3-8 for the PHA, Service Provider, Family, Landlord, and Cost interview guides.)

PHA and Service Provider Staff interviews: These interviews will be done in-person, ideally at the PHA or Service Provider offices.⁵

Family interviews: In-depth interviews will be conducted with selected heads of household. The evaluation contractor will attempt to interview participants at a central location—such as the PHA or provider offices where possible. However, in-home interviews can be arranged for those who can't or don't want to come to a central location. Interviews can also be conducted remotely, if necessary.

Landlord interviews: In-depth interviews will be conducted with selected landlords with properties in opportunity neighborhoods who have had some engagement, either via outreach or recruitment efforts, with mobility services staff. The evaluation contractor will work with PHA and mobility services provider staff to identify landlords who meet this description. From a list of all such landlords, the evaluation contractor will select a sample that is twice as large as the number of interviews needed, randomize the list, and then contact landlords in the order they appear on the list to participate, stopping when enough landlords have been recruited. While not part of the sampling criteria, it is likely that some of the landlords interviewed may be property owners while others may be property managers and that some of the landlords interviewed will have rented to a study participant or another HCV family, while others will not have. The evaluation contractor will interview landlords at a central location—such as the PHA or provider offices, or another location convenient for the landlords.

Procedures for Cost Data Collection. The evaluation contractor, in consultation with HUD, developed an interview guide to collect information needed for the cost-benefit analysis from PHA and service provider staff (please see Instrument 8). Cost data collection will be done during the same site visits as the interview. Cost data collection interviews will be supplemented with information from the service delivery tool and other existing financial data. Please refer to Supporting Statement A for more detail on the instrument development.

Procedures for Administrative Data Collection. As described in Supporting Statement A, the evaluation contractor will request administrative data from HUD and the participating PHAs. To the extent feasible, these requests will be fulfilled using data that HUD already has, thus minimizing burden on the PHAs. To obtain the remaining administrative data needed to complete the study, the evaluation contractor will contact the PHA and arrange to receive an extract of administrative data through a secure electronic transfer tool.

Statistical Impact Analysis for the Core Impact Analysis

Impact Analysis for CMRS versus Control Comparison. The impact analysis for the first two reports to be produced based on the Demonstration focuses on the pairwise comparison of CMRS versus the control condition. The two reports are the Rapid Cycle Evaluation (RCE) Report, reporting on existing voucher and waitlist families enrolled in the first 6 months of the Demonstration, and the final Phase 1 Process and Impact Evaluation Report, reporting on existing voucher and waitlist families enrolled in the first 2 years. The impact analysis at each point will include all households with at least 6 months of date after random assignment. For each report, the evaluation contractor will perform balance tests of the pooled sample and key subgroups (existing voucher families, waitlist families, and site) to confirm that randomization resulted in assignment groups with similar household and neighborhood characteristics at baseline. The sample sizes and minimum detectable effects are specified below in Exhibits B-6 and B-7.

Most outcomes for the impact analysis will be related to moves in the months after random assignment (RA). The outcomes will include:

• Residing in an opportunity area at 6 months after RA (confirmatory outcome)

• Residing in an opportunity area at 2 years after RA

• Leased up or moved within 6 months after RA

• Rental unit characteristics 6 months after RA

• Census tract characteristics 6 months after RA

• Housing assistance payment (HAP) six months after RA

• Expected future outcomes for children based on Opportunity Atlas data

Appendix 1 shows the rental unit characteristics and census tract characteristics to be analyzed in the study along with a limited set of future outcomes for children based on Opportunity Atlas data. Appendix 2 shows the full set of planned impact analyses.

The evaluation contractor will follow the CMTO impact analysis (Bergman et al. 2020) by using an intent-to-treat impact estimation model for our main analysis. This model will take the form of

where is an outcome for family , is the estimated impact of being offered CMRS rather than the control condition, is a vector of family-level characteristics measured at baseline including characteristics of origin neighborhood, are eight site-level dummies (fixed effects; with the ninth site serving as the reference group), is a family-level residual, and , , and are other parameters.

Standard Errors. The model described above will be estimated using ordinary least squares (OLS), which assumes that the outcome data have a normal distribution (i.e., form a bell-shaped curve) with a common variance (i.e., are homoscedastic). There is no reason a priori to expect homoscedasticity, however, since some types of families could have higher variability in their outcomes than other families. Furthermore, many of the outcomes to be estimated are binary; applying OLS to such binary outcomes (i.e., using the linear probability model) induces heteroscedasticity.⁶

To address the potential of heteroscedasticity, the evaluation contractor will compute robust standard errors (i.e., Eicker-White robust standard errors; Angrist & Pischke, 2008) and use these standard errors for hypothesis testing. This is the same approach as used in the CMTO study (Bergman et al., 2020) and the Family Options Study (Gubits et al., 2016).

Statistical Tests for Impacts. The evaluation contractor will use the estimated standard errors to perform tests of statistical significance for the impact estimates. The analysis will test and seek to reject the null hypothesis that assignment to the offer of CMRS produces equivalent outcomes as assignment to the business-as-usual control condition. The level of statistical significance for hypothesis testing will be 0.10. This level of significance has been commonly used in many recent social policy experimental evaluations including in the CMTO study (Bergman et al., 2020).⁷

Pooled and site-specific impacts. The impact analysis will examine effects of the offer of CMRS for the pooled sample and for site-specific samples. There are many reasons why impacts might vary across sites. The targeted number of enrolled families at each site will allow for estimation of site-level impacts.

Subgroup impact estimates. The study’s most important subgroup distinction is that between existing voucher families and waitlist families. The impact analysis will estimate effects for these two subgroups in the pooled sample and in the site-specific samples. The impact analysis will also estimate effects by race.

Multiple Statistical Tests. Because of the large number of statistical tests, the impact analysis will need to address the multiple comparisons problem—that is, the possibility of finding a statistically significant impact by chance, due to the large number of tests being conducted. To address this problem, the evaluation contractor will specify a single outcome—whether a family is leased up in an opportunity area 6 months after random assignment—as the -confirmatory outcome for the initial two reports. However, since HUD is equally interested in the impact on this outcome for existing voucher families, waitlist families, and the pooled sample of both types of families, the evaluation contractor will conduct three confirmatory statistical tests, as shown in Exhibits B-6 and B-7. The evaluation contractor will use a multiple comparisons procedure to adjust the p-values from these three tests. Because of the highly correlated nature of the Phase 1 primary hypotheses (one pooled test and two subgroups tests within the pooled sample), a Westfall-Young resampling approach will be used to adjust the p-values. (Westfall et al., 2011; Gubits et al., 2018). By taking account of correlations between hypothesis tests, this approach will provide greater statistical power than other standard procedures such as Benjamini-Hochberg or Bonferroni-type methods.

Exhibit B-6: Expected Sample Sizes and Minimum Detectable Effects (MDEs) for Lease-up in Opportunity Areas for Phase 1 of the Demonstration, Rapid Cycle Report

Comparison	Sample	CMRS Sample Size	Control Sample Size	MDE for Lease Up in Opportunity Areaa (percentage pts)
CMRS vs. control	All families	842	828	4.9
CMRS vs. control	Existing voucher families	797	797	5.0
CMRS vs. control	Waitlist families	45	31	23.3

Note: Sample sizes allow for a minimum of 6 months of follow-up after RA. MDEs assume 80% power for a two-tailed test significant at the 5% level, with no adjustment for multiple hypothesis testing.

^a For control group comparisons, assumes that 15 percent of the control group will lease up in opportunity areas, as was the case in CMTO (Bergman et al. 2020).

Exhibit B-7: Expected Sample Sizes and Minimum Detectable Effects (MDEs) for Lease-up in Opportunity Areas for Phase 1 of the Demonstration, Phase 1 Impact Report

Comparison	Sample	CMRS Sample Size	Control Sample size	MDE for Lease Up in Opportunity Areaa (percentage pts)
CMRS vs. control	All families	3,366	3,312	2.5
CMRS vs. control	Existing voucher families	3,186	3,186	2.5
CMRS vs. control	Waitlist families	180	126	11.6

Note: Sample sizes allow for a minimum of 6 months of follow-up after RA. MDEs assume 80% power for a two-tailed test significant at the 5% level, with no adjustment for multiple hypothesis testing.

^a For control group comparisons, assumes that 15 percent of the control group will lease up in opportunity areas, as was the case in CMTO (Bergman et al. 2020).

Because of the large number of factors that may influence the success of mobility programs in facilitating moves to opportunity areas, it will be important to examine variation in impact magnitudes according to site and moderating family characteristics. Accordingly, the evaluation contract will separately estimate impacts for a large number of subgroups. Foremost among these subgroups are impacts:

• Among existing voucher families;

• Among waitlist families; and

• By site.
  
  
  

In addition, the evaluation contractor will examine impacts among a large number of subgroups defined by potentially moderating family-level characteristics. These potential moderators are listed in Appendix 3.

For each subgroup, the evaluation contractor will estimate impacts using the same estimation model used for the full sample. In order to test for differences in impacts between two complementary subgroups, the evaluation contractor will add a treatment-subgroup characteristic interaction term to the model (with the subgroup characteristic also included on its own in the model). The test for statistical significance of the coefficient on the interaction term will serve as the statistical test for differences in impacts.⁸ For moderators with more than two subgroups (e.g., race/ethnicity with Black non-Hispanic, Hispanic, White non-Hispanic, and potentially other race/ethnic categories), the evaluation contractor will have a treatment interaction term for every category included in the model. A joint F-test on the coefficients of the interaction terms will serve as the statistical test for variation in impact according to the moderator.

Future Impact Analysis

Phase 2 analyses will estimate the impacts of CMRS versus control with the full five-year enrollment sample and, estimate the impacts of SMRS versus controls and SMRS versus CMRS. It will also add an analysis of the extent to which CMRS and SMRS affect families’ residence in an opportunity area at two years after random assignment, relative to the control condition, as another confirmatory outcome, which will need to be accounted for through a multiple comparisons procedure. In addition, after the enrollment phases of the study, HUD plans to examine the impact of being offered CMRS or SMRS on a range of long-term outcomes, including health and economic outcomes of children and adults. These long-term outcomes will be measured up to 30 years after random assignment in order to capture the long-term effects of neighborhoods on children.

In conducting these analyses, the evaluation contractor will examine impacts for the same subgroups specified above and in Appendix 3. The evaluation contractor will use the same analysis techniques for the Phase 2 analysis as used in the Phase 1 analysis.

Process Study Analysis

The evaluation contractor will conduct a thorough Process Study that meets four objectives: (1) provide context for understanding the quantitative results (e.g., what do PHAs, mobility services providers, and heads of household tell us about their experiences in the Demonstration, and what do heads of household report about their experiences moving, or seeking to move, to an opportunity area?); (2) help assess whether the Demonstration was implemented with fidelity; (3) help identify which of the CMRS services appear to have been most important in facilitating moves to opportunity areas; and (4) explore the extent to which different site contexts—PHA capacity, housing market dynamics, household characteristics, prevalence of racial or source-of-income discrimination, etc.—affect program success. For Phase 1, the qualitative data will contextualize the quantitative results (RQs 1 and 2) and provide exploratory answers to RQs 3-7 (see Exhibit A-1 in Supporting Statement A). The process study analysis will inform the RCE Report and Process and Impact Evaluation Report.

Analysis of interview data will involve organizing, summarizing, and synthesizing the data to support pooled and site-specific findings related to research questions 1 through 7. Findings will provide context for understanding the quantitative results (e.g., what do PHAs, mobility services providers, landlords, and families tell us about their experiences in the demonstration); help assess whether the demonstration was implemented with fidelity; help identify which of the CMRS services appear to have been most important in facilitating moves to opportunity areas; and explore the extent to which site context (e.g., regional partnership or sole agency) and household demonstration and voucher status (e.g., treatment or control household, existing or new voucher holder) affect outcomes. Data also will be analyzed to identify factors affecting program success (e.g., experiences of racial or source-of-income discrimination) and family attributes that might affect neighborhood options and preferences (e.g., age of children or household composition).

Summarizing key themes

The evaluation contractor will develop a thematic-memo template that each site team will use to prepare a summary of key themes identified during data collection. This memo will be prepared shortly after completing a site visit and will provide a high-level summary by theme and identify differences by respondent type. The memos will be required after both rounds of data collection. They will help identify additions or revisions to the coding scheme, provide an overview of variations among the sites, and support swift data analysis for the rapid cycle report.

Coding and Analysis

The evaluation contractor will test audio quality prior to the start of each interview. Once the interview is complete the team will do a quick check of the audio recording to make sure everything recorded properly. Following each site visit, the evaluation contractor will verify audio quality from interviews before transferring recordings to the selected transcription service. Interview transcripts will be imported into NVivo for auto-coding and in-depth thematic analysis. Analysis will differ between the first and second round of data collection because of reporting requirements, timeline and report objectives. After the first round of data collection, analysts will use NVivo’s automated coding function to identify text on key themes documented in thematic memos. Analysis for the first report (the RCE) will focus on producing pooled rather than site-specific findings.

Data analysis for the final phase 1 process and impact evaluation report will involve manual coding and support both a pooled analysis and site-specific findings. To ensure internal consistency of coding, the coding team will test code a transcript, compare codes, and adjust the codebook or coder understanding. The lead coder will conduct periodic consistency checks of coding to ensure ongoing consistency in coding the 612 expected transcriptions.

Once all the transcripts have been coded in preparation for the process and impact evaluation report, analysts will synthesize the themes that emerged and prepare comparative analytic summaries organized by interview topics and key research questions across respondent types and sites, analyzing for patterns and disparities. Summaries will be shared with members of the study team to inform their work, facilitate cross-team discussions, and integrate findings. Interview data also will be a source for illustrative quotes to include in the report.⁹

Analysis of Data on Outreach and Services Delivery

In the RCE report, the evaluation contractor will present summary statistics on implementation outcomes (outreach intensity and type, number of contacts for families or landlords, types of contact made) as well as simple cross-tabulations of data, assessing how the implementation outcomes varied by a limited set of characteristics (e.g. site, PHA, existing or new voucher holder). The report will present these summary statistics in order to assess implementation process overall, generating a high-level descriptive analysis of what is happening across sites and highlighting any significant differences in recruitment or implementation methods and outcomes. Analysis of the data after the second round of implementation site visits will focus on fidelity of CMRS service provision across sites and any emerging differences in participant experiences.

The final Phase 1 process and impact study will include the same descriptive analysis as the RCE report but will expand the cross-tabulation characteristics considered to include broader variables (e.g., race, outreach/household neighborhoods, household size and age of children, household income, control or treatment group, opportunity area, landlord characteristics, etc.). Additionally, this step of the analysis will include correlative testing (Pearson correlations, Chi square tests, and some OLS regressions) to understand whether and how the program’s implementation varied predictably across different situations. For the final report, the data will be analyzed in greater depth to evaluate quality of implementation and whether differences in implementation or process may have influenced differences in the outcomes.

For both the RCE and Phase 1 impact evaluation report, results from the analysis of quantitative data will be used along with those from analysis of interview and survey data to identify elements of the implementation that aided or hindered positive outcomes for families.

Cost Analysis

In the first two Demonstration reports, the cost analysis will determine the per-participant cost of the CMRS. To estimate the per-participant cost of CMRS, the evaluation contractor will develop cost estimates for individual services. These services-level cost estimates will serve as an input, together with conclusions from the process analysis, in a determination of whether some services appear more cost-effective than others.

The cost analysis will use an “ingredients method” approach that builds up cost estimates by identifying all inputs used to provide a program or intervention and assigning a monetary value to each input. These input-level costs are then summed to relevant sub-totals, in this case to individual service estimates, and to an overall total cost. Inputs are typically grouped by category to aid data collection and to provide context when present costs. High-level input categories planned for data collection in the cost analysis are as follows.

• Staffing

• Financial assistance and incentives

• Other direct program costs

• Materials and incidental costs

• Organizational indirect and overhead costs

Costs are expressed as per-unit costs. The evaluation contractor will report these costs in two-ways: per treatment- group member and per household receiving services. Per treatment-group member costs are allocated across all families that are eligible to receive services, whether or not they receive any. Per treatment-group member costs are comparable to impact estimates that include the whole sample of study participants. Per household receiving services costs are allocated across only families that actually receive services. These costs are useful for estimating costs of expanding access to services and for characterizing the intensity of the intervention.

Costs will be estimated as an incremental amount—the additional cost of services received by the treatment group relative to any similar services available to the control group (as reported by PHA and mobility service provider staff in the cost interviews). This added cost approach makes cost estimates comparable to impact estimates, which measure the effect of the additional services provided to the treatment group member. Estimating incremental costs allows the analysis to answer the question of what it costs to achieve the observed impact.

Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems associated with the samples for this information collection request.

Any Use of Periodic (less frequent than annual) Data Collection Cycles to Reduce Burden

With the exception of administrative data collection, which will cover the entire three-year period, the data collection described above will mostly take place only once during the three years covered by this PRA filing. For example, each household who enrolls in the study will complete the Household Roster and Baseline Information Form and Baseline Survey only once during this three-year period. Similarly, each household and landlord who participates in interviews will be interviewed only once during the three-year period. Many of the PHA and service delivery staff who are interviewed will be interviewed only once, though some may be interviewed more than once if needed to complete more than one interview protocol, for example someone may participate in both the PHA staff interview and the PHA cost interview.

3. Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

The evaluation contractor has undertaken extensive efforts to maximize the response rates for the collected data. These efforts are described below, first for the baseline and enrollment and then for the qualitative data collection.

Baseline and Enrollment. All heads of household who agree to participate in the study must complete the Household Roster and Baseline Information Form, provide informed consent, and complete the Baseline Survey in order to move on to the random assignment phase, where they will be randomly assigned either to be offered mobility-related services or to the control group. The evaluation team thus expects to obtain responses from 100 percent of the households who enroll in the study. All baseline data will be collected using a web-based enrollment tool. The enrollment tool will be used by PHA staff to enter the Household Roster and Baseline Information Form data and conduct random assignment. It will also be used by heads of household as they self-administer the Baseline Survey during the enrollment session. The enrollment tool has several features for addressing missing data. First, certain key fields are required—name, social security number (SSN) and date of birth (DOB) for the head of household, for example—thus requiring the PHA staff person conducting enrollment (or an agent assigned to this task by or on behalf of the PHA agent) to enter the data prior to saving the record. To ensure that heads of household do not skip questions inadvertently, the Baseline Survey will require participants to provide a response to each question—even if that response is ‘prefer not to say’ or ‘do not recall’. The enrollment tool also includes a set of built-in reports to monitor data quality. The Site-Wide Data Quality Report for example provides site-specific reports on various data quality metrics, including missing data, values outside expected ranges, potential duplicates.

Families who enroll in person will be given tablets or similar equipment to use to complete the Baseline Survey, or they may choose to use their own personal device. When families are not available in person to enroll in the study, they will be able to complete the enrollment process virtually and complete the Baseline Survey on their own personal computer or phone. If families do not have access to the internet, they will be invited into the PHA office where access will be provided

To maximize response to the Baseline Survey the evaluation contractor will reassure respondents that reported data are aggregated and not attributable to individuals or organizational entities. When done remotely, the enrollment tool will generate automatic email reminders, with the link to the survey, every couple of days over a two-week period to encourage the prospective participant to complete the survey. Finally, an incentive valued at $25 will be provided to families at the end of the enrollment process. If the enrollment session is done in person, the incentive will be provided at the end of the session, along with the random assignment notification materials. If the enrollment session is done remotely, the incentive will be sent by mail, along with the random assignment notification letter.

Qualitative Data Collection Interviews. PHAs are required to participate in the qualitative interviews as a condition of participating in the demonstration, Accordingly, a near 100% response rate is expected from these institutions. A similarly high response rate is expected from the mobility services staff, who are under contract to participate in the Demonstration. (Note that PHA and provider staff can always choose not to answer any question they are asked.) The other qualitative data collection efforts are heavily dependent on gaining cooperation with selected heads of household and landlords. The procedures to select participants for the qualitative interviews are described above in Section 2. This section describes the evaluation contractor’s use of a variety of proven methods to maximize participation and cooperation in the study.

• Maximizing response of household interview participants: The evaluation team will work with PHAs and draw from the enrollment and service provider data to identify households who have leased up using their voucher as well as households that are still searching for housing. Prospective interview participants will receive a letter by mail inviting them to participate in the study. The invitation letter will emphasize the importance of hearing their stories and understanding their experiences. Evaluation staff will call to explain the study in more detail and attempt to set up an appointment to conduct the interview. Prior to beginning the interview, the evaluation contractor will obtain informed consent to participate in the interview from the head of household.

• Recruitment of landlord interview participants: The evaluation team will work with PHA and mobility services provider staff to identify landlords with properties in opportunity neighborhoods who have had some engagement, either via outreach or recruitment efforts, with mobility services staff to recruit for the landlord interviews. Some landlords may be property owners while others may be property managers. It is likely that some of the landlords interviewed will have rented to a study participant or another HCV family, while others will not have. Prospective interview participants will receive a letter inviting them to participate in the study. The invitation letter will emphasize the importance of hearing their stories and understanding their experiences with the Demonstration. Evaluation staff will call to explain the study in more detail and attempt to set up an appointment to conduct the interview.

• The evaluation contractor will send a reminder email and/or place a reminder call or a day or two in advance of a scheduled interview to be sure the respondent is still able to make the interview.

• Respondents will be assured that reported data are aggregated and not attributable to individuals or organizational entities.

• An incentive valued at $40 will be provided to families in the Demonstration for their participation in an in-person interview. Landlord respondents will receive $30 for their participation. These incentives will help to offset the cost of participation in the interview. Incentives will be provided to respondents at the end of the interview if done in-person, otherwise the incentive will be sent by mail following completion of the interview.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

Most of the baseline questions are identical or similar to questions used in previous studies of similar populations led by the evaluation team or other national surveys. As such, they have been thoroughly tested on large samples. The same is true for many questions in the process study and cost study instruments included in this information collection request. The evaluation team relies on review of each instrument by skilled data collection staff and other study team staff, HUD personnel, and the study’s advisory panel to ensure that the instruments are clear, flow well, are as concise as possible, yet collect all the data necessary for analysis that is not otherwise available through other sources. Additionally, before the pilot enrollment phase begins, the evaluation contractor will conduct up to nine pre-tests of each data collection instrument with a small sample of people with similar characteristics. These pre-tests will provide information on the average length of the survey and identify any final modifications to improve the clarity and flow of the instrument. The formal pretest will ensure meaningful access to persons with limited English Proficiency (LEP) and must be conducted in a format accessible to individuals with disabilities, which includes providing effective communication and reasonable accommodations for individuals with disabilities.

The study team will provide to OMB the results from these pre-tests and any associated modifications to the data collection instrument prior to their review.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractors, grantees, or other person(s) who will actually collect or analyze the information for the agency.

HUD’s Office of Policy Development and Research will work with the contractor, Abt Associates and its partners the Urban Institute, MEF Associates, Social Policy Research Associates and Sage Computing to conduct the proposed data collection and analyze the data. Marina L. Myhre, PhD, and Leah M. Lozier, PhD, Social Science Analysts in HUD’s Office of Policy Development and Research, Program Evaluation Division serve as Contracting Officer’s Technical Representatives (COTRs). Their supervisor is Ms. Carol Star. Dr. Myhre can be contacted at (202) 402-5705, Dr. Lozier can be contacted at (202) 402-3013 and Ms. Star can be contacted at (202) 402-6139. The study’s Principal Investigator is Dr. Daniel Gubits from Abt Associates. Dr. Gubits can be reached at (301) 634-1854. Jeffrey Lubell serves as the study’s Project Director and can be contacted at (301) 634-1752. Melissa Vandawalker serves as the study’s Project Manager and can be contacted at (617) 349-2611.

The individuals shown in Exhibit B-8 also assisted HUD in the statistical design of the HCV Mobility Demonstration evaluation.

Exhibit B-8: Individuals Consulted on the Study Design

Name	Telephone Number	Role in Study
Mary Cunningham, Urban Institute	202-261-5764	Co-Principal Investigator
Jill Khadduri, Abt Associates	301-634-1745	Project Quality Advisor
Sam Dastrup, Abt Associates	301-347-5545	Cost Study Lead
Debi McInnis	617-349-2387	Senior Advisor
Diane Levy, Urban Institute	202-833-7200	Implementation Study Lead
Martha Galvez	212-998-6713	Consultant
Stephanie DeLuca	410- 516-8000	Consultant
Ingrid Gould Ellen	212-998-6713	Consultant
Sarah Oppenheimer	617-335-7444	Consultant
Kathy O’Regan	212-998-6713	Consultant

REFERENCES

Angrist, J. D., & Pischke, J.-S. (2008). Mostly harmless econometrics: An empiricist’s companion. Princeton University Press.

Bergman, P., Chetty, R., DeLuca, S., Hendren, N., Katz, L., & Palmer, C. (2020). Creating Moves to Opportunity: Experimental Evidence on Barriers to Neighborhood Choice. NBER Working Paper No. 26164. https://www.nber.org/system/files/working_papers/w26164/w26164.pdf

Chetty, R., Hendren, N., & Katz, L. F. (2016). The Effects of Exposure to Better Neighborhoods on Children: New Evidence From the Moving to Opportunity Experiment. American Economic Review 106(4), 855–902. https://pubs.aeaweb.org/doi/pdfplus/10.1257/aer.20150572

Gubits, D., Shinn, M., Wood, M., Bell, S., Dastrup, S., Solari, C. D., et al. (2016). Family Options Study: 3-Year Impacts of Housing and Services Interventions for Homeless Families. Submitted to the U.S. Department of Housing and Urban Development, Office of Policy Development and Research. Abt Associates. https://www.huduser.gov/portal/sites/default/files/pdf/Family-Options-Study-Full-Report.pdf

Gubits, D., Stapleton, D., Bell, S., Wood, M., Hoffman, D., Croake, S., Mann, D. R., et al. (2018). BOND Implementation and Evaluation: Final Evaluation Report, Volume 2 – Technical Appendices. Submitted to the Social Security Administration, Office of Research, Demonstration, and Employment Support. Abt Associates and Mathematica Policy Research. https://www.ssa.gov/disabilityresearch/documents/BOND%20Deliv%2024e2%20FER%20Vol%202%20Appendix%2020181018.pdf

Mamun, A., Wittenburg, D., Denny-Brown, N., Levere, M., Mann, D. R., Coughlin, R., Croake, S., et al. (2021). Promoting Opportunity Demonstration: Interim Evaluation Report. Submitted to the Social Security Administration, Office of Research, Demonstration, and Employment Support. Mathematica Policy Research. https://www.ssa.gov/disabilityresearch/documents/POD_Interim_Evaluation_Report.pdf

Riccio, J., Verma, N., & Deitch, V. (2019). The Rent Reform Demonstration: Interim Findings on Implementation, Work, and Other Outcomes. Submitted to the U.S. Department of Housing and Urban Development, Office of Policy Development and Research. MDRC. https://www.mdrc.org/sites/default/files/RentReform-InterimFindings_0.pdf

Sanbonmatsu, L., Potter, N. A., Adam, E., Duncan, G. J., Katz, L. F., Kessler, R. C., Ludwig, J., et al. (2012). The Long-Term Effects of Moving to Opportunity on Adult Health and Economic Self-Sufficiency. Cityscape 14(2), 109–136. https://www.jstor.org/stable/41581100

Verma, N., Freedman, S., Tessler, B., Nuñez, S., & Fink, B. (2019). Promoting Work and Self-Sufficiency for Housing Voucher Recipients: Early Findings From the Family Self-Sufficiency Program Evaluation. Submitted to the U.S. Department of Housing and Urban Development, Office of Policy Development and Research. MDRC. https://www.mdrc.org/sites/default/files/Promoting-Work-and-Self-Sufficiency-for-Housing-Voucher-Recipients.pdf

Westfall, P. H., Tobias, R. D., & Wolfinger, R. D. (2011). Multiple Comparisons and Multiple Tests Using SAS^®, 2nd ed. SAS Institute.

Appendix 1: Neighborhood and Housing Unit Outcomes

A. Neighborhood (Census Tract) Characteristics
Distance
	Mean Commute Time in 2000(?) (minutes)
	% Commute < 15 minutes
	Distance to City Hall of Largest City in Commuting Zone (Miles)
Resident demographics
	% Black (recent ACS)
	% Hispanic (recent ACS)
	% White (recent ACS)
	% Foreign-Born (recent ACS)
	% Married (2020)
	% of Children with Single parents (ACS)
	% >= college Education (ACS)
	Population Density (2020, # people per square mile)
Tract Income and Other Characteristics
	Median HH Income (ACS)
	% Labor Force Participation (2020)
	% Poverty (ACS)
	Median Home Value (2020)
	Census Mail Response rate
	Theil Index of Racial Segregation
	# Jobs for No HS Degree, 1 mile Radius
Children's Long-Term Outcomes (from Opportunity Index data)
	Predicted Mean Individual Income Rank (p=25)
	Predicted Mean Household Income Rank (p=25)
	Predicted Mean Household Income Rank for White Children (p=25)
	Teenage Birth Rate for Women (p=25)
	Incarceration Rate (p=25)
Other Indices of Opportunity
	Brandeis COI Overall Child Opportunity Score
	Brandeis COI Educational Subscore
	Brandeis COI Health/Environment Subscore
	Brandeis COI Social/Economic Opportunity Subscore
	HUD transit Index
	Environmental Health Index
B. Unit Characteristics
	Square Feet
	Year Built
	Household Appliance Index
	Baths
	Share with Air Conditioning
	Total Rent Paid to Owner
	Rent Paid by PHA
	Utilities Paid (Estimate by PHAs)
	Total Out of Pocket Expenditures (Tenant)

Appendix 2: List of Impact Analyses

Sample	Pooled Sites/ By Site	Type of Statistical Test	RCE Reporting Timepoint (Expected sample size)	Phase 1 Reporting Timepoint (Expected sample size)	Phase 2 Reporting Timepoint (Expected sample size)
OUTCOME: Reside in Opportunity Area at 6 months after RA
Full sample	Pooled sites	Impact within sample	RCE(1,670)*	P1(6,678)*	P2(13,506)
Existing voucher families	Pooled sites	Impact within subgroup	RCE(1,593)*	P1(6,374)*	P2(12,750)
New voucher families	Pooled sites	Impact within subgroup	RCE(77)*	P1(304)*	P2(756)
Existing vs. New	Pooled sites	Variation across subgroups		P1(6,678)	P2(13,506)
Full sample	By site	Impact within site	RCE(189)	P1(754)	P2(1,524)
Full sample	By site	Variation across sites	RCE(1,670)	P1(6,678)	P2(13,506)
Existing voucher families	By site	Impact within site	RCE(180)	P1(720)	P2(1,440)
Existing voucher families	By site	Variation across sites	RCE(1,593)	P1(6,374)	P2(12,750)
New voucher families	By site	Impact within site		P1(34)	P2(84)
New voucher families	By site	Variation across sites		P1(304)	P2(756)
[Other Subgroups] within Existing voucher families	Pooled sites, Existing voucher families only	Impact within subgroup		P1(varies)	P2(varies)
[Other Subgroups] within Existing voucher families	Pooled sites, Existing voucher families only	Variation across subgroups		P1(6,374)	P2(12,750)
[Other Subgroups] within New voucher families	Pooled sites, New voucher families only	Impact within subgroup		P1(varies)	P2(varies)
[Other Subgroups] within New voucher families	Pooled sites, New voucher families only	Variation across subgroups		P1(304)	P2(756)
[Other Subgroups] within Full sample	Pooled sites	Impact within subgroup	RCE(varies)	P1(varies)	P2(varies)
[Other Subgroups] within Full sample	Pooled sites	Variation across subgroups	RCE(1,670)	P1(6,678)	P2(13,506)
OUTCOME: Reside in Opportunity Area at 2 years after RA
Full sample	Pooled sites	Impact within sample		P1(1,670)*	P2(10,087)
Existing voucher families	Pooled sites	Impact within subgroup		P1(1,593)	P2(9,561)
New voucher families	Pooled sites	Impact within subgroup		P1(77)	P2(526)
Existing vs. New	Pooled sites	Variation across subgroups		P1(1,670)	P2(10,087)
Full sample	By site	Impact within site		P1(189)	P2(1,139)
Full sample	By site	Variation across sites		P1(1,670)	P2(10,087)
Existing voucher families	By site	Impact within site		P1(180)	P2(1,080)
Existing voucher families	By site	Variation across sites		P1(1,593)	P2(9,561)
New voucher families	By site	Impact within site			P2(59)
New voucher families	By site	Variation across sites			P2(756)
[Other Subgroups] within Full sample	Pooled sites	Impact within subgroup		P1(varies)	P2(varies)
[Other Subgroups] within Full sample	Pooled sites	Variation across subgroups		P1(1,670)	P2(10,087)
OUTCOME: Neighborhood and Housing Unit Characteristics at 6 months after RA
Full sample	Pooled sites	Impact within sample	RCE(1,670)	P1(6,678)	P2(13,506)
Existing voucher families	Pooled sites	Impact within subgroup		P1(6,374)	P2(12,750)
New voucher families	Pooled sites	Impact within subgroup		P1(304)	P2(756)
Existing vs. New	Pooled sites	Variation across subgroups		P1(6,678)	P2(13,506)
Full sample	By site	Impact within site	RCE(189)	P1(754)	P2(1,524)
Full sample	By site	Variation across sites	RCE(1,670)	P1(6,678)	P2(13,506)
OUTCOME: Share who moved by 6 months after RA
Existing voucher families	Pooled sites	Impact within subgroup	RCE(1,593)	P1(6,374)	P2(12,750)
Existing voucher families	By site	Impact within site	RCE(180)	P1(720)	P2(1,440)
Existing voucher families	By site	Variation across sites	RCE(1,593)	P1(6,374)	P2(12,750)
[Other Subgroups] within Existing voucher families	Pooled sites, Existing voucher families only	Impact within subgroup		P1(varies)	P2(varies)
[Other Subgroups] within Existing voucher families	Pooled sites, Existing voucher families only	Variation across subgroups		P1(6,374)	P2(12,750)
OUTCOME: Leased-up by 6 months after RA
New voucher families	Pooled sites	Impact within subgroup	RCE(77)	P1(304)	P2(756)
New voucher families	By site	Impact within site		P1(34)	P2(84)
New voucher families	By site	Variation across sites		P1(304)	P2(756)
[Other Subgroups] within New voucher families	Pooled sites, New voucher families only	Impact within subgroup		P1(varies)	P2(varies)
[Other Subgroups] within New voucher families	Pooled sites, New voucher families only	Variation across subgroups		P1(304)	P2(756)
OUTCOME: Receives housing assistance at 6 months after RA
Full sample	Pooled sites	Impact within sample	RCE(1,670)	P1(6,678)	P2(13,506)
Existing voucher families	Pooled sites	Impact within subgroup	RCE(1,593)	P1(6,374)	P2(12,750)
New voucher families	Pooled sites	Impact within subgroup	RCE(77)	P1(304)	P2(756)
Existing vs. New	Pooled sites	Variation across subgroups		P1(6,678)	P2(13,506)
Full sample	By site	Impact within site	RCE(189)	P1(754)	P2(1,524)
Full sample	By site	Variation across sites	RCE(1,670)	P1(6,678)	P2(13,506)
OUTCOME: HAP amount (no HAP=$0) at 6 months after RA
Full sample	Pooled sites	Impact within sample	RCE(1,670)	P1(6,678)	P2(13,506)
Existing voucher families	Pooled sites	Impact within subgroup	RCE(1,593)	P1(6,374)	P2(12,750)
New voucher families	Pooled sites	Impact within subgroup	RCE(77)	P1(304)	P2(756)
Existing vs. New	Pooled sites	Variation across subgroups		P1(6,678)	P2(13,506)
Full sample	By site	Impact within site	RCE(189)	P1(754)	P2(1,524)
Full sample	By site	Variation across sites	RCE(1,670)	P1(6,678)	P2(13,506)

Note: The impact estimation sample sizes for outcomes measured six months after random assignment assume that the Rapid Cycle Evaluation report will include families enrolled in the first six months of enrollment, the Phase 1 Final Report will include families enrolled in the first two years of enrollment, and the Phase 2 Final Report will include families enrolled over all five years of enrollment. For outcomes measured two years after random assignment, the Phase 1 Final Report will include families enrolled in the first six months of enrollment and the Phase 2 Final Report will include families enrolled in the first 3.5 years of enrollment.

*: Confirmatory hypothesis test that will have p-value adjusted by multiple comparison procedure.

Appendix 3: Subgroups Defined by Potentially Moderating Characteristics.

Type of Characteristic	Potentially Moderating Characteristic Subgroup 1 Subgroup 2
HEAD OF HOUSEHOLD CHARACTERISTICS
	Race/ethnicity Black, non-Hispanic White, non-Hispanic Hispanic (Potentially other racial/ethnic groups if sample sizes are sufficient)
	Country of birth Born outside U.S. Born inside U.S.
	Primary Language English isn't Primary Language English is Primary Language
	Length of residence in metro area* Less than 5 years in metro area 5-14 years in metro area 15 or more years in metro area
	Baseline residence in Opportunity Area Started in opportunity area Didn't start in opportunity area
	Income* Lower tercile of site Middle tercile of site Upper tercile of site
	Education (may add categories after assessing baseline data) No college Some college or more
	Employment Working full-time (30 hours or more per week) at baseline Working part-time (1-29 hours or more per week) at baseline Not working at baseline
	Childcare Uses childcare Doesn't use childcare
	Transportation Have access to a car that runs Does not have access to a car that runs
PERCEPTIONS ABOUT NEIGHBORHOOD AND MOVING
	Satisfaction with current neighborhood Satisfied with Current Neighborhood Unsatisfied/indifferent with Current Neighborhood
	Neighborhood safety Feels neighborhood is safe Feels neighborhood is unsafe
	Outlook on staying in neighborhood Sure wants to leave current neighborhood Sure wants to stay in current neighborhood or Indifferent
	Outlook on being racial minority in neighborhood Comfortable about moving to a racially different neighborhood Uncomfortable/indifferent moving to a racially different neighborhood
	Outlook on ability to pay moving expenses Sure could pay moving expenses without program Not sure could pay moving expenses without program
CHILDREN CHARACTERISTICS
	Age of child Any child aged 13 or older All children under age 13
	Number of children More than 2 children 1 or 2 children
NEIGHBORHOOD CHARACTERISTICS
	Neighborhood poverty* Baseline neighborhood below 80 percent of site-level median poverty rate Baseline neighborhood 80-120 percent of site-level median poverty rate Baseline neighborhood above 120 percent of site-level median poverty rate

Note: For most potentially moderating characteristics, the statistical test for variation in impacts will test for difference in impact between subgroups. For starred characteristics, in order to maximize precision, the statistical test for variation will examine the coefficient on an interaction term of the treatment indicator and a continuous measure of the moderating characteristic.

1 Approximately 7,000 additional households are expected to be enrolled in the two years following this initial three-year period; HUD expects to renew its Information Collection Request to extend data collection for this additional period.

2 A small number of waitlist households may be deemed ineligible after random assignment. The consent form notes that and explains that households deemed ineligible will not end up receiving a voucher.

3 Existing voucher families will be assigned to the CMRS and the Control groups at equal rates. Due to the legislative provisions that govern the use of vouchers used by waitlist families, waitlist families will have a somewhat higher probability of being assigned to the CMRS group than the Control group.

4 Some PHAs may choose to designate an agent to administer the Household Roster and Baseline Information Form, use the enrollment tool to determine a household’s random assignment status, and enroll households in the study. Throughout this supporting statement, the term “PHA staff” is used to be inclusive of any such agents, whether or not they are formally employed by the PHA as staff members.

5 Staff interviews and participant interviews will occur under the same 4-5 day site visit. Wherever possible, the evaluation contractor will minimize the time required on site.

6 Angrist, J. D., & Pischke, J.-S. (2008). Mostly harmless econometrics: An empiricist’s companion. Princeton, NJ: Princeton University Press, 47.

7 Recent evaluations that have used a 0.10 level of significance include the Moving to Opportunity Fair Housing Demonstration (Sanbonmatsu et al., 2012; Chetty, Hendren & Katz, 2016), the Family Options Study (Gubits et al., 2016), the Rent Reform Demonstration (Riccio, Verma & Deitch, 2019), the Family Self-Sufficiency Program Evaluation (Verma et al., 2019), and the Promoting Opportunity Demonstration (Mamun et al., 2021).

8 For three characteristics, in order to maximize precision, the evaluation contractor will test for moderation in another manner. These three characteristics—income, census tract poverty rate, and length of residence in the metropolitan area—have continuous measures with thresholds that may vary in meaning across sites. The alternate statistical tests for variation will examine the coefficient on an interaction term of the treatment indicator and the continuous measure of the moderating characteristic.

9 Future demonstration reports will likely also integrate these findings but the timing and content of those reports have not yet been planned.

1