Final - Supporting Statement Part B_MTW Cohort 1

Supporting Statement for Paperwork Reduction Act Submissions

Evaluation of Cohort 1 of the Moving to Work Demonstration Program Expansion
OMB Control # 2528-0328

B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

HUD designed the Evaluation of Cohort 1 of the Moving to Work Demonstration Program Expansion (Cohort 1 Evaluation) to allow for a rigorous measurement of the impacts of MTW designation. Cohort 1 is limited to “small” PHAs, defined as PHAs administering no more than 1,000 housing units across their HCV and public housing programs. The initial application period to determine the interested and eligible PHA pool for Cohort 1 opened in October 2018 and closed in May 2019. Fifty-one PHAs submitted applications by the May 2019 deadline. HUD reviewed each application and determined that 43 PHAs met the eligibility requirements for Cohort 1. These 43 PHAs are located in 30 states and at the time of application ranged in size from 57 units to 952 units, with a median of 506 units.

In November 2019, HUD used random assignment statistical software to assign the 43 PHAs into a treatment group (33 PHAs to be offered MTW designation) and a control group (10 PHAs that would not be offered MTW designation) by geographic region. HUD announced the results of random assignment in August 2020 and invited the PHAs assigned to the treatment group to complete the application for MTW designation. HUD expects to announce the final list of PHAs receiving MTW designation under Cohort 1 in January 2021, after reviewing the completed applications.

Exhibit B-1 provides the sample description, expected sample size, and expected response rate for each data collection source.

All of the data collection in this ICR will be done by Abt Associates (the “research team”). The research team expects to collect the data from key staff at the 43 study PHAs with a 100 percent response rate. The staff to be interviewed include the PHA’s Executive Director, HCV Director, and Public Housing Director (as applicable). All PHAs that applied to Cohort 1 agreed to participate in the evaluation.

Exhibit B-1: Sampling Plan

Primary data collection source	Sample description	Potential respondent universe	Expected respondent sample size per study year	Expected Response Rate
Interviews with MTW PHAs	Up to 3 staff at each of the 33 PHAs in the study’s treatment group	Staff at the 33 PHAs in the study’s treatment group	3 staff per PHA (n=99)	100%
Online Surveys with non-MTW PHAs	1 staff at each of the 10 PHAs in the study’s control group	Staff at the 10 PHAs in the study’s treatment group	1 staff per PHA (n=10)	100%
Interviews with non-MTW PHAs	Up to 2 staff at each of the 10 PHAs in the study’s control group	Staff at the 10 PHAs in the study’s treatment group	2 staff per PHA (n=20)	100%

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This submission does not require a statistical methodology plan, as the sampling of PHA participants is purposive. The study team will work with each PHA’s Executive Director to identify the most relevant staff to answer the study’s research questions. The respondents are limited in number and are not intended to constitute a representative sample of all staff in the study PHAs. In addition, we will not seek to draw statistical inferences from the study data covered in this submission.

Most respondents will only be involved in annual data collection. Some PHA staff may be involved in a limited amount of ad-hoc data collection from the research team in the form of short telephone calls or email exchanges, but these are not formal reporting mechanisms, will not follow a standardized script, and will not be burdensome.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

The study team has subjected every data collection instrument and question to careful scrutiny to make sure it is needed for the evaluation and does not duplicate other existing data sources. The team will also clearly communicate the reason for collecting the data to the respondents and make it as easy as possible for respondents to provide the data.

Although all PHAs applying to Cohort 1 agreed to participate in the evaluation, PHAs may have varying levels of interest in completing the study’s surveys and telephone interviews. The non-MTW PHAs may be least likely to want to participate—knowing that they did not receive the MTW designation they had hoped for and perhaps thinking they have nothing to offer the evaluation. Anticipating some reluctance on the part of PHAs, the study team has tried to minimize the burden of the primary data collection by only asking PHAs for data that is not reported to HUD elsewhere and using the least time intensive approach—simple online surveys and telephone interviews. We also hope to incentivize PHAs’ participation by being very clear up front about what the study team will need from the PHAs and by offering something meaningful to PHAs in return, such as data visualizations, opportunities to comment on draft study findings, and peer learning opportunities.

Each PHA will have a dedicated liaison from the Abt research team for the duration of the evaluation (ideally the same person throughout). The liaison will be the primary point of contact between the PHA and the research team and will be responsible for conducting all data collection activities and achieving high response rates.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

In the first year of data collection, the study team will pretest the telephone interview guides and online survey with three PHAs from the treatment group and two PHAs from the control group. The study team will conduct the data collection activities and ask the respondents for their feedback on the length, content, or structure of the interview or survey. Based on this feedback, the team will make adjustments as needed to the protocols for the remaining data collection.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

HUD contracted with Abt Associates to conduct the data collection. Dr. Laura Peck (Principal Investigator), Dr. Larry Buron (Project Director), and Dr. Judy Geyer (Director of Analysis) developed the statistical aspects of the design. HUD’s Contracting Officer’s Representative (COR), Anne Fletcher, reviewed all the procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call either the HUD COR, Anne Fletcher (202-402-4347) or the Abt Associates Project Director, Larry Buron (301-634-1735).

In addition, Abt Associates has established a panel of Senior Advisors to review the evaluation design, progress, and findings, to maximize the rigor of the evaluation and its value to multiple stakeholders. This panel includes:

• Amy Ginger, Fairfax County Housing and Redevelopment Authority

• Dr. Sandra Newman, Johns Hopkins University

• Chris Kubacki, Phineas Consulting