final-4-27-21- Choice Supporting Statement Part B

Supporting Statement for Paperwork Reduction Act Submissions

Outcomes Evaluation of the Choice Neighborhoods Program

OMB Control # 2528-xxxx

B. Collections of Information Employing Statistical Methods

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

Evaluation Overview

The goal of the evaluation is to answer the following overarching research question: whether public and private dollars were successfully leveraged to 1) replace distressed public and HUD-assisted housing with high-quality mixed-income housing that is well-managed and responsive to the needs of the surrounding neighborhood, 2) improve outcomes for households in the target housing, including employment and income, health, and education, and 3) create the conditions necessary for public and private reinvestment in distressed neighborhoods to improve amenities and assets.

The evaluation will employ quantitative and qualitative methods, including analysis of administrative/secondary data, primary data collection in the form of a large survey of households living in the Choice sites, and interviews and observations from stakeholders regarding the Choice program. In total, Urban expects to field the survey to up to 2,388 Choice residents and contact 252 respondents for qualitative interviews. This information is necessary to the evaluation, specifically to understand differences across Choice sites, over time, in different types of HUD-assisted housing, by grantee type, and for different contextual conditions.

Household Survey

Using the Choice baseline survey as a starting point for survey design, Urban will implement a follow-up survey of residents of the focal developments that captures information about a range of topics, including: housing mobility, housing conditions, neighborhood satisfaction and conditions, services and supports, household composition, physical and mental health, employment, education, public assistance, and child well-being.

The Choice follow-up survey will sample three groups at each of the five sites: Quincy Corridor neighborhood in Boston, Massachusetts; Woodlawn neighborhood in Chicago, Illinois; Iberville/Tremé neighborhood in New Orleans, Louisiana; Eastern Bayview neighborhood in San Francisco, California; and Yesler neighborhood in Seattle, Washington;

At each site, the follow-up survey will sample data from three groups:

1. Households from original target developments;

2. Households in new or revitalized non-replacement units; and

3. Households in new or revitalized replacement units.

Exhibit B-1 shows survey populations, sampling frames, sample sizes, and response rates. To understand how household outcomes may have changed, the contractor will attempt to contact all 753 baseline survey respondents that lived in the target development, with the goal of achieving 465 completes, a 62 percent response rate. In addition, the research team will prepare a random sample of households from the original list of households in the original development that did not participate in the baseline survey to use in the case that they cannot obtain the goal of 465 completes using the 753 baseline survey respondents.

Exhibit B-1: Proposed Survey Populations, Sampling Frames, Sample Sizes, and Response Rates

Group	Residents from original target developments	Residents in new, revitalized non-replacement units	Residents in new, revitalized replacement units	Total
Universe	1,622	1,740	1,695	5,057
Sample	753	1,000	635	2,388
No. of Completes	465	410	312	1,187
Response Rate	62%	41%	49%	50%

The research team will obtain sample contact information for baseline survey respondents from HUD through the Baseline Survey and the Tracking Study Data, which was last updated in 2017. Urban will also request PIC/TRACS administrative data from HUD to update contact information for baseline survey sample members who are still HUD-assisted. The research team anticipates that some of this group will be living in replacement units, while others will be offsite or unassisted. They anticipate a response rate of 62 percent, lower than the 80 percent response rate achieved for this group at baseline.

In addition, Urban will survey households that were not baseline survey respondents but live in the replacement units or non-replacement units in the focal developments. The research team will invite 1,000 potential respondents that live in non-replacement units with an estimated 410 completes for a 41 percent response rate. Urban is assuming that the response rate for the sample in non-replacement units is almost as low as the response rate for the neighborhood sample at baseline (40 percent) because Urban will not necessarily have a name for the head of household. This is somewhat higher than the neighborhood sample because they are living in the new development, where they can ask around about the survey before taking it.

Finally, 635 potential respondents that live in replacement units will be invited, with an estimated 312 completes, for a 49 percent response rate. Urban anticipates this favorable response rate from this group because Urban will have the names of potential respondents.

To establish the sampling frames for the replacement and non-replacement units, Urban will request from HUD or grantees a list of addresses, including unit numbers, for all replacement and non-replacement units. Urban will use those lists to randomly select the sample addresses. The research team plans to stratify the sample by type of income restriction (i.e., affordable up to 50-60 percent AMI, Choice Neighborhoods-funded housing up to 120 percent AMI, and unrestricted, market rate at or above 80 percent AMI).

Urban will request administrative data from HUD or grantees on households in the replacement units. They will request 1) a list of addresses considered to be replacement units, 2) contact information for the head of household in the replacement units, and 3) additional fields on households in the replacement units. These data would provide a sample frame for the replacement unit sample, and will allow them to identify baseline survey respondents in the replacement unit sample, and provide a reference point for assessing whether respondents are similar to or different from other replacement unit households.

Urban will request administrative data from grantees on households in the replacement units. The data from grantees should include names of the head of household, addresses, basic demographic information, and the date of the data. The sample assumes that 21 percent (or 119) of the households in new replacement units will be original households¹ but this estimate may be lower for some sites and higher for others. When the research team receives the PIC/TRAC data from HUD for the baseline residents, the research team will see whether its assumptions are correct. Ultimately, the research team would like a sample of (381) households in the replacement unit sample that are new to the study.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

The research team will survey all target development respondents from the baseline survey. If they do not achieve their goal of 565 completes with that group, they will supplement it with randomly sampled original target development residents who did not participate in the baseline survey. For the non-replacement unit sample, they stratify the list of non-replacement units at each site by type of income restriction (i.e., affordable up to 50-60 percent AMI, Choice Neighborhoods-funded housing up to 120 percent AMI, and unrestricted, market rate at or above 80 percent AMI) and randomly select samples from those groups. For the replacement unit sample, the research team will randomly select the sample from a list of replacement units at each site.

DIR will use a combination of web and computer-assisted telephone interviews (CATI) to collect survey data from the sampled respondents. The data collection period is scheduled to run for a 6-month period. However, survey operations in any one site will be approximately 18 weeks in duration. At the start of data collection in each site, respondents will receive an initial invitation to participate in the self-administered web survey that includes information about the Choice survey, the respondent’s rights as a participant, contact information for DIR’s study-specific toll-free number, and a web link and password for accessing the online version of the survey and a $2 prepaid incentive to encourage participation in the survey. The mailing will include a brief message and phone number for speakers of Spanish, Somali, Cantonese, and Vietnamese to learn more about the study. The mailer will detail additional language assistance options for persons with limited English proficiency. It will also include a correspondence with extra large font to improve accessibility for those with vision impairment, and notify individuals with disabilities that they may request reasonable accommodations in order to participate in the study. The invitation will inform respondents that if they complete the survey within the specified two-week deadline, they will receive an early-bird incentive of $10 in addition to the $50 incentive they propose to offer all participants. Participants will receive a reminder postcard sent to arrive approximately seven days in advance of the early-bird incentive expiration date specified in the initial mailing. The web option will remain available to all respondents for the duration of their data collection period. For respondents that complete the survey on the web, the incentive payment will be sent to the USPS address gathered during the interview, unless the respondent specifies a preferred alternative way to receive the incentive. The survey software used for the web and CATI survey is Section 508 compatible.

After the end of the two-week early bird period, trained and certified DIR telephone interviewers will begin outbound dialing to sample members. Once cases are active for outbound dialing, DIR will dial cases across a range of days (including weekdays and weekends) and times (including morning, afternoon, and evening) to maximize the likelihood of contact. When contact is made, interviewers, trained in gaining cooperation and refusal aversion with the Choice target population, will engage the respondent and conduct the survey. The incentive payment will be sent to the respondent at an address gathered during the interview.

If field interviewers are not allowed to recruit survey participants in person, DIR will take two additional steps. First, the case will be assigned to a DIR interviewer for personalized “hand management” before it has been exhausted by CATI interviewers. Second, up to two additional mailings will be sent to remaining cases. The first mailing will include a letter with an additional $2 incentive and a promise of a $60 post-completion incentive. If they do not respond to that letter, a second reminder letter will be sent emphasizing the limited time to participate and reiterating the offer of the $60 post-completion incentive. Survey participants will receive a $60 incentive payment using the address gathered during the interview or web-based survey.

If field interviewers are allowed to recruit survey participants in person, cases that have not been completed by either web or CATI will be assigned to site-based field interviewers after the available telephone numbers for a case have been exhausted by CATI interviewers. The research team currently anticipate the majority of cases will have moved to the field approximately eight weeks after dialing begins. Field staff will work to locate and interview sample members. Each field interviewer will be assigned a geographically clustered group of cases and beginning with existing contact information, seek to find the sampled respondent. When the existing contact information is not effective, field interviewers will collect information from neighbors, postal employees, and others about where the sample member may be and his or her patterns of visiting the location. Once the sample member is contacted, the field interviewer will conduct the survey. When the field interviewer completes the survey with the sample member in person, the incentive payment will be made at that time. Otherwise, the incentive payment will be sent to the respondent at an address gathered during the interview.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Some of the households in the sample are considered to be a hard-to-reach population because they have moved after being displaced for the project, making it likely that they have experienced multiple moves during the redevelopment and their contact information may not be up to date. Their relationship to the grantee may be strained, affecting their interest in participating in the study. Other households in the sample who are living in replacement and non-replacement units, who are new to the study, may also be considered hard-to-reach because they are not familiar with the redevelopment of the community and may not be compelled to participate in the study. To attain high response rates with this population, Urban will employ a sub-contractor, DIR, to passively track sample participants to improve the accuracy of their contact information. Tracking will begin prior to and continue throughout data collection. Sample members with missing or incorrect contact information will be assigned to the tracking process, which employs a variety of sources to update contact information about hard-to-locate sample members, including alternate contact information from baseline and online nationwide search databases including directory assistance, 555-1212.com, Accurint, Lexis-Nexis, and Intellius.

To increase awareness of the upcoming survey and further encourage sample members to update their contact information, DIR will send a flyer to the full sample with an available USPS address, including addresses obtained from the passive tracking effort. If language information is known or there is a reasonably high share of non-English speakers in a site, DIR will send a double-sided flyer (with English and Spanish, or perhaps Vietnamese, Cantonese, or Somali).

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

DIR will conduct a formal pretest of the follow-up survey using telephone administration. DIR will prepare training materials and train experienced interviewers to administer the pretest. The formal pretest will be conducted over a 2-week period attempting up to 50 sample members of the 2,388 total distributed between the baseline assisted sample and residents in replacement and non-replacement housing. DIR will complete up to nine surveys for the pretest, at least five of whom will be from the baseline assisted residents sample.

Each respondent that completes the pretest survey will receive a $50 incentive payment in the form of either a hardcopy gift card or electronic payment. DIR will manage the delivery of incentives to those respondents completing the pretest. Hard copy incentives typically will be mailed to respondents 3 - 4 weeks after the survey has been completed. Electronic payments will be made available as soon as the interview is completed. DIR will track the incentive payments associated with each respondent and the date that they are issued in an incentive log.

During the pretest, interviewers will complete a pretest evaluation form for all completed and partially completed pretest surveys. The survey leadership team will debrief with the interviewers to determine if there is any language that confused a respondent or if any skips did not flow well throughout the survey; in addition, the leadership team will review each pretest evaluation form. A summary of the results of the pretest evaluations will be included in a Pretest Report that will be submitted to Urban. Modifications to the survey instrument will be considered based on the pretest results.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

HUD has contracted with The Urban Institute and their partners Decision Information Resources (DIR) and Case Western Reserve University’s (CWRU) National Initiative on Mixed-Income Communities. The data collection procedures will be similar to those used in other studies conducted by The Urban Institute. The HUD Contracting Officer’s Representative (COR) reviewed all the procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call either the HUD COR, Leah Lozier (202-402-3013) or the Urban Institute co-Principal Investigators, Diane Levy (202-261-5642) and Brett Theodos (202-261-5865). Other key members of the research team consulted in the statistical aspect of the design are:

Heather Morrison

Director of Data Collection and Management

Decision Information Resources

713-650-1425

Megan Gallagher

Senior Research Associate

Urban Institute

202-261-5253

Tim Triplett

Senior Research Associate and Senior Survey Methodologist

Urban Institute

202-261-5579

1 Relying on Joice, Paul. 2017. “HOPE and Choice for HUD-Assisted Households.” Cityscape, 19 (3): 449-473 and Popkin, Susan J., Diane K. Levy, and Larry Buron. 2009. “Has HOPE VI Transformed Residents’ Lives? New Evidence from the HOPE VI Panel Study,” Housing Studies, 24(4): 477-502.