2528 IHS Supporting Statement

The U.S. Department of Housing and Urban Development (HUD) is undertaking an evaluation of the Impacts of Housing and Services Interventions for Homeless Families to provide research evidence to help federal policymakers, community planners, and local practitioners make sound decisions about the best ways to address homelessness among families. In the remainder of this document, we refer to the study as the Homeless Families Impact Study. The study will compare four combinations of housing and service interventions for homeless families in a rigorous, multi-site experiment, to determine what interventions work best to promote family stability and well-being and, within the limits of statistical power, what sorts of families benefit most from each intervention. The interventions are: 1) permanent housing subsidy without services (Subsidy Only); 2) Community-Based Rapid Re-housing (CBRR), consisting of temporary housing subsidy provided in conventional housing with limited supportive services; 3) temporary housing subsidy provided in facility-based housing with intensive services but no guarantee of a permanent subsidy (Project-Based Transitional Housing-PBTH); and 4) shelter, with whatever services the shelter ordinarily provides to its residents and any other assistance available in the community (Usual Care). This study will also exploit naturally occurring variation in program features within these categories and across sites to explore, non-experimentally, what features of programs seem most responsible for success.

The U.S. Department of Housing and Urban Development has invested considerable resources in strategies to address family homelessness. In HUD’s budget request for 2010, $1.794 billion is proposed for Homeless Assistance Grants, providing rental assistance, emergency shelter, transitional and permanent housing and supportive services to homeless individuals and families.¹ The results of this evaluation will provide evidence to inform policy makers how best to set priorities for those funds and to design eligible activities. Similarly, in response to HUD Continuum of Care funding requirements and in an effort to maximize the effectiveness of limited resources, communities systematically examine their homeless assistance systems to decide which housing and service interventions should be funded. Some cities often expend their own appropriations on interventions for homeless individuals and families. Unfortunately, past research is inadequate to guide federal policy and local practice. While there is a significant amount of research on the characteristics and needs of homeless families and an emergent body of descriptive research on intervention programs and outcomes for families who use them, there is almost no information about the relative effectiveness or cost-effectiveness of different interventions. Senate Report 109-109 for The FY2006 Transportation, Treasury, Judiciary, HUD, and Related Agencies Appropriations bill directed the Department to focus its energies on homeless families and to “undertake research to ascertain the impact of various service and housing interventions in ending homelessness for families”. This study is intended to respond to this mandate.²

A.1.2 Homeless Assistance System Background

This section provides background information on current homeless assistance program models. This background was used to develop the interventions to be tested in the evaluation. The most widely adopted typology of homeless programs is defined in terms of the residential components of the Continuum of Care (CoC): emergency shelter, transitional housing, and permanent supportive housing. Among and within each of these three program types there is significant variation in quality, housing structure and location, privacy and independence for participants, tenure, average and expected lengths of stay, services provided, rules, and expected outcomes. (Locke et al., 2007). This evaluation will provide empirical evidence of the relative effectiveness and costs of these basic models. Emergency and transitional housing are time-limited programs, which rely on families moving on to subsidized or unsubsidized permanent housing. Families in permanent supportive housing programs may also move into mainstream permanent housing, though there is not a specified time frame in which that is expected to occur, and movement is generally expected to be based on family needs. Permanent supportive housing programs offer permanent housing subsidies coupled with services and are available only to families in which at least one parent has a qualifying disability.

Emergency shelters typically serve as the first response to homelessness. This makes shelters a good place to draw a research sample when studying impacts of longer-term transitional housing assistance. Frequently shelters are 24-hour congregate settings, though each family may be provided an individual room or even an apartment. Services vary from basic shelter services (e.g., meals, showers, clothing, and transportation) to minimal case management and referrals to intensive case management augmented by specialized services, such as employment and/or drug or medical treatment. There are approximately 29,949 emergency shelter units (corresponding to 98,287 beds) for homeless families throughout the country (Annual Homeless Assessment Report (AHAR, 2008). Many families have short lengths of stay, leading to high turnover within emergency shelter programs.

Transitional housing offers homeless families housing or rent assistance with supportive services for longer periods, generally six to 24 months. Often families are referred to transitional housing from emergency shelter if shelter workers determine they need more intensive assistance and meet eligibility criteria. Transitional programs follow several models: some are offered in facility-based settings with shared or private rooms or apartments, others are independent units in clustered or scattered site locations where the program maintains the lease and program participants must leave upon completion of the program, while still others are in scattered site community locations where families rent their own apartments with temporary financial assistance from the program. There are approximately 34,621 transitional housing units for homeless families (AHAR, 2008), though there is no national data on the composition of these units across the different models. Stays in transitional housing are longer than those in emergency shelter; AHAR 2008 reports that the median length of stay is 151 nights, as compared with 30 nights in emergency shelter.

As with emergency shelters, services provided through transitional housing vary tremendously from one program to another, though the nature of services is typically more intensive than in shelters. Transitional program services may include childcare, case management and referrals, benefit acquisition and retention, family reunification, education and employment training, mental health and substance abuse treatment, and children’s services. Most transitional housing programs aim to place participants in permanent housing at program completion and may help to broker access to mainstream subsidized housing, but the homelessness system itself rarely funds housing subsidies beyond the temporary rent assistance provided as part of a transitional housing program. Burt (2006) offers a thorough description of the range of transitional housing models.

Permanent supportive housing programs often look relatively similar to the more independent forms of transitional housing, except that there are no time-limits associated with the housing or services. Permanent supportive housing programs funded by HUD require participants to have severe and persistent chronic disabilities to be eligible; thus, under current HUD grant guidelines, a program cannot apply for funds to support permanent rent subsidies and services for non-disabled families. Housing models vary from scattered site apartment units or single-family homes to small-scale group homes to multi-unit developments, such as those funded through the Low-Income Housing Tax Credit program. There are approximately 25,141 permanent supportive housing units for homeless families (AHAR, 2008), almost as many as the number of emergency shelter units and almost as many as the number of transitional units. Continuums often use broad definitions of permanent supportive housing, and equally broad definitions of severe and persistent disabilities. Nonetheless, the figures reported in the Second AHAR represent a very large number of subsidized, supported housing units that target families with severe and persistent disabilities, as defined by HUD.

Other Supportive Services. Many CoCs also have standalone supportive service programs that assist families who are homeless. Service programs may work hand in hand with a residential program by providing employment services or mental health treatment on-site, or they may be completely independent, such as a community-based case management program that provides services to families once they leave a shelter. Supportive service programs may be delivered by a homeless provider, that is, an agency that is primarily focused on assisting people who are homeless, or the programs may be managed by a nonprofit organization that may target a much broader population than just persons who are homeless. The number and type of supportive service programs varies significantly from community to community.

The current study has defined four distinct interventions for assisting homeless families that will be tested using an experimental research design. Families entering emergency shelter who remain for at least seven days will be randomly assigned to one of the designated interventions. The study design relies upon random assignment to existing programs that meet the definitions of the experimental interventions. It is important to recognize that, although practitioners and researchers use shorthand terms such as "transitional housing” or “supportive housing,” these labels do not necessarily reflect uniform approaches. In reality, as Rog and Randolph (2002) note, even when programs of a particular "type" are specifically chosen for study, their characteristics can overlap considerably with other programs that nominally use a "different" model. To circumvent this ambiguity in models, the research team will select programs for study based, not on the nominal models they use or claim to use, but rather on direct assessments of the characteristics of the programs ascertained as part of site selection.

A.2 Information Users

A.2.1 How will the Information Collected be Used?

The information collected for this study will be used by policy makers and local homeless assistance program operators to provide evidence about which types of assistance (combinations of housing and services) are most effective for improving the well being of homeless families. A cost-effectiveness analysis conducted for the study also will assess the relative costs of each intervention in relation to their impacts to provide policymakers with information about how the impacts of the interventions compare to their costs. The data collected at baseline from study participants will be used to describe the population of families seeking assistance from emergency shelter and will be used to verify that random assignment is successful, resulting in well-matched groups without significant differences in demographic characteristics. The information will also be used to define subgroups for analysis and for improving the precision of impact estimates with covariates constructed from baseline variables. Contact information for each sample member gathered at baseline and updated at each tracking interview will be used to maintain contact with each sample member to facilitate the follow up survey at a later date. The follow-up survey, to be conducted under a separate task order contract approximately 18 months after random assignment, will be used to measure outcomes for participating families in several domains: housing stability; self-sufficiency; adult well-being; child well-being; and family preservation. As described in Section A.2.3, the final specifications for the housing and services interventions are still being developed, as are full specifications for outcomes to be measured in the impact analysis. Therefore, the follow-up survey is not included in this request for OMB review, but will be submitted as a new collection at a later date. The modification will also include a detailed protocol for collecting program cost data from the programs that provided the tested interventions in the study sites.

A.2.2 Purpose of the Data Collection

This request for clearance covers the instruments for the baseline and tracking interviews and a key informant interview guide to collect information about housing and services offered by the programs included in the evaluation. In a subsequent phase of the evaluation, a follow-up survey will also be conducted with study participants to measure outcomes of the interventions on five key domains: housing stability; self-sufficiency; adult well-being; child well-being; and family preservation.

The survey instruments submitted here for OMB review consist of a baseline survey to be administered to all families who agree to participate in the study prior to random assignment and a tracking survey to be administered to the same families approximately every six months after random assignment, as part of the participant tracking process. We are also requesting approval for a key informant interview guide that the research team will use to document information about each of the programs included in the sites under each study intervention. This program data will be collected during site visits to each program from individual interviews with program staff and program directors.

This evaluation will offer new evidence concerning the effects of various housing and services interventions for homeless families that will allow policymakers to make informed choices about optimal investment in homelessness assistance. The experimental design will generate data to draw rigorous inferences about the effects of the housing subsidies and services for families, independent of all other factors affecting the lives of study participants. Random assignment serves to ensure that the different intervention groups are well-matched to one another on both observed and unobserved characteristics at the time of their entry into the study. It thus establishes the strongest possible foundation for understanding which of the interventions tested can lead to improved housing stability, self-sufficiency, adult well-being, child well-being, and family preservation.

A.2.3 Who Will Use the Information

The primary beneficiary of the planned survey data collection will be HUD, which will use the information from the baseline and follow-up surveys combined to assess the effects of the various types of assistance packages for homeless families. These data will begin to answer HUD’s questions about impacts of housing assistance and services in all study domains: housing stability; self-sufficiency; adult well-being; child well-being; and family preservation. HUD will also be the primary beneficiary of the key informant interviews, which will provide detailed descriptive information about the programs studied (this information will also be used in estimating and interpreting program impacts). The information obtained through these interviews will help document the characteristics of the services provided to families.

Secondary beneficiaries of this data collection will be those in the public policy and social science research community who are interested in developing policy initiatives to address homelessness among families. Local service providers and decision makers will also use the data to understand how their programs work and to target resources in effective ways. Local program providers will be able to use the study findings on the impacts and cost-effectiveness of the alternative approaches to make decisions about how to focus local resources in the most effective ways. Ultimately, these data will benefit researchers, policy analysts, and policy makers in a wide range of program areas. This project offers the first opportunity to obtain reliable measures of the effects of various housing and services interventions for homeless families. The long-term indirect benefits of this research are therefore likely to be substantial.

A.2.4 Item-by-Item Justification

In developing the surveys, we attempted to balance the need to capture all of the required data against placing undue burden on the respondents, excluding items that—while potentially interesting—are not critical to the measurement of baseline characteristics and outcomes needed to analyze the impacts of the four housing and services interventions. Another goal was to keep the time needed for survey administration to a reasonable duration, thereby limiting respondent burden.

The baseline survey will be the primary source for demographic characteristics, contact information, and baseline status of families included in the research study. The baseline survey will provide data necessary to define the covariates used in the later impact analyses to improve the precision of the impact estimates and to define subgroups of the study sample for analysis of subgroup effects. The baseline interview also provides the source of measures needed to develop the Family Need index, an index of family challenge that will be used in the impact analysis to estimate the impacts of the interventions for families with different characteristics (this index is discussed in more detail below). A final purpose of the baseline interview is to collect contact information for the family head and up to three relatives or friends who are likely to know how to reach the respondent in the future. This contact information is necessary to enhance researchers’ ability to locate respondents for the follow-up survey to measure intervention outcomes. The baseline survey will capture data in the following topic areas:

Housing status prior to shelter entry;
Homelessness history;
Barriers to housing;
Family composition (a roster of adults and children with the family head at baseline and spouse/partner and children not with the family head at baseline, and characteristics of these family members);
Employment and earnings for the family head;
Income sources and total family income;
Health (behavioral health and physical health of the family head);
Substance use;
Foster care history for the family head;
Exposure to domestic violence;
Contact information; and

Demographic characteristics of the family head.

The study design also includes a continuous overall challenge index, to be developed a priori, using baseline characteristics of families in the study. This index will allow an assessment not only of what works best on average over the entire study population, but also of whether some interventions are more effective for families with particular characteristics. The Family Need Index will assess medical and psychosocial challenges and will include such factors as medical conditions, behavioral health problems, domestic violence, placement of children in foster care, and disability, collected on the baseline survey. Culhane, Metraux, Park, Schretzman, and Valente (2007) found that psychosocial risk factors were associated with repeat and episodic use of family shelters but not long shelter stays, in administrative records for four jurisdictions. Families with this pattern represented 2 to 8 percent of all homeless families, across the jurisdictions, but are likely to make up a large proportion of the current sample, because we exclude families who leave shelter on their own in the first week. In addition to predicting episodic use of shelters, this type of challenge index may be particularly important for outcomes of self-sufficiency, family preservation, and adult and child well-being.

The Tracking survey will be administered twice for each family—6 months and 12 months after random assignment, anticipating an 18-month follow-up survey. The purpose of the tracking interview is to verify and update contact information for the respondent and the contact individuals and to update information on family composition, housing program participation and experiences with homelessness. Information on family composition will be used to measure changes in family composition over the follow-up period. Housing information will supplement information to be collected on the follow-up survey about exposure to the interventions and residential stability. The baseline interview and tracking interviews will be translated into Spanish, and other languages as needed.

Program -level data from the emergency shelters in the study sites and the housing and service providers that serve study participants will play an important role in the evaluation. These data will be used to monitor the extent to which programs represent the intended treatment conditions and to understand the nature, quality, and costs of services provided to participants in the study. Any inferences concerning the effectiveness of different treatment conditions must ultimately derive from the careful monitoring of the extent to which programs actually represent the treatment condition it is assumed they represent.

Program-level information will be collected during site visits to each participating site to interview program managers and staff and by reviewing program background documents and materials. We plan to collect program-level data after random assignment is complete, but while study participants are receiving the designated housing and services interventions, approximately 13-16 months after random assignment begins in each site. Information about receipt of services and participation in specific programs by sample members will be collected in the follow-up survey.

The research design includes collecting general information about each program such as program budget, staffing (FTE), type of services offered, and arrangements used to provide services. General information will be documented on a program data collection checklist using information provided by program staff and gathered from background materials. More detailed information will also be documented for the housing assistance and the services provided in the study interventions. For each of the services offered, the research team will document sources of funding, costs of providing the service, description of the service, and measures of quality. The program dimensions to be examined are listed below:

Housing Duration. Is the housing provided time-limited, and if so to what duration? What are average stays in housing?
Housing structure. Is the housing provided primarily congregate, scattered-site, a mixture?
Participant Selectivity. What are the program eligibility requirements? What is the population that the program targets? How strict are the requirements?
Nature of Services. What approach/model does the program use? What does case management include?
Service Intensity. How intense are the services provided? How many consumers do social service staff work with? What is the range of services that are available?
Service Quality. Dimensions of quality include family focus of services, workforce competence, relationship/therapeutic alliance, and structure for providing services.
Staff Attitudes. To what extent do staff see families as having rights to make their own choices or as needing guidance to make wise decisions.
Program restrictiveness. To what extent does failure to cooperate with treatment plans or deviant behavior lead to exclusion from the program?

For each program included in the interventions in each site, we will also collect cost and funding data, to be used to assess the cost-effectiveness of the study interventions. The ultimate goal of the evaluation is to estimate the impact of each of the housing and services interventions compared to Usual Care and to the other more intensive interventions. We can then use that information in conjunction with information on intervention costs (cost per family treated) to examine the cost effectiveness (benefits per dollar spent). Thus at the end of the study, we will be able to present information on impacts and their costs to decision makers in a cost-effectiveness framework. Those decision makers can then weigh the costs and impacts of various policy options.

During initial site visits, the research team will document the sources of cost data for each program and gather basic information about the manner in which financial records are maintained and where key cost items are documented. In later site visits, under a subsequent evaluation task order, the research team will review financial records and meet with program staff to document program budget information on the following: housing subsidy amounts (including operating and capital costs of project-based programs, where relevant); supportive service provision (if relevant to the intervention); and program administration. We will collect information on sources of funding used to support the intervention to validate that the costs are fully documented. To derive appropriate daily unit cost estimates, we will also collect general program information, such as program unit capacity, typical occupancy/enrollment rates, service types, and definition of a service unit. A detailed cost data collection protocol to be used to collect this cost data will be submitted with the modified OMB Supporting Statement when the follow-up survey instrument is submitted for OMB review.

The Follow-up survey (not included in this request for review) will be used to measure outcomes of the housing and services interventions for homeless families. The survey will collect information on homelessness and housing stability between random assignment and the interview date; employment outcomes, including employment rates, and wages of family heads who work during the same period; family preservation, adult well-being, and child well-being. These topics will be included in the follow-up survey:

Housing Stability

Homelessness during follow-up period
Residential moves during the follow-up period
Housing satisfaction, affordability, quality
Housing hardship (difficulty paying for rent/utilities)

Self Sufficiency

Employment during the follow-up period
Earnings during the follow-up period
Food security
Education and training

Family Preservation

Composition of the family
Stability of family composition
Child separations
Foster care placements

Child Well-being measured from Parent reports about focal child³

School attendance, performance
School mobility
Health status
Behavioral strengths and challenges

Services Received during Follow up Period

Services needed and received
Relationship to service provider
Choice in housing and services

Adult Well Being measured for Custodial parent

Physical health
Substance use
Behavioral health symptoms
Depression
Trauma symptoms
Exposure to violence

The study’s informed consent form is presented in Appendix A and the baseline survey instrument is in Appendix B. The tracking instrument is presented in Appendix C, and the key informant interview guide for program-level data is presented in Appendix D. The item-by-item justifications for the baseline and tracking survey instruments are provided in Appendices E and F. Appendix G presents the Federal Register Notice for the evaluation.

A.3 Improved Information Technology

Improved information technology will be used in this evaluation in three distinct ways:

to maintain all demonstration data in a single location;
to assist the ongoing sample tracking and locating efforts; and

to facilitate collection of the survey data in standardized and accurate ways that also accommodate the confidential collection of sensitive data.

The Homeless Families Impact Study will generate a substantial amount of data, including interviews with enrolled families at baseline, random assignment records, tracking interviews, and the follow-up survey. A study data base will be developed to manage the various sources of data. The database will house information from administrative data from HUD and other agencies that may provide data, baseline, tracking, and follow-up survey data, random assignment output, and data about each program included in each sites’ study interventions.

A.3.1 Information Technology and Sample Tracking

Maintaining contact with participating families over the course of the study is essential to ensure a high response rate to the follow-up survey and to collect interim information on housing stability, family composition, and employment.

Tracking families who are homeless is challenging. Passive tracking methods often used in panel studies are not likely to be as effective with this population as in other studies. For example, homeless families are not likely to file a change of address card with the Postal Service as they move from shelter to permanent residence, nor are they likely to maintain a landline phone number. Also, without an initial residence, we will have to rely on the shelter address(es) and the family’s pre-shelter address as the foundation for tracking. The pre-shelter address will provide some information as to neighborhoods where the families may have ties (we will also ask if families have an email address or cell phone). Another challenge is that some families in the study will have experienced violence or trauma either prior to becoming homeless or while on the streets or in a shelter. This may make them wary of providing the types of information needed to maintain a solid tracking database. Establishing the legitimacy of the study and trust in the local site assistant who will conduct the tracking interviews is critical to the success of this study.

Exhibit A-1 summarizes plans for tracking. Passive tracking for this study will include periodic searches of proprietary databases. We will also retain any address or phone number information provided through administrative records such as Homeless Management Information System (HMIS) data and, for the treatment groups, program data collected by the providers of the interventions. All tracking updates collected will be maintained in the study database.

Active tracking will begin with the baseline survey when we determine previous residential addresses (most recent prior to shelter entry) and three relatives or friends that we can consult for future locating efforts. We will conduct in-person locating efforts for study participants every three months after random assignment, alternating between a tracking letter and an in-person locating and a tracking interview.

Exhibit A-1.
Recommended Tracking Sources and Frequency

Tracking Sources and Frequency

Source	Who Is Covered?	Timing
Passive Methods
NCOA^a	All program participants in all of the intervention groups	Semi-annually, beginning 6 months after random assignment
SSA death index	All program participants	Semi-annually, prior to contacting participants
Phone number data	All program participants	Quarterly, beginning 3 months after random assignment
Homeless Management Information System (HMIS) data^b	All program participants	Quarterly, beginning 3 months after random assignment
Active Methods
Baseline survey^c	All program participants	Immediately before random assignment
In person locating efforts (tracking letters with in-person follow-up)	All program participants, on a rotating basis.	Quarterly, beginning 3 months after random assignment.
Tracking Interviews	All program participants, on a rotating basis.	Every 6 months as part of the in-person locating.

a. National Change of Address (NCOA) System of the U.S. Postal Service.

b. HMIS will be used where available.

c. The 18-month follow-up survey will occur under a subsequent task order.

A.3.2 Information Technology and Survey Administration

Information technology assists in the survey data collection in three ways:

Design and management of the sample;
Survey administration; and
Survey data management.

Each of the study surveys will be administered using computer-assisted personal interviewing (CAPI) technology. The CAPI technology ensures that the survey data is of high quality. Data quality is enhanced in three key ways. First, CAPI technology controls the flow of the interview, ensuring that skip patterns are followed properly. It also allows the interviewer to both confirm responses (to minimize data entry errors) and check the logic of some responses by establishing allowable range of values. CAPI technology also allows interviewers to easily record verbatim responses to open-ended questions. Further, it records the current status of each case to facilitate monitoring of response rates and prompt resolution of problems if necessary.

A.4 Duplication of Similar Information

The purpose of the baseline and tracking survey for the Homeless Families Impact Study is to obtain baseline and follow-up information about the status and well-being of families experiencing homelessness in the study sites who enroll in the evaluation. Families who reside in emergency shelters in the designated study sites will be the sample for this evaluation. Information about these respondents' homelessness history, pre-shelter housing, barriers to locating housing, employment status, family composition, income and income sources, physical and mental health, substance use, educational attainment, foster care history, and other characteristics is not available through any other source. These data are essential for establishing baseline characteristics for the study sample and the basis for measuring impacts of the interventions to which each family will be randomly assigned.

Duplication will also be avoided in this study by use of the centrally maintained database, which will link all data collected in the baseline survey and during the subsequent active and passive tracking efforts. This eliminates the need to ask about personal characteristics or background factors for families on the follow -up survey. In addition, information collected on the first tracking interview will be stored in the study database and referenced during subsequent tracking interviews. In this way, later tracking interviews will involve verification and correction of previously-provided contact information rather than collecting each item anew.

A.5 Small Businesses (Involvement of Small Entities)

Respondents for this data collection include families who volunteer to participate in the evaluation and local homeless assistance programs that operate the programs included in the interventions being studied. Some of the homeless assistance programs will be private, non-profit organizations. Estimates of the reporting burden to these entities, to provide information to the researchers about the features and costs of their programs are provided in Section A.13. Data on program features and costs are needed to document the content of the housing and services interventions. Local programs will not be asked to develop new reporting procedures or data collection methods, but will instead meet with the research team to answer questions about their program operations and costs. If local programs already produce program summaries or reports that contain the information needed for the evaluation, the research team will collect the needed information from these existing reports, thereby reducing the time needed to answer the study questions and the reporting burden.

A.6 Less Frequent Data Collection

The baseline and tracking survey data collection efforts and efforts to collect program-level data are essential to conducting the analysis of the impacts of the housing and services interventions. Less frequent data collection would jeopardize HUD’s ability to conduct the impact analysis and would hinder HUD’s ability to maintain up-to-date contact information for respondents in order to conduct the follow-up survey after 18 months. The program-level data on the nature of the housing and services offered to program participants is critical to understanding the differences in program implementation across sites and to interpreting impact estimates.

By definition homeless families are mobile and more difficult to track over time than families who do not experience homelessness. They have unstable housing patterns and traditional tracking methods are less effective in providing updated location data. It is imperative that we maintain frequent contact with this study population in order to keep them engaged in the study. Less frequent contact with study participants could result in lower response rates to the follow-up survey, which in turn would jeopardize the estimation of intervention impacts.

A.7 Special Circumstances

The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public, General Information Collection Guidelines). There are no circumstances that require deviation from these guidelines.

A.8 Federal Register Notice/ Consultation Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, the Department of Housing and Urban Development (HUD) published a notice in the Federal Register on July 13, 2009. The docket number was FR-5287-N-02 and the document number is FR Doc. E9-16547. The Federal Register Notice appeared on pages 33455-33456. The authority is Section 3506 of the Paperwork Reduction Act of 1995, 44 U.S.C, Chapter 35, as amended. The notice provided a 60-day period for public comments, and comments were due by September 11, 2009. No comments were received. A copy of the notice is shown in Appendix G.

The Homeless Families Impact Study design was developed and is being implemented with the assistance of Abt Associates Inc., the prime contractor. Several subcontractors and consultants have collaborated with the Abt team to develop the study design. Key members of the Abt team include Dr. Stephen Bell, Dr. Jill Khadduri, Mr. Jacob Klerman, Ms. Michelle Wood, Ms. Brooke Spellman, and Ms. Mary Joel Holin. Dr. Marybeth Shinn (Vanderbilt University), Dr. Dennis Culhane (University of Pennsylvania), Dr. Martha Burt (Urban Institute), Dr. Ellen Bassuk (Center for Social Innovation/National Center on Family Homelessness), Dr. Beth Weitzman (New York University), and Dr. Larry Orr also worked with Abt Associates to develop the study design.

HUD has collaborated on the design of the evaluation with the Abt Associates team throughout all phases of the study to date. The purpose of such consultation is to ensure the technical soundness and usefulness of the data collection instruments in carrying out the aims of the evaluation.

A.9 Payments to Respondents

Incentive payments are a powerful tool for maintaining low attrition rates in longitudinal studies. Respondents completing the baseline survey will receive $35 for their time. Program participants will also receive a $10 incentive payment each time they complete a tracking interview. We also recommend paying a $5 incentive to all sample members who return their inter-wave tracking letters to us or call into the toll-free line to update their contacting data. This modest incentive shows participants that we value and appreciate the time they take to respond to requests.

The use of incentive payments for the baseline and tracking surveys in the Homeless Families Impact Study is proposed to help ensure a high response rate. For the tracking surveys, a high response rate is essential for maintaining up to date locating information for study participants to ensure the research team is able to locate sample members and complete the follow up survey. Low response rates to the tracking interviews increase the risk of losing contact with sample members and of failure to complete the follow up survey.

The Impacts of Housing and Services Interventions for Homeless Families panel is small and avoiding attrition is essential to the success of the study. The size of the panel will be 3,000 families. This population size will permit detection of impacts in the likely size range only if panel attrition is kept very low (precision of the impact estimates is discussed in Section B.2. Even with no attrition, only fairly large effects can be detected. Therefore, we believe it is absolutely necessary to take every possible step to minimize panel attrition over the study follow-up period. This minimal attrition rate is the core justification for an incentive system for the baseline and tracking interviews. The need to maintain the panel is further complicated by the housing instability likely in this study population.

We also believe that the study population for this evaluation is likely to respond positively to incentive payments. Previous research has shown that sample members with certain socio-economic characteristics are significantly more likely to become survey respondents when incentive payments are offered. In particular, sample members with low incomes and/or low educational attainment have proven responsive to incentives, as have minority group members. These characteristics are expected to be heavily represented in this study panel (Duffer et al. 1994); Educational Testing Service (1991).

Based upon these considerations and prior research experience, we believe that that the use of incentives will improve substantially the probability of panel retention and the viability and power of this experimental research study. The Homeless Families Impact Study represents the first experimental research project HUD has initiated to test the effects of various interventions to assist homeless families; the Department thus places a high level of importance on ensuring that the study panel remains of sufficient size so that the intended statistical measures can be used to draw firm policy conclusions.

A.10 Confidentiality

The subjects of this information collection and the nature of the information to be collected require strict confidentiality procedures. The information requested under this collection is protected and held confidential in accordance with 42 U.S.C. 1306, 20 CFR 401 and 402, 5 U.S.C.552 (Freedom of Information Act), 5 U.S.C. 552a (Privacy Act of 1974) and OMB Circular No. A-130 (copies are included as Appendix H). As required by 5 U.S.C. 552a (Privacy Act of 1974), HUD will publish a Systems of Record Notice (SORN) in the Federal Register. Detailed plans for informed consent and data security procedures are described below.

A.10.1 Informed Consent

All potential participants should be able to make a genuinely informed decision about study participation. Vigorous outreach with a clear message and strong supporting materials will be used to ensure that those assigned to the interventions tested through the study understand the opportunities available and are likely to take advantage of the intervention’s benefits.

However, those who participate actually face little risk by agreeing to be part of the evaluation. The outreach effort will emphasize this fact. Families who reside in emergency shelter for at least seven days will be invited to enroll in the evaluation. The informed consent of each sample member will be obtained through a signed consent form, the “Participation Agreement,” which describes the evaluation, the process of random assignment, and the information requirements of the evaluation. As shown in Appendix A of this submission, this form also indicates to the applicant that participation is voluntary.

The explanation of the study provided to all families eligible for the evaluation will include a brief description of the interventions available through the study—some of which might or might not be available to a family that does not participate in the study and others that are only available through participation in the study. It will be important to make it clear that some families who volunteer and are randomly assigned will receive a permanent housing subsidy (through the SUB intervention). Families must also be told that, through the lottery system used to assign people to programs in the study, there is a possibility that the family may be assigned to the Usual Care group and will not be selected to receive one of the study interventions. Such families can remain in the emergency shelter and can be referred to or seek assistance on their own from other program providers in the community, including housing assistance from the local housing agency. Families assigned to the Usual Care group will not be given referrals to any of the programs that are part of the study, but can be referred to other providers including transitional housing providers that are not in the study, following whatever is the usual practice of the emergency shelter.

A research site liaison, a local staff member hired and trained by the research team, will conduct intake and random assignment in each study site. The research site liaison will describe the other implications of participating in the study, which relate mostly to data collection. Volunteers will need to complete the baseline interview and must agree to being contacted in the future for tracking and for the follow-up interview. We also want families who agree to participate in the study to grant the researchers permission to access information about them from other administrative records systems, like HUD’s Homeless Management Information System (HMIS), and HUD’s Public and Indian Housing’s Information Center (PIC), and from other providers of subsidized housing programs in order to monitor receipt of housing assistance. Permission to access data from other sources will also be necessary in the event that other types of administrative data are collected for other purposes, such as measuring outcomes under future task orders.

The family head will also be told that some of the special programs being offered have eligibility requirements and that families will be assigned only to services that they are eligible to receive. The research liaison will answer questions the family head may have and then ask if (s)he would like to volunteer for the study. Those who agree will sign the informed consent statement and complete the baseline interview.

A.10.2 Data Confidentiality Protections

The data collected in the surveys for the Homeless Families Impact Study as well as any administrative data collected from HMIS or HUD’s Public Housing Information Center (PIC) data system (this system provides records on the receipt of housing assistance through the public housing or Housing Choice Voucher programs) will be used only for the purposes of evaluating the housing and services interventions tested in the evaluation. Mailings to potential respondents and all in-person introductions (e.g. at the time of the baseline and tracking interviews) will include assurances that participation is voluntary, that all information will be kept confidential, and that the respondents' answers will be reported only in aggregate form.

An assurance of confidentiality is included in the Participation Agreement where participants will provide informed consent (see Appendix A). An assurance of confidentiality also will be made to all respondents as part of the introduction to each of the surveys. Abt employees and telephone and field interviewers sign a pledge of confidentiality as a condition of employment. Separate data files will be maintained for questionnaire responses and identifying information; linking will be possible by a common identification number. For both survey data and corresponding administrative data on sample members, computer security will be maintained by passwords known only to project staff members that require access to these files.

In addition, the Contractor strongly recommends that HUD consider applying for an NIMH Confidentiality Certification for the Homeless Families Impact Study. This certification strengthens the privacy protections otherwise applicable to such research, by virtue of the language in the Public Health Service Act Section 301(d),⁴ which says:

The Secretary may authorize persons engaged in biomedical, behavioral, clinical, or other research...to protect the privacy of individuals who are the subject of such research by withholding from all persons not connected with the conduct of such research the names or other identifying characteristics of such individuals. Persons so authorized to protect the privacy of such individuals may not be compelled in any Federal, State or local civil, criminal, administrative, legislative, or other proceedings to identify such individuals.

The certification should be requested for all data obtained through the evaluation of alternative housing and services interventions for homeless families, including primary data collected solely for purposes of this evaluation, as well as any administrative data collected in support of the evaluation. In addition, all design documents, random assignment protocols, and analysis files must be protected. The study’s data collection plan, this OMB statement, and the proposed survey instruments are also subject to the review and approval of Abt Associates’ and Vanderbilt University’s Institutional Review Boards. The IRB’s approval is required in order to obtain NIMH certification.

A.10.3 Data Storage and Handling of Survey Data

To ensure data security and enhance data quality, the survey data collection will be done using Computer Assisted Personal Interviewing (CAPI) technology. Survey data will be collected using the Confirmit CAPI System. The Confirmit CAPI System has the following security features:

Data on the CAPI console is encrypted with Rijndael algorithm (256 bit key).
CAPI data transfers use Web Services Enhancements (WSE 3.0) for security. The messages sent and received from the console are encrypted. WSE 3.0 provides AES128 + RSA 1.5 as default algorithms for symmetric encryption and key-wrap. The contractor has also implemented Secure Conversation with an X509 certificate (which uses 1024 bit key).

In addition to the standard security features offered through the CAPI software, the contractor has implemented the following enhancements:

Use of PGP whole disk encryption on all CAPI laptops, and
The file transfers are made to servers running SSL.

Once the surveys are completed, data will be transferred from the CAPI system to the study’s database. Transfer to the database will be done in a secure manner, using a FIPS-certified encryption algorithm.

A.11 Sensitive Questions

The baseline survey includes questions about history of homelessness, household income and other financial circumstances but not about spending on personal items that would be considered private or sensitive. The interviews also include questions about physical and emotional health, substance use, and history of felony conviction, items that can be considered sensitive. These items are necessary to evaluate the effects of the housing and services interventions being tested and will be treated as confidential information to be used only for the purposes of the study, as will all information collected in the interviews. Respondents will be reminded during the interviews that their responses will be kept confidential, to encourage their candid responses. Respondents will also be reminded that they can refuse to answer any questions that may make them uncomfortable with no repercussion.

The program-level data to be collected about the programs and services provided by the interventions is not individual data and does not contain any sensitive questions.

A.12 Burden Estimates (Total Hours and Wages)

The baseline survey data collection for the Homeless Families Impact Study will be implemented beginning late in calendar year 2009 or early in calendar year 2010. Tracking interviews will begin approximately six months after the start of random assignment and baseline data collection in each site.

Exhibit A-2 shows the estimated respondent burden for the baseline and tracking interviews. It shows the average time, in hours, estimated to be spent by demonstration participants who complete each of the interviews. It also shows the minimum and maximum estimates for the length of the surveys, depending on the individual circumstances of the respondents (primarily the size of their families, whether they are currently working, and the extent of their homelessness histories). The estimates of burden assume a total sample size of 3,000 families (250 families in 12 sites).

Exhibit A-2 also shows the estimate of burden hours and costs on the key informant interviews to gather program-level data. It shows the time, in hours, estimated to be spent by local program staff and program directors to complete the interviews needed to gather program-level data. The total burden of data collection from these staff is 900 hours over a period of six months during 2010-2011.

Exhibit A-2.
Estimated Respondent Burden Hours and Costs

Form	Respondent Sample	Number of Respondents	Average Time to Complete (Minimum, Maximum) In Minutes	Frequency	Total Burden (hours)
Baseline Survey	All enrolled families (N=3,000)	3,000	40 (35, 50)	1	2,000
Tracking Interview	All enrolled families (N=3,000)	3,000	10 (8, 15)	2	1,000
Tracking Letters	All enrolled families (N=3,000)	3,000	5 (3, 10)	3	750
Follow-up Survey*	All enrolled families (N=3,000)		TBD	TBD	TBD
Key Informant Interviews	Staff from programs providing services in the studied interventions	300 (up to 25 respondents in each site)	60	3 responses per respondent to collect all needed program information	900
TOTAL Burden Hours					4,650

*Burden estimates for the follow-up survey will be submitted in a modification to this Supporting Statement.

Using the average times, the total burden of the Impacts of Housing and Service Interventions for Homeless Families data collection from survey respondents is 4,650 hours, not counting the follow-up survey, over a period of 24 months during 2009-2011. The total burden is reflected as burden hours, and no separate cost burden has been calculated.

A.13 Capital Costs (Maintenance of Capital Costs)

This data collection effort involves no recordkeeping or reporting costs for respondents other than the time burden to respond to questions on the data collection instruments as described in item A.12 above. There is no known cost burden to the respondents.

A.14 Costs to the Federal Government

The estimated cost to the federal government of the planned baseline data collection activities for the Impact of Housing and Services for Homeless Families evaluation is $810,432. The estimated costs of the planned tracking effort are $834,876, and the estimated cost for key informant interviews is $364,352. These are subtotals of the total cost of the first phase of the evaluation, $4,499,298. The first phase evaluation costs include costs associated with research design, site recruitment and engagement, and analysis of interim data and preparation of an interim study report. The first phase does not include costs associated with follow-up survey data collection, analysis of intervention impacts and costs, and preparation of final impact report.

Exhibit A-3 shows the costs to the federal government of the planned baseline survey data collection, participant tracking, and program-level data collection activities. HUD’s current evaluation contractor, Abt Associates, prepared these estimates. These costs are entirely federal costs.

Exhibit A-3.
Estimated Costs to the Federal Government

Line Item	Cost to the Federal Government	Total Cost
Participant Surveys
Baseline Survey	$810,432	$810,432
Tracking Interviews	$834,876	$834,876
Follow-up Survey	TBD—Not included in this request	TBD—Not included in this request
Key Informant Interviews to document program-level data	$364,352	$364,352
Key Informant Interviews to collect program cost data	TBD—Not included in this request	TBD—Not included in this request
Total	$2,009,660	$2,009,660

A.15 Program or Burden Changes

This request for clearance does not involve a change in burden due to any program changes or adjustments. It concerns a new data collection not previously submitted to OMB for review. This is a new information collection that will increase the public reporting burden.

A.16 Publication and Tabulation Dates

The data collected for the Homeless Families Impact Study will be analyzed, tabulated, and reported to HUD by the evaluation contractor, Abt Associates Inc, and Abt’s team of subcontractors and consultants.

A.16.1 Time Schedule for Analysis and Reporting

Collection of baseline survey data from study participants is expected to begin with the baseline survey in late 2009 or early 2010 and will end approximately 12 months later after all sample members are enrolled in the study. The baseline survey data set will be cleaned and appended to the study database on a rolling basis. The analysis of these baseline data will be carried out in the following months, with an interim report on the evaluation completed in September 2011. The schedule breaks down as follows:

Baseline Data Collection: 12 months beginning late 2009 or early 2010

Participant Tracking: 18 months for each participant beginning with the baseline survey

Program-level Data Collection: January – May 2011

Baseline Data Analysis: April 2011 through July 2011

Interim Report: September 2011

The data collected from the sites about the programs that implement the study interventions, along with information from participants collected through the baseline survey and tracking, will be used to develop the Interim Report at the end of the study period. The report will include the following components: 1) a description of the study sites and interventions being studied; 2) descriptive information about the interventions and the services and housing provided to families; 3) results of participant enrollment including final sample sizes; 4) description of the baseline characteristics of families; 5) a summary of the study procedures at each site; and 6) plans for follow-up surveys with the families. The Interim Report will not include an analysis of program impacts, which will be based on longer term follow-up data collected in future task orders. However, by the time the Interim Report is developed, we will be able to assess limited information on preliminary outcomes such as residential stability and changes in household composition over time by intervention group based on data collected as part of the participant tracking effort.

A.16.2 Analytic Techniques, Tabulations, and Reporting

The ultimate goal of the study is to estimate the impact of each of the housing and services interventions compared to Usual Care and to the other more intensive interventions. This initial task order includes only randomization, baseline data collection, and participant tracking, but no collection of data on outcomes or estimation of impacts. Collection of data on outcomes, the analysis of that data, and cost-effectiveness analysis will be conducted under a separate contract task order. The reporting to be done using the baseline data on study participants and housing and services program data will consist of descriptive analyses of the characteristics of the study sample and the intervention programs. Exhibit A-4 displays a template for how participant baseline data can be presented in the interim report.

Exhibit A-4.
Baseline Characteristics of Families in the Impacts of Housing and Services Interventions for Homeless Families Evaluation

Characteristic	All Sample Members Combined (N=3,000)
Study Site
Demographic Characteristics of the family head (custodial parent)
Gender
Marital Status
Age at Random Assignment
Race/ethnicity of family head
Educational Attainment
Veteran Status
Pre Shelter Housing
Type of housing
Barriers to Housing
Types of barriers reported
Homelessness History
Age at first incidence of homelessness
Number of times homelessness in lifetime and as an adult
Total time spent homeless
Employment Status
Employment status at baseline
Average hourly wage for those working
Reason not working if not employed at baseline
Family Composition
Number of family members in shelter
Number of adults and children separated from family at baseline
Age of youngest child
Employment status of adults in family
Disability status of family members
Previous felony conviction for adults
Adult Health
Incidence of health conditions
General health status
Adult Mental Health
PTSD symptoms
Depression symptoms
Adult Substance Use
Alcohol use
Drug use
Other
History of foster care (family head)
History of domestic violence victimization (family head)

A.17 Expiration Date

All data collection instruments created for the Impact of Housing and Services Interventions for Homeless Families evaluation will display prominently the expiration date for OMB approval.

A.18 Certification Statement

This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

Part B: Collection of Information Employing Statistical Methods

B.1 Identification of Appropriate Respondents

B.1.1 Sample Recruitment and Random Assignment

The study design will be a randomized experiment. We will recruit at least 2,400 homeless families who have been in emergency shelter for at least 7 days across 12 sites. (As noted earlier, the sample size may be increased to 3,000 families across the 12 sites). We will exclude families who leave shelter in less than 7 days because the more intensive interventions considered in this study are not considered appropriate for families with such transitory needs. We expect shelters to continue to provide all services and referrals they ordinarily provide to help families leave shelter up until the point of random assignment. Families will then be assigned, as close to the 7-day mark as is feasible, to the Subsidy Only (SUB), Community-Based Rapid Re-housing (CBRR), Project-Based Transitional Housing (PBTH), or Usual Care (UC) interventions.

Our design also recognizes that not all families are eligible for all interventions. Consistent with this consideration, families will be screened as to their eligibility for each intervention and for each specific service provider in their site (although we will attempt to recruit sites where all families are eligible for most or all interventions), and will be randomly assigned only to interventions for which they are eligible. As long as one provider within each experimental intervention at a given site will accept a family with a particular profile, that family will be eligible for that intervention. Exhibit B-1 shows the random assignment model we plan to use to allocate families to interventions, assuming all four interventions are included in the study design.

As shown at the top of the exhibit, the population of interest for this study is all families who have been in an emergency shelter for at least 7 days and who have at least one child 15 or younger. This restriction is included because child outcomes are important to the study, and we will not have a large enough sample to consider outcomes for youth who become young adults in the course of the follow-up period. Hence the age restriction to 15 and younger.

The study is not designed to capture the experiences of families who seek assistance directly from transitional housing programs without first entering emergency shelters. The design relies on emergency shelters as the point of intake for families in the study.

Exhibit B-1.
Random Assignment Plan

In each site, this population will be identified and randomly assigned in one of two ways. In sites where all such families qualify for assistance from at least one provider of each intervention, families will be randomly assigned to all four interventions, as shown in the left-hand stream of the diagram. In sites where some families are ineligible for all programs that make up a particular intervention, we will randomly assign those families only to the interventions for which they are eligible. The right-hand stream in Exhibit B-1 shows the random assignment design for families who are eligible to receive subsidies, with or without intensive services (interventions SUB and CBRR), but are not eligible to receive transitional housing (intervention PBTH) because no transitional housing provider in the site will accept them. This diagram and the resulting analysis can be generalized to the situation where some families are not eligible for other interventions, but for simplicity we illustrate the case where restrictions apply only to Transitional Housing. We assume that all families will be eligible for the emergency shelter (intervention UC). Note that both streams could be operative in the same site; i.e., families who are eligible for all interventions would be assigned as in the left-hand stream, while those who are not eligible for transitional housing would be assigned as in the right-hand stream. Should we choose a site with no transitional housing programs, all families in that site would be randomly assigned to three interventions, as in the right-hand stream in Exhibit B-1.

As we describe below, this design assures that comparisons of interventions will involve well-matched groups in each intervention, even when some families are ineligible for a particular intervention program. The design thus assures that any observed differences in outcomes are caused by the differential treatment families receive, and not by any pre-existing differences among the families.

Although assignment to interventions will be at random, within interventions families need not be assigned at random to service providers that represent the intervention. Assignment can be made on the basis of family characteristics, as is currently done. Thus, for example, if one or more of the transitional housing programs in a site specialize in families with a particular profile (only families with domestic violence issues, or only families where the mother has been clean and sober for some period), then among families randomly assigned to Transitional Housing, only those that fit that program will be assigned to that service provider. If a site has vouchers available only to veterans, then among families randomly assigned to the SUB intervention, only families that include a veteran will be assigned to veteran housing. This preserves and studies programs as they currently operate.

With the addition of subsequent Task Orders in which follow-up data are collected, the design will provide rigorous experimental answers with sufficient statistical power for the following broad questions:

What is the relative effectiveness of homeless interventions in ensuring housing stability of homeless families?
Are the same interventions that are effective for short-term housing stability of homeless families effective for longer-term housing stability as well?
What is the relative effectiveness of homeless interventions in ensuring the well-being of homeless parents and self-sufficiency of homeless families?
Do some interventions promote family preservation and benefit children’s well-being, in particular, more than other interventions?

The overarching research question for this study is the extent to which housing and/or intensive services influence housing stability, family well-being, and other non-housing outcomes. The study design we propose will provide empirical evidence on each of these effects, separately and in tandem. Many families leave shelter on their own, but little is known about what happens to them in terms of either residential stability or other outcomes. By including a Usual Care group that does not receive a dedicated subsidy or targeted intensive services, we will understand not only the impacts of interventions relative to no special services, but also whether interventions that explicitly address homelessness produce superior results to temporary shelter and the mainstream poverty assistance system but no additional specialized assistance. In this section, we describe the specific impact estimates that we will generate to answer these questions.

B.1.2 Universe of Households and Survey Samples

The study sample will comprise families, defined as at least one adult and one child, who experience homelessness, receive assistance at an emergency shelter, and remain in the shelter for at least seven days. Exhibit B-2 summarizes the definition and sample sizes for all of the random assignment groups. We believe that the sample sizes assumed here are realistic, both in terms of the likely number of families who will enroll in shelter in the study sites and in terms of the local communities’ ability to implement the study interventions.

Exhibit B-2.
Definition and Size of Randomly Assigned Groups in the
Impact of Housing and Services Interventions for Homeless Families Evaluation

Group	Intervention Definition	# Assigned per Site	Total # Assigned
SUB	Subsidy only; defined as deep, permanent housing subsidy that may include housing related services but no supportive services.	50-63	600-750
CBRR	Community-Based Rapid Re-housing: Time-limited housing subsidy that may also include housing-related services and limited supportive services	50-63	600-750
PBTH	Project-Based Transitional Housing: Time-limited housing subsidy coupled with supportive services	50-63	600-750
UC	Usual Care: Other assistance available in the community	50-63	600-750
	Total, all Intervention Groups	2,400 - 3,000	2,400 - 3,000

B.2 Administration of the Survey

B.2.1 Sample Design

Up to 3,000 eligible families that agree to participate in the study will complete a baseline survey prior to random assignment. All randomly assigned families that completed a baseline interview will be included in the participant tracking. For the follow-up survey (not included in this request for OMB clearance), interviews will be attempted with all members of the research sample (3,000 families). Therefore, no sampling is required for the tracking or follow-up surveys.

Data to analyze the impacts of the housing and services interventions will come primarily from the follow up survey, which will be submitted for OMB review under a separate supporting statement. Key topics to be included in the follow-up survey are related to housing stability (incidence of homelessness in the follow-up period, use of shelter, type of housing situations); self-sufficiency (employment and earnings over the follow-up period, income and receipt of public assistance); family preservation (changes in family composition over the follow-up period, placement of children into foster care); adult well-being (physical and behavioral health); and child well-being (academic performance; school attendance, health, and behavioral health) for a focal child, defined as one child, selected at random from among children age 15 years old or younger who resided with the family head at baseline.

B.2.2 Estimation Procedures

As described in Section A.16 above, the baseline data to be collected for the evaluation will be used to describe the study sample, to define subgroups for analysis, and to provide baseline measures of outcomes to use as covariates in impact estimates to improve precision. With properly designed and implemented random assignment, treatment-control comparisons of raw mean outcomes provide unbiased estimates of impact. Use of regression analysis to control for baseline characteristics that affect the outcome improves the precision of the estimates while preserving their unbiased character. The estimates of precision presented in the next section assume such regression adjustments, with precision gains based on those obtained in similar studies, including the studies of the Effects of Housing Vouchers on Welfare Families and the Moving to Opportunity demonstration conducted previously by HUD.

B.2.3 Degree of Accuracy Required

The baseline data collected here will not be used to estimate impacts. However, the research team has estimated the minimum detectable effects for this evaluation that will be available through the impact analysis. The analysis of statistical power is presented here.

Power Calculations for Binary Outcomes

In this section, we consider statistical power to estimate impacts of interest. Specifically, we report minimum detectable effects (MDEs). MDEs are the smallest true effects of an intervention that researchers can be confident of detecting as statistically significant when analyzing samples of a given size. The power analyses are computed assuming exactly equal numbers of families are assigned to the four treatments. In fact, the ex ante exclusion of some families from the Transitional Housing intervention due to ineligibility for the PBTH programs included in the interventions in a site implies slightly lower precision for comparisons involving PBTH and slightly higher precision for comparisons not involving PBTH. Lacking an estimate of how common the exclusions will be, it did not seem worthwhile to provide more detailed power calculations.

Our analysis indicates that the proposed design will have sufficient statistical power to detect impacts of the magnitude we might expect to occur for two of the central outcomes of the study—housing stability and child separation from the family. As discussed below, we will be able to detect effects on these outcomes as small as 8 percentage points for the sample overall and as small as 13.6 percentage points for a subgroup comprising one-third of the study sample.

Exhibit B-3 shows the MDEs for the study sample as a whole using the full sample of 2,250 which is 75 percent response of the full sample of 3,000 respondents) and for subgroups of the stated size. The MDEs presented are the minimum detectable differences in outcomes (in percentage points) between two randomly assigned groups with 80 percent power when we perform a two-sided⁵ statistical test at 10 percent level of significance, assuming a regression R² of 0.10⁶ and no finite population correction.⁷ The differences are shown for various average outcome levels for the reference group, the “usual care” sample.

The last column of Exhibit B-3 shows that for a mean Usual Care group outcome of 0.5, the MDE for all sites pooled is 7.9 percentage points. This means that if the true effect of SUB, CBRR or PBTH compared to UC is to change the prevalence rate of an outcome measure—such as return to shelter housing, or percent of families whose head is a leaseholder at 18-month follow-up—from 50 percent to under 42 percent (for return to shelter) or above 58 percent (for leaseholding), we would have an 80 percent likelihood of obtaining an impact estimate that is statistically significant. If the true effect is less than 8 percentage points, there is a lower likelihood that differences between assignment groups will be statistically significant, though many might still be detected.

Our hypothesis is that the interventions to be tested in relation to the Usual Care intervention—all involving housing assistance or subsidy of some sort—will have fairly large effects on housing stability. Drawing on the longitudinal HMIS analysis of shelter utilization (AHAR, 2008; Culhane et al., 2007), we estimate that of families who remain in shelter for at least seven days without any special assistance, approximately 50-60 percent find housing that keeps them from returning within a multi-year follow-up period. There is substantial potential for the proposed interventions to expand this percentage, by using subsidies to eliminate the risk of shelter return for many families in the other 40-50 percent of the population. Housing subsidies remain available to families many months after first receipt, during which time they should provide a sufficiently stable and improved housing option compared to shelters that, for most families, precludes the need for returns to shelter. Research in St. Louis, Philadelphia, and New York City (Stretch & Krueger, 1993; Culhane 1992; Shinn et al., 1998) tends to support this projection. For example, in St. Louis just 6 percent of families who left shelter with a housing voucher returned, compared to 33 percent of those without subsidized housing.⁸ Housing stability differed by more than 60 percent between those who received a subsidy (80 percent in stable housing at five years) and those who did not (18 percent stable at five years) in the New York study. Thus, we conclude that an MDE of 8 percentage points assures confident detection of the type of impact on housing stability we would expect from the tested interventions.

Exhibit B-3.
Minimum Detectable Effects for Prevalence Estimates for the
Entire Sample and by Subgroups

Sample	Expected Total Number of Families Randomly Assigned	Number of Completed Follow-up Survey Interviews		MDE if Mean Outcome for the Usual Care Group is:
Sample	Expected Total Number of Families Randomly Assigned	Intervention Groups A, B, C (each)	“Usual Care” Group	0.1 (or 0.9)	0.3 (or 0.7)	0.5
All Sites	2,400	450	450	4.7 pp	7.2 pp	7.9 pp
67% Subgroup	1,600	300	300	5.8 pp	8.8 pp	9.6 pp
33% Subgroup	800	150	150	8.2 pp	12.5 pp	13.6 pp

A similar conclusion holds for the prevalence of child separation from the family during the follow-up period. This is likely to be a less common occurrence, making the column of Exhibit B-3 labeled “MDE if Mean Control Group Outcome is: 0.3” likely the most relevant one.⁹ Here, a slightly smaller true impact of 7.2 percentage points can be detected with 80 percent assurance. MDEs for subgroups are of course larger for both outcomes shown in the exhibit. As noted elsewhere, the study will be best equipped to explore how impacts differ by family characteristics using the Family Need Index whose role in producing larger or smaller impacts can be examined without dividing the sample into pieces.¹⁰

Power Calculations for Earnings

Exhibit B-4 shows the MDEs for earnings impacts for the entire sample and for subgroups. These MDE’s are based on the adult earnings outcomes from the Moving To Opportunity (MTO) Demonstration (Orr, et al., 2003), a study of families who were living in distressed (i.e., barely better than emergency shelters) public housing or private assisted housing projects in high poverty neighborhoods at baseline. Across all sites, the proposed design will be able to detect differences between groups (e.g., between the transitional housing and Usual Care random assignment groups) of $1,149 or more in annual earnings. Given that only two of the interventions tested have a partial focus on the labor market—though better, more stable housing may enable steadier employment and resulting greater earnings—the study design is weaker for detecting these effects. On the one hand, it is by no means assured that even an intervention directly focused on employment and training could produce an earnings impact of over $1,100 per year. On the other hand, a true impact substantially smaller than this amount—say, an impact on annual earnings of $600—would have little potential to move families out of poverty and hence may not be important to detect with high confidence.

Exhibit B-4.
Minimum Detectable Effects for Annual Earnings Impacts for the Entire Sample and by Subgroups

Sample	Expected Total Number of Families Randomly Assigned	Number of Completed Follow-up Survey Interviews		MDE (dollars)
Sample	Expected Total Number of Families Randomly Assigned	Treatment Groups 1-3 (each)	“Usual Care” Group	MDE (dollars)
All Sites	2,400	450	450	1,149
67% Subgroup	1,600	300	300	1,407
33% Subgroup	800	150	150	1,990

Notes: (1) The MDE’s are based on calculations which assume that two-sided tests are used at the 10 percent significance level, the desired power is 80 percent, and the regression R2 is identical to the MTO adult annual earnings impact regression. (2) All MDE’s assume a 75% survey response rate, with no finite population correction. (3) The variance of earnings is derived from the standard error of the ITT impact estimate for the experimental group (n=1,729) vs. the treatment group (n=1,310) in the MTO Demonstration: $254.

B.2.4 Procedures with Special Populations

In this study we may encounter interview respondents whose first language is Spanish. We will translate each of the survey instruments into Spanish, for administration in the language most comfortable for the respondent. The participation agreement also will be made available in Spanish.

For respondents who are most comfortable in other languages or in sign language, the Contractor will use translators to carry out the interviews. These might be family members or staff members of community agencies. The contact and advance letters will both provide a TTY number for use by respondents who have hearing impairments. Any calls to request materials in other languages will be noted, so that appointments with those respondents can be scheduled with a translator included.

B.3 Maximizing the Response Rate

For the baseline survey, all families who agree to participate in the evaluation must complete the baseline interview to pass through random assignment and thereby access the housing services that led to their decision to volunteer for the study. We therefore expect a response rate of 100 percent for the baseline survey.

B.3.1 Sample Control During the Data Collection Period

During the data collection period for the participant tracking component of the study, non-response levels and the risk of non-response bias will be minimized in the following ways:

The Contractor will recruit interviewers skilled at working with this population. Interviewers will receive additional training in working with special populations and assistive technologies.
The Contractor will use trained interviewers who are skilled at maintaining rapport with respondents, so that the number of break-offs and the incidence of item nonresponse will be kept low.

Respondents will have a choice of time for the data collection.
Additional field tracking and locating steps will be taken, as needed, when sample members are not found at the phone numbers or addresses previously collected.
The use of the Abt Associates Field Management System will permit interactive sample management and electronic searches of historical tracking and locating data.
For the follow-up survey, the Contractor’s survey director and field supervisors will manage the sample release and monitor response rates in a manner that allows us to work the sample groups for each of the study interventions evenly.

By these methods, the Contractor anticipates being able to achieve a 75 percent response rate for the follow-up survey.

B.4 Test of Procedures

Abt Associates conducted a pretest of the baseline survey instrument in August 2009, with a total of nine respondents. Pretest respondents for the baseline survey were selected from homeless families residing in one of two emergency shelter or transitional housing programs in the Boston area. The pretest allowed the Contractor to test the appropriateness of language level and word usage in the questionnaire and to confirm the estimates of interview length. Experienced interviewers conducted the pretests. The Contractor prepared a pretest report for HUD, describing the results of the pretest. Only minor modifications were recommended in the baseline instrument, to clarify the language, and to provide additional probes and explanations for interviewers to ensure that the purpose of the questions is clear for respondents. On the basis of the pretest, the Contractor did not recommend modifications to the question order or skip patterns. The pretest showed that the items flowed smoothly and logically for respondents. In addition, the pretest confirmed an average interview length of 40 minutes, which is the basis for the burden estimates in this supporting statement.

B.5 Individuals Consulted on Statistical Aspects of the Design

The individuals shown in Exhibit B-5 assisted HUD in the statistical design of the evaluation.

Exhibit B-5.
Individuals Consulted on the Study Design

Name	Telephone Number	Role in Study
Dr. Stephen Bell Abt Associates Inc.	301-634-1721	Co-Principal Investigator
Dr. Marybeth Shinn Vanderbilt University	615-322-8735	Co-Principal Investigator
Dr. Jill Khadduri Abt Associates Inc.	301-634-1745	Project Quality Advisor
Mr. Jacob Klerman Abt Associates Inc.	617-520-2613	Analysis Task Leader
Dr. Martha Burt Consultant to Abt Associates Inc.	202-261-5551	Project Advisor
Dr. Dennis Culhane University of Pennsylvania	215-746-3245	Project Advisor
Dr. Ellen Bassuk, Center for Social Innovation and National Center on Family Homelessness	617-467-6014	Project Advisor
Dr. Beth Weitzman New York University	212-998-7446	Project Advisor
Dr. Larry Orr Consultant to Abt Associates Inc.	301-467-1234	Project Advisor

Inquiries regarding the statistical aspects of the study's planned analysis should be directed to:

Dr. Stephen Bell Co-Principal Investigator Telephone: 301-634-1721

Dr. Marybeth Shinn Co-Principal Investigator Telephone: 615-322-8735

Mr. Jacob Klerman Analysis Task Leader Telephone: 617-520-2613

References

Burt, M.R. (2006). Characteristics of transitional housing for homeless families: Final report. Prepared for the US Department of Housing and Urban Development.

Cowal, K., Shinn, M., Weitzman, B.C., Stojanovic, D., & Labay, L. (2002). Mother-child separations among homeless and housed families receiving public assistance in New York City. American Journal of Community Psychology, 30, 711-730.

Culhane, D. P., Metraux, S., Park, J.M., Schretzman, M. & Valente, J. (2007). Testing a typology of family homelessness based on patterns of public shelter utilization in four U.S. jurisdictions: Implications for policy and program planning. Housing Policy Debate, 18(1), 1-28.

Culhane, D. P. (1992). The quandaries of shelter reform: An appraisal of efforts to “manage” homelessness. Social Service Review, 66, 428–440.

Duffer, Allen P. et al., "Effects of Incentive Payments on Response Rates and Field Costs in a Pretest of a National CAPI Survey" (Research Triangle Institute, May 1994).

Locke, G., Khadduri, J. & O’Hara, A. (2007). Housing models (Draft).

Rog, D.J. & Randolph, F.L. (2002). A multisite evaluation of supported housing: Lessons learned from cross-site collaboration. New Directions for Evaluation, 94, 61-72.

"National Adult Literacy Survey Addendum to Clearance Package, Volume II: Analyses of the NALS Field Test" (Educational Testing Service, September 1991), pp. 2-3.

Orr, L.L., Feins, J., Jacob, R., Beecroft, E., Sanbonmatsu, L., Katz, L., Liebman, J. & Kling, J. (2003). Moving to Opportunity interim impacts evaluation: Final report. Cambridge, MA: Abt Associates Inc. and National Bureau of Economic Research.

Shinn, M., Weitzman, B. C., Stojanovic, D., Knickman, J. R., Jiminez, L., Duchon, L., James, S., & Krantz, D.H. (1998). Predictors of homelessness from shelter request to housing stability among families in New York City. American Journal of Public Health, 88 (10), 1651-1657.

Stretch, J. J. & Kreuger, L. W. (1993). A social-epidemiological five year cohort study of homeless families: A public/private venture policy analysis using applied computer technology. Computers in Human Services, 9(3-4), 209-230.

U.S. Department of Housing and Urban Development (2008). The Third Annual Homeless Assessment Report (AHAR) to Congress.

1 See: www.hud.gov/budgetsummary2010 accessed on May 15, 2009

2 Senate Report 109-109 to accompany HR 3058. July 26, 2005 (page 176). The report is available at http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=109_cong_reports&docid=f:sr109.pdf, accessed on June 11, 2009.

3 A focal child will be randomly selected from among all children present with the respondent at baseline who is 15 years old or younger. The reason for this age restriction is to ensure that by the time of the follow-up survey the focal child is not older than 18.

4 42 U.S. Code Section 241(d).

5 While one-sided tests would decrease MDE’s, we believe one-sided tests are inappropriate because we care about negative impacts; i.e., they are in a substantive sense not equivalent to a finding of no impact. To see this consider comparing Transitional Housing to Subsidy Only. There a negative point estimate implies that one of the interventions is worse than the other. We care about that, above and beyond the idea that the other intervention is not better.

6 Since we will estimate regression-adjusted impact estimates, we assume an amount of explanatory power for the regressions. An R² of 0.10 is conservatively assumed. This is the pseudo-R² for the general health outcome probit regression in the Effects of Housing Vouchers on Welfare Families evaluation. Outcomes with higher regression R²’s will have smaller MDE’s.

7 Applying the finite population correction (FPC) would reduce the MDE’s. However, we believe not applying the FPC more accurately represents our uncertainty as to results holding true in future similar applications of the intervention approaches.

8 Note that this observational pattern is not a direct measure of the impact of subsidized housing on shelter return. Likely the families who exited shelter with subsidies differed from the without-subsidy group on other factors that led to their better outcomes. But even if the difference in unadjusted shelter return rates exaggerates the true impact of a subsidy by an extreme amount—say, 2 or 3 times—the observed 27 percentage point difference would mean an impact of 9 to 13 percentage points.

9 We note that Cowal, et al, (2002), finds 44 percent. In as much as that estimate applies here, we will have slightly lower power.

10 “Challenge score” can be entered into the impact regression equation interacted with indicator variables for the different random assignment groups to see if the magnitude of effect from being assigned to a particular service package changes as the degree of family challenge rises, and the equation then estimated using all the data. Impacts on categorical subgroups will be estimated by splitting the sample and doing separate analyses for each category.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Abt Associates Inc.
File Modified	0000-00-00
File Created	2021-02-03