0320_OMB Supporting Statement-A_2.21.14

Evaluation of SAMHSA homeless PROGRAMs

Supporting Statement

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting approval for an extension from the Office of Management and Budget (OMB) for data collection activities for the Evaluation of SAMHSA Homeless Programs, including the Grants for the Benefit of Homeless Individuals (GBHI) and Services in Supportive Housing (SSH) programs. These activities include administration of the following surveys:

• Client Interview – Baseline (Attachment 1)

• Client Interview – 6-Month Follow-up (Attachment 2)

• Stakeholder Survey (Attachment 3)

This data collection is approved under OMB number 0930-0320, which expires March 31, 2014. SAMHSA’s GBHI/SSH programs are authorized under Section 506 and 520A of the Public Health Service Act. GBHI/SSH programs also address Healthy People 2020 Objectives: Mental Health and Mental Disorders: Treatment Expansion (Focus Areas: MHMD-8, 9, 10 and 12) and Substance Abuse: Screening and Treatment (Focus Area: SA- 8) and support SAMHSA’s Strategic Initiatives of Data, Outcomes and Quality and Recovery Support.

Homelessness affects more than 3.5 million people in the United States (National Law Center on Homelessness & Poverty, 2009) and about 38% of those homeless are alcohol dependent and 26% abuse other drugs (Burt et al., 1999; National Coalition for the Homeless, 2009). In a general homeless population, about 32% of men and 36% of women are estimated to have co-occurring mental and addictive disorders (North, Eyrich, Pollio, & Spitznagel, 2004). Overall, in a national sample, about three-quarters (74%) reported any alcohol, drug, or mental health problem in the year before shelter admission (Burt et al., 1999). The literature is replete with evidence suggesting that homelessness, substance use, and mental illness are closely associated and that the prevalence rates for the latter two problems are high (Hiday, Swartz, Swanson, Borum, & Wagner, 1999; Mallett, Rosenthal, & Keys, 2005; Shelton, Taylor, Bonner, & Bree, 2009; Vangeest & Johnson, 2002). Several populations, including veterans, families, victims of trauma, and criminal justice populations, are at particular risk for homelessness and alcohol and drug abuse (Greenberg & Rosenheck, 2008; HUD, 2009; McNeil, Binder & Robinson, 2005; Moore, Gerdtz, & Manias, 2007; National Law Center on Homelessness and Poverty, 2009; Rukmana, 2008; Veterans Administration, 2009). Eighty-five percent of the chronic homeless—those who have either been continuously homeless for one year or more or have had at least four episodes of homelessness in the past three years—have co-occurring mental and addictive disorders (Joseph & Langrod, 2004). Across two national samples, the National Survey of Homeless Assistance Providers and Clients and the U.S. Department of Housing and Urban Development’s (HUD) 2008 Annual Homeless Assessment Report, between 10% and 23% of respondents were veterans (Burt et al., 1999; U.S. HUD, 2009). The U.S. Department of Veterans Affairs (2009) estimates that about 45% of homeless veterans have mental illness, more than 70% suffer from alcohol or other drug abuse problems, and many are comorbid for these conditions. Persian Gulf and Middle East–era returning veterans are at increased risk for homelessness compared with prior service era veterans (Kline et al., 2009). Substance use, mental illness, and victimization are primary predictors of homelessness in longitudinal studies (Shelton et al., 2009; van den Bree et al., 2009) and criminal justice involvement has a bidirectional relationship with homelessness (Caton, Wilkins, & Anderson, 2007; Greenberg & Rosenheck, 2008; Martell, Rosner, & Harmon, 1995).

Effectiveness of substance abuse treatment (e.g., Modified Therapeutic Communities, Motivational Enhancement Therapy, Service Outreach and Recovery) in producing abstinence and a number of positive outcomes like employment stability, treatment adherence, and reduced unprotected sex has been well established (Ball et al., 2007; Borsari & Carey, 2000; Brown & Miller, 1993; Conrad et al., 1997; Drake, Yovetich, Bebout, Harris, & McHugo, 1997; Kertesz, Crouch, Milby, Cusimano, & Schumacher, 2009; Miller, Benefield, & Tonigan, 1993; Project MATCH Research Group, 1997; Rosenblum, Magura, Kayman, & Fong, 2005; Stephens, Roffman, & Curtin, 2000), as have the effects of integrated treatment for co-occurring disorders (e.g., Integrated Dual Disorders Treatment) on substance abuse, mental health, hospitalization, violence, and homelessness (Drake, McHugo, & Noordsy, 1993; Mueser, Drake, & Miles, 1997).

Housing interventions are most effective when combined with other services (Caton et al., 2007; Nelson, Aubry, & Lafrance, 2007), however, it is unclear which combination of housing models, services, and treatment yields the most robust outcomes with respect to housing stability, substance use, psychiatric symptomology, employment, and other important outcomes. In addition to combining treatment and housing strategies, various common structural characteristics or services, systems and program organization have been found in effective programs. In their review, Cheng and Kelly (2008) describe structural characteristics generally found in effective programs: interagency coalitions; interagency service delivery teams; interagency management information systems and client tracking systems; interagency agreements that formalize collaborative relationships; interagency application for funds; uniform application, eligibility criteria, and intake assessments; and co-location of services.

Gaps in the research include: studies that include non-HUD–funded programs to better describe prevalence of substance abuse, mental illness, and co-occurring problems; evaluation of subgroups of homeless individuals within a single study (with similar definitions, measures, and procedures); information on the needs of subpopulations across the continuum of homelessness; implementation and effectiveness of EBP’s specifically in homeless populations; fidelity of treatment and housing models implemented; cost-effectiveness in complex sites that employ multiservice interventions; and the value that comprehensive initiatives, such as those implemented by GBHI/SSH grantees, add to the overall treatment systems. There is also a dearth of empirical studies that look at performance measurement of homeless programs, benchmarking, and efficiency measures. Finally, there are few multisite studies of the sustainability of programs after cessation of federal funding and factors associated with sustainability.

Treatment providers know how to serve many of these individuals successfully and cost-effectively, but barriers to effective treatment exist; even after entering substance abuse treatment programs, many people who are homeless do not complete them. Recognizing the enormous societal costs of these persons’ failure to get needed treatment services, SAMHSA was funded by Congress to establish GBHI/SSH, competitive, discretionary grant programs initiated in 2001 with the following goals: (1) to link substance use and mental health treatment services with housing programs and other services, (2) to expand and strengthen treatment services for people who are homeless who also have substance use disorders, mental disorders, or co-occurring substance use and mental disorders; and (3) to increase the number of homeless people who are placed in stable housing and who receive treatment services for alcohol, substance use, and co-occurring disorders.

Between 2001 and 2008, 182 GBHI/SSH grants were awarded to provide services to the target population. An additional 127 Grantees were funded between 2009 and 2012. Some Grantees serve priority populations, including criminal justice populations, chronically homeless persons, returning veterans, and chronic public inebriates; others focus on serving families, women, native Alaskans, Native Americans/Indians, other minority populations, or youth. Although all are required to, at a minimum, provide outreach, case management, substance abuse or co-occurring disorders treatment (integrated, sequential, or parallel), and wraparound and recovery services, many augment these services by adopting or adapting additional evidence-based practices (EBPs) from one of the SAMHSA toolkits (e.g., Assertive Community Treatment (ACT), Illness Management and Recovery, Supportive Employment), the Center for Substance Abuse Treatment’s (CSAT) Treatment Improvement Protocols, or the National Registry of Evidence-Based Programs and Practices (NREPP). The models for service delivery vary and include primarily by referral, direct provision of treatment and other services, or a mix of direct service provision and referral to other community-based organizations. Service models are implemented in an array of settings, including on the street through outreach; in drop-in settings, shelters, and hospitals; at medical, substance abuse, or mental health clinics; in residential treatment communities; or in any of these settings or other non-office settings through mobile crisis units or ACT teams.

All clients are assessed by Grantees at intake, 6-months follow-up to intake and at program discharge with the CSAT Government Performance and Results Act (GPRA) Client Outcome Measures for Discretionary Programs (OMB control number 0930-0208) or the Center for Mental Health Services (CMHS) National Outcomes Measures (NOMS) (OMB control number 0930–0285). This data is provided to SAMHSA by the Grantees via the web and stored in the Services Accountability Improvement System (SAIS) or the Transformation Accountability (TRAC) system. The GBHI/SSH 2004-2011 grantee cohorts have served 48,255 individuals (Broner, Trudeau, & Embry, 2012). Per the FY2010 President Obama’s budget, outcomes data available for a subset of clients served by through 91 active GBHI Grantees indicated that individuals demonstrate: 1) 122% increase in employment or engaging in productive activities; 2) 166% increase in persons with a permanent place to live in the community; 3) 52% increase in no past months substance use; and 4) 36% improvement in no/reduced alcohol or illegal drug related health, behavioral or social consequences.

The SAMHSA GBHI/SSH evaluation represents the most comprehensive assessment of GBHI/SSH ever undertaken and will provide evidence on the effects of GBHI/SSH project activities on client outcomes, treatment services, and treatment systems. This information will allow SAMHSA to determine the extent to which GBHI/SSH has met the objectives of implementing a program to provide substance abuse and integrated co-occurring mental and addictive disorders treatment and other wraparound services to meet the needs of homeless individuals and end homelessness among those with substance use and co-occurring problems. To achieve this overarching goal, the evaluation will identify the barriers, challenges, and facilitators of successful GBHI/SSH project implementation. This evaluation will also examine the feasibility, utility, and sustainability of future GBHI/SSH cohorts and make recommendations to SAMHSA of ways to improve future initiatives within the GBHI/SSH portfolio.

The purpose of the evaluation is formative with an intent to identify and measure post-program participation findings across the broad array of outcomes expected to be influenced by the range of services provided by GBHI/SSH Grantees either directly or through referral. These services, which are provided following assessed need, include treatment for substance abuse and mental health disorders (which includes screening, assessment, and active treatment), outreach, case management, and wraparound services (which can include, for example, relapse prevention, crisis care, education or vocational services, transportation, medical care, housing readiness training, benefits application, housing application, peer support services) and aftercare. As much as possible, the intent is for each of these services to be based on models and practices that have an evidence base in peer-reviewed literature. Thus, outcomes directly relevant to GBHI/SSH include those related to substance use, mental health, employment and education, as well as to additional behavioral outcomes and housing. The evaluation does not intend to draw causal inferences with respect to GBHI/SSH program participation and outcomes, but to measure (more explicitly than the current GPRA/NOMS allow) a variety of outcomes directly related to the specific services included in GBHI/SSH programs. These are programmatic outcomes that are used to monitor the provision of services, understand the way services are tailored to clients with different needs and to better understand how the implemented service models match the models described in the efficacy and effectiveness literature.

This evaluation will examine structural, process, outcome, and cost components. The first component, structure, encompasses the resources available in a treatment delivery system; it can apply to individual practitioners, groups of practitioners, and to organizations and agencies. They represent the capacity to deliver quality care, but not the care itself. Process, the second component, represents what is done to and for the client. Process measurement can also focus on individual practitioners, groups, organizations, agencies or systems of care. Measures of structure and process will characterize the grantee organization and its partnerships, the system within which the program is embedded, the grantee’s relationships with stakeholders, the target population, services provided and received, program planning and implementation. These measures will provide important information on the nature of GBHI/SSH programs and will be used in outcome models as variables that are expected to be associated with client and program outcomes.

The outcome evaluation is the third component, which focuses on addressing the utility of GBHI/SSH programs, or, the value in terms of the benefits produced. The outcome evaluation, using a formative, not causal approach, will focus on the program effects of GBHI/SSH programs on client outcomes, accounting for grantee characteristics, treatment system and community contexts.

The fourth and final component of the evaluation is an economic evaluation of GBHI/SSH. The economic evaluation connects significantly to all other aspects of the evaluation by incorporating results from the process, structure, and outcome evaluations. The economic evaluation questions are focused on measuring the cost and cost-effectiveness of GBHI/SSH at the client, grantee, and system levels; on obtaining cost metrics that allow GBHI/SSH and GBHI/SSH components to be compared with other services; and on determining factors that affect the cost and cost-effectiveness of GBHI/SSH.

In summary, we present the evaluation framework and SAMHSA’s intent for the evaluation to address SAMHSA’s questions in terms of this formative evaluation. The purpose of the evaluation is formative with an intent to identify and measure post-program participation findings across the broad array of outcomes expected to be influenced by the range of services provided by GBHI/SSH Grantees either directly or through referral. These are programmatic outcomes that are used to monitor the provision of services, understand the way services are tailored to clients with different needs and to better understand how the implemented service models match the models described in the efficacy and effectiveness literature (see Attachment 4).

2. Purpose and Use of Information

The purpose of the Client Interview – Baseline and the Client Interview – 6-Month Follow-up is to collect client-level data that can be utilized to assess program impact on client outcomes and to provide descriptive information about clients. The data collected through the Stakeholder Survey will provide descriptive information about stakeholders involved with the GBHI/SSH programs and their relationship with the grantees. The information collected through all three surveys will provide the data necessary to conduct a complete structure, process, outcome, and cost evaluation, as described above. Detailed descriptions and purpose of the surveys are presented in the following paragraphs.

Client Interview – Baseline & Client Interview – 6-Month Follow-up:

The Client Interview – Baseline and 6-Month Follow-up were developed to assess program impact on client outcomes on the basis of consultation with SAMHSA; discussion and feedback during the Evaluation Expert Panel meeting; written comments received after expert panelist review of the preliminary evaluation plan, draft data collection tables, and an extensive literature review; and a review of protocols used in SAMHSA’s Homeless Families and Supportive Housing initiatives as well as by HUD. Additional areas highlighted for measurement include co-occurring mental disorders, history of homelessness, housing (placement/satisfaction), perception of coercion and choice in treatment and housing, readiness for change, service need, perception of care, and client burden. Three additional domains were also consistently advocated by the expert consultants: services received, trauma, and veterans’ service era and combat information.

The Client Interview – Baseline and the Client Interview – 6-Month Follow-up are composed of the following sections:

• Military Service Questions—Given the high prevalence of homelessness among returning veterans and differentially by service era (Kline et al., 2009), baseline collection of military service was recommended. These questions are included to collect basic information about the military background of clients, specifically branch of service, years of service, and service in a combat zone. This information is adapted from the CMHS Jail Diversion and Trauma Recovery Evaluation. This information is collected for descriptive purposes and is only collected at baseline.

• Employment—One question was developed to assess employment in the previous six month period to assess the impact of treatment services on this outcome measure. Employment is viewed as important in the ability to attain and maintain housing (Burt et al., 1999; Pickett-Schenk et al., 2002; Shaheen & Rio, 2007). This item will be asked at baseline and 6-month follow-up.

• Criminal Justice Involvement—Two questions were developed to assess number of arrests and number of nights incarcerated for the previous 6‑month period. Criminal justice involvement has been strongly associated with homelessness and with substance use (e.g., Greenberg & Rosenheck, 2008). These items will be asked at baseline and 6-month follow-up.

• Co-occurring disorders—At the request of SAMHSA, questions were developed to capture self-report data on severity and extent of co-occurring disorders among clients served by GBHI/SSH grantees. Those with co-occurring mental disorders are at increased risk for homelessness and co-occurring mental disorders are prevalent among chronically homeless individuals (e.g., Drake et al., 1997; Joseph & Langrod, 2004). This information will be both descriptive and used in sub-group analyses of client outcomes. These questions will only be asked at baseline.

• Housing and Homeless History—These questions assess the client’s current residence and residential history in the past 6 months including places stayed, time spent homeless, and problems encountered finding housing. The questions on past 6-month residential history are adapted from the HUD “Life After Transitional Housing” Study (Burt, 2009). Other questions include age of first homeless episode and the frequency and length of time a client has been homeless in the past three years, the amount of time necessary to measure homeless chronicity. These questions have been adapted from the CMHS/CSAT Homeless Families Study. Questions about homelessness in the past three years will be asked at baseline only. Questions about current residence and residential history in the past 6 months will be asked at both baseline and 6-month follow-up.

• Housing Satisfaction and Choice—This measure provides information on client satisfaction with various aspects of his or her housing, as well as, the amount of choice the client had over the place where he or she currently resides, which both have been associated with positive client outcomes (Greenwood, Schaefer-McDanile, Winkel, & Tsemberis, 2005; Srebnik, Livingston, Gordon, & King, 1995; Tsemberis, Moran, Shinn, Asmussen, & Shern, 2003). The measure was developed for the CMHS Supportive Housing Initiative. This set of questions will be asked at the baseline and the 6-month interviews.

• Perception of Housing Coercion—These questions were modified from Robbins, Callahan, and Monahan’s (2009) study of perceived coercion to treatment and client housing satisfaction among clients in Housing-First and Supportive Housing Programs. These questions are included to assess explicit treatment requirements and the extent to which clients feel they must participate in services to remain in their housing. These will be asked at the baseline and 6-month interviews.

• Readiness to Change Questionnaire (RTCQ)—This was developed by Rollnick et al. (1992) to use in brief interventions among problem drinkers. The questionnaire will be used to assess readiness to change among individuals who abuse alcohol and other drugs. These measures will be asked at both baseline and 6-month interviews and will allow the contractor to compare changes over time as well as include as a mediator variable in the client outcome analyses.

• Services Needed and Received—This section is designed to obtain information from the client’s perspective on the types of services he or she needed and types of services he or she received. These questions are adapted from the CMHS/CSAT Homeless Families Initiative and the CMHS Jail Diversion and Trauma Recovery Evaluation. These additional questions will allow the contractor to document all services received by clients, including those services provided outside of the GBHI/SSH programs; document differences between client and program reporting; and assess whether and how service receipt changed over time. These data will be important in improving ability to test whether treatment produces abstinence and housing stability. These items will be asked at the baseline and 6-month interviews.

• Perception of Care—These questions include a subset of items from the full Mental Health Statistics Improvement Program (MHSIP) Consumer Survey to assess cultural sensitivity to care, quality of treatment, general satisfaction, and the degree to which services focus on consumer recovery and self-management (Ganju, 1999). The MHSIP Consumer Survey was designed to obtain the subjective evaluation from the consumer on issues related to access, quality, appropriateness and outcomes. The questions have been adapted by the CMHS NOMS. This information will be both descriptive and used as a mediator in outcome analyses. These questions will be asked at the baseline and 6-month interview.

• Treatment Choice—These questions are included to determine the extent to which clients feel coerced into treatment participation. The types of coercion covered include: income benefits, housing benefits, child custody, court ordered-treatment, and abstinence from substance use. There is also one question designed to assess whether clients are aware of other similar services in their community. Although developed specifically for this evaluation, the literature indicates these are areas of coercion for substance abuse treatment clients (Robbins et al., 2009). These items will be assessed at baseline and 6-month follow-up.

• Client Treatment Burden—These questions are posed to determine the financial burden treatment could potentially place on an individual. In addition to economic impediments, there are other practical impediments to participation in treatment services (Tucker et al., 2004; Rapp et al., 2006). The sources of financial burden were adapted from the Client Drug Abuse Treatment Cost Analysis Program (DATCAP) which was designed to assess the costs incurred by patients who attend inpatient or outpatient treatment services (DATCAP.com). This information will be collected at the 6-month interview only and will be used in the cost evaluation.

• Abbreviated Posttraumatic Stress Disorder Checklist (PCL-C)—Expert panelists recommended measuring trauma symptoms given that trauma is prevalent in the homeless population (e.g., Browne & Bassuk, 1997; Goodman, 1991; Bassuk et al., 1996; Burt et al., 1999; HUD, 2009; Shelton et al., 2009) and without intervention consistently predicts negative substance abuse, employment, housing and criminal justice outcomes. This 6-item measure is an abbreviated version of the PCL-C (Weathers, Litz, Huska, & Keane, 1994) which was developed to use as a screening instrument by primary care doctors (Lang & Stein, 2005). This information will be collected at both baseline and 6-month follow-up to assess changes in trauma symptoms.

A chart is presented in Attachment 4 for each of the Client Interview questions noting the domain relevant to the item or measure, justification for use in the evaluation, the population for which the measures were developed, additional literature citations and, as relevant, the corresponding OMB approval number for the items used in previous OMB approved cross-site evaluations and SAMHSA performance monitoring measures.

The target population for the Client Interview – Baseline and the 6-Month Follow-up is all accepted and enrolled clients receiving services under the GBHI/SSH grants in the 2009-2012 cohorts.

Stakeholder Survey:

The Stakeholder Survey is a 22-item questionnaire that will be administered, per voluntary consent, via the web, to GBHI/SSH grantee stakeholder partners for projects funded from 2009 - 2012. This is the main method through which the contractor will collect primary data from stakeholders. The questionnaire is designed to address SAMHSA’s GBHI/SSH evaluation objectives regarding service provision, impact on local treatment systems, implementation lessons learned and project sustainability. The questions specifically gather background information about the partner agency, the services provided, and experience partnering on the implementation and sustainability efforts of the local GBHI/SSH program. This information is necessary to (a) assess important aspects of the GBHI/SSH program related to partnering, (b) measure characteristics of the local treatment system in which the grantee is located, and (c) identify moderating or mediating variables of client outcomes. Questions regarding partner agency characteristics and services offered were developed for the evaluation based on a review of GBHI/SSH grantee documents submitted to SAMHSA (e.g., grantee applications) and per SAMHSA review and feedback. Implementation and collaboration questions were adapted from the cross-site evaluation survey of Weed and Seed funded by the Department of Justice (Trudeau, Barrick, & Roehl, 2010) and from the SAMHSA CMHS Jail Diversion TCE cross-site qualitative study on program sustainability (Broner, 2010a, 2010b). Experience of the contractor in implementing this type of survey is described in Attachment 4.

3. Use of Information Technology

The Client Interview – Baseline and the Client Interview – 6-Month Follow-up are designed primarily as a paper and pencil interview. The interview form will use electronically scannable TeleForm technology to reduce data entry burden and errors. The client interview will be administered onsite by either the grantee program or the grantee’s local evaluator. Once the interview is complete, the administrator will place the completed survey into a sealed, postage-paid envelope and return it to the contractor. Once received by the contractor, the form will be scanned into a dataset. Scanning these forms will eliminate the need for data entry, thereby reducing cost and the potential for data error. Further discussion of the TeleForm is provided in Attachment 4.

The Stakeholder Survey will be administered via the web. Each survey respondent will be issued a username and password to access the web-based survey for their program. To complete the survey, each respondent will login to a secure web-based form to fill out the survey. The web-based survey will reduce burden on the respondent and minimize potential for measurement error. For example, skip patterns and automatic data quality checks (e.g., range checks) can be coded into the online survey form to improve data quality.

4. Effort to Identify Duplication

SAMHSA monitors the performance of GBHI/SSH programs by requiring the grantees to collect and submit data through the GPRA (OMB No. 0930-0208) and NOMS (OMB No. 0930–0285). The Client Interview – Baseline and the Client Interview – 6-Month Follow-up cover some of the same domains as the GPRA and NOMS data (e.g., employment and criminal justice) but there is no duplication of data that will be collected from the Client Interview – Baseline and the 6-Month Follow-up that can be obtained from the GPRA data. The GPRA data cover a previous 30-day timeframe which is not robust enough to accurately assess the impact of treatment services on outcome measures or establish best practices, which are both primary objectives of the evaluation. The Client Interview – Baseline and the 6-Month Follow-up questions are unique from the GPRA and NOMS questions in that timeframes are extended from assessing the previous 30 days to assessing the previous six months. This timeframe extension was strongly endorsed by expert panelists at the Evaluation Expert Panel meeting.

The contractor conducted an extensive literature review to confirm that the data collected through the CSAT GBHI Client Interview – Baseline, 6-Month Follow-up, and the CSAT GBHI Stakeholder Survey would not be duplicative of any ongoing national or state-level data collection efforts. Panelists at the Evaluation Expert Panel meeting and contractor staff who have expertise in SAMHSA technical assistance have also confirmed this data collection will not be duplicative. Data collected in this evaluation is not available from other sources and will be unique because of the scale and breadth of the initiative’s implementation: nationwide, across a spectrum of provider settings, and across a broad cross-section of populations.

5. Involvement of Small Entities

The contractor has designed the client interview to include only the most pertinent information needed to be able to effectively carry out this evaluation. Grantees include state agencies, local services providers, and tribal organizations and some may be small entities; however, there will not be a significant impact on these small entities. GBHI/SSH grantees are required by SAMHSA to administer a baseline and 6-month follow-up GPRA or NOMS interview to all clients admitted to their GBHI/SSH program. The contractor will ask the administrator of the GPRA/NOMS interview to also administer the Client Interview – Baseline and the Client Interview – 6-Month Follow-up immediately following the GPRA/NOMS interview. Since they will already be interviewing the client and are receiving funds for data collection under the GBHI/SSH grant, the additional interview will not add a significant amount of burden to the grantees.

6. Consequences If Information Collected Less Frequently

Client Interview – Baseline & Client Interview – 6-Month Follow-up: A client-level interview will be administered on a voluntary basis to clients who receive services under GBHI/SSH. Only those clients who complete the initial baseline interview will be asked to complete a 6-month follow-up interview. Data collection at these follow-up points is necessary to measure the short- and longer-term outcomes of GBHI/SSH.

Following up at six months is optimal for producing useful outcome data. Waiting until six months after the initial receipt of services allows enough time for effects of GBHI/SSH to develop, including changes in housing status and stability, substance use behavior, mental health symptoms, and secondary outcomes, such as criminal justice involvement, employment, and trauma symptoms.

Stakeholder Survey: This is a one- time collection.

7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required by 5 CFR1320.8(d) was published in the Federal Register on January 13, 2014 (79 FR 2188). No comments were received.

SAMHSA has made extensive use of experts in the area of homeless research, including current and previous GBHI/SSH grantees, to provide guidance on the design and analysis plan of the evaluation. An expert panel meeting was held in December 2009 to review the various aspects of the evaluation, including the preliminary evaluation plan, data collection procedures, economic analysis methods, and literature review. An additional expert panel meeting was held in July 2012. The experts provided feedback on all aspects of the evaluation and their comments and suggestions were incorporated into the development of the surveys. The list of experts is provided in Exhibit 1.

Exhibit 1: Expert Panel Members

Expert	Affiliation	Contact Information
Margarita Alegría, Ph.D.	Harvard Medical School Director and Professor of Psychiatry Center for Multicultural Mental Health Research, Cambridge Health Alliance 120 Beacon Street, 4th Floor Somerville, MA 02143	Phone: 617-503-8447 E-mail: [email protected]
Peggy Bailey, M.A.	Corporation for Supportive Housing Senior Policy Advisor 1731 Connecticut Ave, NW, 4th Floor Washington, DC 20009	Phone: 917-596-6337 E-mail: [email protected]
Gary Bond, Ph.D.	Dartmouth Psychiatric Research Center Professor of Psychiatry Rivermill Commercial Center 85 Mechanic Street, Suite B4-1 Lebanon, NH 03766	Phone: 603-448-0263 E-mail: [email protected]
Brian Dates, M.A.*	Southwest Counseling Solutions Director of Evaluation and Research 1700 Waterman Detroit, MI 48209	Phone: 313-841-7442 E-mail: [email protected]
Anne Fletcher	U.S. Department of Housing & Urban Development, Office of Policy Development & Research Social Science Analyst 451 7th Street, SW Room 8120 Washington, DC 20140	Phone: 202-402-4347 E-mail: [email protected]
Linda Frisman, Ph.D.	Connecticut Department of Mental Health and Addiction Services Director of Research 410 Capitol Avenue, MS # 14 RSD Hartford, CT 06134	Phone: 860-418-6788 Email: [email protected]
Louis Kurtz, M. Ed.*	Division of Behavioral Health Kentucky Department for Behavioral Health Development and Intellectual Disabilities Acting Division Director 100 Fair Oaks Lane 4E-D Frankfort, KY 40621	Phone: 502-564-4456 E-mail: [email protected]
William McAllister, Ph.D.	Institute for Social and Economic Research and Policy Senior Research Fellow Center for Homelessness Prevention Studies, Associate Director Columbia University 420 West 118th St, MC 3355 New York, NY 10027	Phone: 212-854-5781 Email: [email protected]
Stephen Metraux, Ph.D.*	Department of Health Policy & Public Health University of the Sciences in Philadelphia Associate Professor 600 South 43rd Street Philadelphia, PA 19104-4495	Phone: 215-596-7612 E-mail: [email protected]
Roger H. Peters, Ph.D.	Department of Mental Health Law and Policy, University of South Florida Louis de la Parte Florida Mental Health Institute Professor 13301 N. Bruce B. Downs Blvd. Tampa, Florida 33612-3807	Phone: 813-974-9299 E-mail: [email protected]
Laura Prescott, B.A.	President and Founder Sister Witness International 50 Union Street Greenfield, MA 01301	Phone: 413-212-4343 E-mail: [email protected]
Alan Rabideau	Consumer and Family Member Consultant 112 Kincheloe Drive Kincheloe, MI 49788	Phone: 906-495-7158 E-mail: [email protected]
Michael Rowe, Ph.D.	Program for Recovery & Community Health Department of Psychiatry, Yale University School of Medicine Associate Clinical Professor & Co-Director319 Peck Street, Building 1 New Haven, CT 06513	Phone: 203-764-8690 E-mail: [email protected]
David L. Shern, Ph.D.	President and Chief Executive Officer Mental Health America 2000 North Beauregard, Sixth Floor Alexandria, VA 22311	Phone: 703-813-1747 E-mail: [email protected]
David Smelson, Psy.D.*	University of Massachusetts Medical Center, Professor of Psychiatry Translational Research Edith Nourse Rogers Memorial Veterans Hospital, Director 55 Lake Avenue North Worcester, MA 01655-0002	Phone: 508-856-3768 x 15122 E-mail: [email protected]
Mary E. Smith, Ph.D.	Decision Support, Research and Evaluation Division of Mental Health, Illinois Department of Human Services Associate Director 160 North LaSalle Street, 10th Floor Chicago, IL 60601	E-mail: [email protected]
Sally J. Stevens, Ph.D.*	Southwest Institute for Research on Women Executive Director Department of Gender and Women’s Studies, Professor College of Social and Behavioral Sciences, University of Arizona 925 North Tyndall Avenue Tucson, AZ 85721-0438	Phone: 520-626-9558 E-mail: [email protected]
Sam Tsemberis, Ph.D.*	Pathways to Housing, Inc. Founder and CEO Columbia University Medical Center, Clinical Assistant Professor of Psychiatry 55 West 125th Street, 10th Floor New York, NY 10027	Phone: 212-289-0000 x1101 E-mail: [email protected]
* Also current and prior GBHI/SSH grantees

9. Payment to Respondents

Client Interview – Baseline & Client Interview – 6-Month Follow-up: GBHI/SSH clients, who are homeless individuals with substance use and/or mental health disorders, are typically a hard-to-reach transient population. To increase response rates, all clients who agree to participate in the client interview at baseline will receive a cash equivalent incentive worth a $10 value (e.g., gift card). Participants who complete the baseline will be asked to complete a 6-month follow-up interview. Clients who agree to participate in the 6-month follow-up will receive a cash equivalent incentive worth a $25 value (e.g., gift card). Respondents will not be penalized if they wish to skip questions or stop the interview during either the baseline or 6-month follow-up. Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. Research has shown improved response rates when remuneration is offered to respondents. Results from the 2001 National Household Survey on Drug Abuse (NHSDA) incentive experiment were reported by Wright, Bowman, Butler, & Eyerman (2005); key conclusions from their analyses are summarized below:

The $20 and $40 incentive payments each produced about a 10-point gain in overall response rates when compared with the $0 control group. The overall response rate was significantly higher for $40 than the $20 incentive within many of the subgroups addressed in the analysis. Both incentive payment groups more than paid for themselves due to decreased costs of follow-up and more productive screening resulting from the improved response rates. Incentives motivate (or obligate) respondents to admit to substance use that they might not have admitted without the incentive.

During the current data collection, grantees have also noted similar findings, indicating that client enrollment has increased when compared to other prior GPRA/ NOMS enrollment, and that the incentives are a primary reason for successful follow-up rates.

Stakeholder Survey: No cash incentives or gifts will be given to respondents.

10. Assurance of Confidentiality

Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all study components. The contractor is developing the evaluation surveys and analyzing the data and has extensive experience protecting and maintaining the privacy of respondent data.

Client Interview – Baseline & Client Interview – 6-Month Follow-up: The process of administering the Client Interview – Baseline and 6-Month Follow-up is designed to protect client privacy, reduce client discomfort and burden, and ensure that the collected data are of the highest quality. Grantee staff will collect the Client Interview – Baseline and 6-Month Follow-up data immediately following the administration of the SAMHSA-required GPRA or NOMS interview. Under the current data collection, the contractor held training webinars with all GBHI/SSH grantees to detail the steps involved in administering the client interview and the procedures to follow to ensure protection of respondent’s rights and safeguarding of client data. Grantee programs will be provided with a Client Interview Script (Attachment 5), a Client Interview Consent Form (Attachment 6), a Client Interview Process and Procedures Guide, a Question-by-Question Guide, and a Frequently Asked Questions (FAQ) Guide.

To begin the Client Interview – Baseline or 6-Month Follow-up, the interview administrator (hereafter referred to as ‘administrator’) will provide the client with a brief introduction to the interview and ask the client if they will agree to hear more. If the client agrees to proceed, the administrator will read the informed consent for the client interview to the client, who will sign it if he or she understands and agrees with its contents. The consent form will explain the purpose of the cross-site evaluation and the interview, describe the interview length and procedures, describe risks or benefits and steps the evaluation is taking to protect the client’s privacy, inform the client of the incentive, and inform them that the interview is voluntary and that he or she may refuse to answer a question or stop the interview at any point without penalty. The consent form will also include the OMB approval expiration dates, the statement of survey burden, and the statement that the study is federally sponsored. This process will take place in a private location to protect client privacy. The administrator will write the GBHI/SSH site ID number, the client’s GPRA/NOMS ID number, and the Interviewer ID number on the first page of the interview. This is the only identifying information the evaluation will have access to; the evaluation will not know the client’s name or be able to connect client interview answers to a particular client.

The Client Interview – Baseline and 6-Month Follow-up each have two parts. In the first part of each interview, the administrator will read the questions to the client and mark the answers on the scantron form. This part of the interview is comprised of sections related to military service, employment, criminal justice, co-occurring disorders, housing and homeless history, housing satisfaction and choice, perception of housing coercion, readiness to change, services needed and received, client treatment burden, and trauma symptoms. The second part of the Client Interview – Baseline and 6-Month Follow-up includes sections related to perception of care, treatment coercion, and treatment choice. These sections will be completed by the client without the administrator present. The client will be provided information about the kinds of questions they will be answering and assistance in the correct way to use the scantron. The client will again be reminded he or she can refuse to answer questions or stop the interview completely. He or she will also be instructed not to write any identifying information on the form, like their name. If a client is illiterate, the administrator can assist the client in two ways. First, before the client answers anything, the administrator can explain how to answer yes/no questions or Likert scale questions by pointing out what those answers look like or explain which directions imply ‘better’ or ‘worse’. Second, the administrator may remain in the room with the client but in a location that prevents the administrator from seeing the client’s responses. While in the room the administrator may read each question to the client using a blank copy of the instrument that is not the instrument the client is filling out. As needed, the administrator may remind the client of the answer format and may point out what the answer options look like using the blank instrument. In the event this happens, the administrator will be instructed to follow two rules: 1) consistently remind the client to protect or hide their instrument or answers while the administrator is helping them using the blank instrument and 2) always point out or describe all possible answer choices for a given question to reduce the potential for bias. Once the client completes this portion of the survey, he or she will place the survey into a tamper proof/evident, postage-paid envelope and return it to the administrator who will mail both sections to the contractor for processing. Once received, they will both be scanned into a secure dataset.

All clients who complete the Client Interview – Baseline will be asked to participate in the Client Interview – 6-Month Follow-up. If they agree, the client will be given another informed consent outlining the same content as the baseline consent form. Again, they will be informed that participation is voluntary and they will not be penalized for non-participation. The 6-month follow-up will be administered by the grantee staff in the same scantron format as the baseline following the same procedures outlined above. Client interviews will be identified only with the client GPRA/NOMS number which will be necessary to link the baseline data with the 6-month follow-up data and to link the GPRA/NOMS data with the Client Interview – Baseline and 6-Month Follow-up data; no personally identifying information will be given to the contractor.

Stakeholder Survey: The contractor will obtain limited contact information for stakeholders, including full name and e-mail address, to notify them of the survey. Stakeholders will be contacted through e-mail and issued a username and password to access the web-based survey for their grantee program. Each respondent will login to a secure web-based form to complete the survey. They will also be given the grantee program’s identification number which they will be asked to enter during the web survey. This will be the only identifying information linked to the stakeholder’s responses which will be used to link the responses to the appropriate grantee program. The stakeholders will be required to give electronic informed consent (Attachment 7) before they begin answering questions. At no point will survey responses be linked to a specific stakeholder.

For all data collection activities, the contractor will use passwords to safeguard all project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. Contractor staff will also be trained on handling sensitive data and the importance of privacy. In addition, the Client Interview – Baseline and 6-Month Follow-up, the consent form, and the client interview script have been reviewed and approved by the contractor’s Institutional Review Board (IRB) (Federal Wide Assurance Number 3331), approval #13244. The Stakeholder Survey is under IRB review. In keeping with 45 CFR 46, Protection of Human Subjects, the GBHI/SSH procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent. The IRB-approved model informed consents meet all Federal requirements for informed consent documentation. This template will be customized by each grantee to obtain informed consent for participation in the study. Any necessary changes to the surveys will be reviewed by the contractor’s IRB.

Data from the GBHI/SSH client interviews will be safeguarded in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.

11. Questions of a Sensitive Nature

Client Interview – Baseline & Client Interview – 6-Month Follow-up: The client interviews, by necessity, will collect sensitive information about homelessness, substance abuse, mental health, and criminal justice involvement as these are all outcomes of interest to SAMHSA. Also, the Evaluation Expert Panel strongly endorsed including a measure regarding trauma symptoms. The client interview will ask clients about trauma symptoms they may be experiencing but they will not be asked about specific traumatic events. If these questions cause any distress for the client, the interview administrator will connect them with someone from the grantee program who they can speak with. Also, two sections included in both the Client Interview – Baseline and the 6-Month Follow-up interviews, Perception of Care and Treatment Choice, will be self-administered to eliminate discomfort a client may feel in giving their feedback about the program to program staff. Sensitive information of this nature is always regarded as private, and privacy for clients in federally assisted treatment programs is assured through strict adherence to Federal Regulation 42 CFR, Part 2. All client interviews will be conducted in a private space and the administrator will first obtain consent for participation. Respondents will be informed about the purpose of the data collection and that responding to all interview questions is voluntary. They will be assured that they may stop taking the interview at any time without forfeiting the incentive and without penalty from the grantee program. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them. Respondents’ names or other personally identifying information will not be linked to data collected.

Stakeholder Survey: No sensitive information will be collected from the grantee stakeholders.

12. Estimates of Annualized Hour Burden

Estimate the annualized hour burden of the collection of information from clients. The total client sample size for the Client Interview data collection effort is estimated to be a maximum of 7,356 respondents based on grantee target enrollment numbers. The baseline survey is expected to have a response rate of 80%, therefore resulting in 5,885 respondents completing the baseline survey. The 6-month follow-up survey is expected to have a response rate of 80% of the baseline sample, leaving 4,708 respondents with baseline and follow-up data. Exhibit 2 presents estimates of annualized burden based on preliminary testing. As evidenced from the testing, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information, the total estimated time to complete the baseline survey is 20 minutes. The 6-month follow-up survey drops two small sections from the baseline survey but adds one longer section and it is estimated that it will take 24 minutes to complete.

Exhibit 2. Cross-Site Data Collection Burden for the Client Interview – Baseline, Client Interview – 6-Month Follow-up, & Stakeholder Survey

Instrument/Activity	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Burden Hours	Hourly Wage	Total Respondent Costa
Baseline data collection (Clients)	5,885	1	5,885	.33	1,942	$20.76	$40,316
6-month follow-up data collection (Clients)	4,708	1	4,708	.40	1,883	$20.76	$39,091
Client Subtotal	10,593		10,593		3,825	$20.76	$79,407
Stakeholder Survey	648	1	648	.28	181	$31.88	$5,770
TOTAL	11,241		11,241		4,006		$85,177

^aTotal respondent cost is calculated as hourly wage × time spent on survey × total number of responses.

Estimate the annualized hour burden of the collection of information from grantee stakeholders. The total stakeholder sample size is estimated to be 648 (approximately 5 responses per 127 GBHI/SSH grantee sites). The stakeholder web survey is estimated to take 17 minutes to complete. Exhibit 2 presents estimates of annualized burden based on preliminary testing.

Estimate the annualized cost burden to the respondent for the collection of information from clients. There are no direct costs to respondents other than their time to participate in the interview. The total cost of the time respondents spend completing these surveys is $79,407 (number of total baseline client respondent hours plus follow-up respondent hours × $20.76, the estimated average hourly wages for adults as published by the Bureau of Labor Statistics (2008) inflated to 2010 value). The annualized cost is approximately $19,852.

Estimate the annualized cost burden to the respondent for the collection of information from stakeholders. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend completing these surveys is $5,770 (number of stakeholder respondent hours × $31.88, the estimated average hourly wages for individuals working in professional managerial occupations as published by the Bureau of Labor Statistics (2008) inflated to 2010 value). The annualized cost is approximately $1,442.

13. Estimates of Annualized Cost Burden to Respondents

There are no respondent costs for capital or start-up or for operation or maintenance.

14. Estimates of Annualized Cost to the Government

The estimated cost to the government for the data collection is $261,220.  This includes approximately $250,000 for cost of materials, programming, incentives, trainings, contractor labor, housing and maintaining data, and approximately $2,805 per year represents SAMHSA costs to manage/administer the survey for 2% of one employee (GS-15).  The annualized cost is approximately $65,305.

15. Changes in Burden

There is no burden change.

16. Time Schedule, Publications, and Analysis Plan

Time Schedule: Exhibit 3 outlines the key time points for the study and for the collection of information. The requested period also allows for training and start-up activities associated with the preparation for data collection.

Exhibit 3. Time Schedule for Entire Project

Activity	Time Schedule
Obtaining OMB approval for extension of data collection	Spring 2014
Client Interview - Baseline and 6-month Follow-up Implementation Continues	Spring 2014
Stakeholder Survey Data Collection Begins	Spring 2014
All Client Interview Data Collection Ends	Fall 2015
Stakeholder Survey Data Collection Ends	Fall 2015
Data analysis	Ongoing (2014 – 2016)
Dissemination of findings Interim reports, presentations, manuscripts, final report	Ongoing (2014 – 2016)

Publications: The evaluation is designed to produce knowledge about the implementation and impact of GBHI/SSH programs. It is therefore important to prepare and disseminate reports, concept papers, documents, and oral presentations that clearly and concisely present project results so that they can be appreciated by both technical and nontechnical audiences. The contractor will:

• Produce rapid-turnaround analysis papers, briefs, and reports;

• Prepare and submit monthly technical progress reports, semi-annual briefings and annual progress reports;

• Prepare special reports in concert with SAMHSA and expert panel input. For example, the contractor plans to submit a “portrait” of the GBHI/SSH grantee and client characteristics;

• Prepare final cross-site findings report, including an executive summary;

• Deliver presentations at professional and federally sponsored conventions and meetings; and

• Disseminate reports and materials to entities inside and outside SAMHSA.

Analysis:

Client Interview – Baseline & Client Interview – 6-Month Follow-up:

The outcome evaluation component focuses on examining the utility of future GBHI/SSH cohorts through the review of planned and actual outcomes. Within the context of the social-ecological framework, the outcome evaluation will focus on the effects of the GBHI/SSH programs on client outcomes, accounting for grantee characteristics and treatment system and community contexts. Hierarchal linear models (HLM) will be used to estimate the mean change in client-level outcomes between baseline and follow-up. HLM is appropriate for these analyses because this modeling approach allows the contractor to control for the clustering of clients within grantee. Within the HLM framework, the contractor will adjust for client characteristics. These adjusted mean changes will provide a rigorous, yet easy-to-understand, assessment of program impact. Separate analyses will address the impact of grantee characteristics (such as program model) on client-level outcomes, controlling for client characteristics. As appropriate, subgroup analyses will be conducted in which the data will be stratified by program type or client type to assess whether outcomes differ among the different types of programs or for different types of client (e.g., veterans or women).

As described in Attachment 4, the supplemental data from the Client Interview – Baseline and 6-Month Follow-up will be combined with data from the GPRA/NOMS measures, information gathered about grantee program components, and data from secondary sources such as SAIS GPRA, the National Survey of Substance Abuse Treatment Services (N-SSATS; OMB No. 0930-0106) and the Treatment Episode Data Set (TEDS; OMB No. 0930-0335) to develop a comprehensive portrait of the GBHI client populations, the needs of these populations, the services provided to address those needs, and the outcomes across a multitude of domain areas for those participating in GBHI/SSH programs. These supplemental data will provide mediating and moderating variables, as well as information on client characteristics not covered by the GPRA/NOMS measures. The areas addressed by the supplemental data collection include service need, burden, satisfaction/perception of care, the form of care or individually tailored care, model adaption, homelessness, housing (placement/safety/perceived choice/perceived value), readiness for change, and co-occurring mental disorders. Three additional domains (services, trauma and veteran’s service era and combat information) were added in response to the recommendations of the expert panel and SAMHSA and confirmed with the GBHI/SSH Grantees. The additional services data will improve our ability to describe the relationships between treatment plans and abstinence and housing stability including measuring the extent to which models of matching services to needs are being used and the appropriate dosage of services as described in the literature for these models. As GPRA/NOMS data includes administrative data on services received only at discharge, it impossible to assess whether and how service receipt changes over time using only GPRA/NOMS data alone. The GPRA/NOMS data does not collect this information from the client or address perceived need and service matching. The supplemental data will address this limitation. Additionally, the panelists recommended measuring trauma symptoms given that trauma is prevalent in the homeless population (e.g., Browne & Bassuk, 1997; Goodman, 1991; Bassuk et al., 1996; Burt et al., 1999; HUD, 2009; Shelton et al., 2009) and without intervention consistently predicts negative substance abuse, employment, housing and criminal justice outcomes. Finally, given the high prevalence of homelessness among returning veterans and differentially by service era (Kline et al., 2009), along with there being several Grantee programs focused solely on veterans, baseline collection of veteran service era was recommended.

The contractor conducted a literature review that helped advance the thinking about likely influences on client-, grantee-, and system-level outcomes (Broner et al., 2010). As the data collection and analysis plans were developed, information from the review was used to strengthen the evaluation’s ability to provide insightful findings on what works for whom, under what approaches, and in what systems and contexts. At the client level, demographic characteristics (sex, age, race or ethnicity), parental status, educational attainment, veteran status (for recent cohorts), disability, social supports, and involvement with the criminal justice system can be important with respect to understanding the appropriateness and expected effectiveness of specific approaches. Client differences in substance abuse, mental illness, and co-morbidity are of central importance to GBHI/SSH. This data collection and analyses will allow the evaluation to describe how client populations differ on these factors across study sites and test whether these factors are associated with differential program choices, components and successful provision of services, including housing the clients. For example, by collecting gender at the client level, the evaluation will assess whether programs are better able to provide appropriate services for female clients than for male clients. Clients will also differ in their levels of participation, program completion, and treatment compliance. Information from the supplemental data collection will enhance the GPRA/NOMS discharge data, allowing the evaluation to estimate what client characteristics are significantly associated with participation at 6-month follow-up and to test whether participation mediates the programs’ ability to carry out full services objectives.

The outcome evaluation component focuses on addressing the “utility” element of the evaluation’s Objective 1, which per SAMHSA’s RFA is to “examine the feasibility, utility, and sustainability of future Treatment of Homeless cohorts through the review of planned and actual outcomes.” The outcome evaluation will focus on the changes in client outcomes that are associated with differences in grantee models. The findings will be framed in a pre-post quasi-experimental design that will allow the evaluation to examine the relationship of outcomes to both intent-to-treat and service receipt. HLM will be used to estimate the mean change in client-level outcomes between baseline and follow-up. HLM is appropriate for these analyses because this modeling approach allows the evaluation to control for the clustering of clients within grantee. Within the HLM framework, the evaluation will adjust for client characteristics and other contextual factors. These adjusted mean changes will provide easy-to-understand estimates of possible program impact. Although these estimates are not intended to be causally interpreted, the evaluation intends to compare them to estimates for similar models and populations in the scientific literature to confirm that they are within ranges that would be expected, conditional on the level of adherence to the models that is observed for each grantee. These estimates form a baseline for exploring how program decisions and characteristics alter service delivery and outcomes. In this way, variation among the Grantees will serve as experimental variation for analyzing ‘key ingredients’ of models for achieving different outcomes, such as linking clients to certain types of housing. As appropriate, subgroup analyses will be conducted in which the data will be stratified by program type or client type to assess whether outcomes differ among the different types of programs or for different types of client (e.g., veterans or women). Sample data analysis shells are presented in Attachment 4.

Stakeholder Survey:

Stakeholder responses to the web-survey will provide crucial information on grantee structure and process including information on barriers, solutions and innovative strategies for successful implementation. Systematic qualitative analyses will be conducted using the software package ATLAS.ti. Key results will be identification of prominent themes regarding model choice and implementation success across the grantees. Rank ordered themes will be presented as well as descriptive statistics on web survey responses (means, medians, ranges). When appropriate for answering evaluation hypotheses, pairwise or ANOVA/ANCOVA statistical tests will be conducted to determine differences in responses across programs and model types. Finally, web survey responses will provide program-level independent variables that will be incorporated into the client-level outcome model described above. Specifically, the contractor will test whether these program elements are significant moderators or mediators of client outcomes.

17. Display of Expiration Date

The OMB approval expiration date will be displayed.

18. Exceptions to Certification for Statement

There are no exceptions to the certification statement. The certifications are included in this submission.

21