cooperative agreements to benefit homeless individuals (CABHI) evaluation client & stakeholder Surveys
Supporting Statement
The Substance Abuse and Mental Health Services Administration’s (SAMHSA’s) Center for Mental Health Services (CMHS) and Center for Substance Abuse Treatment (CSAT) are requesting a revision from the Office of Management and Budget (OMB) for data collection activities under OMB No. 0930-0320, which expires on May 31, 2017. This collection was previously known as the Cross-Site Evaluation for the Grants for the Benefit of Homeless Individuals (GBHI), but is now known as the CABHI Evaluation Client & Stakeholder Surveys. These data collection activities, focused on project clients and stakeholders, will be conducted for SAMHSA’s evaluation of the CABHI services grant program, which is scheduled through September 2020.
Client Interview: Baseline (Attachment 1)
Client Interview: 6-Month Follow-up (Attachment 2)
Stakeholder Survey Wave 1 (Attachment 3)
Stakeholder Survey Wave 2 (Attachment 4)
Stakeholder Survey Wave 3 (Attachment 5)
The CABHI grant program is authorized under Sections 506, 509, and 520A of the Public Health Service Act, as amended. The program also aligns with SAMHSA’s Recovery Support strategic initiative and addresses Healthy People 2020 Objectives: Mental Health and Mental Disorders (Topic Area HP 2020-MHMD) and Substance Abuse (Topic Area HP 2020-SA).
SAMHSA’s mission is to reduce the impact of substance abuse and mental illness on America’s communities; one important element of that is meeting the treatment, housing, and support service needs of people who are homeless and have substance use disorders, mental disorders, or both. From 2001 through 2010, SAMHSA’s CSAT funded projects through the Grants for the Benefit of Homeless Individuals (GBHI) grant program, which focused on expanding and strengthening treatment services for people who are homeless or at risk of homelessness and have substance use disorders, mental health disorders, or both. From 2007 through 2010, SAMHSA’s CMHS also funded projects with a focus on homeless populations through the Services in Supportive Housing (SSH) grant program, which provided services to individuals and families experiencing chronic homelessness in coordination with existing Permanent Supportive Housing (PSH) programs and resources. In 2011, CMHS and CSAT began jointly funding the CABHI grant program to build on the success of the GBHI and SSH programs.
CABHI emphasizes the value of providing access to permanent housing and supportive services for people who are chronically homeless and have mental disorders, substance use disorders, or both. It enhances and extends the GBHI and SSH efforts by focusing on the development and expansion of local infrastructures that integrate treatment and services for mental and substance use disorders, permanent housing, and other critical services for individuals experiencing chronic homelessness.
Over the past 5 years, the CABHI program has continued to grow and evolve. CABHI grants were originally awarded to community-based entities (hereafter referred to as “Communities”) through 2012. Beginning in 2013 through 2015, CABHI grants were awarded to state agencies (i.e., State Mental Health Authorities or Single State Agencies for Substance Abuse) with an additional focus on statewide planning for integrated services. In 2016, SAMHSA funded CABHI grants at three levels:
state (up to $1.5 million per year),
local government (up to $800,000 per year), and
community (up to $400,000 per year).
The varying levels of 2016 CABHI and potential future grantees are united by the goal of enhancing and expanding infrastructure and capacity for mental health and substance abuse treatment and related support services for individuals experiencing chronic homelessness or veterans, families, or youth experiencing homelessness as a result of these conditions. This is accomplished through the provision of PSH, behavioral health treatment, and recovery support services, and enrollment in health insurance, Medicaid, or other mainstream benefit programs.
To ensure that project clients receive comprehensive and coordinated services, grantees must devote at least 70% of grants funds to providing the following types of treatment and support services: outreach and engagement, case management, behavioral health treatment services, trauma-informed services, peer support, family-driven and youth-guided frameworks, collaboration among providers, and support aimed at service and treatment retention. Although CABHI funds cannot be used to fund housing directly, grantees must connect their clients to permanent housing.
The grantees must also engage in several activities to address the CABHI program’s focus on infrastructure development (e.g., developing a statewide plan to sustain partnerships across public health and housing systems), using up to 20% of grant funds for states and 10% for local governments and communities. In line with prior CABHI cohorts, grantees must also have a governing body composed of stakeholders across service systems that supports the goal of improving infrastructure and is charged with monitoring project implementation and progress (i.e., steering committees, State Interagency Councils on Homelessness).
The data collection activities for which SAMHSA is seeking revised approval were developed for an evaluation of the 2009 through 2012 GBHI, SSH, and CABHI grant cohorts, which included 127 grantees. The SCI and Stakeholder Survey were successfully implemented, providing data critical to the achievement of the evaluation’s objectives. This evaluation was recently concluded with key results showing that clients needed many types of services and grant projects were successful in meeting those needs, key client outcomes (substance abuse, mental health, homelessness, and arrests) improved significantly, and clients who received more services tended to report greater improvements in outcomes. The evaluation also found that, to a large extent, grantees successfully implemented services in accord with grant requirements, numerous types of organizations successfully implemented the grants, and partnerships and collaboration were important elements of these grants.
In 2016, SAMHSA funded a 4-year cross-site evaluation of the 30 CABHI grant projects initially funded in 2016. The evaluation may also include CABHI grant projects funded in 2017. The primary task of the CABHI evaluation is to conduct a comprehensive process and outcome evaluation, addressing questions related to the implementation of the CABHI grant projects and the extent to which they were able to meet the program’s goals.
Process evaluation primarily represents what is done to and for the client; this aspect of the evaluation will also include a focus on structure, or the resources available in the service delivery system, which represent the capacity to deliver quality care, but not the care itself. The CABHI evaluation process measures include characteristics of the grantee organization and its partnerships; the system within which the project is embedded; relationships with stakeholders; characteristics of the target population; services received, including implementation of evidence-based practices (EBPs); staffing patterns; costs of services; barriers and facilitators of project implementation; and project sustainability efforts.
The outcome evaluation will focus on outputs, which are the most immediate or proximal results of project activities (e.g., changes in partner collaboration, the number of clients enrolled in mainstream benefits), and client outcomes, particularly those related to behavioral health and homelessness and housing instability. Statistical analyses (described in Section A.16) will be used to examine changes in client outcomes from baseline to follow-up and to explore whether changes in outcomes were associated with client characteristics, receipt of services, service models, or other project or grantee characteristics. The evaluation questions (EQs) that will be addressed by the CABHI evaluation are listed in Exhibit 1.
Exhibit 1. CABHI Evaluation Questions by Domain
Systems |
EQ1. What organizations, agencies, and individuals (federal, state, and local) are involved in state or community Interagency Councils on Homelessness? What is the nature of the collaborations? (Process) |
EQ2. What are the barriers to or facilitators of state- and community-level collaboration and partnership development? How are they addressed? (Process) |
EQ3. To what extent do CABHI activities lead to enhanced coordination and collaboration across mental and physical health providers, housing providers, and other organizations and agencies? (Outcome) |
EQ4. Does the type of partner dyad (e.g., mental health treatment providers, housing providers) influence the degree to which coordination and collaboration are enhanced? |
EQ5. What is the impact of collaboration across federal, state, and local agencies on CABHI grant activities, processes, and outcomes? (Outcome) |
EQ6. How do state- and local-level systems change in response to CABHI activities? (Outcome) |
(continued)
Exhibit 1. CABHI Evaluation Questions by Domain (continued)
Expansion and Access to Services |
EQ7. Who provided (project staff), what services (modality, type, intensity, duration), to whom (individual characteristics), in what context (system, community), and at what cost (facilities, personnel, dollars)? (Process) |
EQ8. What strategies were used to maintain fidelity to any EBP chosen or intervention across providers over time? (Process) Was greater fidelity reported for certain strategies or EBPs? |
EQ9. How many individuals were reached through the program? (Process) What were their characteristics (demographics, illnesses, housing status, and history)? How did the number of individuals reached and their characteristics compare with grantee targets? Was success in meeting or exceeding enrollment targets associated with grantee configurations or strategies? |
EQ10. To what extent were services based on the needs of specific service recipients (e.g., trauma-focused services)? (Process) |
EQ11. To what extent do CABHI projects lead to increased access to stable, permanent housing? (Outcome) Were certain grantee configurations, models or strategies, services or service bundles (including the use of EBPs), or target populations associated with greater increases? |
EQ12. To what extent do CABHI projects lead to increased access and enrollment in mental health services and supports? (Outcome) Were certain grantee configurations, models or strategies, services or service bundles (including the use of EBPs), or target populations associated with greater increases? |
EQ13. To what extent do CABHI projects lead to increased access and enrollment in private insurance, Medicaid, or other benefits (e.g., SSI/SSDI, TANF, SNAP)? (Outcome) |
EQ14. What funding strategies did state or community Interagency Councils on Homelessness develop or adopt for sustainability of mental health, substance use, housing, and other supports and services? (Process) |
EQ15. How did the grantees use evaluation findings and performance measurement to support improved collaboration with stakeholders and support sustainability? (Process) |
(continued)
Exhibit 1. CABHI Evaluation Questions by Domain (continued)
Participant Outcomes |
EQ16. To what extent do CABHI projects lead to improved housing stability? (Outcome) |
EQ17. Where comparisons are possible and appropriate, how do CABHI grant participants compare with similar nonparticipants on HMIS measures (e.g., housing tenure)? (Outcome) |
EQ18. To what extent do CABHI projects lead to improved behavioral health, daily life and functioning (e.g., social connectedness, relationships, employment/education, criminal justice involvement), enrollment in benefits, and decreased substance use? (Outcome) |
EQ19. What project/contextual factors were associated with outcomes? (Outcome) Were certain grantee configurations, models or strategies, services or service bundles (including the use of EBPs), or target populations associated with greater increases? |
EQ20. What individual factors were associated with outcomes, including race/ethnicity and sexual identity (sexual orientation/gender identity)? (Outcome) |
To address these evaluation questions, it is critical to gather accurate data on participant outcomes and stakeholder involvement in and perspectives on grantee projects. Building on the previous evaluation efforts, the SCI and Stakeholder Survey have been reviewed and updated. To improve upon the approach in the previous evaluation, the Stakeholder Survey will be collected at multiple time points, with careful coordination of the timing of different data collections so as not to overburden the grantees.
The SCI complements and expands upon the required client-level CSAT Government Performance and Results Act (GPRA) Client Outcome Measures for Discretionary Programs data collection (OMB No. 0930-0208). All clients are assessed by grantees at intake, 6-month follow-up to intake, and at program discharge; the SCI is implemented with the GRPA at intake and 6-month follow-up to intake. This approach was successfully used for the GBHI, SSH, and CABHI grantees in the previous evaluation with over 7,000 completed surveys at intake and over 5,000 completed surveys at the 6-month follow-up.
As previously described, the 2016 CABHI grantees were funded at three levels: states, local governments, and communities. Within the states (n=3), sub-recipients are contracted to implement the project and provide services at the local level; for two states, that is the county level, and for one state that is the Department of Housing and Urban Development (HUD) Continuum of Care (CoC) level. The state sub-recipients will be the entities that will enroll and provide services to clients, while the states themselves will act in oversight and coordinating roles. Therefore, the state sub-recipients will be viewed by the evaluation similarly to local government and community grantees as they share much the same function. The number of states and sub-recipients, local governments, and communities funded by CABHI in 2016 is listed in Exhibit 2. Potential future cohorts will also be funded at the same three levels and similar numbers (e.g., 30 grantees across the three levels).
Exhibit 2. The 2016 CABHI Grantees
Funding Level |
Number |
State |
3 |
State sub-recipient |
9 |
Local Government |
12 |
Community |
15 |
The purpose of the SCI – Baseline and 6-Month Follow-up is to collect client-level data that can be used to assess the association between project characteristics and client outcomes and to provide descriptive information about clients. The data collected through the Stakeholder Survey will provide descriptive information about stakeholders involved with the CABHI projects and their relationship with the grantees. The information collected through all three surveys will provide the data necessary to answer the evaluation questions listed above. Detailed descriptions and purpose of the surveys are presented in the following paragraphs.
The SCI – Baseline and 6-Month Follow-up were developed for the Homeless Programs evaluation to enhance the data gathered in the GPRA, which produced a more comprehensive understanding of participant outcomes and their association with project characteristics. The SCI was developed in consultation with SAMHSA staff and an expert panel (described in Section 8) that endorsed including the following domains: co-occurring mental health disorders, trauma, history of homelessness, housing (placement/satisfaction), perception of coercion and choice in treatment and housing, service need and receipt, and perception of care. Based on the experience gained during the previous evaluation and the new CABHI evaluation questions, revisions have been made to the SCI, which are summarized in Exhibit 3. A full description of the questions included in the SCI Baseline and 6-Month Follow-up is provided below. The target population for the SCI is all accepted and enrolled clients receiving services under the CABHI grants in the 2016 cohort and potential future cohorts if funded.
Exhibit 3. Changes to the SCI Baseline and 6-Month Follow-up
Section |
Included
in original SCI Baseline and |
Revisions
made for CABHI SCI Baseline and |
Military Service |
Yes |
Section was removed (these questions are now on the GPRA instrument) |
Employment |
Yes |
Section was removed |
Housing and Homeless History |
Yes |
Section has been expanded to collect a more detailed housing history including when and why a client moved, lost or gained housing. |
Perception of Housing Coercion |
Yes |
Section updated and streamlined for CABHI clients but no substantive changes made. |
Housing Satisfaction and Choice |
Yes |
No changes have been made. |
Criminal Justice |
Yes |
One question on the number of nights in jail has been removed (this is now captured in the Housing & Homeless History section). |
Treatment History, Needs and Services |
Yes |
Treatment history has been updated with 5 additional mental health diagnoses to align with the new DSM-5. |
Readiness to Change |
Yes |
Section was removed. |
K6 |
No |
Section was added to assess serious mental illness, which is an enrollment criterion for the grant program. |
Posttraumatic Stress Disorder Checklist |
Yes |
No changes have been made. |
(continued)
Exhibit 3. Changes to the SCI Baseline and 6-Month Follow-up (continued)
Section |
Included
in original SCI Baseline and |
Revisions
made for CABHI SCI Baseline and |
Services Needed and Receipt |
Yes |
Section has been updated by revising the type of services clients are asked about. The new list reflects the services used most frequently by Homeless Program clients. |
Benefits Assistance and Insurance |
No |
Section was added; one of the key components of the grant is to assist clients with benefits/insurance enrollment. |
Number of Children |
No |
Question was added; SAMSHA has identified families as a key target population. |
Perception of Care |
Yes |
Five questions were added to update the Perception of Care tool. |
Treatment Choice |
Yes |
No changes made. |
SCI 6-Month Follow-up Only |
||
Program Status and Services |
No |
Section was added. |
Client Burden and Costs |
Yes |
No changes made. |
DSM = Diagnostic and Statistical Manual of Mental Disorders.
The SCI Baseline and 6-Month Follow-up are composed of the following sections:
Homelessness and Housing
Housing and Homeless History—These questions assess the client’s current residence and residential history in the past 6 months, including places stayed, time spent homeless, and problems encountered finding housing. Detailed housing history questions identify why participants moved or moved out of specific housing situations.
Perception of Housing Coercion—These questions were modified from Robbins, Callahan, and Monahan’s (2009) study of perceived coercion to treatment and client housing satisfaction among clients in Housing-First and Supportive Housing Programs. These questions are included to assess explicit treatment requirements and the extent to which clients feel they must participate in services to remain in their housing.
Housing Satisfaction and Choice—This measure provides information on client satisfaction with various aspects of his or her housing and the amount of choice the client had over the place where he or she currently resides, which both have been associated with positive client outcomes (Greenwood, Schaefer-McDanile, Winkel, & Tsemberis, 2005; Srebnik, Livingston, Gordon, & King, 1995; Tsemberis, Moran, Shinn, Asmussen, & Shern, 2003). The measure was developed for the CMHS Supportive Housing Initiative.
Criminal Justice Involvement—Three questions were developed to assess number of arrests, bookings, and times held in custody for the previous 6‑month period. Criminal justice involvement has been strongly associated with homelessness and with substance use (e.g., Greenberg & Rosenheck, 2008). These items will be asked at baseline and 6-month follow-up.
Treatment History, Needs, and Services—Eleven questions assess a client’s treatment and diagnostic history. Measures of treatment and diagnostic history provide a fuller understanding of a client’s condition and are a key moderator in client outcome models.
The K6 scale was designed to discriminate cases of serious mental illness (SMI) from non-cases. The tool was developed with support from the U.S. government’s National Center for Health Statistics for use in the redesigned U.S. National Health Interview Survey (NHIS) and is described in detail in Kessler et al. (2003).
Abbreviated Posttraumatic Stress Disorder Checklist (PCL-C)—Expert panelists recommended measuring trauma symptoms given that trauma is prevalent in the homeless population (e.g., Bassuk et al., 1996; Browne & Bassuk, 1997; Burt et al., 1999; Goodman, 1991; HUD, 2009; Shelton et al., 2009) and without intervention consistently predicts negative substance abuse, employment, housing, and criminal justice outcomes. This 6-item measure is an abbreviated version of the PCL-C (Weathers, Litz, Huska, & Keane, 1994), which was developed to use as a screening instrument by primary care doctors (Lang & Stein, 2005). This information will be used to assess changes in trauma symptoms.
Services Needed and Received—This section is designed to obtain information from the client’s perspective on the types of services he or she needed and types of services he or she received. These questions are adapted from the CMHS/CSAT Homeless Families Initiative and the CMHS Jail Diversion and Trauma Recovery Evaluation. These questions will allow the contractor to document the types of services received by clients and assess whether and how service receipt changed over time. These data will be important in improving ability to test whether treatment is associated with abstinence and housing stability.
Benefits Assistance and Insurance—This section is designed to obtain information from the client on the types of mainstream benefits (e.g., SSI/SSDI, TANF, SNAP) they receive, may apply to receive, and if they receive Medicaid or other type of insurance. Obtaining access to mainstream benefits is vital for the long term recovery and stability of chronically homeless individuals and is a key outcome for CABHI project to measure.
Number of Children—One question asks the number of children living with the client as CABHI grantees focus on homeless families.
Perception of Care—These questions include a subset of items from the full Mental Health Statistics Improvement Program (MHSIP) Consumer Survey to assess cultural sensitivity to care, quality of treatment, general satisfaction, and the degree to which services focus on consumer recovery and self-management (Ganju, 1999). The MHSIP Consumer Survey was designed to obtain the subjective evaluation from the consumer on issues related to access, quality, appropriateness and outcomes. This information will be descriptive and used as a mediator in outcome analyses.
Treatment Choice—These questions are included to determine the extent to which clients feel coerced into treatment participation. The types of coercion covered include income benefits, housing benefits, child custody, court ordered-treatment, and abstinence from substance use. There is also one question designed to assess whether clients are aware of other similar services in their community. Although developed specifically for this evaluation, the literature indicates these are areas of coercion for substance abuse treatment clients (Robbins et al., 2009).
The following questions are only asked at the SCI 6-Month Follow-Up:
Program Status and Services—At the 6-month follow-up, a set of 8 questions are asked to better document four types of treatment services received directly by clients from CABHI projects over the last 6 months. Clients will also be asked to identify specific evidence based services they received while involved in a CABHI project. Project staff will help clients identify services they have received to reduce client burden and increase accuracy.
Client Burden and Costs—These questions ask clients the amount of time they spend traveling to and attending treatment and any out-of-pocket costs, such as co-pays, fees, travel costs, and child care costs, incurred to attend treatment.
The following sections were removed from the SCI based on experience during the Homeless Programs evaluation and changes to the GPRA.
Military Services and Employment—Questions regarding veteran status and details on employment are now included on the GPRA forms.
Readiness to Change—These questions were found to be not useful in assessing change over the 6-month period during the Homeless Programs evaluation.
The Stakeholder Survey has been revised and separated into three waves with one wave to be completed each year of the grant for a total of three waves. This change in approach allows the evaluation team to ask stakeholders and partners specific questions on collaboration, services, and implementation as those activities occur instead of at the end of the grant project. Exhibit 4 lists the changes made to the Stakeholder Survey. The surveys will be administered, per voluntary consent, via the web, to CABHI grantee stakeholder partners. The questionnaire is designed to address SAMHSA’s CABHI evaluation objectives regarding service provision, impact on local treatment systems, and partner collaboration. Stakeholders and partners will be identified by the evaluation team using grantee proposals and reports and confirmed with the grantee. Each year, the evaluation team will compile an updated stakeholder list for review by the grantee project director. Once confirmed, the stakeholder point of contact (someone directly involved with the CABHI project) will be contacted by e-mail and asked to participate in the web-survey.
Exhibit 4. Changes to the Stakeholder Survey
Section |
Included in Original Stakeholder Survey |
Revisions made for CABHI Stakeholder Survey |
Wave(s) |
Stakeholder’s Personal Involvement |
Yes |
No changes made. Repeat respondents in Waves 2 and 3 will see responses provided in the previous wave and have the option to update the information as needed. |
Waves 1, 2, and 3 |
Stakeholder Organization |
Yes |
Section has been updated with revised questions on organizational type and service provision. Repeat respondents in Waves 2 and 3 will see responses provided in the previous wave and have the option to update the information as needed. |
Waves 1, 2, and 3 |
Treatment and Case Management Services |
Yes |
Added one question on providing treatment with a psychiatrist or psychologist. |
Wave 2 |
Client/Consumer Choice in Treatment |
Yes |
No changes made. |
Wave 2 |
Wraparound and Support Services |
Yes |
Added question on legal services for family re-unification services and moved medical care questions to new section. |
Wave 2 |
Health Care Services |
No |
Added new section focused on eight health care services commonly provided by grantee stakeholders and partners. |
Wave 2 |
Housing Services |
Yes |
No changes made. |
Wave 2 |
Collaboration with Other Organizations |
Yes |
Three new questions have been added to help identify how stakeholders and partners work together on specific activities. Collaboration will be measured twice to better understand how CABHI projects impacted collaboration. |
Waves 1 and 3 |
Implementation of the SAMHSA Project |
Yes |
No changes made. |
Wave 3 |
For all waves, questions regarding partner agency characteristics and services offered were developed for the evaluation based on a review of CABHI grantee documents submitted to SAMHSA (e.g., grantee applications) and per SAMHSA review and feedback. Implementation and collaboration questions were adapted from the cross-site evaluation survey of Weed and Seed funded by the Department of Justice (Trudeau, Barrick, & Roehl, 2010) and from the SAMHSA CMHS Jail Diversion TCE cross-site qualitative study on program sustainability (Broner, 2010a, 2010b).
The added collaboration measures in Waves 1 and 3 are based on a measure of inter-professional team collaboration developed by Mellin and coauthors (2010). The new measures will collect information about collaborative processes and partnerships at the state and community levels and to examine the networks involved in successful information sharing and collaborations across homeless-serving agencies and the homeless persons they serve.
The SCI – Baseline and 6-Month Follow-up are designed primarily as a paper and pencil interview. The interview form will use electronically scannable TeleForm technology to reduce data entry burden and errors. The client interview will be administered onsite by either the grantee program or the grantee’s local evaluator. Once the interview is complete, the administrator will place the completed survey into a sealed, postage-paid envelope and return it to the contractor. Once received by the contractor, the form will be scanned into a dataset. Scanning these forms will eliminate the need for data entry, thereby reducing cost and the potential for data error.
The Stakeholder Survey will be administered via the web. Before any web-based data collection begins, SAMHSA will secure a system authorization to operate, which includes a security assessment and privacy impact assessment. Each survey respondent will be issued a username and password to access the web-based survey for their program. To complete the survey, each respondent will login to a secure web-based form to fill out the survey. The web-based survey will reduce burden on the respondent and minimize potential for measurement error. For example, skip patterns and automatic data quality checks (e.g., range checks) can be coded into the online survey form to improve data quality.
SAMHSA monitors the performance of CABHI projects by requiring the grantees to collect and submit data through the GPRA (OMB No. 0930-0208). The SCI Baseline and 6-Month Follow-up cover some of the same domains as the GPRA and NOMS data (e.g., housing and criminal justice) but there is no duplication of data that will be collected from the SCI Baseline and the 6-Month Follow-up that can be obtained from the GPRA data. The GPRA data cover a previous 30-day time frame that is not robust enough to accurately assess the association between treatment services and outcome measures, which are both primary objectives of the evaluation. The SCI Baseline and the 6-Month Follow-up questions are unique from the GPRA and NOMS questions in that time frames are extended from assessing the previous 30 days to assessing the previous 6 months. This timeframe extension was strongly endorsed by expert panelists.
The contractor conducted an extensive literature review to confirm that the data collected through the SCI Baseline, 6-Month Follow-up, and the Stakeholder Survey would not be duplicative of any ongoing national or state-level data collection efforts. Data collected in this evaluation are not available from other sources and will be unique because of the scale and breadth of the initiative’s implementation: nationwide, across a spectrum of provider settings, and across a broad cross-section of populations.
Participation in this evaluation will not impose a significant impact on small entities. CABHI grantees may include state agencies, tribal organizations and other jurisdictions, local governments, and community service providers. Some of the community service providers may be small entities; however, the CABHI data collection instruments are designed to include only the most pertinent information needed to be able to carry out the evaluation effectively, and their impact will not be significant.
CABHI grantees are required by SAMHSA to administer a baseline and 6-month follow-up GPRA interview to all clients admitted to their CABHI projects. The contractor will ask the administrator of the GPRA interview to also administer the SCI Baseline and 6-Month Follow-up immediately following the GPRA interview. Since they will already be interviewing the client and expected by SAMHSA to support the CABHI evaluation as a part of accepting grant funding, the additional interview will not add a significant amount of burden to the grantees.
SCI Baseline and 6-Month Follow-up: A client-level interview will be administered on a voluntary basis to clients who receive services through a CABHI project. Only those clients who complete the initial baseline interview will be asked to complete a 6-month follow-up interview. Data collection at these follow-up points is necessary to measure short- and longer-term client outcomes.
Following up at 6 months is optimal for producing useful outcome data. Waiting until 6 months after the initial receipt of services allows enough time for effects of CABHI services to develop, including changes in housing status and stability, substance use behavior, mental health symptoms, and secondary outcomes, such as criminal justice involvement, employment, and trauma symptoms.
Stakeholder Survey: The Stakeholder Survey will be administered once per year to stakeholders involved in CABHI projects. Each year, stakeholders and partners will complete the survey’s corresponding wave (e.g., Wave 1 is completed in Year 1). This data collection frequency allows the evaluation to collect relevant data during the grant’s early implementation, full implementation, and close-out phases. Further, annual data collection is expected to improve data quality as respondents will have less recall bias (e.g., questions on collaboration prior to the grant will be easier to answer in the first year than the third).
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on March 14, 2017 (82 FR 13641). No comments were received.
SAMHSA has made extensive use of experts in the area of homeless research, including current and previous CABHI grantees, to provide guidance on the design and analysis plan of the evaluation. Expert review of the SCI was obtained during its original development in December 2009 and expert panel meetings were held regularly to review analysis of the SCI data during the evaluation of previous CABHI, SSH, and CABHI projects. The experts provided feedback on all aspects of the evaluation and their comments and suggestions were incorporated into the development of the surveys. The list of experts is provided in Exhibit 5. To support adapting the SCI for CABHI grantees, expert panel members provided further review and comment during a December 2016 meeting.
Exhibit 5. Expert Panel Members
Expert |
Affiliation |
Contact Information |
Margarita Alegría, PhD**
|
Harvard
Medical School |
Phone:
617-503-8447 |
Peggy Bailey, MA |
Corporation
for Supportive Housing |
Phone: 917-596-6337
|
Gary Bond, PhD |
Dartmouth
Psychiatric Research Center Professor of Psychiatry |
Phone: 603-448-0263
|
Brian Dates, MA* |
Southwest
Counseling Solutions |
Phone: 313-841-7442
|
(continued)
Exhibit 5. Expert Panel Members (continued)
Expert |
Affiliation |
Contact Information |
Anne Fletcher |
U.S.
Department of Housing & Urban Development, Office of Policy
Development & Research |
Phone: 202-402-4347
|
Linda Frisman, PhD |
Connecticut
Department of Mental Health and Addiction Services |
Phone: 860-418-6788
|
Louis Kurtz, MEd*,** |
Division
of Behavioral Health |
Phone: 502-564-4456 E-mail: [email protected]
|
William McAllister, PhD** |
Institute
for Social and Economic Research and Policy |
Phone: 212-854-5781 E-mail: [email protected]
|
(continued)
Exhibit 5. Expert Panel Members (continued)
Expert |
Affiliation |
Contact Information |
Stephen Metraux, PhD*,** |
Department
of Health Policy & Public Health |
Phone: 215-596-7612
|
Roger H. Peters, PhD |
Department
of Mental Health Law and Policy, University of South
Florida |
Phone: 813-974-9299
|
Laura Prescott, BA |
President
and Founder |
Phone: 413-212-4343
|
Alan Rabideau |
Consumer
and Family Member Consultant |
Phone: 906-495-7158
|
Michael Rowe, PhD |
Program
for Recovery & Community Health |
Phone: 203-764-8690
|
(continued)
Exhibit 5. Expert Panel Members (continued)
Expert |
Affiliation |
Contact Information |
David L. Shern, PhD |
President
and Chief Executive Officer |
Phone: 703-813-1747 |
David Smelson, PsyD* |
University
of Massachusetts Medical Center, Professor of Psychiatry |
Phone: 508-856-3768
x 15122
|
Mary E. Smith, PhD |
Decision
Support, Research and Evaluation |
E-mail: [email protected]
|
Sally J. Stevens, PhD*,** |
Southwest
Institute for Research on Women |
Phone: 520-626-9558
|
Sam Tsemberis, PhD*
|
Pathways
to Housing, Inc. |
Phone: 212-289-0000 x1101 E-mail: [email protected]
|
*Prior GBHI, SSH, and CABHI grantees
**Participated in December 2016 meeting.
SCI Baseline and 6-Month Follow-up: CABHI clients, who are homeless individuals with substance use and/or mental health disorders, are typically a hard-to-reach transient population. To increase response rates, all clients who agree to participate in the client interview at baseline will receive a gift card worth a $15 value. Clients who complete the baseline will be asked to complete a 6-month follow-up interview. Clients who agree to participate in the 6-month follow-up will receive a gift card worth a $30 value (e.g., gift card). Respondents will not be penalized if they wish to skip questions or stop the interview during either the baseline or 6-month follow-up. Survey research literature suggests that monetary incentives have a strong positive effect on response rates and no known adverse effect on reliability. Research has shown improved response rates when remuneration is offered to respondents. Results from the 2001 National Household Survey on Drug Abuse (NHSDA) incentive experiment were reported by Wright, Bowman, Butler, and Eyerman (2005); key conclusions from their analyses are summarized below:
The $20 and $40 incentive payments each produced about a 10-point gain in overall response rates when compared with the $0 control group. The overall response rate was significantly higher for $40 than the $20 incentive within many of the subgroups addressed in the analysis. Both incentive payment groups more than paid for themselves due to decreased costs of follow-up and more productive screening resulting from the improved response rates. Incentives motivate (or obligate) respondents to admit to substance use that they might not have admitted without the incentive.
During the Homeless Programs evaluation, grantees noted similar findings, indicating that client enrollment has increased when compared to other prior GPRA enrollment, and that the incentives are a primary reason for successful follow-up rates.
Stakeholder Survey: No cash incentives or gifts will be given to respondents.
Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all study components. CABHI evaluation team members have extensive experience protecting and maintaining the privacy of respondent data. All data will be securely stored on a protected server.
SCI Baseline and 6-Month Follow-up: The process of administering the SCI Baseline and 6-Month Follow-up is designed to protect client privacy, reduce client discomfort and burden, and ensure that the collected data are of the highest quality. Grantee staff will collect the SCI data immediately following the administration of the SAMHSA-required GPRA interview. The contractor will provide training and technical assistance to all CABHI grantees to detail the steps involved in administering the client interview and the procedures to follow to ensure protection of respondent’s rights and safeguarding of client data. Grantee programs will be provided with a Client Interview Script, a Client Interview Consent Form, a Client Interview Process and Procedures Guide, a Question-by-Question Guide, and a Frequently Asked Questions (FAQ) Guide.
To begin the SCI Baseline or 6-Month Follow-up, the interview administrator (hereafter referred to as “administrator”) will provide the client with a brief introduction to the interview and ask the client if they will agree to hear more. If the client agrees to proceed, the administrator will read the informed consent for the client interview to the client, who will sign it if he or she understands and agrees with its contents. The consent form will explain the purpose of the cross-site evaluation and the interview, describe the interview length and procedures, describe risks or benefits and steps the evaluation is taking to protect the client’s privacy, inform the client of the incentive, and inform them that the interview is voluntary and that he or she may refuse to answer a question or stop the interview at any point without penalty. The consent form will also include the OMB approval expiration dates, the statement of survey burden, and the statement that the study is federally sponsored. This process will take place in a private location to protect client privacy. The administrator will write the CABHI site ID number, the client’s GPRA ID number, and the Interviewer ID number on the first page of the interview. This is the only identifying information the evaluation will have access to; the evaluation will not know the client’s name or be able to connect client interview answers to a particular client.
The SCI Baseline and 6-Month Follow-up each have two parts. In the first part of each interview, the administrator will read the questions to the client and mark the answers on the Scantron form. This part of the interview comprises sections related to housing and homelessness history, criminal justice involvement, treatment history, needs and services, benefits assistance and insurance, and number of children. The second part of the SCI-Baseline and 6-Month Follow-up includes sections related to perception of care, treatment coercion, and treatment choice. These sections will be completed by the client without the administrator present. The client will be provided information about the kinds of questions they will be answering and assistance in the correct way to use the Scantron. The client will again be reminded he or she can refuse to answer questions or stop the interview completely. He or she will also be instructed not to write any identifying information on the form, like their name. If a client is illiterate, the administrator can assist the client in two ways. First, before the client answers anything, the administrator can explain how to answer yes/no questions or Likert scale questions by pointing out what those answers look like or explain which directions imply “better” or “worse.” Second, the administrator may remain in the room with the client but in a location that prevents the administrator from seeing the client’s responses. While in the room, the administrator may read each question to the client using a blank copy of the instrument that is not the instrument the client is filling out. As needed, the administrator may remind the client of the answer format and may point out what the answer options look like using the blank instrument. In the event this happens, the administrator will be instructed to follow two rules: (1) consistently remind the client to protect or hide their instrument or answers while the administrator is helping them using the blank instrument and (2) always point out or describe all possible answer choices for a given question to reduce the potential for bias. Once the client completes this portion of the survey, he or she will place the survey into a tamper proof/evident, postage-paid envelope and return it to the administrator who will mail both sections to the contractor for processing. Once received, they will both be scanned into a secure dataset.
All clients who complete the SCI Baseline will be asked to participate in the SCI 6-Month Follow-up. If they agree, the client will be given another informed consent outlining the same content as the baseline consent form. Again, they will be informed that participation is voluntary, and they will not be penalized for non-participation. The 6-month follow-up will be administered by the grantee staff in the same Scantron format as the baseline following the same procedures outlined above. Client interviews will be identified only with the client GPRA number, which will be necessary to link the baseline data with the 6-month follow-up data and to link the GPRA data with the SCI Baseline and 6-Month Follow-up data; no personally identifying information will be given to the contractor.
Stakeholder Survey: The contractor will obtain limited contact information for stakeholders, including full name and e-mail address, to notify them of the survey. Stakeholders will be contacted through e-mail and issued a username and password to access the web-based survey for their grantee program. Each respondent will login to a secure web-based form to complete the survey. They will also be given the grantee program’s identification number, which they will be asked to enter during the web survey. This will be the only identifying information linked to the stakeholders’ responses, which will be used to link the responses to the appropriate grantee program. The stakeholders will be required to give electronic informed consent before they begin answering questions. At no point will survey responses be linked to a specific stakeholder.
For all data collection activities, the contractor will use passwords to safeguard all project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. Contractor staff will also be trained on handling sensitive data and the importance of privacy. In keeping with 45 CFR 46, Protection of Human Subjects, the CABHI procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent. The informed consents meet all federal requirements for informed consent documentation. This template will be customized by each grantee to obtain informed consent for participation in the study. The contractor’s Institutional Review Board (IRB) has deemed the CABHI evaluation data collection efforts as program evaluation and not research. Any necessary changes to the surveys will be reviewed by the contractor’s IRB.
Data from the CABHI client interviews will be safeguarded in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.
SCI Baseline and 6-Month Follow-up: The client interviews, by necessity, will collect sensitive information about homelessness, substance abuse, mental health, and criminal justice involvement as these are all outcomes of interest to SAMHSA. The client interview will ask clients about trauma symptoms they may be experiencing, but they will not be asked about specific traumatic events. If these questions cause any distress for the client, the interview administrator will connect them with someone from the grantee program who they can speak with. Also, two sections included in both the SCI Baseline and the 6-Month Follow-up interviews, Perception of Care and Treatment Choice, will be self-administered to eliminate discomfort a client may feel in giving their feedback about the program to program staff. Sensitive information of this nature is always regarded as private, and privacy for clients in federally assisted treatment programs is assured through strict adherence to Federal Regulation 42 CFR, Part 2. All client interviews will be conducted in a private space and the administrator will first obtain consent for participation. Respondents will be informed about the purpose of the data collection and that responding to all interview questions is voluntary. They will be assured that they may stop taking the interview at any time without forfeiting the incentive and without penalty from the grantee program. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them. Respondents’ names or other personally identifying information will not be linked to data collected.
Stakeholder Survey: No sensitive information will be collected from the grantee stakeholders.
Estimate the annualized hour burden of the collection of information from clients. The total client sample size for the Client Interview data collection effort is estimated to be a maximum of 5,827 respondents based on grantee target enrollment numbers. The 6-month follow-up survey is expected to have a response rate of 80% of the baseline sample, leaving 4,662 respondents with baseline and follow-up data. Exhibit 6 presents estimates of annualized burden based on preliminary testing. As evidenced from the testing, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information, the total estimated time to complete the baseline survey is 25 minutes. The 6-month follow-up survey adds one section on project status, services received, and cost and is estimated to take 30 minutes to complete.
Exhibit 6. Annual Cross-Site Data Collection Burden for the Client Interview – Baseline, Client Interview – 6-Month Follow-up, & Stakeholder Survey
Instrument/ Activity |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Hourly Wage |
Total Respondent Costa |
Baseline data collection (Clients) |
5,827 |
1 |
5,827 |
0.42 |
2447 |
$7.25 |
$17,743 |
6-month follow-up data collection (Clients) |
4,662 |
1 |
4,662 |
0.5 |
2331 |
$7.25 |
$16,898 |
Client Interview Subtotal |
5,827b |
|
10,489 |
|
4,778 |
|
$34,641 |
Stakeholder Survey |
780 |
1 |
780 |
0.41 |
320 |
$36.95 |
$11,824 |
Total |
6,607b |
|
11,269 |
|
5,098 |
|
$46,465 |
aTotal respondent cost is calculated as hourly wage × time spent on survey × total number of responses.
bEstimated number of total unique respondents.
Estimate the annualized hour burden of the collection of information from grantee stakeholders. The total stakeholder sample size is estimated to be 780 (approximately 10 responses per 78 potential CABHI grantees and sub-recipients as each cohort of CABHI grantees is expected to have 39 grantees and sub-recipients). The stakeholder web survey is estimated to take 25 minutes to complete. Exhibit 6 presents estimates of annualized burden based on experience during the Homeless Programs evaluation.
Estimate the annualized cost burden to the respondent for the collection of information from clients. There are no direct costs to respondents other than their time to participate in the interview. The total time respondents spend completing these surveys is 4,778 hours which includes baseline client respondent hours plus follow-up respondent hours. Multiplying by $7.25, the federal minimum wage, the total cost per year is approximately $34,641.
Estimate the annualized cost burden to the respondent for the collection of information from stakeholders. There are no direct costs to respondents other than their time to participate in the study. The total time respondents spend completing these surveys is 320 hours. Multiplying by $36.95, the estimated average hourly wages for individuals working in professional managerial occupations as published by the Bureau of Labor Statistics (2008) inflated to 2016 value, the total cost per year is approximately $11,824.
There are no respondent costs for capital or start-up or for operation or maintenance.
The annualized cost to the government is approximately $1,450,552. The estimated 4-year total cost to the government for the data collection is $5,802,208. This includes approximately $1,438,574 per year (or $5,754,295 total) for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and learning laboratories; contractor labor for managing data collection; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $11,978 per year (or $47,912 total) represents SAMHSA costs to manage/administer the data collection and analysis for 10% of one employee (GS-14-4, $119,776 annual salary).
Currently there are 4,006 total burden hours in the OMB inventory. SAMHSA is requesting 5,098; an increase of 1,092 hours. This is due to the response burden times have been revised to reflect real-world experience during the Homeless Programs evaluation. Additionally, for the Stakeholder Survey, up to 78 grantees are expected to participate each year in the evaluation. Exhibit 7 shows the changes in burden for each data collection tool.
Exhibit 7. Annual Burden Changes
Instrument |
Original Number of Respondents |
Original Burden Hours |
New Number of Respondents |
New Burden Hours |
Explanation |
SCI Baseline |
5,885 |
1,942 |
5.827 |
2,447 |
Hours per response was increased to 0.42 from 0.33 |
SCI 6-Month Follow-up |
4,708 |
1,883 |
4,662 |
2,331 |
Hours per response was increased to 0.5 from 0.4 |
Stakeholder Survey, All Waves |
648 |
181 |
780 |
320 |
Hours per response was increased to 0.41 from 0.28. Up to 78 grantees will participate in the evaluation each year compared to 25 for the Homeless Programs Evaluation |
Time Schedule: Exhibit 8 outlines the key time points for the study and for the collection of information. The requested period also allows for training and start-up activities associated with the preparation for data collection.
Exhibit 8. Time Schedule for Entire Project
Activity |
Time Schedule |
Obtaining OMB approval for extension of data collection |
Spring 2017 |
SCI Baseline and 6-month Follow-up Implementation |
Spring 2017 |
Stakeholder Survey Data Collection Begins |
Spring 2017 |
All Client Interview Data Collection Ends |
Spring/Summer 2019 |
Stakeholder Survey Data Collection Ends |
Winter 2018 |
Data analysis |
Ongoing (2017 – 2020) |
Dissemination
of findings |
Ongoing (2017 – 2020) |
Publications: The evaluation is designed to produce knowledge about the implementation and impact of CABHI projects. It is therefore important to prepare and disseminate reports, concept papers, documents, and oral presentations that clearly and concisely present project results so that they can be appreciated by both technical and nontechnical audiences. The contractor will
produce rapid-turnaround analysis papers, briefs, and reports;
prepare and submit quarterly technical progress reports, annual briefings, and annual progress reports;
prepare special reports in concert with SAMHSA and expert panel input (e.g., the contractor plans to submit a “portrait” of the CABHI grantee and client characteristics);
prepare final cross-site findings report, including an executive summary;
deliver presentations at professional and federally sponsored conventions and meetings; and
disseminate reports and materials to entities inside and outside SAMHSA.
For SCI data, the primary analytic approach will be multilevel latent growth models with random intervention effects. Analyses will test the statistical significance of changes in outcomes from baseline to follow-up and test whether putative predictors of changes in outcomes are indeed significantly related to changes in outcomes. In its simplest form, this framework can be considered hierarchical linear modeling (HLM) in which client outcomes are reflected in their pre-post change and random effects are used for clients within projects. For dependent variables that require different distributional treatment (e.g., binary or count outcomes), the simple form of the model is GLMM (generalized linear mixed models). Within these, we can estimate overall outcomes and test whether client or project characteristics or other factors influence outcomes (moderation). We anticipate exploring a variety of moderating or predictive factors, including
baseline client severity in terms of substance use and mental health;
grantee characteristics, such as organization type (likely constructed from latent class analysis described below);
project characteristics, such as service bundles (also based on latent class analysis), degree of use of EBPs, and stage of implementation over the life of the project; and
community/system factors like breadth and strength of collaboration of the project with community partners and providers.
Within this framework, if feasible, we will estimate models to assess how a client’s outcomes change given the number and type of actual services received.
An additional feature of this approach would be the incorporation of random intervention effects at the grantee level. Instead of an overall effect of services that would be fixed across grantees, each grantee would have its own estimated effect. This is done to allow for—and predict—variation in treatment (model, service, or EBP) effects that depend on the designs that each grantee chose, which would presumably yield different effect sizes across grantees.
Stakeholder responses to the web-based surveys will provide crucial information on grantee implementation, including models of service delivery, collaboration among partners, and perceived effectiveness. Analysis of stakeholder survey data will take three main forms.
First, descriptive statistics will be used to portray how grantees implement the grant projects, including the configurations of partners from various disciplines or types of organizations, how various partners deliver services in different areas (e.g., substance abuse and mental health treatment, housing and housing supports, and other supports), how well and how frequently partners collaborate, and how stakeholders perceive local implementation of the grant (e.g., the strategies and approaches used, the extent of involvement by key players). The stakeholder survey will typically include multiple respondents within each grant. For some analyses, responses on a given variable will be averaged to form a single value for the grant project. For other analyses, the variability among respondents within each grant project will be of interest.
Second, analyses will assess the extent to which the frequency of collaboration changes. Data from Wave 1 will be used to look at change in collaboration from before the grant to the early stages of the grant. Later, data from Wave 3 (which will be administered in the final year of the grants) will be added to look at changes in collaboration throughout the grant. Some analyses of changes in collaboration will use average levels of collaboration across all respondents within each grant project; other analyses will focus on certain subgroups of partners (e.g., do housing partners report great increases in collaboration) or partner dyads (e.g., between housing partners and substance abuse treatment providers).
Third, data from the stakeholder survey will be used in analyses of the extent to which various aspects of grant implementation are associated with greater improvement in client outcomes. Web survey responses will provide program-level independent variables that will be incorporated into the client-level outcome model described above. For example, do grant projects in which partners report higher levels of collaboration achieve better improvements in outcomes than grant projects with lower levels of collaboration? As a first step toward these models, methods such as latent class analysis and factor analysis will be used to reduce the complexity of the underlying data to facilitate use in analyses assessing associations between implementation and outcomes.
The OMB approval expiration date will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | doc prep |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |