PATH_Evaluation_OMB_SS_PartA 8.28.17
PATH_Evaluation_OMB_SS_PartA 8.28.17.docx
Evaluation of the Projects for Assistance in Transition from Homelessness (PATH) Program
OMB: 0930-0381
EVALUATION OF THE PROJECTS FOR ASSISTANCE IN TRANSITION FROM HOMELESSNESS (PATH)
SUPPORTING STATEMENT
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for the federally mandated National Evaluation of the Projects for Assistance in Transition from Homelessness (PATH). The data collection activities described in this package include five new data collection activities which include web surveys, telephone interviews and site visits with guided discussions. The five new data collection activities are the:
State PATH Contact (SPC) Web Survey
PATH Intermediary Web Survey
PATH Provider Web Survey
PATH Site Visit Discussion Guides
PATH Telephone Interview Guide
The PATH grant program, created as part of the Stewart B. McKinney Homeless Assistance Amendments Act of 1990, is administered by SAMHSA’s CMHS’ Homeless Programs Branch. The PATH program is authorized under Section 521 et seq. of the Public Health Service (PHS) Act, as amended. The program also aligns with SAMHSA’s Recovery Support strategic initiative and addresses Healthy People 2020 Mental Health and Mental Disorders Topic Area HP 2020-MHMD.
Background of the PATH Program
Since 1991, the SAMHSA PATH program has funded the 50 states, the District of Columbia, Puerto Rico, and four U.S. Territories (the U.S. Virgin Islands, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands) (referred to as PATH grantees). For Fiscal Year (FY) 2016, $61,260,644 was available for PATH grantee funding (SAMHSA, 2016). Funding is allocated based on a formula detailed in Section 524 of the original authorizing legislation (Sections 521–535 of the PHS Act) that determines some state’s share based on the ratio of the state population living in urbanized areas compared with the total U.S. urban population. Some states and territories (e.g., District of Columbia), receive a minimum allotment of $300,000 while the U.S. Virgin Islands, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands each receive $50,000. States (but not the territories) are required to match federal PATH funds with at least $1 in cash or in-kind services for every $3 in federal funds. Grantees can use up to 20 percent of their PATH funds to provide limited housing assistance and no more than four percent for administrative expenses (SAMHSA, 2016).
The program’s goals are to strengthen and increase referrals and linkages to permanent housing that support recovery for people who are homeless or at imminent risk of homelessness and have serious mental illness (SMI) or co-occurring SMI and substance use disorders. PATH funds are available to provide a range of allowable services to enable members of this target population to: “secure safe and stable housing, improve their health and live a self-directed, purposeful life” (SAMHSA, 2016). Additionally, the FY 2016 Funding Opportunity Announcement (FOA) encourages PATH grantees to prioritize services for veterans, to work to reduce disparities and to migrate their reporting to the Homeless Management System (HMIS) by the end of the state FY 2016. HMIS is a locally administered, electronic data collection tool that stores person-level information about individuals who access the homeless service system (Freeman, 2010). Participation in HMIS is required by the U.S. Department of Housing and Urban Development (HUD) from local Continuums of Care (CoC), a competitively awarded program to help ensure that communities are addressing homelessness comprehensively with resources from other agencies (Freeman, 2010). PATH grantees make grants to local, public and non-profit organizations to provide the PATH allowable services, shown in Table 1.
Table 1. PATH Allowable Services
PATH Allowable Services |
Outreach services |
Screening and diagnostic treatment services |
Habilitation and rehabilitation services |
Community mental health services |
Alcohol or drug treatment services |
Staff training—including the training of individuals who work in shelters, mental health clinics, substance abuse programs, and other sites where homeless individuals require services |
Case management services |
Supportive and supervisory services in residential settings |
Referral for primary health services, job training, educational services, and relevant housing services |
Housing services as specified in Section 522 (b) (10) of the PHS Act, including: Minor renovation, expansion, and repair of housing; Planning of housing; Technical assistance in applying for housing assistance; Improving the coordination of housing services; Security deposits; Costs associated with matching eligible homeless individuals with appropriate housing situations; and one-time rental payments to prevent eviction. |
Overview of the PATH Evaluation
In 2016, SAMHSA funded a 4-year evaluation of the PATH program through September 2020. While the PATH evaluation is funded under the Evaluation of SAMHSA’s CABHI Program, it is a separate and distinct evaluation and includes two evaluation tasks. The first evaluation task is to meet the mandates of Section 528 of the PHS Act which requires the SAMHSA Administrator to evaluate the expenditures of PATH grantees at least once every three years to ensure they are consistent with legislative requirements and to recommend changes to the program design or operations. The second evaluation task is to conduct additional data collection and analysis to further investigate the sources of variation in key program output and outcome measures that are important for program management and policy development.
Shown in Figure 1 is a simple logic model for the evaluation of the PATH program. Examined will be the: context or inputs (independent variables) that may impact how the PATH programs (at the grantee and provider level) function. This includes the populations served, resources that are used to perform program activities and contextual variables that may influence program operation (mediating variables); throughputs or the mechanisms that are in place for program activities and outputs to occur; activities and outputs of the PATH programs (at the grantee and provider level); and outcomes of the PATH program.
Figure 1. Simple Logic Model for the PATH Evaluation
The PATH evaluation will include structure/process and outcome evaluation components. Measures of structure and process will be used to characterize the grantees/providers, the systems within which the PATH program is embedded, the grantees’/providers’ relationships with other stakeholders, the target population identified for services, services provided and received, program planning and implementation, and monitoring by grantee and intermediary PATH staff. The costs associated with grant services and activities will also be captured by the evaluation.
The outcome evaluation will focus on the outputs and the outcomes of the PATH Program. The outputs of the PATH program include: the number of persons receiving PATH-funded services, outreached/contacted and enrolled; the number of services provided; and the number of referrals provided. The outcome evaluation will be limited, given limitations in available data and will include the number of persons referred to and attaining substance use treatment, primary health services, job training, educational services, housing services, housing placement assistance, income assistance, employment assistance and medical assistance.
As previously noted, the PATH evaluation includes two components to address separate tasks of the evaluation. The two evaluation tasks are described below in more detail.
PATH Triennial Evaluation Component. The first evaluation task, which is referred to as the PATH Triennial Evaluation Component, will be conducted to meet the mandates of Section 528 of the PHS Act. This component, at a minimum, must determine:
Are services funded with PATH monies appropriate?
Are services well administered?
Have outcome and process goals been achieved? Measures include:
The number of homeless persons contacted
The percentage of eligible contacted homeless persons with SMI who are subsequently enrolled in services
The percentage of enrolled homeless persons receive community mental health services
Six evaluation questions (EQs) for the PATH Triennial Evaluation Component cover the three interrelated domains: EQ1 through EQ5 address structures and processes; and EQ6 focuses on outcomes. The EQs are shown in Table 2.
Table 2. Evaluation Questions (EQs) for the PATH Triennial Evaluation Component
Structure/Process |
EQ1: In what contexts do grantees (states/providers) operate? |
EQ2: How are grantees (states/providers) defined by key characteristics? |
EQ3: How were programs implemented and barriers and challenges overcome? |
EQ4: What services models were provided, why and how? |
EQ5: What costs were associated with grant services and activities? |
Outputs/Outcomes |
EQ6: What are the outputs and outcomes of the programs? |
The most recent PATH Triennial Evaluation Report was finalized in 2016. The next PATH Triennial Evaluation Report needs to be finalized in 2019.
PATH Enhanced Evaluation Component. The second evaluation task, which is referred to as the PATH Enhanced Evaluation Component is to conduct additional data collection and analysis to further investigate the sources of variation in key output and outcome measures. The PATH Enhanced Evaluation Component is intended to capitalize on, and build upon, the substantial body of evidence obtained in the most recent triennial evaluation results presented in the 2016 PATH Evaluation Report. This enhanced evaluation is a response to the substantial variation in outcomes across local-area providers revealed by preliminary data analyses. Table 3, illustrates this variability for a selection of measures.
Table 3. Descriptive Statistics for Key Measures
|
(A) |
(B) |
(C) |
(D) |
(E) |
|
N |
Valid |
528 |
521 |
521 |
510 |
462 |
Missing |
0 |
7 |
7 |
18 |
66 |
|
Mean |
365.81 |
71.72 |
74.71 |
62.52 |
65.21 |
|
Standard Deviation |
554.44 |
23.49 |
23.27 |
27.23 |
25.68 |
|
Minimum |
.00 |
3.19 |
2.83 |
.00 |
.00 |
|
Maximum |
4872.00 |
100.00 |
100.00 |
100.00 |
100.00 |
|
Per-centiles |
20 |
82.93 |
51.86 |
55.52 |
36.46 |
41.18 |
40 |
147.80 |
70.05 |
71.58 |
58.48 |
61.02 |
|
60 |
262.47 |
82.71 |
85.72 |
74.53 |
75.48 |
|
80 |
467.07 |
93.85 |
100.00 |
90.12 |
90.38 |
|
Data Source: PATH Annual Report Data 2013-2015, data are averaged across all years of data reported by a provider (2013-2015)
* Defined as a person who spent the prior night in a place not intended for human habitation or in a short-term shelter
The EQs for the Enhanced Component of the PATH evaluation are shown in Table 4 and are focused on explaining sources of variation in key output and outcome measures of the PATH program.
Table 4. Evaluation Questions (EQs) for the PATH Enhanced Evaluation Component
Outputs/Outcomes |
EQ1: What explains variations in the percentage outreached/contacted individuals who were eligible? |
EQ2: What explains variations in the percentage of eligible individuals who became enrolled? |
EQ3: What explains variations in the percentage of outreached/contacted individuals who were literally homeless? |
EQ4: What explains variations in the percentage of outreached/contacted individuals who were literally homeless who became enrolled in the PATH program? |
EQ5: What explains variations in the percentage of outreached/contacted individuals who were veterans? |
EQ6: What explains variations in the percentage of veterans outreached/contacted who became enrolled in the PATH program? |
EQ7: What explains variations in the percentage of enrolled individuals who were assisted through SSI/SSDI Outreach, Access, and Recovery (SOAR)? |
EQ8: What explains variations in the percentage of providers who met annual targets (i.e., numbers outreached/contacted, enrolled and who were literally homeless)? |
EQ9: What explains variations in levels and effectiveness of collaboration with the CoCs and other community resources? |
EQ10: What explains variations in effective use of available data to manage programs and monitor quality? |
EQ11: What explains variations in the percentage of referrals made and attained? |
The PATH evaluation will use web surveys, telephone interviews and site visits to facilitate the collection of information regarding the structures and processes in place at the grantee and provider level (described in Section A.2). Primary data collection will allow the investigation of these areas using data from key PATH stakeholders (administrators, direct care staff, and consumers).
The outputs and outcome data will be obtained from grantee applications and providers’ intended use plans (IUPs) which provide some data for previous FY, and from PATH annual report data, which is also required by Section 528 of the PHS Act. The PATH grantees are required to provide annual data in four areas: budget and organizational context, numbers of persons served by the PATH program, types of services provided with program funds, and basic demographic and clinical characteristics of program consumers. The data are submitted by PATH provider organizations through a web-based data collection system. The 2016 PATH Evaluation Report reported data for the period of 2010 through 2012. The next stage of the evaluation will examine data from annual reports for the years 2013 through 2015. The collection of the PATH annual report data to be included in the evaluation was approved under OMB No. 0930-0205.
The number of PATH grantees and providers for the period of 2013 through 2015 are shown in Table 5.
Table 5. The Number of PATH Grantees and Providers, 2013-2015
|
2013 |
2014 |
2015 |
Grantees |
56 |
56 |
56 |
Providers* |
499 |
492 |
489 |
*Source: 2013-2015 PATH annual data reports, PATH funded provider data
Purpose and Use of Information
The primary users of the data to be collected and reported for the PATH evaluation are staff in SAMHSA/CMHS’ Homeless Programs Branch. The information to be collected will be used for two primary purposes: 1) To meet the mandates of Section 528 of the PHS Act which requires the SAMHSA Administrator to evaluate the expenditures of PATH grantees at least once every three years to ensure they are consistent with legislative requirements and to recommend changes to the program design or operations; and 2) To collect information that helps explain and better understand variations among providers on key program measures that are important for program management and policy development. SAMHSA requested $64.6 million for the PATH program in its FY 2017 Budget Request.
Described below are the five new data collection instruments that are the focus of this OMB request and that fall into three categories: a) web surveys; b) site visit guides; and b) telephone interview guides.
a) Web Surveys
Web-based surveys will be utilized to capture detailed, structured information from PATH grantees, intermediaries and providers. Three web surveys have been developed for the PATH evaluation.
State PATH Contact (SPC) Web Survey: The SPC Web Survey will be utilized to capture detailed, structured information from the SPCs or a comparable staff person from all 56 grantees. The SPC Web Survey will collect information regarding: the grantee organization; the SPC (role, length of time as SPC, time spent working on PATH, other responsibilities); types of organizations and roles of intermediaries within the PATH program; populations served; the PATH allowable or eligible services provided and whether they are a priority service (i.e. that is prioritized within the PATH program or that is a focus of the PATH program); selection, monitoring and oversight of PATH providers; sources for match funds; provision of training and technical assistance; implementation of Evidence Based Practices (EBPs) and innovative practices including SOAR; data reporting, use of data and HMIS; collaboration, coordination and involvement with CoCs and with other agencies, state and national organizations; and ratings of PATH Program features (e.g., fostering of interagency collaboration) (see Attachment 1). This survey will be administered once during the study period.
PATH Intermediary Web Survey: Some PATH grantees provide PATH funds through intermediary organizations (i.e., County agencies or regional behavioral health authorities). A representative of the intermediary organizations that are involved with the PATH program will be asked to complete the PATH Intermediary Web Survey. The PATH Intermediary Web Survey will collect comparable information to what will be collected through the SPC Web Survey regarding: the intermediary organization; the survey respondent (role, length of time in role, time spent working on PATH, other responsibilities); roles within the PATH program; populations served; the PATH allowable or eligible services provided and whether they are a priority service (i.e. that is prioritized within the PATH program or that is a focus of the PATH program); selection, monitoring and oversight of PATH providers; sources for match funds; provision of training and technical assistance; implementation of EBPs and innovative practices including SOAR; data reporting, use of data and HMIS; collaboration, coordination and involvement with CoCs and with other agencies, state and national organizations; and ratings of PATH Program features (e.g., fostering of interagency collaboration) (see Attachment 2). This survey will be administered once during the study period.
PATH Provider Web Survey: The PATH Project Director (PD) or comparable person for all PATH providers will be asked to participate in the PATH Provider Web Survey. On average, there are close to 500 PATH providers each year (see Table 5). The PATH Provider Web Survey will collect information regarding: the organization and program (e.g., type of organization, funding history); the respondent (role, time spent working on PATH, other responsibilities); sources for match funds; populations served; outreach methods utilized; services provided; implementation of EBPs and innovative practices including SOAR; housing types available to PATH clients; provision and receipt of training and technical assistance; data reporting, use of data and HMIS; collaboration, coordination and involvement with CoCs and other organizations; and ratings of PATH Program features (e.g., fostering of interagency collaboration) (see Attachment 3). This survey will be collected once during the study period.
b) Site Visit Guides
Following the same procedures used successfully in the prior PATH Triennial Evaluation, site visits will be conducted with a purposive sample of PATH grantees and providers to collect more nuanced information than will be possible with the web survey. Sites will be selected that represent a wide array of characteristics including: geographic area and federal allocation size. The site visits will be utilized to collect information regarding: provider and state characteristics; practices and priorities; context within which the grantees and providers operate; and services available within the areas the providers operate. Focus groups will be held with current or former consumers of the PATH program to obtain consumer perspectives regarding the impact of the programs.
We have selected categories of individuals for interviews during the site visits. These individuals fill a number of different roles and each will be able to provide a different perspective on the PATH program. Seven discussion guides were developed to conduct semi-structured interviews with stakeholders from the PATH programs during the site visits (see Attachment 4). The sessions to be conducted during the site visits are described below.
Opening Session with State Staff: An Opening Session with the SPC and other relevant grantee staff (e.g., Directors, SPC supervisors) as determined by the SPC to get an understanding of the grantee agency, the context for the PATH program and how the PATH program fits in with the wider state homeless efforts. Discussed will be the successes, barriers, and strategies for: collaborating and network building with federal and national resources, other state agencies and programs, and local provider organizations; and implementation of HMIS and SOAR across the State (see Attachment 4.1).
SPC Session: A session with the SPC to gather detailed information about the grantee’s management and oversight of the PATH program, strategies related to technical assistance and training, use of data for quality assurance and will include discussion of successes, barriers, and strategies for collaboration and coordination across the state and provider systems (see Attachment 4.2).
State and Provider Stakeholder Session: A session with staff from other agencies or divisions (e.g., staff involved with a Statewide HMIS system) and from the intermediary organizations that provide oversight and monitoring of the PATH program. At the provider level, sessions will take place with staff from other agencies (e.g., subcontractor staff, CoC staff) that are stakeholders of the provider’s PATH programs to understand services provided, how services are coordinated, and facilitators and challenges to service delivery (see Attachment 4.3).
Opening Session with Provider Leadership Staff: A session with the leadership and other relevant staff (e.g., CEO, Chief Operating Officer, Program Directors) from the provider organization to get an understanding of the agency and context for the PATH program. These interviews will provide us with the opportunity to understand how the PATH program operates within the context of the larger provider organization. Discussed will be the successes, barriers, and strategies for: collaborating and network building with other service providers in their local area related to serving PATH consumers and use of HMIS and SOAR with PATH consumers (see Attachment 4.4).
PATH Provider Project Director (PD) Session: A session with the PATH project director or comparable staff at the PATH provider agency, to gather detailed information regarding the operation of PATH program (see Attachment 4.5).
PATH Direct Care Staff Session: A session with PATH outreach workers, case managers, and treatment staff/providers (e.g., clinicians or nurse practitioners) to understand services provided, how services are coordinated, and facilitators and challenges to service delivery (see Attachment 4.6). These staff have the closest contact with program consumers and are in the best position to provide insight into the day to day challenges faced and the successes achieved on a day to day basis.
PATH Consumer Focus Group: A focus group with consumers (project participants) to understand their experience with homelessness or being at-risk for homelessness, services received through the PATH program and other service providers including assistance in obtaining benefits (SSI, SSDI, Medicare, and Medicaid), and level of satisfaction with PATH services (see Attachment 4.7).
Site visits will occur once during the study period. Five grantees will be visited and during each visit, and within each grantee, up to two providers will be visited. The site visits will last between two and three days, which will depend on the number of providers visited.
c) Telephone Interviews
As part of the Enhanced evaluation component of the PATH evaluation telephone interviews will be conducted with a sample of SPCs, Intermediaries and providers. The telephone interviews will be conducted to explore through open-ended questions in greater detail, explanations for variations among providers in measures that are important for program management and policy development.
The intent of the Enhanced Evaluation component is to explore potential sources of variation in program performance across a variety of measures identified as important by SAMHSA (see Table 4). A purposive sample of respondents will be selected for this data collection. After review of data from the Annual Reports, grantees’ applications, providers’ IUPs and the web surveys, we will identify those measures that exhibit considerable variation in performance. A small number of providers who exhibit high scores on a particular measure(s) as well as providers who score markedly lower on that measure(s) will be selected for these interviews. The interview will focus on those modules of the interview guide that are most relevant for understanding variation on these particular measures.
One telephone interview guide has been developed for the PATH evaluation.
PATH Telephone Interview: The PATH Telephone Interview will be utilized to explore with SPCs and staff from the intermediaries and provider organizations, policies, procedures and practices that may explain variations in the measures identified for the PATH Enhanced Evaluation Component (see Table 4). The PATH Telephone Interview will explore barriers, facilitators and lessons learned in implementing the PATH program (Attachment 5). This telephone interview will be administered once during the study period.
Data from the web surveys, site visits and telephone interviews will be analyzed along with data from secondary sources including grantees’ annual applications and PATH providers’ IUPs, PATH annual report data, data from the U.S. HUD Annual Homeless Assessment (AHAR) Report, and census data. The secondary data sources will provide contextual data which will be included in both components of the PATH evaluation as appropriate. The contractor has received the grantees’ annual applications and providers’ IUP and the 2013-2015 PATH annual report data from SAMHSA. AHAR data are publicly available at https://www.hudexchange.info/programs/hdx/guides/ahar/#reports and census data is publicly available at https://www.census.gov/2010census/data/.
Shown in Table 6 are the data sources that will be utilized to address the EQs for the Triennial Evaluation Component of the PATH Evaluation.
Table 6. Data Sources for Addressing the Evaluation Questions in the Triennial Evaluation Component of the PATH Evaluation
EQ |
Data Sources |
||||||
Web Surveys |
Site Visits |
Telephone Interviews |
Applications /IUPs |
Annual Data |
AHAR Data |
Census Data |
|
EQ1: In what context do grantees (states/providers) operate? |
|
|
|
|
|
|
|
EQ2: How are grantees (states/providers) defined by key characteristics? |
|
|
|
|
|
|
|
EQ3: How were programs implemented and barriers and challenges overcome? |
|
|
|
|
|
|
|
EQ4: What services models were provided, why and how? |
|
|
|
|
|
|
|
EQ5: What costs were associated with grant services and activities? |
|
|
|
|
|
|
|
EQ6: What are the outputs and outcomes of the programs? |
|
|
|
|
|
|
|
Shown in Table 7 are the data sources that will be utilized to address the EQs for the Enhanced Evaluation Component of the PATH Evaluation.
Table 7. Data Sources for Addressing the Evaluation Questions in the Enhanced Evaluation Component of the PATH Evaluation
EQ |
Data Sources |
||||||
Web Surveys |
Site Visits |
Telephone Interviews |
Applications /IUPs |
Annual Data |
AHAR Data |
Census Data |
|
EQ1: What explains variations in the percentage of individuals outreached/contacted who were eligible? |
|
|
|
|
|
|
|
EQ2: What explains variations in the percentage of eligible individuals who became enrolled? |
|
|
|
|
|
|
|
EQ3: What explains variations in the percentage of individuals outreached/contacted who were literally homeless? |
|
|
|
|
|
|
|
EQ4: What explains variations in the percentage of individuals outreached/contacted who were literally homeless who became enrolled in the PATH program? |
|
|
|
|
|
|
|
EQ5: What explains variations in the percentage of individuals outreached/contacted who were veterans? |
|
|
|
|
|
|
|
EQ6: What explains variations in the percentage of veterans outreached/contacted who became enrolled |
|
|
|
|
|
|
|
EQ7: What explains variations in the percentage of enrolled individuals who were assisted through SOAR? |
|
|
|
|
|
|
|
EQ8: What explains variations in the percentage of providers who met annual targets? |
|
|
|
|
|
|
|
EQ9: What explains variations in the level and effectiveness of collaboration with the COCs and other community resources? |
|
|
|
|
|
|
|
EQ10: What explains variations in effective use of available data to manage programs and monitor quality? |
|
|
|
|
|
|
|
EQ11: What explains variations in the percentage of referrals made and attained? |
|
|
|
|
|
|
|
The SPC, Intermediary and Provider web surveys will be administered via an on-line data collection system. Before any Web-based data collection begins, SAMHSA will secure a system authorization to operate, which includes a security assessment and privacy impact assessment.
Using a web instrument allows for automated data checks as well as for skip procedures which will reduce the burden among respondents and possibility of data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will help respondents give valid responses and ensure that responses follow the expected format. Responses will generate skip patterns for later questions in the instrument, where the respondents only complete relevant sets of questions based on their previous responses and do not see others. Using a web-based system also provides the capability to send automatic email reminders to grantees when surveys have not been completed.
The web based system will comply with the requirements of Section 508 of the Rehabilitation Act to permit accessibility to people with disabilities.
Telephone Interviews and In-person Interviews (Site Visits)
The other two data collection instruments submitted for OMB clearance will be used by evaluation project staff during telephone interviews, site visit discussions and focus groups. Evaluation staff will read the questions to the respondents and a note-taker will record the responses. With respondent consents, the interviews will be recorded as back-up to the note-taker. The interview recordings will be stored on a secure, password protected computer and server and will be deleted once the interview responses are considered final.
There are two mandatory components in the PATH legislation. The first component is that all PATH funded entities must prepare and submit an annual report on program accomplishments (persons served, PATH-eligible services provided, referrals provided, demographics for PATH consumers) and how PATH dollars are spent (dollar amount (PATH federal and match funds) for services dedicated to persons who are homeless or at risk of homelessness and serious mental illness, number staff and full time equivalent (FTE) staff supported by PATH federal and match funds). This data collection is approved under OMB No. 0930-0205.
The second mandatory component is a triennial evaluation of the PATH program. The previous PATH triennial evaluation was approved under Evaluation of Programs to Provide Services to Persons Who Are Homeless with Mental and/or Substance Use Disorders (Homeless Programs)—OMB No. 0930-0339. Prior to that approval, the triennial evaluation of the PATH program was approved under OMB No. 0930-0332.
The data collection proposed for this evaluation is not available elsewhere, is not duplicative, and is critically valuable for assessing the PATH program consistency with legislative requirements and to recommend changes to the program design or operations. The five data collection instruments developed for the triennial PATH evaluation are unique and differ from what is collected via the PATH annual report and build on data utilized for the last triennial evaluation which was conducted under OMB No. 0930-0320 and OMB No. 0930-0339 and which have been revised to exclude the PATH program.
The data collection proposed for this evaluation does not have a significant impact on small entities. Most of the data will be collected from PATH program management, provider staff, and consumers involved in the program. Some of the PATH providers may be small entities; however, the information to be collected will not have significant impact on the providers and is needed to fulfill the statutory requirement and planning needs of SAMHSA/CMHS.
Failure to collect the proposed data will prevent SAMHSA/CMHS from meeting its obligation under Section 528 of the authorizing legislation, which calls for a triennial evaluation to evaluate the expenditures of PATH grants to ensure that they are consistent with legislative requirements and to recommend changes to the program design or operations.
The proposed data collection complies with 5 CFR 1320.5(d) (2).
Consultation Outside the Agency
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on June 26, 2017 (82 FR 28871). No comments were received.
A Technical Panel was convened to provide guidance on the design of the PATH Enhanced Evaluation Component of the PATH Evaluation. Feedback from the Technical Panel was incorporated in revisions made to the design of the evaluation. The Technical Panel was comprised of SPCs and an expert with expertise with homelessness, Veterans Administration, CoCs and HMIS. The Technical Panel will be convened to provide guidance during data reporting. The list of Technical Panel members and their affiliation is provided in Table 8.
Table 8. Technical Panel Members
Member Name |
Affiliation |
Bob Currie |
SPC Tennessee Department of Mental Health 426 Fifth Avenue, North, Nashville, TN 37244 Phone: (615) 532-4651 Email: [email protected] |
Robert Snarr |
SPC Utah Dept. of Human Services Division of Substance Abuse and Mental Health 195 North 1950 West Salt Lake City, UT 84116 Phone: (801) 538-4080 Email: [email protected] |
Melodie Pazolt |
SPC Washington Department of Social and Health Services Division of Behavioral Health and Recovery Services P.O. Box 45330 Olympia, WA 98504 Phone: (360) 725-0487 Email: [email protected] |
Rhonda Thissen |
SPC Virginia Department of Behavioral Health and Developmental Services 1220 Bank Street Richmond, VA 23218 Phone: (804) 786-2316 Email: [email protected] |
Mark Johnston |
Consultant Mark Johnston Consulting Phone: (540) 272-272629 Email: [email protected] |
The data collection instruments included under this request, are modifications of instruments utilized in the last PATH triennial evaluation and which were approved under OMB No. 0930-0339. The instruments were utilized with grantee and provider staff and consumers of the PATH programs.
Payment to Respondents
The PATH evaluation includes the collection of data from consumers of homeless services. Focus groups will be held with consumers during site visits to PATH programs. Focus group participants will be offered a nominal incentive of a $20 gift card. The local provider will select the type of gift card that is appropriate for that specific community. The gift cards will be provided as a courtesy for the time and effort spent participating in the site visit interviews.
No other respondents for the proposed data collection activities will be paid for participating in the evaluation.
To ensure the privacy and protection of respondents’ rights, the data collection protocol and instruments will be submitted for review and approval by an Internal Review Board (IRB).
The following procedures will be in place to ensure the security and protection of all data collected. All respondents will provide consent to participate in the evaluation. Respondents will be informed that: the data being collected is sponsored by SAMHSA; the purpose and uses of the evaluation results; that their participation is voluntary and that they do not have to participate in the data collection activities and can skip questions; and that all individual responses will be kept private and that data will be aggregated so that responses will not be identifiable by individual or organizational names. Data from the online surveys will be collected through a dedicated Windows-based secured server that utilizes industry standard secure SSL encryption and firewall protection against unauthorized access to data. Access to the online surveys and data will be username and password protected. Data collected from the site visits and telephone interviews will also be stored in files that are password protected and access will be limited to individuals who have a need to work on them. All hardcopies of forms and notes will be kept in secure, locked cabinets.
Most of the data being collected through this data collection effort through the web surveys, telephone interviews, and the site visit discussions is not of a sensitive nature. Some data will be collected from current or former consumers of the PATH programs during focus groups that may be of a sensitive nature. The purpose of the data collection is to gather information on topics related to homelessness, mental health and substance abuse which are important topics to SAMHSA.
All data collection tools will be reviewed by the contractor’s IRB (FWA #16073) and data collection for each tool will not begin until it is approved or exempted. Informed consent will be obtained from all respondents, including the consumers participating in the consumer focus group (see Attachment 4). The informed consent for consumers will be provided to PATH providers by email prior to the visit. At the beginning of the focus group, the contractor will read the consent form to the consumers and respond to questions. Signed consents forms will be collected before the focus group formally begins and the tape recorder is turned on. The consent forms for all data collection activities, including the focus groups are included in Attachments 1-5 and include the points noted above in Section A.10.
Estimates of Annualized Hour Burden
The estimated burden for data collection is 1,228.5 hours over the four-year evaluation period. Using May 2015 National Occupational Employment and Wage Estimates from the Bureau of Labor Statistics, U.S. Department of Labor (http://www.bls.gov/oes/current/oes_nat.htm), the estimated total cost to respondents is $33.033.33. For the SPCs, intermediary and program management staff for PATH providers we utilized the mean hourly salary for Social and Community Service Manager ($33.38) (https://www.bls.gov/oes/current/oes119151.htm ). For PATH program staff (outreach workers/case managers) we utilized the mean hourly salary for Community and Social Service Occupations ($22.19) (https://www.bls.gov/oes/current/oes210000.htm). Utilized for the consumers was the federal minimum wage of $7.25 (https://www.dol.gov/whd/minimumwage.htm ).
Table 9a provides the basis of the resulting estimates of the hour burden of collection of information, based on the proposed protocols and survey instruments. The hourly burden estimates are based on previous experience with similar versions of these instruments used in the prior PATH Evaluation.
Table 9a. Data Collection Burden
Instrument/ Activity |
Number of Respondents |
Responses per Respondent |
Total Responses |
Hours per Response |
Total Hour Burden |
Hourly Wage Rate |
Total Hour Cost ($) |
Web Surveys |
|||||||
SPC Web Survey |
561 |
1 |
56 |
1 |
56 |
$30.38 |
$1,701.28 |
PATH Intermediary Web Survey |
282 |
1 |
28 |
1 |
28 |
$30.38 |
$850.64 |
PATH Provider Web Survey |
5003 |
1 |
500 |
1 |
500 |
$30.38 |
$15,190.00 |
Telephone Interviews |
|||||||
SPC Telephone Interviews |
284 |
1 |
28 |
1 |
28 |
$30.38 |
$850.64 |
PATH Intermediary Telephone Interviews |
145 |
1 |
14 |
1 |
14 |
$30.38 |
$425.32 |
PATH Provider Telephone Interviews |
606 |
1 |
60 |
1 |
60 |
$30.38 |
$1,822.8 |
Site Visit Interviews |
|||||||
Opening Session with State Staff |
257 |
1 |
25 |
2 |
50 |
$30.38 |
$1,519.00 |
SPC Session |
58 |
1 |
5 |
2 |
10 |
$30.38 |
$303.80 |
State Stakeholder Session |
259 |
1 |
25 |
1.5 |
37.5 |
$30.38 |
$1,139.25
|
Opening Session with Provider Leadership Staff |
5010 |
1 |
50 |
2 |
100 |
$30.38 |
$3,038.00 |
PATH Provider PD Session |
1011 |
1 |
10 |
2 |
20 |
$30.38 |
$607.60 |
PATH Provider Direct Care Staff Session |
5012 |
1 |
50 |
2 |
100
|
$22.19 |
$2,219.00 |
Provider Stakeholder Session |
5013 |
1 |
50 |
1.5 |
75 |
$30.38 |
$2,278.50 |
Consumer Focus Groups |
10014 |
1 |
100 |
1.5 |
150 |
$7.25 |
$1,087.50 |
Total |
1,001 |
- |
1,001 |
- |
1,228.5 |
- |
$33.033.33 |
1 1 respondent X 56 SPCs =56 respondents
2 1 respondent X 28 Intermediaries= 28 respondents
3 1 respondent X 500 PATH providers=500 respondents
4 1 respondent X 28 SPCs =28 respondents
5 1 respondent X 14 Intermediaries= 14 respondents
6 1 respondent X 60 PATH providers=60 respondents
7 5 respondents X 5 site visits=25 respondents
8 1 respondent X 5 site visits=5 respondents
9 5 respondents X 5 site visits =25 respondents
10 5 respondents X 10 site visits (2 providers per state) =50 respondents
11 1 respondent X 10 site visits (2 providers per state) =10 respondents
12 5 respondents X 10 site visits (2 providers per state) =50 respondents
13 5 respondents X 10 site visits (2 providers per state) =50 respondents
14 10 respondents X 10 site visits (10 Consumers per provider (2 providers per state) =100 respondents
Table 9b. Data Collection Burden Summary
Instrument/Activity |
Number of Respondents |
Total Responses |
Total Hour Burden |
SPC Web Survey |
56 |
56 |
56 |
PATH Intermediary Web Survey |
28 |
28 |
28 |
PATH Provider Web Survey |
500 |
500 |
500 |
PATH Site Visit Discussion |
315 |
315 |
542.5 |
PATH Telephone Interviews |
102 |
102 |
102 |
Total |
1,001 |
1,001 |
1,228.5 |
Estimates of Annualized Cost Burden to Respondents
There are no costs to respondents associated with either (a) capital or startup efforts or (b) operation and maintenance of services.
Estimates of Annualized Cost to the Government
The estimated cost to the government for data collection is $697,912. This includes approximately $650,000 for contractor costs and $47,912 costs for SAMHSA to manage the survey for 10% of one employee (GS-14-4, $119,776 annual salary). The annualized cost is approximately $232,637.
This is a new collection of information.
Time Schedule, Publications, and Analysis Plan
Time Schedule
Planning for the PATH Evaluation began in September 2016. It is anticipated that data collection using the web surveys will begin within 8 weeks of receiving OMB approval and data collection is expected to last through Year 3 of the project (November 2017-June 2019). Table 10 outlines the data collection time schedule for the PATH evaluation.
Table 10. Time Schedule for Data Collection
Activity |
Month/Year |
Project Year |
Web Surveys |
||
|
September-October 2017 |
Year 2 |
|
November-December 2017 |
Year 2 |
|
December 2017-January 2018 |
Year 2 |
|
January- February 2018 |
Year 2 |
Site Visits |
||
|
February- March 2018 |
Year 2 |
|
May-August 2018 |
Year 2 |
Telephone Interviews |
||
|
September-November 2018 |
Year 3 |
|
January-February 2019 |
Year 3 |
|
March-June 2019 |
Year 3 |
Year 2: September 2017-September 2018 (Option Year 1)
Year 3: September 2018 – September 2019 (Option Year 2)
Year 4: September 2019 – September 2020 (Option Year 3)
Table 11 outlines the data analysis and reporting time schedule for the PATH evaluation.
Table 11. Time Schedule for Data Analysis and Reporting
Activity |
Month/Year |
Evaluation Year |
Data Analysis |
||
|
September 2017- April 2019 |
Year 2-3 |
|
September 2018- April 2020 |
Year 2-3 |
Reporting |
||
|
July 2019 |
Year 3 |
|
August 2019 |
Year 3 |
|
July 2020 |
Year 4 |
|
August 2020 |
Year 4 |
Year 2: September 2017-September 2018 (Option Year 1)
Year 3: September 2018 – September 2019 (Option Year 2)
Year 4: September 2019 – September 2020 (Option Year 3)
Analysis Plan
Data analysis will take place throughout the project. The analysis plan for both components of the PATH evaluation reflect the multifaceted and comprehensive nature of the objectives, evaluation questions, and data sources. Our analyses will include qualitative and quantitative techniques and reflect the logic model and ecological frame presented in Section A.1. and in Figure 1. The questions to be answered by the evaluation are noted in Section A.1. Below we discuss the analysis plans for the PATH Evaluation.
Qualitative Data. Qualitative data for the PATH evaluation will come mostly from the Site Visit Guides and the Telephone Interviews. Notes from the site visits and telephone interviews will be inductively analyzed with the assistance of software such as NVivo or Dedoose, using a grounded theory approach (Charmaz 2006). Analysis will be done concurrently with data collection. The qualitative analysis team will meet daily to discuss emerging themes and to confirm reliability and validity of the emerging coding structure. Emerging themes will be mapped onto the associated evaluation questions. The qualitative analysis will be conducted to identify: facilitators to implementation and operation of the PATH program; the types of barriers encountered during implementation and strategies developed to deal with barriers; collaboration among systems, providers and other community agencies and organizations; and to understand consumer’s experience with homelessness or being at-risk for homelessness, services received through PATH program and other service providers and levels of satisfaction with PATH services.
Quantitative Data. Quantitative analyses for the PATH Triennial Evaluation Component will be primarily descriptive with an emphasis on characteristics that vary across grantees and providers. Frequencies, percentages and means and correlations will be the primary quantitative tools for these. Some variables will be continuous; others will be discrete. Characteristics that are measurable and meaningfully comparable across grantees and providers will be used to create continuous or categorical variables that will be included in statistical models.
Descriptive statistics will be utilized to characterize the grantees and providers, the contexts and systems within which the PATH programs are embedded, the grantees and providers’ relationships with stakeholders and other community providers, the target population identified for services, services provided and received, program planning and implementation, and the monitoring by grantee and intermediary staff.
Mixed-Method Approach for PATH Enhanced Evaluation Component. The measures selected for the enhanced evaluation component are quantitative while a large portion of the hypothesized sources of variation in the selected measures is derived from narrative reports and other qualitative sources of information. This necessitates a mixed-methods approach that combines qualitative and quantitative analytic techniques.
The first step will be a qualitative analysis of the narrative information on sources of variation. This step will include (a) identification of dimension(s) (or constructs) along which variation in outputs/outcomes is hypothesized; (b) classification of responses along the identified dimension(s) into meaningful categories; and (c) quantification of the categories by creating ordinal or nominal variables suitable for quantitative analysis.
The quantitative portion of the enhanced evaluation will start with a data reduction technique such as latent class analysis (LCA) to identify a smaller set of latent constructs underlying the potentially large number of manifest variables created by the qualitative analysis outlined above. The results of LCA (e.g. posterior probabilities of membership in the latent classes) can then be used to model each of the 11 measures selected for enhanced analysis. The choice of model will account for the clustering of local-area providers within states/territories using a multilevel approach. Providers will constitute level one and states/territories will be level two. The models will test hypotheses about the effects of both level one and level two predictors on the selected measures as well as possible interactions within and across levels.
Table 12 below summarizes the output/outcome measures selected for further investigation in the PATH Enhanced Evaluation Component and their hypothesized sources of variation. The last column lists the data sources that will be used to test the indicated hypotheses.
Table 12. PATH Enhanced Evaluation Components Measures, Hypothesized Sources of Variation, and Associated Data Sources
Output/Outcome Measure |
Hypothesized Sources of Variation |
Data Sources |
1. The percentage of outreached/contacted individuals who were PATH eligible |
|
|
2. The percentage of eligible individuals who became enrolled |
|
|
3. The percentage of outreached/contacted individuals who were literally homeless* |
|
|
4. The percentage of outreached/contacted literally homeless individuals who became enrolled |
|
|
5. The percentage of outreached/contacted individuals who were veterans |
|
|
6. The percentage of outreached/contacted of veterans who became enrolled |
|
|
7. The percentage of enrolled individuals who were assisted through SOAR |
|
|
8. The number of annual targets met or exceeded by the provider |
|
|
9. Level and effectiveness of interagency collaboration (full operational definition under development) |
|
|
10. Effective use of available data to manage programs and monitor quality (index under development) |
|
|
11. The percentage of referrals that resulted in services received |
|
|
KEY AHAR: Annual Homeless Assessment Report IUP: Local-area providers’ Intended Use Plans |
||
* Definition of literal homelessness may vary across grantees, provided it is not more restrictive than the statutory definition.
See Attachment 7 for sample tables that were reproduced from the 2016 PATH Evaluation Report. Comparable tables will be utilized in the PATH evaluation. The attachment also includes a blank table shell for the enhanced component of the evaluation.
Publications and Dissemination
Reporting will take place during Year 3 (September 2018 – September 2019) and Year 4 (September 2019 – September 2020) of the project. The primary products of the project will be two final reports: the PATH Triennial Evaluation Component Report and the PATH Enhanced Evaluation Component Report. The final reports will focus on findings, conclusions, and recommendations for policy. The evaluation team will also participate in an in-person briefing for SAMHSA staff.
17. Display of Expiration Date
The OMB expiration date will be displayed.
18. Exceptions to Certification for Statement
The collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Teresita Camacho-Gonsalves |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |
© 2024 OMB.report | Privacy Policy