Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for four new data collection instruments for the National Evaluation of the Projects for Assistance in Transition from Homelessness (PATH):
PATH Site Visit Interview Guide: State PATH Contact
PATH Provider Site Visit Interview Guide: Supervisor/Administrator
PATH Provider Site Visit Interview Guide: Outreach Worker/Case Manager
PATH Consumer Focus Group Discussion Guide
The PATH program was created by Congress to help States and Territories provide flexible community-based services for individuals experiencing serious mental illnesses and homelessness (or are at imminent risk of homelessness). The goal of the program is to link persons who are homeless and have serious mental illness (or co-occurring serious mental illness and substance use disorders) to services that facilitate access to treatment to improve mental health functioning and to other services that support ongoing stability. Public Law 101-645, 42 USC 290cc-21 et seq., the Stewart B. McKinney Homeless Assistance Amendments Act of 1990 (section 521 et seq. of the PHS Act, hereafter referred to as “the Act”) established the PATH program and assigned SAMHSA/CMHS responsibility for making monetary allotments. Every fiscal year, CMHS awards grants to each of the States, the District of Columbia, the Commonwealth of Puerto Rico, the Virgin Islands, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands.
This proposed new data collection will provide SAMHSA/CMHS with relevant and standardized information needed for program planning and performance. The current evaluation will answer the following questions:
1. Are services well administered?
2. Have process goals been achieved?
Archival data will be the primary source of data to answer these questions and to help determine relative intervention effectiveness and best practices. Results will enable the Government to better address gaps in service availability and to access and identify environmental/contextual factors that affect process and performance. The use of primary data collection through site visits (which include focus groups) will facilitate the identification of successes, barriers to success, and effective and ineffective program implementation strategies. The data collection will also indicate the perceptions of key PATH stakeholders at the State and local levels related to PATH operations, services, and effectiveness. This knowledge can help further improve PATH program operations.
Section 522 of the Stewart B. McKinney Homeless Assistance Amendments Act of 1990 (section 521 et seq. of the PHS Act) specifies that grantee States and Territories must expend their payments solely for making grants to political subdivisions of the State and to nonprofit private entities (including community-based veterans’ organizations and other community organizations) for the purpose of providing specified services. A wide range of eligible services is identified in the legislation, including outreach; screening and diagnostic treatment; habilitation and rehabilitation; community mental health; alcohol and drug treatment; staff training; case management; supportive and supervisory services in residential settings; referrals for services; and housing services.
Section 528 of the Act specifies that not later than January 31 of each fiscal year, a funded entity will “prepare and submit to the Secretary a report in such form and containing such information as the Secretary determines to be necessary for: (1) securing a record and a description of the purposes for which amounts received under Section 521 were expended during the preceding fiscal year and of the recipients of such amounts; and (2) determining whether such amounts were expended in accordance with the provisions of this part.”
To address this legislation, the PATH program requires States to provide annual data in four main areas: budget and organizational context, numbers of persons served by the PATH program, the types of services provided with program funds, and basic demographic and clinical characteristics of program consumers. Data in the annual report are used to provide the minimum necessary process and outcome measures required for the mandated triennial report to Congress. This submission is for a collection of contextual, process, and outcome information to evaluate the national PATH program. Section 528 of the PHS Act specifies that the Administrator of the Substance Abuse and Mental Health Services Administration shall evaluate at least once every 3 years the expenditures of grants under this part by eligible entities in order to ensure that expenditures are consistent with the provisions of this part. The evaluation shall include recommendations regarding changes in program design or operations. Collectively, the data used to conduct the evaluation, including this new data collection effort, will address the similarities and differences in PATH intervention models used by States and Territories and the differential effectiveness of these models with different homeless populations. By obtaining the perspectives of the PATH administrators and service providers, SAMHSA can gain a better understanding of the PATH program and provide improved guidance to States, Territories, and local service providers to better serve their clients.
The primary purpose of this data collection is to evaluate the PATH program. The data collected with the proposed instruments will:
Supplement analyses and provide better-defined answers regarding program effectiveness in response to this congressionally mandated evaluation.
Inform the PATH program by identifying best practices and promising program models for delivering PATH services to the homeless and those at risk of homelessness.
Enable CMHS to document and assess the accomplishments and lessons learned by the PATH grantees and providers.
Enrich SAMHSA’s understanding of how the States and Territories administer the PATH formula grant by providing qualitative data from those providing the services and those who are recipients of the PATH-funded services.
Improve guidance to assist States, Territories, and local service providers with better tools to better serve their clients and thereby improve client outcomes.
Contribute greatly toward enhancing the mandated triennial report to Congress by offering the new and added perspective of the grantees and local providers engaged in the PATH program administration and in the field.
The data gathered from onsite interviews and focus groups along with detailed analysis of archival data such as census data, the Department of Housing and Urban Development’s (HUD) Homeless Management Information System (HMIS), and other contextual data will be used to provide the mandated triennial report to Congress. For example, the proposed data collection will expand on previous efforts by providing more detailed information on the effects of funding on implementation and performance. Similarly, new priority populations have been included in the initiative (e.g., veterans). However, we know little to nothing about how reprioritizing homeless subpopulations has affected implementation or the effectiveness with which dollars are spent. Finally, States and Territories are beginning to change their assessment and reporting systems to be consistent with HUD/HMIS. Some say this change has had a dramatic effect on assessment and reporting, while others have reported a relatively smooth transition. SAMHSA/CMHS needs to better understand how and where these changes have had an impact and how implementation has been affected to better provide relevant guidance and help grantees improve.
The following section describes the four instruments in detail, how each will be administered (and to whom and why), and what can be gained from this activity. Face-to-face interviews will take place during site visits.
Site Visit Interview Guides
Four site visit protocol and semistructured interview/discussion guides were developed for the purpose of obtaining qualitative data from the field. A total of 10 States will be randomly selected based on selection criteria of geographic region, urbanicity, poverty levels, and PATH funding levels. The SAMHSA Project Officer will approve site selection plans.
One site visit will be conducted with an SPC from each of the 10 States selected. SPCs from selected sites will help the MANILA research team identify providers in their State who will be asked to participate in the interviews with the supervisor/administrator and outreach worker/case manager. After site selection and prior to the site visit, SPCs will be asked to recommend several possible interviewees from each category who are knowledgeable about and involved in PATH program activities. SPC recommendations will be based on providers’ experience as service providers and with advocacy for homeless issues. Once the recommendations are made, the research team will contact one outreach worker/case manager and one supervisor/administrator from the list of possible interviewees for the interviews. When possible, interviewees will be selected randomly from the list of recommended providers.
MANILA site visitors will conduct one to two focus groups with consumers of homelessness services during each site visit. PATH providers will be asked to recommend consumers who are familiar with the agency, who will be open to sharing their experiences with the agency and homelessness, and who are unlikely to exhibit erratic behavior. This will reduce the likelihood of having unstable or unmanageable participants and increase the likelihood of having high-functioning participants who will be cooperative during the focus group. This approach should also increase the reliability and validity of the data collected.
The following four data collection interview guides were pilot-tested with four SPCs and eight providers (supervisors/administrators and outreach workers/case managers) at four local provider sites to (1) ensure clarity and comprehensiveness and (2) assess the burden placed on staff to participate in this data collection activity. Providers interviewed also reviewed focus group discussion guides and provided feedback. The focus group guides were also pretested with three different clients. Following receipt of OMB clearance, site visits will be arranged and administered using the interview and focus group discussion guides described below in 10 States with 10 SPCs, 10 PATH provider sites, and 10–12 clients at each provider site. All interviewees and focus group participants will be assured that their responses will be maintained to preserve their anonymity to the extent the law allows. No personal names will be shown on the notes taken during the focus group discussions. All respondents will be informed that response to any particular question is voluntary. In group discussions, names will not be recorded or attached to particular speakers. Responses will be aggregated to the extent possible so individual answers will not be identifiable. If audiotaping (with interviewees’ permission) is conducted during these interviews, the audio records will be erased after the interview notes have been created and verified.
PATH Site Visit Interview Guide: State PATH Contact. After providing some basic information (history with organization, job responsibilities, role in PATH, and length of time in position), the SPC will be asked about his/her understanding of PATH. He or she will be asked more specifically about his/her perceptions of Federal expectations and training and technical assistance activities related to homelessness, service coordination, collaboration issues, and housing concerns. The SPC will then be asked about recommendations for changes to the PATH program. The interviewee will be asked to review and sign an informed consent form, which includes information about the study and what participation in it entails. Signed consent forms will be stored in a secure location at the contractor’s offices. This interview guide appears in Attachment A.
PATH Provider Site Visit Interview Guide: Supervisor/Administrator. After providing some basic information (history with organization, job responsibilities, role in PATH, length of time in position, homeless services offered, and staffing supported by PATH), the respondent will be asked to share his or her understanding of the PATH program and Federal expectations and to describe how his or her staff determine eligibility for PATH services, how they work and collaborate with other service providers to address clients homelessness, and how they work to help clients transition to mainstream services and housing. He or she will be asked his or her perceptions of the availability of training and the role of the consumer in the organization and in treatment. The respondent will also be asked about emerging trends regarding homelessness. The interviewee will be asked to review and sign an informed consent form, which includes information about the study and what participation in it entails. Signed consent forms will be stored in a secure location at the contractor’s offices. This interview guide appears in Attachment B.
PATH Provider Site Visit Interview Guide: Outreach Worker/Case Manager. After providing some basic information (history with organization, job responsibilities, PATH role, length of time in position), the respondent will be asked to share his or her understanding of the PATH program and Federal expectations and to describe services to PATH clients, what collaboration services are available to address homelessness, and how transition and housing and cultural competency are addressed in the organization. The interview will also assess availability of staff training. The respondent will be asked about emerging trends regarding homelessness, likes and dislikes, and recommendations for changes to PATH. The interviewee will be asked to review and sign an informed consent form, which includes information about the study and what participation in it entails. Signed consent forms will be stored in a secure location at the contractor’s offices. This interview guide appears in Attachment C.
PATH Consumer Focus Group Discussion Guide. Focus groups will be conducted using a facilitated discussion format. The participants will be asked about services received and their experiences with the agency providing the services. Participants will need to sign a consent form that includes an agreement that any personal information that may come out during the discussions will be protected by the evaluators and SAMHSA to the extent of the law. Signed consent forms will be stored in a secure location at the contractor’s offices. Incentives, determined with the provider agency to be appropriate for participation, will be offered. Monetary value of incentives will not exceed U.S. $20. The focus group discussion guide appears in Attachment D.
No data will be submitted electronically. All new data collection instruments submitted for OMB clearance will be used during face-to-face interviews and focus groups on site.
The PATH legislation has two mandatory components. One component specifies that all funded entities must prepare and submit an annual report of what was accomplished and how PATH dollars were spent. This data collection is approved under OMB No. 0930-0205. The second component specifies a mandated triennial evaluation of the PATH program. This is the first time that primary data collection is proposed for the required evaluation, requiring clearance for the following interview guides.
The four site visit interview guides/focus group discussion guides (SPC, PATH Provider Supervisor/Administrator, PATH Provider outreach worker/case manager, and Focus Group participant) developed to collect data for the PATH evaluation are unique. State and local PATH contacts, the PATH Technical Panel, and knowledgeable SAMHSA/CMHS staff have been consulted as to whether the information collected is available elsewhere. All have confirmed the data are unique and valuable. Therefore, the proposed data collection does not duplicate other efforts.
This data collection effort does not have a significant economic impact on a substantial number of small entities. Data will be collected from State PATH program and provider staff. Data will not be collected from a substantial number of small entities. The information collected will be the minimum needed to fulfill the statutory requirement and the planning needs of SAMHSA/CMHS.
Failure to collect the information on this proposed schedule would prevent the PATH program evaluation from meeting its obligations to more clearly determine the process and broad impact of the program. Not having these data would, in large part, prevent SAMHSA/CMHS from improving guidance to PATH grantees regarding improving the efficiency and effectiveness of their efforts. Data will be collected only once during the contract period. Each respondent will be informed that participation is voluntary.
The proposed data collection complies fully with all guidelines of 5 CFR 1320.5.
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on June 8, 2011 (Volume 76, p. 33324). No comments were received during the comment period.
Consultations with various sources outside SAMHSA on the evaluation design, sample design, data sources, and data collection instruments occurred during the research design phase and will continue to take place as needed. The purpose of these consultations is to ensure the integrity of the study design and the relevance of the data collection activities and to maximize the likelihood that the findings of this study will generate valuable information regarding the evaluation of the PATH program.
During the research design phase, the SAMHSA Project Officer, with the support of the contracting agency on this project, convened a Technical Panel of experts. The panel discussed the purpose of the evaluation and reviewed data collection instruments (e.g., data collection guides) to ensure they were written using plain, coherent, und unambiguous terminology and were understandable to those who will be responding to them. The panel offered valuable recommendations regarding the evaluation design, the measures that should be assessed, and the processes involved to obtain the data necessary for a meaningful evaluation.
Technical Panel
SAMHSA staff and Technical Panel members who have provided guidance on the PATH evaluation design and related data collection instruments are listed below:
C. H. Hank Balderrama
Washington State Department of Social and Health Services
Division of Behavioral Health and Recovery
Denise Barber
Florida Department of Children and Families
Mental Health Program Office
Monica Bellamy
Housing Coordinator
Michigan Department of Community Health
Community Living
Administration for Children and Families
Charles Boyle
Bureau Chief, Division of Mental Health and Addiction
Indiana Department of Mental Health and Addiction
Bernie Bluhm
Program Planning and Review Specialist
New Hampshire Department of Health and Human Services
Bureau of Homeless and Housing Services
Michael R. Newman
New York State Office of Mental Health
Housing Services Unit
Michael Shank
Virginia Department of Behavioral Health and Developmental Services
Office of Mental Health Services
Gary Travis
Mental Health Program Consultant
Department of Human Services
Adult Mental Health Division
Sheldon Wheeler
Department of Health and Human Services
Adult Mental Health Services
Dana Woolfolk
National Coalition for the Homeless Board Member
Faces of Homeless Speakers Bureau/NCH
Federal Representatives
Pamela Fischer
Social Science Analyst
Government Project Officer for the PATH Evaluation
Department of Health and Human Services, SAMHSA/CMHS
Deborah Stone
Public Health Advisor
Alternate Task Order Officer for the PATH Evaluation
Department of Health and Human Services, SAMHSA/CMHS
MANILA Staff
Betsy McDonel Herr
Project Director, Year 1
MANILA Consulting Group, Inc.
Lisa Kleppel
Project Director, Years 2–3
MANILA Consulting Group, Inc.
Paul Brounstein
Technical Director
MANILA Consulting Group, Inc.
Lawrence Rickards
Expert Consultant
MANILA Consulting Group, Inc.
Sarah Paige Fuller
Consultant
MANILA Consulting Group, Inc.
The national evaluation of the PATH program includes focus groups of consumers of homelessness services who are either homeless or at risk of homelessness. These consumers are a “hard-to-reach” population for data collection purposes. As part of CMHS’s strategy to obtain a high response from consumers of homeless services who will be offered participation in the focus group, each person who participates will be offered a nominal incentive of a gift card in the value of $20. The local provider will select a type of card that is appropriate for that specific community. The card can be considered a courtesy for the time and effort spent participating in the focus group.
The use of incentives is based on a review of the literature, which shows that incentives, even when small in monetary terms, are effective in increasing response rates (Armstrong, 1975; Church, 1993; Goyder, 1994).1 Such studies also show that rather than negatively affecting data quality, the quality of the data is improved because there are fewer instances of item nonresponse and more comments to open-ended questions (James & Bolstein, 1990; Brennan, 1992; Shettle & Mooney, 1999).2
No other respondents for these data collection activities will be paid for participating in the evaluation. Participation in the PATH triennial national evaluation is completely voluntary.
All individual data will be collected in accordance with the Privacy Act of 1974 (5 U.S. Code [U.S.C.] 552a), SAMHSA Participant Protection requirements, and other Federal and Department of Health and Human Services [HHS] regulations on the protection of human subjects (e.g., 5 U.S.C. 301; 42 U.S.C. 289(a)).
The research team obtained Institutional Review Board (IRB) approval in August 2011 so that human subject protections are assured. No respondent identifiers will be made available from the research. When reporting data, the research team will use organization codes rather than organization names, and the data will be aggregated, so the responses will not be identifiable by individual or organization.
Each respondent will be assured that the project will protect the privacy of all respondents to the full extent of the law. The consent form that all interviewees and focus group participants must sign states that participation in the study is strictly voluntary and that individuals have the right to refuse to participate. Respondents will be assured the information will be reported only in aggregate form in reports, that their names and other personal identifiers will not be associated with their answers, and that no one will have access to the information except as may be required by law, regulation, or subpoena, or unless permission is given by the respondent. Hardcopy forms will be held in a locked area for receipt and processing.
A complete Information Technology/Security Plan, consistent with the National Institute of Standards and Technology Special Publication 800-18, Revision 1, Guide for Developing Security Plans for Federal Information Systems, was submitted to SAMHSA in February 2010.
No questions of a sensitive nature are being collected through this data collection effort.
The estimated burden for data collection is 205 hours. Using May 2010 National Occupational Employment and Wage Estimates from the Bureau of Labor Statistics, U.S. Department of Labor (http://www.bls.gov/oes/current/oes_nat.htm#b11-0000), the estimated total cost to respondents is $4,405.
Table 1 provides the basis of the resulting estimates of the hour burden of collection of information, based on the proposed protocols. The basis for these burden estimates is as follows:
Site Visit Interview Guides for SPCs and PATH Providers
Interview with SPCs: Pilot-testing indicates that interviews will require approximately 1.1 hours (65 minutes), though the actual individual range of time required was 45 to 85 minutes (.75 to 1.42 hours). Ten SPCs will be interviewed. Total burden for these interviews is expected to be 11 hours.
Interviews with PATH Supervisors and Providers: Pilot-testing indicates that interviews will require approximately 40 minutes (.67 hours), though the actual individual range of time required during pretesting was from 30 to 45 minutes (.5 to .75 hours). Two provider staff at each of 10 sites (n = 20) will be interviewed, thereby incurring 14 hours of burden.
Total burden incurred for SPC administrators and PATH providers is expected to be approximately 25 hours.
PATH Client Focus Group Discussions: Ninety-minute client focus groups for up to 12 consumers will be held at each provider site visit. The number of focus groups held will be determined by timing, availability of clients, and logistics.
Total burden incurred for all focus group participants is expected to be approximately 180 hours.
Estimates of response burden and costs appear in Table 1 below.
Table 1. Annual Burden
PATH Evaluation |
Number of Respondents |
Responses/ Respondent |
Total Responses |
Hours/ Response |
Total Hour Burden |
Hourly Wage Rate |
Total Hour Cost ($) |
Site Visit Interviews (10 sites) |
|
|
|
|
|
|
|
State PATH Contact |
10* |
1 |
10 |
1.1 |
11 |
$35 |
$385 |
Provider Staff - Supervisor/ Administrator |
10** |
1 |
10 |
.67 |
7 |
$30 |
$210 |
Provider Staff - Outreach Worker/Case Manager |
10*** |
1 |
10 |
.67 |
7 |
$30 |
$210 |
Consumer Focus Group Discussion |
120**** |
1 |
120 |
1.5 |
180 |
$20 |
$3,600 |
Total |
150 |
|
150 |
|
205 |
|
$ 4,405 |
* 1 respondent x 10 sites = 10 total respondents
** 1 respondent x 10 sites = 10 total respondents
*** 1 respondent x 10 sites = 10 total respondents
**** Up to 12 respondents x 10 sites = 120 respondents
There are no costs to respondents associated with either (a) capital or startup efforts or (b) operation and maintenance of services.
The estimated cost to the Federal government of conducting the National Evaluation of the PATH program is based on the Government’s contracted cost of the data collection and related study activities along with the personnel cost of government employees involved in oversight and/or analysis. For the data collection activities for which OMB approval is currently being requested, CMHS personnel are expected to spend approximately 100 hours on activities related to data collection at a rate of $40/hour. This results in an estimated $4,000 in personnel time incurred by the Government. Most of these costs will be incurred in 2011–2012 when data collection is expected to be under way. The PATH evaluation contract (No. 283-07-4004) is charged with interview instrumentation, collection, analysis, and preparing and printing a final report summarizing the data. The total cost to the Government is $767,909. When annualized, the cost to the Government amounts to $260,000 (which includes an estimated $4,000 per year for GPO time) per year.
This is a new data collection.
Pilot site visits were completed and revisions made to the instruments in early January 2011. The OMB clearance package was submitted to SAMHSA/CMHS in late January 2011. It is anticipated the evaluation team will be in the field within 6 weeks of final OMB clearance to conduct the first site visits.
The evaluation team will write annual and final reports that include a synthesis of the evaluation findings. The reports will include qualitative and quantitative analyses of data collected and graphic and tabular displays of the key findings.
The primary product of the evaluation will be a final report in September 2012. This report will cover findings from the data collection activities conducted each year and for all data collection activities at the end of the 3-year contract.
The full time line for the evaluation appears in Table 2 below.
Table 2. Evaluation Time Line
Evaluation Activity |
Schedule |
Establish Technical Panel |
Year 1 |
Convene Technical Panel |
Year 1 |
Develop Evaluation Plan |
Year 1 |
Obtain and code all 2009 State applications and IUPs |
Years 1 and 2 |
Obtain and begin analysis of third-party data sources (e.g., census, ACS, HUD) |
Year 2 |
Obtain and begin analysis of annual report of outcomes for 2009 initiative |
Year 2 |
Develop site visit protocols and data collection instruments |
Year 2 |
Pilot-test all data collection instruments |
Year 2 |
Refine all instruments based on feedback from pilot-testing |
Year 2 |
Submit OMB statement with all proposed data collection instruments |
Year 2 |
Obtain OMB approval to start data collection activities |
Year 3 |
Select site visit locations |
Year 3 |
Arrange logistics and conduct 10 site visits involving SPCs, PATH providers, and consumers |
Year 3 |
Analyze and synthesize all data |
Year 3 |
Draft outline for final report for GPO and Technical Panel approval |
Year 3 |
Submit draft final report for review and feedback |
Year 3 |
Incorporate feedback into final report |
Year 3 |
Submit final report of findings and recommendations to GPO |
Year 3 |
The expiration date will be displayed on all PATH materials with OMB approval.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
This evaluation is based on a mixed methodology. The quantitative data include both secondary source information collected annually as part of the PATH program and related data from other Federal data collections. The qualitative evaluation component will use data extracted from site applications and data collected from site visit assessments.
The PATH program has a universe of 56 funding jurisdictions: 50 States, the District of Columbia, Puerto Rico, and 4 U.S. Territories. Each funding jurisdiction or site has local PATH-funded providers that provide direct services. The number of local PATH-funded providers varies based on the state/territory and is based on the poverty level of the population in that region. The number of locally funded PATH providers remains relatively invariant over time since the resources for PATH formula grants have also remained relatively constant over time. PATH program characteristics and outcome data will be extracted from applications and reports from all States and Territories and providers. No sampling is required; all members of the potential respondent universe will participate in this part of the assessment. Samples drawn from the potential universe of respondents will be required for primary data collections using 10 site visits, which will include visits to SPCs and PATH provider agencies (to interview supervisors/administrators and outreach workers/case managers, and to conduct focus groups with consumers) in those States selected for site visits.
Site Visit Interview Guides
The universe for potential site visits does not include the four U.S. Territories or Puerto Rico because of both cost and limited generalizability from findings drawn from these sites to the continental United States where most funds (98.1%) are expended. Puerto Rico and the other Territories also have unique homelessness issues as a result of their geography, culture, systems of care and other differences that would provide limited information about the overall PATH program. The cost of site visits relative to potential return on investment would be prohibitive. However, Puerto Rico and the four U.S. Territories will be included in data extraction efforts.
Pilot-testing of all instruments took place during site visits to four states at the beginning of Year 2. Four SPCs and eight local providers within the four States were selected to review all questions and provide feedback on the types of questions asked as well as the comprehensiveness and clarity in all data collection instruments. Level of effort and burden were assessed, and instruments were revised to incorporate user feedback.
During the third year of the contract, 10 site visits will be conducted. These visits will focus on identifying PATH implementation models and challenges and best practices at the State and local provider level. Focus groups with consumers and interviews with SPCs and providers will identify successes, barriers to success, and perceptions of PATH operations, services, and effectiveness. Ideally, site visits will provide clear and tangible data concerning what works, what is needed, and how to improve PATH operations.
Ten sites have been selected from the remaining 47 funded sites—the 50 States and the District of Columbia excluding the 4 first-year site visit locations. The 10 sites were selected from a stratified random sample of the 47 eligible PATH initiative State sites. The stratification placed the sites in a matrix using region, urbanicity, poverty level, funding level, and special populations served as potential selection criteria. The 10 sites were chosen randomly from the completed sampling matrix to provide good representation of the breadth of PATH initiatives. Final site selection will be approved by the SAMHSA Task Order Officer and the PATH Technical Advisory Panel.
The 10 site visits will include an interview with the SPC and 2 in-person interviews with locally funded PATH providers (one supervisor/administrator and one outreach worker/case manager). When providers are spread over a large geographic area, interviews may need to be held by audio/video conference calling. There will be 1–2 focus groups including a total of up to 12 PATH clients. These groups will be held at the provider site during the in-person site visit. Clients will be selected from those receiving services at the provider location on the final 2 days of the 3-day assessment. A convenience sample of clients will be used. Service providers will be asked to refer consumers to one of the site visitors who will solicit their participation. Participation will be incentivized. The incentive will have less than a U.S. $20 value and will be approved by the provider prior to soliciting consumer participation.
State PATH Contact and PATH Provider Interviews
A total of 25% of PATH sites (14 of 56 sites) will receive a site visit; this includes the 4 pilot site visits that have already been completed and 10 site visits during Year 3 that have not yet been conducted. With an examination of one-quarter of the sites, the site visits will provide a wide range of first-hand qualitative data collected through interviews and focus groups.
As part of the pretesting effort during Year 2, each site and the corresponding SPC was purposely selected from a stratified sampling frame. The stratification was based on the sites’ region, level of urbanicity, and past success as viewed by the PATH Technical Panel. The PATH Technical Panel, SAMHSA Task Order Officer, and contractor staff selected four sites that have solid experience in PATH implementation that were willing to provide feedback on the initiative process and functioning and on data collection protocols and instrumentation. Stratification ensured that site visit assessments reflected a broad a range of sites based on program context and focus.
During Year 3, site selection (10 sites) for the formal assessment was made from a stratified random sample. The stratification placed the sites in a matrix based on the State/region, level of urbanicity, poverty level, funding, and special populations served. The 10 sites were chosen randomly within the different elements. Each site visit will include interviews with SPCs and key staff implementing PATH at the State level, local PATH-funded providers, and PATH-funded service recipients. These participants will be identified for participation using a snowball sample to ensure they are the most qualified of the possible participants. The qualifications include experience as either a service provider or a service recipient and experience with advocacy for homelessness issues. The SPC will recommend the local PATH-funded providers. These providers will then recommend PATH-funded service recipients. There are no known deficiencies with the sampling frame, and there are no issues with timeliness or completeness. The focus of these in-person assessments is to provide qualitative information regarding the process, experience, and challenges of program implementation, service provision, and receipt of services. The large sample of sites from a stratified random sampling matrix should assure the selection of sample that is representative of the population on factors that have been found to differentiate implementation process (e.g., rural versus urban).
Focus Groups With PATH Clients
Each site visit will include 1–2 focus groups with up to a total of 12 PATH-funded clients/consumers. These participants will be identified for participation using a purposive sample to ensure they are the most qualified of the possible participants. The qualifications will include experience as a service recipient and diagnosis of a mental illness. Providers will identify clients to participate from among those present on site and receiving services.
Up to 12 consumers will be interviewed at each site assessment in 1 or 2 groups, depending on timing, availability of clients, and logistics. Each focus group is time-limited (90 minutes).
Site visit participants include SPCs, local PATH-funded providers, and PATH clients. To make the site visits less burdensome and to increase response rates, the scheduling and location of the interviews will accommodate the participants’ requirements. The scheduling of the site visits and related interviews will be based on the SPCs’ and providers’ needs. The location of the interviews and focus groups will be based on the site’s geography and the needs of the providers. Telecommunications, electronic mail, and audio/video discussions via the Internet will be used to reduce the travel burdens when necessary.
The participation of PATH clients in the focus groups will likely require some incentive. The PATH client focus groups will be held on site during the in-person assessment. Service providers will be asked to refer consumers to participate. The participants will receive an incentive that has a value of less than U.S. $20 and will be approved by the provider prior to soliciting consumer participation.
Both SAMHSA/CMHS PATH Government Task Order Officers and the PATH Technical Panel reviewed all materials used in the initial four pilot-tests. Their comments and concerns were addressed in the revisions made to the data collection protocols and instrumentation used in more formal pilot-testing. Pilot-tests of the instruments occurred during the Year 2 site visits. The four initial site visits resulted in feedback from a total of four SPCs and eight providers. The four initially selected sites represented a purposeful sample. Selections were made to ensure that sites represented key program discriminators (e.g., region, urbanicity, perceived need, innovation, effectiveness in implementation) and to realize the best potential for receiving feedback on protocols, instrumentation, and relevant PATH program information. The SAMHSA PATH Government Task Order Officers and the PATH Technical Panel worked to identify the best sites for pilot-testing data collection protocols and instrumentation.
Site Visit Interview Guides
Pilot sites were chosen randomly using criteria such as geographic distribution, urbanicity, poverty levels, and levels of PATH funding. Prior to each site visit, all respondents were sent copies of the interview questions to be asked during the site visit.
At each of the Year 2 pilot site visits the SPC and two local PATH-funded providers were interviewed and asked to review and comment on the focus group discussion guides, as appropriate. These respondents were asked not just topics for inclusion but also appropriate language and potential sensitivities of consumer focus group respondents. Feedback from these interviews has been incorporated in the finalized instruments attached to this submission.
The contractor for the evaluation has had ample experience with qualitative and quantitative data collection and analysis. SAMHSA staff and the Technical Panel have been an important part of the evaluation process providing feedback concerning the evaluation plan, determination of sites for pilot tests, and data collection procedures. See Table 3.
Table 3. Evaluation Personnel
Name |
Title |
Organization |
Contact |
Contract Staff |
|||
Lisa Kleppel, MPH |
Project Director |
MANILA Consulting Group, Inc. |
571-633-9797 x 209 |
Paul Brounstein, PhD |
Technical Director |
MANILA Consulting Group, Inc. |
571-633-9797 x 234 |
Julie Gloudemans, MS, PhD Candidate |
Senior Program Analyst |
MANILA Consulting Group, Inc. |
571-633-9797 x 277 |
Shauna Harps, PhD |
Deputy Project Director |
MANILA Consulting Group, Inc. |
571-633-9797 x 247 |
SAMHSA Staff |
|||
Pamela Fischer, PhD |
Government Project Officer |
SAMHSA, CMHS |
240-276-1901 |
Deborah Stone, PhD |
Social Science Analyst |
SAMHSA, CMHS |
240-276-2411 |
Dorrine Gross |
Public Health Advisor |
SAMHSA, CMHS |
240-276-1898 |
List of Attachments
Attachment A —PATH Site Visit Interview Guide: State PATH Contact
Attachment B —PATH Provider Site Visit Interview Guide: Supervisor/Administrator
Attachment C —PATH Provider Site Visit Interview Guide: Outreach Worker/Case Manager
Attachment D —PATH Consumer Focus Group Discussion Guide
1 Armstrong, J. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39(1): 111–116; Church, A. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1): 62–79; Goyder, J. (1994). An experiment with cash incentives on a personal interview survey. Journal of the Market Research Society, 36(4): 360–366.
2 James, T., & Bolstein, R. (1990) . The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54(3): 346–361; Brennan, M. (1992). The effect of monetary incentives on mail survey response rates: New data. Journal of the Market Research Society, 34(2): 173–177; Shettle, C., & Mooney, G. (1999). Monetary incentives in U.S. Government surveys. Journal of Official Statistics, 15(2): 231–250.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | MEMORANDUM |
| Author | Monica Capizzi |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |