cooperative agreements to benefit homeless individuals (CABHI) evaluation client & stakeholder Surveys
Supporting Statement
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
SCI Baseline and 6-Month Follow-up: The targeted universe for the Supplemental Client Interview (SCI) is all enrolled and accepted clients who receive services under the CABHI grants. Eligibility for the receipt of CABHI services is limited to homeless individuals with substance use disorders, mental health disorders, or co-occurring substance use and mental health disorders. Based on the target enrollment numbers for the grantees, the expected total number of clients receiving services is 5,827.
Stakeholder Survey: Stakeholder names will be generated based on data extraction from SAMHSA documents (e.g., grantee applications) and reviewed through follow-up with grantees for all CABHI grantee programs. The numbers of grantee partnerships vary but CABHI grantees and sub-recipients are expected to have around 10 essential partners and stakeholders participating on a grantee’s Interagency Council on Homelessness and Steering Committees. These stakeholders will be contacted through email and provided a link to the web survey with a username and password for secure log-in.
SCI Baseline and 6-Month Follow-up:
As described in Section A.6, the SCI will collect data from individuals at baseline and at 6-month follow-up. Data collection at the follow-up point is necessary to measure the short- and longer-term outcomes of the CABHI programs implemented by the grantees. Because measuring these outcomes is one of the primary objectives of the CABHI initiative, less frequent data collection would greatly endanger the utility of the CABHI initiative to all clients.
The process of administering the SCI Baseline and 6-Month Follow-up is designed to protect client privacy, reduce client discomfort and burden, and ensure that the collected data are of the highest quality. CABHI grantee staff will collect the SCI Baseline and 6-Month Follow-up data immediately following the administration of the SAMHSA-required GPRA interview. These grantee interviewers are trained interviewers who have received training on interview administration, participant engagement, participant protection, and tracking procedures, from the grantee as well as from SAMHSA per OMB approved procedures developed for the GPRA measures (OMB No. 0930-0208). Under the current data collection, the contractor will also hold training sessions with all grantees to detail the steps involved in administering the client interview and the procedures to follow to ensure protection of respondent’s rights and safeguarding of client data. Grantee programs will be provided with a Client Interview Script, a Client Interview Consent Form, a Client Interview Process and Procedures Guide, a Question-by-Question Guide, and a Frequently Asked Questions (FAQ) Guide.
To begin the SCI Baseline or 6-Month Follow-up, the administrator will provide the client with a brief introduction to the interview and ask the client if they will agree to hear more. If the client agrees to proceed, the administrator will read the informed consent for the client interview to the client, who will sign it if he or she understands and agrees with its contents. The consent form will explain the purpose of the cross-site evaluation and the interview, describe the interview length and procedures, describe risks or benefits and steps the evaluation is taking to protect the client’s privacy, inform the client of the incentive, and inform them that the interview is voluntary and that he or she may refuse to answer a question or stop the interview at any point without penalty. The consent form will also include the OMB approval expiration dates, the statement of survey burden, and the statement that the study is federally sponsored. This process will take place in a private location to protect client privacy. The administrator will write the CABHI site ID number, the client’s GPRA ID number, and the Interviewer ID number on the first page of the interview. This is the only identifying information the evaluation will have access to; the evaluation will not know the client’s name or be able to connect client interview answers to a particular client.
The SCI Baseline and 6-Month Follow-up each have two parts. In the first part of each interview, the administrator will read the questions to the client and mark the answers on the scantron form. This part of the interview is comprised of sections related to military service, employment, criminal justice, co-occurring disorders, housing and homeless history, housing satisfaction and choice, perception of housing coercion, readiness to change, services needed and received, client treatment burden, and trauma symptoms. The second part of the SCI Baseline and the 6-Month Follow-up includes sections related to perception of care, treatment coercion, and treatment choice. These sections will be completed by the client without the administrator present. The client will be provided information about the kinds of questions they will be answering and assistance in the correct way to use the scantron. The client will again be reminded he or she can refuse to answer questions or stop the interview completely. He or she will also be instructed not to write any identifying information on the form, like their name. If a client is illiterate, the administrator can assist the client in two ways. First, before the client answers anything, the administrator can explain how to answer yes/no questions or Likert scale questions by pointing out what those answers look like or explain which directions imply ‘better’ or ‘worse’. Second, the administrator may remain in the room with the client but in a location that prevents the administrator from seeing the client’s responses. While in the room the administrator may read each question to the client using a blank copy of the instrument that is not the instrument the client is filling out. As needed, the administrator may remind the client of the answer format and may point out what the answer options look like using the blank instrument. In the event this happens, the administrator will be instructed to follow two rules: 1) consistently remind the client to protect or hide their instrument or answers while the administrator is helping them using the blank instrument and 2) always point out or describe all possible answer choices for a given question to reduce the potential for bias. Once the client completes this portion of the survey, he or she will place the survey into a tamper proof/evident, postage-paid envelope and return it to the administrator who will mail both sections to the contractor for processing. Once received, they will both be scanned into a secure dataset.
All clients who complete the SCI Baseline will be asked to participate in the SCI 6-Month Follow-up. If they agree, the client will be given another informed consent outlining the same content as the baseline consent form. Again, they will be informed that participation is voluntary and they will not be penalized for non-participation. The 6-month follow-up will be administered by the grantee staff in the same scantron format as the baseline following the same procedures outlined above. Client interviews will be identified only with the client GPRA number which will be necessary to link the baseline data with the follow-up data and to link the GPRA data with the SCI Baseline and the 6-Month Follow-up. The contractor will not have any contact with the clients between baseline and follow-up. However, follow-up success will benefit from the fact that CABHI grantees are providing case management or other services that keep them in ongoing engagement with the clients. Furthermore, they are conducting their own administrative data collection that requires them to maintain contact with the clients.
Stakeholder Survey: The contractor will obtain limited contact information for stakeholders, including full name and e-mail address, to notify them of the survey. Stakeholders will be contacted through e-mail and issued a username and password to access the web-based survey. Each respondent will login to a secure web-based form to complete the survey. They will also be given the grantee project’s identification number which they will be asked to enter during the web survey. This will be the only identifying information linked to the stakeholder’s responses which will be used to link the responses to the appropriate grantee program. The stakeholders will be required to give electronic informed consent before they begin answering questions.
SCI Baseline and 6-Month Follow-up: The ability to gain the cooperation of potential respondents is key to the success of this endeavor. All grantees are required by SAMHSA to administer the GPRA interview to 100% of clients who enter treatment under the CABHI grants. In addition, a minimum of 80% of clients must also receive the 6-month follow-up GPRA interview. In order to increase the likelihood of client response and ease the burden placed on both client and grantee, the SCI will be administered immediately following the GPRA interview. The contractor anticipates a 100% response rate for the SCI Baseline and a 20% attrition rate for the SCI 6-Month Follow-up. The contractor will employ several strategies to maintain high response rates in the SCI Baseline and 6-Month Follow-up:
Stress the importance of the project as well as the contractor’s commitment to respondent privacy.
Train survey staff for handling sensitive information collection in a respectful manner.
Administer the survey immediately following the administration of the SAMHSA-required GPRA interview.
Offer cash equivalent incentives (e.g., gift cards) for survey response.
Stakeholder Survey: To recruit participants for the stakeholder survey, the contractor will ask grantee project directors to review and confirm a list key partner agencies or organizations and will then contact the confirmed stakeholders and partners to ask that they participate in the survey. This list will be reviewed each year prior to the administration of the survey. Under the Homeless Programs Stakeholder Survey, the evaluation team achieved a response rate of 60% (1,353 respondents of 2,278 invited), with an average of 8 respondents in each of the169 sites. As in that study, it is anticipated that all nominees will have access to the web because they represent agencies or organizations and their involvement in partnering with the grantee is part of their job – and can therefore access the web via a computer in their office. Although web survey respondents will not be provided incentives, stakeholders and partners are often actively involved in the partnership and therefore are typically motivated to share their experiences and perspectives. To be successful and useful, the stakeholder web survey does not need to achieve response rates at the same level of the client interview. The main consideration is that some partners from each site respond; it is not necessary that all, or even most, partners in a site respond. The contractor will use several strategies to achieve sufficient response rates in the stakeholder survey:
Ask grantee project directors to inform their stakeholders and partners about the survey and encourage them to participate.
Send stakeholders and partners an initial email invitation that explains the study and its importance, why they are being asked to participate, how they can contact the contractor for additional information, and how to access the web survey.
Send reminder emails to non-respondents and, if approved by SAMHSA, ask grantee project directors to also encourage non-respondents to participate.
Keep the survey to a reasonable length that encourages participation and will not lead to “word of mouth” comments among nominees that discourage participation.
If needed, allow respondents some other way to take the survey other than over the web (e.g. mailed hard copy or conducted over the telephone).
SCI Baseline and 6-Month Follow-up: Based on real-world experience gained from the Homeless Program’s SCI Baseline and 6-Month Follow-up the baseline interview, including informed consent, takes approximately 25 minutes to complete. The 6-month follow-up interview, including informed consent, takes approximately 30 minutes.
Under the original OMB approval and subsequent extension, baseline interviews were conducted with over 7,000 clients and 6-month follow-up interviews were conducted with over 5,000 clients in the Homeless Programs projects. The interview procedures have worked well, with high participation rates. Grantee staff who administered the client interviews reported no substantial problems but rather that the interviews worked well and were well-received by clients.
Stakeholder Survey: Based on real-world experience gained from the Homeless Program’s Stakeholder Survey, on average, each Stakeholder Survey Wave is expected to take 25 minutes to complete including time for informed consent and instructions. The web survey contains a number of skip patterns and response times will vary, for example, based on the services offered by the stakeholder.
Under the original OMB approval and subsequent extension, 1,353 respondents completed the Homeless Program’s Stakeholder Survey during which the survey procedures functioned well and no substantial problems were reported.
As noted in Section A.8, SAMHSA has consulted extensively with an expert panel who will continue to provide expert advice throughout the course of the evaluation. In addition, the contractor team is comprised of several experts who will be directly involved in the data collection and statistical analysis. Also, contractor in-house experts will be consulted throughout the program on various statistical aspects of the design, methodological issues, economic analysis, database management, and data analysis. Exhibit 1 provides details of these team members and advisors.
Expert |
Affiliation |
Contact Information |
CABHI Evaluation Staff |
||
James Trudeau, Ph.D. Project Director |
Principal Scientist Center for Justice, Safety, and Resilience RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709 |
Phone: 919-485-7751 E-mail: [email protected] |
Arnie Aldridge, Ph.D. Deputy Project Director |
Research Economist Behavioral Health Economics Program RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709 |
Phone: 919-990-8389 E-mail: [email protected] |
Antonio Morgan-Lopez, Ph.D. Data Analysis Task Lead |
Senior Research Quantitative Psychologist Risk Behavior and Family Research RTI International 3040 Cornwallis Road Research Triangle Park, NC 27709 |
Phone: 919-316-3436 E-mail: [email protected]
|
SAMHSA Advisors |
||
Kirstin Painter, Ph.D. Contracting Officer’s Representative |
Public Health Advisor CMHS, SAMHSA 5600 Fishers Lane, Room 14E89D Rockville, MD 20857 |
Phone: 240-276-1932 Email: |
Sarah Ndiangui, M.P.H. Alternate Contracting Officer’s Representative |
Public Health Advisor CSAT, SAMHSA 5600 Fishers Lane, Room 13E85B Rockville, MD 20857 |
Phone: 240-276-2918 Email: |
References
Bassuk, E. L., Weinreb, L. F., Buckner, J. C., Browne, A., Salomon, A., & Bassuk, S. S. (1996). The characteristics and needs of sheltered homeless and low-income housed mothers. Journal of the American Medical Association, 276, 640–646.
Borsari, B., & Carey, K. B. (2000). Effects of a brief motivational intervention with college student drinkers. Journal of Consulting and Clinical Psychology, 68, 728–733.
Browne, A., & Bassuk, S. S. (1997). Intimate violence in the lives of homeless and poor housed women: prevalence and patterns in an ethnically diverse sample. American Journal of Orthopsychiatry, 72, 261–277.
Burt, M. R., Aron, L. Y., Douglas, T., Valente, J., Lee, E., & Iwen, B. (1999). Homelessness: Programs and the people they serve. Washington, DC: Interagency Council on the Homeless.
Goodman, L., Saxe, L., & Harvey, M. (1991). Homelessness as psychological trauma. American Psychologist, 46, 1219–1225.
Kline, A., Callahan, L., Butler, M., St. Hill, L., Losonczy, M. F., & Smelson, D. A. (2009). The relationship between military service era and psychosocial treatment needs among homeless veterans with a co-occurring substance abuse and mental health disorder. Journal of Dual Diagnosis, 5, 357–374.
Mellin, E. A., Bronstein, L., Anderson-Butcher, D., Amorose, A. J., Ball, A., & Green, J. (2010). Measuring interprofessional team collaboration in expanded school mental health: Model refinement and scale development. Journal of Interprofessional Care, 24(5), 514-523.
Shelton, K., Taylor, P., Bonner, A., & van den Bree, M. (2009). Risk factors for homelessness: Evidence from a population-based study. Psychiatric Services, 60, 465–472. doi:10.1176/appi.ps.60.4.465
Srebnik, D., Livingston, J., Gordon, L., & King, D. (1995). Housing choice and community success for individuals with serious and persistent mental illness. Community mental health journal, 31(2), 139-152.
Wright, D., Bowman, K., Butler, D., & Eyerman, J. (2005). Non-response bias from the national household survey on drug abuse incentive experiment. Journal of Economic and Social Measurement, 30, 219–231.
LIST OF ATTACHMENTS
Attachment 1: Client Interview – Baseline
Attachment 2: Client Interview – 6-Month Follow-up
Attachment 3: Stakeholder Survey, Wave 1
Attachment 4: Stakeholder Survey, Wave 2
Attachment 5: Stakeholder Survey, Wave 3
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Orme, Stephen |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |