SAMHSA Response to OMB

Attachment 4: SAMHSA Response to Original OMB Comments (3/3/2011)

Evaluation of SAMHSA Homeless Programs

Client Interview and Stakeholder Survey

Response to OMB February 22, 2011 Comments (Submitted March 3, 2011)

Summary Response

The following is a response to OMB review comments received on February 22, 2011, in response to the application for data collection for the Cross-Site Evaluation for the Grants to Benefit Homeless Individuals (GBHI) Program that was submitted December 30, 2010. We have listed each of the questions and comments in bold and provided our response. We also have attached, in response to the reviewers’ requests, the baseline and 6-month client surveys amended to respond to reviewer comments and the IRB approvals for stakeholder, client and other data collection efforts. To aid in review of response, we begin with a brief summary response for each Comment.

1. SAMHSA GBHI program background (Part A, Section 1). We provide a background of the SAMHSA GBHI program.

2. Evaluation design and “outcomes” (Part A, Section 2). We describe the evaluation framework and SAMHSA’s intent for the evaluation to address SAMHSA’s questions in terms of this formative evaluation. In brief, the purpose of the evaluation is formative with an intent to identify and measure post-program participation findings across the broad array of outcomes expected to be influenced by the range of services provided by GBHI Grantees either directly or through referral. These are programmatic outcomes that are used to monitor the provision of services, understand the way services are tailored to clients with different needs and to better understand how the implemented service models match the models described in the efficacy and effectiveness literature.

3. Web capabilities for stakeholder survey (Part A, Section 3). We describe the extensive experience of the contractor in developing and administering similar web stakeholder surveys, past and potential response rates, and grantee stakeholder feedback on the proposed stakeholder survey.

4. Incentives (Part A, Section 9). We agree that the incentives are cash-equivalent. We note the language on the client interview consent of “non-cash coupon” (Supporting Statement Attachment 6) and note IRB’s request for this level of specificity to not confuse clients or create an expectation that they will be receiving cash from the GPRA interviewer. We are willing to use language that satisfies OMB’s recommendation for clarity, our IRB, and SAMHSA policies.

5. Interviewers, Confidentiality, GBHI client survey and GPRA survey, IRB (Part A, Section 10). Interviewers: We provide the additional requested information on the interviewers explaining that the Grantee interviewers are trained interviewers trained both by Grantees and by SAMHSA for administration of the CSAT GPRA Client Outcome Measures and tracking. We also collected information from each of the 25 GBHI GBHI sites regarding work performed, background, training, and feasibility of implementing the GBHI supplemental client survey along with the CSAT GPRA Client Outcome Measure (OMB control number 0930-0208). We describe how these interviewers, while employed by the Grantee, are trained interviewers who engage in GPRA interviewing and tracking of clients in accordance with SAMHSA’s procedures that were OMB approved. We also provide the results of a two-site pilot test of the client survey that included cognitive testing, with active clients per the request of OMB. The pilot test demonstrated the feasibility of the survey in terms of format, content, and administration procedures for both clients and the interviewer. Per OMB’s suggestion we have added additional instructions on the survey to remind interviewers to display “Show Cards” that are also in training materials; we attach the GBHI client survey with those suggestions within the bounds of the directions from the standardized measures.

Confidentiality: We agree with the reviewers that the surveys are not anonymous and point to the Supporting Statement sections that emphasize the contractor’s procedures to ensure client privacy (section A.10, pages 11-13 ).

GBHI survey: We provide the reviewers with the requested information regarding the CSAT GPRA Client Outcome Measures (OMB control number 0930-0208), purpose as approved by OMB, and how the GBHI survey would be administered according to GPRA procedures following the GPRA administration during the same session. We also describe our positive results from our pilot study of the surveys, which found that the procedures proposed are feasible regarding client and interviewer burden, response time, and response sets. 

IRB: In response to reviewer request we have attached the human subjects approval per the contractor’s Internal Review Board (IRB). We also emphasize the procedures that were undertaken, that there were no IRB concerns noted, that the consent and procedures were viewed as protecting the client and agency. We also point to the discussion of the IRB and findings that was included in the Supporting Statement (section A.10, pages 13-14).

6. GBHI client survey burden and response rate (Part A, Section 12). Reviewers requested a pilot of the survey with active clients out of concern that those in recovery and who were employed as contractors would not be representative of those who may not be in recovery and in treatment. The pilot to test the GBHI client survey content, procedures, burden and feedback (through cognitive testing) was implemented in two sites with active clients. Findings of the pilot indicated that current GBHI clients responded within the average time of that previously reported in the Supporting Statement (section A.12, pages 14-15 and section B.4, pages 21-22); burden included consent, survey, self-administration, looking up of client information, and receipt of self-administered information. OMB also asked about response rates. All clients approached for the pilot study responded positively and agreed to enroll in the pilot, however, we continue to propose that response rates over time will be at least 80% for the GBHI survey given the current rates of GPRA consent at baseline and 6-month follow-up (87.6% for GBHI Grantees per SAIS 6-month follow-up data). As it is proposed to administer both the GPRA survey and GBHI client survey in the same setting reducing burden and cost to clients and staff GPRA interviewers, we also piloted the survey with and without the GPRA survey in one setting. Our findings were that data quality was not reduced, clients did not experience the surveys as redundant and clients and interviewer did not experience an undue burden. The approach of combining the two survey administrations also has precedent in previous SAMHSA evaluations that were OMB-approved (Access to Recovery, OMB control number 0930-0299; Screening, Brief Intervention, Brief Treatment and Referral to Treatment Cross-site Evaluation, OMB control number 0930-0282; Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services, OMB control number 0930-0317).

OMB also requested a burden estimate for tracking. Given that the majority of clients remain in the program over six months, based on SAIS GPRA discharge data (OMB control number 0930-0208) for the GBHI program, that programs remain in contact with clients post-program completion or drop out (based on discharge numbers and on site visit and telephone interviews with each of the 25 sites conducted in the fall and winter 2010 – 2011), and that the GPRA interviewers are required to attain at least 80% response rate for the 6-month (with current GBHI rates at 87.6%), the additional burden for the GBHI survey would be negligible. This is also part of the rationale to administer both surveys at the same time, to minimize the burden of the cross-site evaluation on Grantees, their staff and their clients.

7. Analysis plans, duplication and non-response (Part A, Section 16). Per the reviewers request we include a brief description of the analysis plans and three sample data shell tables. We discuss how the GBHI client (and stakeholder) surveys were developed to not duplicate current SAMHSA performance collection requirements as also described in the Supporting Statement (section A.4, page 8). We also address the non-response issues. Specifically, administration of baseline surveys will be continual as new clients enroll in the program each month. Therefore, 6-month follow-up interviews will always follow the baseline date but no sooner than 180 days and no later than 210 days for an individual client (as is the OMB-approved CSAT GPRA protocol with which we are coordinating, OMB control number 0930-0208). No interviews will be attempted outside of these windows and those clients who do not complete an interview will be considered ‘non-respondent’. Since baseline interviews will cease in Spring of 2014, no follow-up interviews will be conducted after Fall 2014.

8. Survey Frame and Sampling. (Part B, Section 1). We address the question of why we are conducting a census rather than employing sampling methods, based on sample sizes, the needs of our evaluation, power, and the possible negative impact of selecting a subset of clients who will be aware of the individuals which were selected.

9. GBHI client survey (Part B, Section 2). Reviewers stated a set of concerns regarding the “complexity” of the survey for the population, given use of Likert scales and matrices and time-based questions, and requested detailed information on the sources and use of these types of scales and formats with the population. In summary, we provided information that supports that the majority of scales developed for this population include Likert scales (E.g., Bernstein et al., 1994; Bliese et al., 2008; Broner et al., 2002; Conrad et al., 2001; Davis et al., 2009; Derogatis & Melisaratos, 1983; Eisen et al., 2000, 2004; Gardner et al., 1993; Gulcur et al., 2007; Greenwood, Schaefer-McDanile, Winkel, & Tsemberis, 2005; Keane, Newman, & Orsillo, 1997; Koenig et al.,1993; Lang & Stein, 2005; Lehman, 1988; Lehman et al., 1991; McEvoy et al., 1989; Overall & Gorham, 1988; Peters & Wexler, 2005; Peters et al., 2008; Robbins et al., 2009; Rollnick et al., 1992; Ruggiero, Del Ben, Scotti, & Rabalais, 2003; Saltstone, Halliwell, & Hayslip, 1994; Shern et al., 1994; Skinner, 1982; Spies et al., 2007; Srebnik, Livingston, Gordon, & King, 1995; Storgaard, Nielson, & Gluud, 1994; Tsemberis, Moran, Shinn, Asmussen, & Shern, 2003; Weathers, Litz, Huska, & Keane, 1994; Zung, 1979). Similarly, time-based questions are commonly used and have been demonstrated as reliable and cognitively appropriate for assessment with psychiatric, substance abuse, criminal justice, veteran and homeless populations (for example, Banks, McHugo, Williams, Drake, & Shinn, 2002;  Broner et al., 2002; Broner et al., 2004; Brown et al., 1993; Burt, 2009; Carey, 1997; Clark & Rich, 2003; Dartmouth Psychiatric Research Center, 1997; Ehrman & Robbins, 1994; Fischer, Shinn, Shrout & Tsemberis, 2008; Milby, Wallace, Ward, Schumacher, & Michael, 2005; North, Eyrich, Pollio, & Spitznagel, 2004; Peters & Wexler, 2005; Peters et al., 2008; Sacks, Drake, Williams, Banks & Herrell, 2003; Smith, North, & Spitznagel, 1992;  Sobell & Sobell 1992; Sobell & Sobell, 1995; Spies et al., 2007; Tsemberis, Gulcur, & Nakae, 2004; Tsemberis et al., 2007—many of which are also likert response scales). Time-based and likert scale (matrix) questions focused on housing satisfaction, psychiatric symptoms and functioning (including trauma), drug use and attitudes, service choice, burden and satisfaction, quality of life, social supports, homeless and residential history, military history, education and employment, and so forth, have been included in current and prior OMB approved evaluations and SAMHSA-wide client-level program performance data collection (For example, SAMHSA Targeted Capacity Expansion Grants for Jail Diversion Programs, OMB control number 0930-0277; SAMHSA Homeless Families, OMB control number 0930-0223; HUD Life After Transitional Housing, OMB control number 2528-0239; National Outcomes Performance Assessment of the Collaborative Initiative to Help End Chronic Homelessness  , OMB control number 0930-0247; Access to Recovery, OMB control number 0930-0299; Safe Dates, OMB control number 0920-0783; School Violence, OMB control number 1850-0814; SAMHSA CMHS NOMs Adult Consumer Outcome Measures for Discretionary Programs, OMB control number 0930-0285; SAMHSA CSAP Participant Outcome Measures for Discretionary Programs, OMB control number 0930-0208).

Per the request of the reviewers we cross-walk each survey question and identify the domain, justification, some of the literature in which data from these measures is published, and the OMB control number for those measures that have been previously included in a relevant OMB approved survey. We also highlight for reviewers that measurement choice was the result of an extensive literature review and expert panel meeting that included national researchers, consumers, policy makers, other advocates and government agencies as described in the Supporting Statement (section A.2, page 4 and section A.8, pages 9-11). We add information based on our site visits and conversation with the 25 GBHI Grantee sites subsequent to submission of the Supporting Statement that provide support for this survey. We also present results of the cognitive testing and burden estimates (consistent with the tables submitted in the Supporting Statement, section A.12, page 14-15) of the survey with current active clients in two sites responsive to the OMB request for a “full dry run” and cognitive testing with active clients.  

GBHI Client Survey and Stakeholder Survey Response to OMB Questions

Part A

Section 1:

We would like some additional background on how the program works and how a client gets involved in the program. If SAMHSA can provide an already-written background document that answers all of these questions, then they do not need to be answered separately.

Response: The Center for Substance Abuse Treatment (CSAT) at the Substance Abuse and Mental Health Services Administration (SAMHSA) was funded by Congress to establish the Grants for the Benefit of Homeless Individuals (GBHI) program, sometimes also referred to as the Treatment for Homeless program. GBHI is a competitive, discretionary program initiated in 2001 to reduce homelessness and maintain housing through provision of treatment and other services to youth, adults and families with substance use and co-occurring mental disorders. The GBHI program includes general Grantees focused on intervening in homelessness and services for supportive housing (SSH) Grantees who provide supports for those previously homeless to maintain their housing and sobriety. The program’s goals are to (1) link substance use and mental health treatment services with housing programs and other services, (2) expand and strengthen treatment services for people who are homeless who also have substance use disorders, mental disorders, or co-occurring substance use and mental disorders; and (3) increase the number of homeless people who are placed in stable housing and who receive treatment services for alcohol, substance use, and co-occurring disorders.

Between 2001 and 2008, GBHI awarded 169 grants to provide services to the target population. An additional 25 Grantees were funded in 2009. Some Grantees serve priority populations, including criminal justice populations, chronically homeless persons, returning veterans, and chronic public inebriates; others focus on serving families, women, native Alaskans, Native Americans/Indians, other minority populations, or youth. Although all are required to, at a minimum, provide outreach, case management, substance abuse or co-occurring disorders treatment (integrated, sequential, or parallel), and wraparound and recovery services, many augment these services by adopting or adapting additional evidence-based practices (EBPs) from one of the SAMHSA toolkits (e.g., Assertive Community Treatment (ACT), Illness Management and Recovery, Supportive Employment), CSAT’s Treatment Improvement Protocols, or the National Registry of Evidence-Based Programs and Practices (NREPP). The models for service delivery vary and include primarily by referral, direct provision of treatment and other services, or a mix of direct service provision and referral to other community-based organizations. Service models are implemented in an array of settings, including on the street through outreach; in drop-in settings, shelters, and hospitals; at medical, substance abuse, or mental health clinics; in residential treatment communities; or in any of these settings or other nonoffice settings through mobile crisis units or ACT teams.

As described below, all clients are assessed by Grantees at intake, 6-months follow-up to intake and at program discharge with the CSAT GPRA Client Outcome Measures (OMB control number 0930-0208) which data is provided to SAMHSA via the web and stored in the Services Accountability Improvement System (SAIS). Since the inception of the GBHI program, CSAT homeless grants have served 33,171 individuals, a majority minority men aged 18 to 54 (Le Fauve, 2009). In 2010 the active portfolio has served over 22,000 individuals. Per the FY2010 President Obama’s budget, outcomes data available for a subset of clients served by the program through 91 active GBHI Grantees was cited indicating that individuals demonstrate: 1) 122% increase in employment or engaging in productive activities; 2) 166% increase in persons with a permanent place to live in the community; 3) 52% increase in no past months substance use; and 4) 36% improvement in no/reduced alcohol or illegal drug related health, behavioral or social consequences.

1. How likely are clients to remain in the program once they are admitted?

Response: Client retention is high among these programs. This is due in large part to the fact the programs are designed using evidence-based practices to intensively engage and retain an at-risk population that otherwise has limited resources. Among the 25 GBHI program Grantees that are the focus of the cross-site evaluation, typical program participation varies from 6 months to 24 months, based on grantee proposal reviews and telephone conversations with program directors and site visits conducted during the fall and winter 2010 -2011. The program length was designed by each grantee to address the needs of each grantee’s target population. A review of prior GBHI cohort CSAT GPRA Client Outcome Measures data (OMB control number 0930-0208) submitted to SAMHSA via the Services Accountability Information System (SAIS) indicates that clients of these earlier grantee programs remained in the programs an average of 139 days. These data also report high six month follow-up rates for clients who begin the programs, 87.6% as of February 28, 2011.

2. Does someone follow-up with respondents if they leave the program early?

Response: Yes. Grantees are required by SAMHSA to conduct a 6-month follow-up GPRA survey with at least 80% of their clients. If clients leave the program prior to six months, the grantee still has to follow-up with them. The current 6-month follow-up rate for all GBHI Grantees is 87.6%. Grantees are trained in tracking and retention of clients to assist their efforts in following-up with clients through face-to-face trainings provided quarterly and a series of on-line courses. If grantee interviewers require further assistance, they may also request telephone or on-site technical assistance in client retention, tracking and follow-up techniques. Grantees also conduct a discharge interview when a client is discharged from or has left the program regardless of calendar time since enrollment.

3. Does someone follow-up with clients after they complete the program?

Response: Grantees are required to complete a GPRA survey with at least 80% of their clients six months following the baseline interview. If a client completes the program before the 6-month period ends, the grantee is required to attempt to contact them and complete the 6-month interview. The current 6-month follow-up rate for all GBHI Grantees is 87.6%. Grantees also conduct a discharge interview when a client is discharged from or has left the program regardless of calendar time since enrollment.

4. Are you planning to follow any of the clients who drop out of the program during the survey period?

Response: Yes. The GBHI client interview will be administered by trained GPRA interviewers following the GPRA interview during the same meeting. The procedures for follow-up for the GBHI client interview mirror those for the CSAT GPRA Client Outcome Measures approved by OMB (control number 0930-0208). GBHI clients are required to be followed during this period whether or not they are still enrolled. We note again that these programs have a low drop-out rate.

5. Who are the stakeholders and what is their relationship to both the grantee and the client?

Response: Stakeholders are individuals, groups of individuals or organizations who are affected by these programs, who have an interest in the impact of these programs and who may offer collaboration or support for these programs and their clients. Examples of current stakeholders include substance abuse treatment providers, mental health treatment providers, housing providers, vocational service providers, medical care providers, HIV/AIDS services organizations, housing authorities, veterans agencies, state/city/county policy makers, state-/county-/city-wide initiatives to end homelessness, advisory boards, consumer boards, and advocacy groups. These stakeholders may have a direct relationship with both the grantee and the clients as partners to the grantee who provide services to the clients. The relationship can also be direct only through the grantee. For instance, a housing authority that funds housing will have contact with the grantee and require reports from the grantee on a continual basis, but will not have direct contact with the clients themselves.

6. Does the stakeholder know and interact with the grantees?

Response: The level of interaction between stakeholders and grantees varies based on information collected during conference calls with and site visits to the 25 GBHI Grantees. Stakeholders may have an active or passive relationship with the grantee programs. Part of the rationale of collecting data from these stakeholders is to better understand how these programs interact with the stakeholders in their communities and more broadly in the system.

7. What about with the clients?

Response: The client knows and interacts with the grantee staff, as well as with contractors and other service provider staff from stakeholder agencies. The client is unlikely to know the stakeholders per se but is usually aware of their agencies.

8. Are the grantees known to the clients?

Response: Yes, the Grantees are the programs that serve the clients. They provide services to the clients, either through direct provision or through linkage to another organization. Grantees are required by SAMHSA to provide, either directly or through referral, direct treatment for substance abuse and mental health disorders (which includes screening, assessment, and active treatment), outreach, case management, and wraparound services (which can include, for example, relapse prevention, crisis care, education or vocational services, transportation, medical care, housing readiness training, benefits application, housing application, peer support services). All clients are screened and accepted by the grantee program. Only once they are accepted by the grantee are they a “GBHI client.” The Grantees employ interviewers to screen, assess, administer the CSAT GPRA Client Outcome Measures (OMB control number 0930-0208) at baseline and follow-up points and to track clients while active, following or if leave the program, as described above.

9. Do the grantees and the clients ever interact during the course of the program?

Response: Yes. Grantee staff interact with the clients during the course of the program both providing direct services and helping the client to link to other community services. Grantees also hire GPRA interviewers to screen, assess and track clients who will interact with the clients over the course of program involvement and likely beyond during aftercare, or if they have dropped out of the program.

Section 2:

1. We are concerned with the “outcome” portion of the evaluation. It lacks the rigor of a counterfactual or control group. Therefore, it is inappropriate for SAMHSA to ascribe causality, impact or effect of the program, as it appears that SAMHSA may wish to do (e.g., page 3, “focus on the effects…on client outcomes”). For a formative evaluation, it can be appropriate to measure much more modestly than is proposed here a limited number of low burden client outcomes to see if the program shows promise (i.e., the absence of any positive outcomes would suggest a program with little or no promise). For simple grantee monitoring, it can be useful to determine grantee adherence to specified practices, which should have a very modest, if any, client component. Please clarify which of these SAMHSA aspires to do.

Response: The purpose of the evaluation is formative with an intent to identify and measure post-program participation findings across the broad array of outcomes expected to be influenced by the range of services provided by GBHI Grantees either directly or through referral. These services, which are provided following assessed need, include, as detailed above, treatment for substance abuse and mental health disorders (which includes screening, assessment, and active treatment), outreach, case management, and wraparound services (which can include, for example, relapse prevention, crisis care, education or vocational services, transportation, medical care, housing readiness training, benefits application, housing application, peer support services) and aftercare. As much as possible the intent is for each of these services to be based on models and practices that have an evidence base in peer-reviewed literature. Thus, outcomes directly relevant to the GBHI program include those related to substance use, mental health, employment and education, as well as to additional behavioral outcomes and housing. As correctly noted by the reviewers, the evaluation does not intend to draw causal inferences with respect to the GBHI program participation and outcomes, but to measure (more explicitly than the current GPRA measures allow) a variety of outcomes directly related to the specific services included in the GBHI programs. These are programmatic outcomes that are used to monitor the provision of services, understand the way services are tailored to clients with different needs and to better understand how the implemented service models match the models described in the efficacy and effectiveness literature. The additional questions place a time burden of 20 minutes on respondents (as described in the Supporting Statement, section A.12, pages 14-15 and section B.4, pages 21-22 and per survey testing with active clients in two sites subsequent to submission of the Supporting Statement) and will provide invaluable information to guide future programming efforts to aid homeless individuals with substance use or mental health disorders.

2. Describe the use of the information gained by the comparison of the self administered Part II of the baseline survey that contains several overlapping questions with the baseline survey. What quality indicators are in place for analysis of contradicting responses?

Response: The SV portion module of Part I of the survey is not self administered and contains questions about service needs and then actual receipt of needed services. The PC and the TCC (Part II) are self-administered and are intended to measure clients’ perception of the grantee program and the services they received. The PC produces a measure of overall client satisfaction, as well as the program’s culture with respect to empathy and attentiveness to clients and client empowerment. The goal of the TCC is not intended to gauge client satisfaction per se but is designed to capture the overall ‘philosophy’ or ‘style’ of the program model that is being implemented by the grantee. Specifically, some models, particularly for housing a population like this, use approaches that are prescriptive or coercive or part of a mandate from the criminal justice system. In order to assess where a grantee model fits in that dimension, the clients’ perceptions of how choice is presented and the requirements or stipulations surrounding other services are used to populate this scale. These client provided data are a capstone to or are part of the triangulation process that uses multiple data sources to evaluate these models (e.g., qualitative data from grantee interviews and documents, other administrative data, etc.)

Although there appears to be some overlap between the PC and the TCC for some questions they are qualitatively different questions. In the case of the PC, for example, the client is asked if they would switch providers if other providers were available. The similar question in the TCC asks whether switching to another provider would be permitted for that client (e.g., court –ordered to a particular provider). We believe we should maintain the integrity of the instruments and use them for their distinct goals. A quality review of these data is an important exercise for surveys like these. We intend to have boolean cross-checking using SAS or STATA for all elements that might yield internal inconsistency and either use logical imputation when possible or suppress scale items for which there is no clear solution. Fortunately, the properties of most scales are robust to a small number of missing items including for example the use of Likert matrix questions as part of a self-administered interview is consistent with other OMB approved studies such as Access to Recovery (OMB Control Number 0930-0299.

Section 3:

1. Have you tested the web capabilities and response rates for the stakeholder survey?

Response: We have not formally tested the web-based version and response rate for the stakeholder survey as it has not yet been approved and the cost of doing so would be prohibitive prior to OMB approval. However, as part of the site visits held in the fall and winter 2010-2011members of the evaluation team have met with stakeholders at each of the 25 GBHI Treatment for Homeless Grantees where we reviewed and discussed the stakeholder survey, including the survey questions, web-based format, length of the survey, and likelihood of response. The feedback we received was positive without suggestions for modifying the content, layout or procedures. The length of time estimated to take the survey, 17 minutes (as described in the Supporting Statement, section A.12, pages 14-15 and section B.4, page 22)was endorsed as non-burdensome. Also, the stakeholders’ familiarity with SAMHSA and the purpose of the types of programs it funds and interest in community services was viewed as a positive to attaining a high response rate within our anticipated response rate. As described in the Supporting Statement (p. 17) we conservatively estimated a 50% response rate for prior completed Grantee Stakeholders and an 80% response rate for current Grantees Stakeholders producing an average response rate of 65% across all years of GBHI program funding. The contractor, RTI International, has extensive experience in both developing and administering web-based surveys and surveying stakeholders with program models similar to the GBHI Treatment for Homeless program. The below table highlights a few of these studies and describes their applicability to the stakeholder survey for this evaluation.

Study Name	Relevance/Experience	OMB Control Number
Evaluation of the National Weed and Seed Strategy	Included a Web-based survey of stakeholders such as agency representatives or involved residents in 169 sites (1,353 respondents) achieving a 59% response rate. Please note that many of these 169 sites had already completed their funding cycle and thus had no direct mandate to participate in the evaluation unlike GBHI where we have ongoing relationships with all 25 sites and, as a part of their receiving funding, are required to participate in the cross-site evaluation.	N/A
Law Enforcement Forensic Processing Study	Multi-mode survey, including web-based, of 2,000 law enforcement agencies achieving a 73% response rate.	1121-0320
Safe Schools/Healthy Students Cross-site Evaluation	Included a survey of partnership members and key partners in 97 grantee sites. During four waves of data collection the average response rate for these surveys was 65%.	1121-0247

Section 9:

Incentives:

1. Grocery cards are a cash equivalent incentive. Please change the description in A9. We only consider items that are not easily exchanged for money, like a ruler or a magnet, as non-cash incentives. A gift card to a grocery store could easily be exchanged for cash and therefore should be considered a cash incentive.

2. Please clarify when the client receives the incentive.

Response: Question 1. We agree we are proposing cash equivalent incentives. All of the 25 Grantees requested cash equivalent incentives such as Target gift cards in discussions about the planned cross-site evaluation. The Grantees’ rationale for cash-equivalent incentives was twofold. First, all Grantees indicated that these types of incentives would be administratively easier to receive and track than other types of incentives. Second, the Grantees indicated that the clients would prefer something that they could use toward food and housing items. As described in section A.9 on page 11 of the Supporting Statement, incentives have been demonstrated to increase response rates and panel retention in surveys about sensitive behaviors without being overly coercive (Cottler et al., 1996) . We are also comfortable that using gift cards does not alter either the response rate or increase the probability of increasing substance use (Festinger et al., 2008). The language on page 9 of the Supporting Statement should have read: To increase response rates, all clients who agree to participate in the client interview at baseline will receive an incentive worth a $10 value (e.g., gift card). Participants who complete the baseline will be asked to complete a 6-month follow-up interview. Clients who agree to participate in the 6-month follow-up will receive an incentive worth a $25 value.”

The client consent form submitted on December 30, 2010 with the Supporting Statement (Attachment 6) reads “non-cash coupon worth $10.00 today; you will also receive a non-cash coupon worth $25.00 at the 6 month follow-up interview…”. Would OMB like this changed by deleting the words “non-cash” so the informed consent reads “coupon worth…”. The downside is that it is perhaps misleading in that clients may expect a “cash” coupon then; this level of specificity, “non-cash coupon”, was encouraged by the IRB.

Question 2. As noted in the submitted informed consent submitted with the Supporting Statement (Attachment 6), incentives will be provided at the baseline administration of the GBHI Client Interview and at the administration of the 6-month follow-up interview. Clients will not be penalized for skipping items or terminating an interview before full completion. If a client wishes to stop the interview, he or she will still receive the incentive.

Section 10:

Interviewers:

We are concerned that you are using grantees as interviewers for two reasons:

1. While it may be useful to have the grantees administer the client surveys because they are familiar with the clients, there is also a conflict of interest since the client will be evaluating the program with a grantee present.

2. We are also concerned because survey administration is not an easy task and while you will provide training to the grantees prior to the evaluation, they are not professional interviewers and they may not be as comfortable or skilled at collecting the required data.

Response:

Question 1. While we appreciate the concern that using Grantees as interviewers runs the risk of a potential conflict of interest, it is one of the limitations of this type of performance assessment. Grantees employ trained interviewers whose job it is to administer the CSAT GPRA Client Outcome Measures interviews, other program assessments, and track clients that are active and those who have dropped out of the program for 6-month and discharge interviews This method of grantee collected data is the primary method that can generate data on a large number of subjects from all 25 grantee programs given the level of available resources. Further as these data are used for clinical and performance assessment by Grantees, to collect them twice (once for grantee use in performance assessment and once for evaluation) would be duplicative and burdensome. OMB has approved multiple evaluations with grantee only or a mix of grantee and independent assessors depending on the grantee, for example: Targeted Capacity Expansion Grants for Jail Diversion Programs (OMB control number 0930-0277), Homeless Families (OMB control number 0930-0223), National Outcomes Performance Assessment of the Collaborative Initiative to Help End Chronic Homelessness, (OMB control number 0930-0247), ; Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services (OMB control number 0930-0317). The assessment of the program by clients is, however, client self-administered to reduce response bias consistent with an approach taken by the OMB approved Access to Recovery (OMB control number 0930-0299) cross-site evaluation survey that includes Likert-matrix client self-administered satisfaction and detailed program feedback.

Question 2. We did not sufficiently describe in our Supporting Statement the Grantee interviewers who will conduct these interviews. Although the interviewers will be Grantee staff, they are “professional interviewers” that have been trained in interviewing and tracking in conjunction with their role conducting the SAMHSA CSAT GPRA Client Outcome Measures for Discretionary Programs (OMB control number 0930-0208), as well as through Grantee sponsored training. In preparation for the proposed data collection activities, we conducted telephone and face-to-face discussions during site visits with the grantee GPRA interview staff, as well as with managers, supervisors and local evaluators. We have confirmatory information following submission of the Supporting Statement about the level of the interviewers and the interviewers’ assessment of the proposed interviews and interview procedures. Based on these discussions with each of the 25 CSAT GBHI Treatment for the Homeless Grantees, each of the 25 sites has GPRA interviewers who have been trained by SAMHSA through face-to-face and on-line courses on interviewing and tracking of clients for follow-up interviews. Each of these interviewers has also received additional training from the Grantees in administering screening and assessment surveys and in engaging and tracking clients. These interviewers while employed by the Grantee are responsible for administrating the CSAT GPRA and other interviews, assigning ID numbers for on-line submission to SAMHSA for client-level data collection, providing incentives if the Grantee provides incentives, and tracking and following up with clients during and following program completion or drop-out for client, per procedures approved by OMB (control number 0930-0208). Grantees had prepared their staff for potential participation in client-level and other data collection prior to award, based on the SAMHSA GBHI Treatment for Homeless Request for Application (RFA) language that if a cross-site evaluation was awarded Grantees have agreed to participate in the cross-site evaluation activities.

The proposed GBHI client baseline and 6-month follow-up interviews were carefully reviewed by the cross-site evaluation expert panel and SAMHSA staff (see Supporting Statement section A.8, page 9); include the use of standardized instruments and measures previously approved by OMB for other cross-site evaluations including the population of focus for the GBHI cross-site evaluation (see Supporting Statement section A.2, pages 4-7 and below); and were reviewed with GPRA interviewers along with supervisory, management and local evaluation staff from each of the 25 GBHI sites. Following a semi-structured site visit protocol, each question was reviewed, along with procedures. Questions were asked regarding any potential redundancy with current GPRA questions, which questions or domains should be included or removed, and what additional questions should be considered. All 25 sites, consistent with the expert panel and prior SAMHSA review, endorsed all questions and associated domains as non-redundant; further grantee respondents thought the survey was particularly additive for the Treatment for Homeless program in terms of describing their clients, the areas of focus for clients and services, activities for a sufficient period that would capture both Grantee services and client activities, beliefs and behaviors, and that this would help to better describe the individual grant programs and overall GBHI program. While Grantees had suggestions for additional domains, they did not recommend replacing the questions proposed; adding more questions was viewed by the cross-site evaluation contractor as burdensome. Further, the Grantees felt that the format of the questions and type of questions were consistent with other questions clients answer without difficulty or confusion and believed that accurate information would be obtained from time-based questions. Finally, two sites voluntarily pre-tested the interview with a total of 4 clients during the original response period for comments/feedback and confirmed that there were no difficulties in overall administration, in response to individual questions, in response to Likert matrices or time-based questions, or completing self-administered sections. Grantees noted that clients were used to these types of interviews and to both interviewer administered and self-administered questionnaires. The findings of these two Grantees were consistent with our findings of the two-site pilot test of the client baseline and 6-month surveys and of with feedback from the non-grantee interviewer who had conducted our pilot.

We recommend using a neutral third party interviewer and full dry run of the survey protocol and administration with potential respondents. While complete interviewer materials and training is important with any survey, this survey requires special conditions and procedures and as little as possible should be left up to the Grantees to make up on the fly. This is true for professional interviewers, but is especially important since you are proposing to use non-professional interviewers for your evaluation. A dry run of the evaluation with actual clients will help assure both the interviewers and the survey team that the evaluation protocol works and collects the desired information.

Response: In preparation for client-level data collection, if approved by OMB, we conducted a “full dry run of the survey protocol” (both baseline and 6-month follow-up client interviews) and conducted cognitive testing with clients. We administered each of the client protocols to 8 respondents for baseline and 6-month follow-up. We also tested whether receiving a pre-survey of GPRA questions would increase time, reduce responsiveness or change the quality of the responses, varying whether the clients received the surveys alone on in conjunction with the GPRA survey. The process also included full informed consent procedures, directions, Part I and Part II questionnaires including clients sealing their responses in the envelop and returning them to the interviewer. The interviewer was a non-grantee interviewer trained on the GPRA interview. The clients were active clients at two current SAMHSA sites. The time burden was consistent with that noted in section A.12 on pages 14-15 and section B.4 on pages 21-22 of the Supporting Statement and will not result in a revision of the burden estimate table. The results of this pilot showed an average of 4 minutes for reviewing client information and the consent procedures, 14 minutes to administer Part I, and 3 minutes for client self-administered completion of Part II for both baseline and 6 month followup. No client had suggestions for removal of items or evidenced difficulty or slowness on particular items; all clients had positive comments on the survey indicating satisfaction about being heard. A majority of the clients felt that by responding to the questions that the grantee programs would be better informed about the services clients needed and better able to describe client gains.

These data, along with the information from current GPRA interviewers who reviewed and pre-tested the interview also in two sites, with four clients, during the response/comment period, indicate that the survey could be successfully administered.

Moreover, the survey is comprised primarily of standardized measures or measures that are or have been used for other SAHMSA cross-site evaluations (OMB numbers 0930-0277, 0930-0223, 2528-0239, 0930-0285; 0930-0247; and see chart of measures below) and whose data has been published in peer reviewed publications (for example, Broner et al., 2009; Burt, 2009; Burt, 2010; Case et al., 2009; Davis et al., 2009; Naples et al., 2007; Rog et al., 1995; Rog & Buckner, 2007; Rog et al., 1995; Steadman & Naples, 2005). We do agree with OMB reviewers the inclusion of instructions on the interview itself is useful tool. For each standardized instrument, or previously OMB approved measure, we included the measures instructions to maintain the integrity of the measure (see Supporting Statement Appendix 6) in addition to emphasizing this in training materials for interviewers. We will add instructions to each of the Likert scale matrix questions for Part I, “present Show Card” as a reminder to the GPRA interviewer.

Confidentiality:

1. The client survey is not anonymous because the survey requires identifying information on it, i.e., a GBHI client number. An anonymous survey is one that requires no identifying information on it and cannot link an individual in any manner back to this form.

2. How is this number assigned and is it something that the client would know or would it be looked up by the interviewer?

Response: We agree with OMB that this survey is not anonymous. In a review of the submitted Supporting Statement and materials we could not find the use of the word ‘anonymous’. The information provided to the cross-site evaluation is provided via an ID number assigned by the Grantee interviewer. The GPRA interviewer at each of the 25 sites assigns a number to accepted clients. This “GPRA number” is used for submission of the SAMHSA CSAT GPRA Outcomes Measures (OMB control number is 0930-0208) via their on-line SAIS system and is kept by the Grantee not the client. This number would be placed on the supplemental client interview form by the GPRA Interviewer. The cross-site evaluation will then link this number to the GPRA data for each client. Linking to the already collected GPRA data minimizes the need to avoid duplicate collection of information. For example, demographic information, substance use and attitudes, general psychiatric symptoms (not-trauma), physical health (including HIV), sexual risk, and family supports is not collected on the supplemental baseline and 6-month surveys as these data are already collected via the CSAT GPRA survey.

GBHI survey:

1. How does the GBHI survey fit with the GPRA survey operations? While we can understand wanting to administer these two surveys together in order to simplify procedures, we are concerned about this might affect the results in GBHI.

2. What is the purpose of the GPRA survey and what kind of questions does it ask? Can you also send us either the OMB control number and/ or a copy of the GPRA questionnaire?

3. How long on average does it take to administer the GPRA survey?

4. What incentives are provided by GPRA, if any? Are they provided at the same time as the incentive is provided for GBHI?

Response:

Question 1. The GBHI client survey as described in section A.2 on pages 4-7 of the Supporting Statement would be administered following the GPRA survey as described in section B.2 on pages 18-20 of the Supporting Statement. The decision to conduct the GBHI client interview in conjunction with the GPRA survey was based on the recommendation of the 15-member expert panel and SAMHSA staff, as well as at the request of the 25 GBHI Grantees as confirmed during site visits held during the fall and winter 2010-2011. During the piloting of the baseline and 6-month GBHI client interviews in two sites with active clients, we varied the survey administration, administering it alone and with the GPRA and found no differences in response rates, response times, or consistency of responses. The different procedures also did not affect the consistently positive response to the GBHI survey or to the GPRA by the clients. Moreover, the approach of combining the two survey administrations has precedent in previous SAMHSA evaluations that were OMB-approved (Access to Recovery, OMB control number 0930-0299; Screening, Brief Intervention, Brief Treatment and Referral to Treatment Cross-site Evaluation, OMB control number 0930-0282; Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services, OMB control number 0930-0317)

Question 2. The CSAT GPRA Client Outcome Measures survey was OMB approved and reauthorized in 2009 (control number 0930-0208). The GPRA survey was developed and is used by SAMHSA to comply with the Government Performance and Results Act (GPRA) of 1993 which requires all federal departments to develop strategic plans, set performance targets related to their strategic plan on an annual basis, report annually the degree to which they met these goals, conduct regular evaluations of their programs and use the results to explain successes and failures on the basis of performance monitoring data. The GPRA survey serves as a performance measure for all of SAMHSA’s discretionary services grants. The CSAT GPRA survey collects information at baseline, 6-month and discharge on client demographic characteristics and drug (by type of drug and overall) and alcohol use, education, employment, criminal justice activity, HIV, sexual risk behavior, current housing, social connection, and general (not trauma) psychiatric symptoms for the 30 days prior to interview administration. As noted in section A.4 on page 8 of the Supporting Statement the GBHI client survey was constructed to complement and not be redundant with the CSAT GPRA. The additional information proposed for the GBHI supplemental survey is specifically relevant to Treatment for Homeless Grantees to describe these programs and their clients at baseline and 6-month follow-up.

Question 3. The GPRA survey was OMB approved and reauthorized in 2009 (control number 0930-0208). The total estimated burden for administering the CSAT GPRA Client Outcome Measures is 21 minutes including instructions, review of documents and client information prior to interviewing, and so forth. The average time for the CSAT GPRA survey administration is approximately 10 minutes.

Question 4. The GPRA allows for payment of $20.00 incentives for follow-up interviews. Grantees may choose to provide an incentive up to a $20 value for completion of the 6-month follow-up GPRA survey only, though it is not a requirement. Examples of incentives that clients may receive include gift cards, coupons, toiletries, movie tickets, transportation vouchers, etc. If the grantee has chosen to provide the client with an incentive for completing the GPRA survey, then yes they would receive the incentive at the same time as the incentive provided for the 6-month follow-up GBHI survey. However, based on discussion with the 25 Grantees during site visits, clients will not receive double incentives, for the baseline and 6-month interviews.

IRB review:

1. Was this collection reviewed by an IRB?

2. If so, provide the certificate of approval plus information about their findings and recommendations. Specifically, we are interested in knowing if they felt the consent form provided enough protection both for the clients and for the agency and if they had any additional concerns about gaining consent from this population.

Response: Questions 1 and 2. The data collection plan, procedures, surveys, consents and scripts were reviewed and approved by RTI International’s IRB, a federally assured IRB (Federal Wide Assurance Number 3331). The consent forms for the client and stakeholder surveys submitted with the Supporting Statement (see Appendices 6 & 7) includes the RTI IRB approval. The surveys and accompanying materials, procedures and design were approved by the IRB. No concerns were identified. The IRB approved the consent forms as providing sufficient information for protection of clients, stakeholders and agency. In response to reviewer request, we have attached the approvals relevant to the stakeholder survey and to the client surveys. This information is also available in the original Supporting Statement; please see the Supporting Statement, section A.10, pages 13-14 and excerpted below:

“In addition, the three interviews, all informed consents and the client interview script have been reviewed and approved by the contractor’s Institutional Review Board (IRB) (Federal Wide Assurance Number 3331), review #12612. In keeping with 45 CFR 46, Protection of Human Subjects, the CSAT GBHI procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent. The IRB-approved model informed consents meet all Federal requirements for informed consent documentation. This template will be customized by each grantee to obtain informed consent for participation in the study. Any necessary changes to the surveys will be reviewed by the contractor’s IRB.

Data from the CSAT GBHI client interviews will be safeguarded in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.” (P. 13-14, Lines 510-523, Supporting Statement)

Section 12:

1. Can you provide us with background information on why you expect an 80% response rate for both administrations of the client survey?

2. We would also like to see additional support for the 20 minute total estimated completion time, especially with respondents who currently represent the target population, instead of individuals who in the past represented these individuals. We are concerned that contractors, even those whom were formerly homeless, may not serve as a good proxy for your target population.

3. Can you provide information to support estimated burden in follow up time needed for non-respondents.

Response:

Question 1. All clients who are enrolled in the GBHI program complete the GPRA survey. Client retention is also high among these programs. This is due in large part to the fact the programs are designed using evidence-based practices to intensively engage and retain an at-risk population that otherwise has limited resources. This is also due to Grantees having dedicated trained interview staff to conduct interviews and track clients as described previously. Grantees are required to complete a GPRA survey with at least 80% of their clients six months following the baseline interview. If a client completes the program before the 6-month period ends, the grantee is required to attempt to contact them and complete the 6-month interview per the OMB approved GPRA procedures protocol (0930-0208). Grantees also conduct a discharge interview when a client is discharged from or has left the program regardless of calendar time since enrollment. Among the 25 GBHI program Grantees that are the focus of the cross-site evaluation, typical program participation varies from 6 months to 24 months, based on grantee proposal reviews and telephone conversations with program directors and site visits conducted during the fall and winter 2010 -2011. The program length was designed by each grantee to address the needs of each grantee’s target population. A review of prior GBHI cohort CSAT GPRA Client Outcome Measures data (OMB control number 0930-0208) submitted to SAMHSA via SAIS indicates that clients of these earlier grantee programs remained in the programs an average of 139 days. These data also report high six month follow-up rates for clients who begin the programs, 87.6% as of February 28, 2011. All clients approached for the pilot study responded positively and agreed to enroll in the pilot. We believe that our estimates of at minimum 80% enrollment and retention remains accurate as a projection over the course of the project and is consistent with the description in the Supporting Statement (section A.12, pages 14-15 and section B.3, pages 20-21).

Question 2. In preparation for client-level data collection, if approved by OMB, we conducted a “full dry run of the survey protocol” (both baseline and 6-month follow-up client interviews) and conducted cognitive testing with clients. We administered each of the client protocols to 8 respondents for baseline and 6-month follow-up. We also tested whether receiving a pre-survey of GPRA questions would increase time, reduce responsiveness or change the quality of the responses, varying whether the clients received the surveys alone on in conjunction with the GPRA survey. The process included full informed consent procedures, directions, Part 1 and Part II questionnaires including clients sealing their responses in the envelop and returning them to the interviewer. The interviewer was a non-grantee interviewer trained on the GPRA survey. The clients were active clients at two current SAMHSA sites. The time burden was consistent with that noted in section A.12 on pages 14-15 and section B.4 on pages 21-22 of the Supporting Statement The results of this pilot showed an average of 4 minutes for the consent procedures, 14 minutes to administer Part I, and 3 minutes for completion of Part II for both baseline and 6 month followup. No client had suggestions for removal of items or evidenced difficulty or slowness on particular items; all clients had positive comments on the survey indicating satisfaction about being heard. A majority of the clients felt that by responding to the questions that the grantee programs would be better informed about the services clients needed and better able to describe client gains. As described previously, two GBHI Grantees also administered the surveys to four clients during the feedback/comment period, and found similarly that the survey performed well, within the allotted time, and engendered positive responses from clients about the types of questions asked. As it is proposed to administer both the GPRA survey and GBHI in the same setting reducing burden and cost to clients and staff GPRA interviewers, we also piloted the survey with and without the GPRA in one setting. Our findings were that data quality was not reduced, clients did not experience the surveys as redundant and clients and interviewer did not experience an undue burden. The approach of combining the two survey administrations also has precedent in previous SAMHSA evaluations that were OMB-approved (Access to Recovery, OMB control number 0930-0299; Screening, Brief Intervention, Brief Treatment and Referral to Treatment Cross-site Evaluation, OMB control number 0930-0282; Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS Services, OMB control number 0930-0317).

Question 3. As the GBHI client survey will not require additional follow-up or tracking over and above the GPRA, which follow-up burden is included in the approved measure (0930-0208) and as the Grantee is required to maintain 80% retention as a condition of funding, tracking burden would be negligible. Purposefully, to reduce Grantee and client burden and be cost effective we proposed that the GBHI client interview be supplemented to the GPRA interview session and as noted above, is the procedure for several OMB approved SAMHSA evaluations (control numbers: 0930-0299; 0930-0282; 0930-0317).

Section 16:

1. What kind of statistics and reports are planning to use this data for once it is collected? These uses can be both internal to the GBHI program, internal to SAMHSA, and external to the public or other interested parties. Include a detail discussion of your analysis plan and table shells of your expected analysis.

2. Would this information be collected even if SAMHSA was not conducting this evaluation and other ways this data might be used outside of the GBHI program?

3. Can you add dates for the nonresponse follow-up to the schedule?

Response:

Question 1. The GBHI cross-site evaluation supplemental data will be combined with data from the CSAT GPRA Client Outcome Measures, information gathered from the Grantees describing their program components, and data from secondary sources such as SAIS GPRA and Technical Assistance data, the National Survey of Substance Abuse Treatment Services (N-SSATS) and the Treatment Episode Data Set (TEDS) to develop a comprehensive portrait of the GBHI client populations, the needs of these populations, the services provided to address those needs, and the outcomes across a multitude of domain areas for those participating in GBHI programs. These supplemental data will provide mediating and moderating variables, as well as information on client characteristics not covered by the CSAT GPRA survey. The areas addressed by the supplemental data collection include service need, burden, satisfaction/perception of care, the form of care or individually tailored care, model adaption, homelessness, housing (placement/safety/perceived choice/perceived value), readiness for change, and co-occurring mental disorders. Three additional domains (services, trauma and veteran’s service era and combat information) were added in response to the recommendations of the expert panel and SAMHSA and confirmed with the GBHI Grantees. The additional services data will improve our ability to describe the relationships between treatment plans and abstinence and housing stability including measuring the extent to which models of matching services to needs are being used and the appropriate dosage of services as described in the literature for these models. As CSAT GPRA data includes administrative data on services received only at discharge, it impossible to assess whether and how service receipt changes over time using only GPRA data alone. The GPRA data does not collect this information from the client or address perceived need and service matching. The supplemental data will address this limitation. Additionally, the panelists recommended measuring trauma symptoms given that trauma is prevalent in the homeless population (e.g., Browne & Bassuk, 1997; Goodman, 1991; Bassuk et al., 1996; Burt et al., 1999; HUD, 2009; Shelton et al., 2009) and without intervention consistently predicts negative substance abuse, employment, housing and criminal justice outcomes. Finally, given the high prevalence of homelessness among returning veterans and differentially by service era (Kline et al., 2009), along with there being several Grantee programs focused solely on veterans, baseline collection of veteran service era was recommended.

We conducted a literature review that helped advance our thinking about likely influences on client-, grantee-, and system-level outcomes (Broner et al., 2010). As we developed our data collection and analysis plans, we used information from the review to strengthen the evaluation’s ability to provide insightful findings on what works for whom, under what approaches, and in what systems and contexts. At the client level, demographic characteristics (sex, age, race or ethnicity), parental status, educational attainment, veteran status (for recent cohorts), disability, social supports, and involvement with the criminal justice system can be important with respect to understanding the appropriateness and expected effectiveness of specific approaches. Client differences in substance abuse, mental illness, and co-morbidity are of central importance to GBHI. Our data collection and analyses will allow us to describe how client populations differ on these factors across study sites and test whether these factors are associated with differential program choices, components and successful provision of services, including housing the clients. For example, by collecting gender at the client level, we will assess whether programs are better able to provide appropriate services for female clients than for male clients. Clients will also differ in their levels of participation, program completion, and treatment compliance. Information from the supplemental data collection will enhance the CSAT GPRA information from the SAIS discharge data to allow us to estimate what client characteristics are significantly associated with participation at 6 month follow-up and to test whether participation mediates the programs’ ability to carry out full services objectives.

The outcome evaluation component focuses on addressing the “utility” element of the evaluation’s Objective 1, which per SAMHSA’s RFA was is to “examine the feasibility, utility, and sustainability of future Treatment of Homeless cohorts through the review of planned and actual outcomes.” The outcome evaluation will focus on the changes in client outcomes that are associated with differences in grantee models. The findings will be framed in a pre-post quasi-experimental design that will allow us to examine the relationship of both intent-to-treat and service receipt from to outcomes. HLM will be used to estimate the mean change in client-level outcomes between baseline and follow-up. HLM is appropriate for these analyses because this modeling approach allows us to control for the clustering of clients within grantee. Within the HLM framework, we will adjust for client characteristics and other contextual factors. These adjusted mean changes will provide easy-to-understand estimates of possible program impact. Although these estimates are not intended to be causally interpreted, we do intend to compare them to estimates for similar models and populations in the scientific literature to confirm that they are within ranges that we would expect, conditional on the level of adherence to the models that we observe for each grantee. These estimates form a baseline for exploring how program decisions and characteristics alter service delivery and outcomes. In this way, variation among the 25 Grantees will serve as experimental variation for analyzing ‘key ingredients’ of models for achieving different outcomes, such as linking clients to certain types of housing. As appropriate, subgroup analyses will be conducted in which the data will be stratified by program type or client type to assess whether outcomes differ among the different types of programs or for different types of client (e.g., veterans or women).

Example table shells are provided below.

The first table shell describes patient transportation needs and receipts by several different grantee strata and for a given set of services. Specifically, we calculate proportions of patients answering affirmatively to questions SV11a and SV11b on the client survey for each relevant cell. Cells are constructed by the kind of service the client might need to receive, whether a grantee or grantee collaborator are providing the service, the geographic scenario for the client with respect to the service, and the type of transportation offered as part of the program.

Table 1. (Sample Table) Proportions (st. dev) of Clients Reporting Transportation Needs and Receipt (SV11a and SV11b): Stratified by Combinations of Transportation Options for Primary Services and Grantee Organizational Characteristics

In-person Services

Provided by Grantee Organization

Provided by non-Grantee Organization (e.g., local clinic, unemployment office, DMV)

Primary Transportation for Logistics for Service Provision

Typical Residence of Consumers at main stage of Continuum of Care

SATX OP

MHTX OP

Clinical CM

Medical TX

Housing or other Wraparound CM

SATX OP

MHTX OP

Clinical CM

Medical TX

Housing or other Wraparound CM

Direct Provision of Transportation to Central Location

On-site Central Location

proportion (st. dev)

Off-site Cluster

Scatter-site

Subsidized Transportation to Central Location

On-site Central Location

Off-site Cluster

Scatter-site

Service Provided at Residence

On-site Central Location

Off-site Cluster

Scatter-site

Service Provided at Central – No Transportation Support

On-site Central Location

Off-site Cluster

Scatter-site

Note: Data are based on client survey responses at 6-month follow-up

The second table is an example of a model of what characteristics are associated with a client’s housing status at both baseline and follow-up. In addition to client characteristics, we plan to include yet tobe determined independent variables that capture the grantee’s program. Although not causal, the estimate for the indicator of 6-month follow-up provides a benchmark for comparing how well these models may have performed compared with published estimates from rigorous effectiveness studies.

Table Shell 2. Multinomial Logit Estimates of the of Housing Status

	Shelter	Residential SA Treatment	Transitional Housing	Permanent, Independent Housing
Follow-Up Indicator	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Female	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Hispanic	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Black	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Asian	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
American Indian	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Native Hawaiian/ Pacific Islander	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Alaska Native	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Other Race/Ethnicity	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Age	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Illicit Substance Use (Days)	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Mental Illness (Scale)	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Indicator of Client Reporting Transportation Needs	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Vector of Grantee Model Characteristics	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)

Notes: [MODEL INFORMATION]. N = # clients in rectangular analysis sample. P-values are given in parentheses below coefficient estimates. Reference category for dependent variable is “No Housing”. Male and White are reference categories for gender and race indicators. Random effects for clients nested within grantee random effects are used if a Hausman Specification Test does not preclude a random effects specification.

The second table also provides context for the third table in which an additional grantee characteristic, transportation efficacy, is included in a moderator analysis of housing status after six additional months in the program.

Table Shell 3. Moderating Influence of a Program’s Transportation Strength on its Ability to Link Clients to Housing, Estimates from a Multinomial Logit Model of Housing Status

	Shelter	Residential SA Treatment	Transitional Housing	Permanent, Independent Housing
Transportation Efficacy (Scale)* Follow-Up Indicator	0.000	0.000	0.000	0.00
	(0.000)	(0.000)	(0.000)	(0.00)
Transportation Efficacy (Scale)	0.000	0.000	0.000	0.00
	(0.000)	(0.000)	(0.000)	(0.00)
Follow-Up Indicator	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Female	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Hispanic	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Black	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Asian	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
American Indian	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Native Hawaiian/ Pacific Islander	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Alaska Native	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Other Race/Ethnicity	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Age	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Illicit Substance Use (Days)	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Mental Illness (Scale)	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Indicator of Client Reporting Transportation Needs	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)
Vector of Grantee Model Characteristics	0.00	0.00	0.00	0.00
	(0.00)	(0.00)	(0.00)	(0.00)

Notes: [MODEL INFORMATION]. N = # clients in rectangular analysis sample. P-values are given in parentheses below coefficient estimates. Reference category for dependent variable is “No Housing”. Male and White are reference categories for gender and race indicators. Random effects for clients nested within grantee random effects are used if a Hausman Specification Test does not preclude a random effects specification. Transportation Efficacy is a continuous scale derived from qualitative data analyses. The interaction term in the first row represents the moderating influence of a program’s transportation characteristics on its ability to link clients to housing.

The findings will be used by the GBHI program to refine the program parameters for funding opportunities (e.g., by providing additional details on the types of services appear to improve outcomes for specific types of populations). The findings will be used by SAMHSA to inform the Homeless portfolios in CSAT and CMHS, to assure that the best evidence is incorporated into the funding decisions. The results will be of importance to the GBHI Grantees and other providers of services for homeless individuals with substance use or mental disorders in terms of providing guidance for future directions. The results may also be of importance to consumer and consumer advocacy programs in providing information about these types of programs, areas of improvement and by the types of clients served.

Question 2. This information would not be collected if SAMHSA were not conducting this evaluation. As noted above, the findings have the potential to inform program development for all programs providing services to homeless individuals with substance use or mental disorders and not simply programs supported with SAMHSA funds.

Question 3. Yes. Specifically, administration of baseline surveys will be continual as new clients enroll in the program each month. Therefore, 6-month follow-up interviews will always follow the baseline date but no sooner than 180 days and no later than 210 days for an individual client (as is the OMB-approved CSAT GPRA protocol with which we are coordinating, OMB control number 0930-0208). No interviews will be attempted outside of these windows and those clients who do not complete an interview will be considered ‘non-respondent’. Since baseline interviews will cease in Spring of 2014, no follow-up interviews will be conducted after Fall 2014.

Part B

Section 1:

Survey frame and sampling procedures:

1. How are clients selected to be a part of the evaluation, assuming that clients are accepted into the program on a rolling basis?

2. Please justify statistically why you decided to conduct a census of all clients instead of a sample. This information should be framed within a discussion of the level of precision you are hoping to achieve in your analysis for key estimates (which you should specifically identify).

Response:

Question 1. All clients accepted into the 25 GBHI programs during the baseline enrollment period will be included in the evaluation.

Question 2. We are attempting to collect data from all new enrollments in the programs, i.e., a census of the programs’ clients during the latter 3.5 years of program operations and during the period in which they have fully implemented their programs. Because this is a formative evaluation, our focus is on analyzing these programs and better understanding their processes and degree of implementation in order to inform future SAMHSA initiatives and in order to determine the feasibility of different approaches whose effectiveness has been studied in previous literature. Our first reason for choosing a census rather than sampling is the small number of total clients we are expecting for each of the 25 grantees (<125). Any sampling of this universe rather than a census unnecessarily creates a framework of ‘inference’ that is arbitrary. In other words, we are evaluating these programs per se. Other than ‘non-response’, any statistics we calculate to describe the programs will be actual population parameters describing what occurred rather than sampling statistics in which we could make only inference about what actually happened. This is a common evaluation approach and has precedent in the OMB approved patient and client surveys of SBIRT and ATR, respectively (Access to Recovery, OMB control number 0930-0299; Screening, Brief Intervention, Brief Treatment and Referral to Treatment Cross-site Evaluation, OMB control number 0930-0282). A second reason for the census is the consideration that each program provides many different combinations of services to clients who also vary substantially in their characteristics and needs. We wish to obtain the most comprehensive data possible describing these program outputs. Third, in any given month, these programs are often enrolling less than 10 new clients. These programs are intensive in nature having a great deal of contact with clients at central program locations. Moreover, clients are often housed in the same facilities and group counseling sessions that can include all the current clients. In other words, clients are often very familiar with each other and share information about the programs. Sampling out of these small, close groups leads to two risks we wish to avoid. The risk of linking a respondent to their responses increases when everyone knows who the small number of respondents were in a given time period. Moreover, grantees have noted that clients are very sensitive to fairness and could respond negatively to the programs or fellow clients when they are not randomly selected for the survey and the incentive.

We do note that for a conservative expected population size (2750), we have greater than .8 power to detect differences in dichotomous variables at the .05 significance level that are 2.5 percentage points or larger. These calculations assumed a null proportion of .50 (assuming the largest variance possible for a dichotomous variable), alternative hypotheses of <.475 or > .525 and intracluster correlation of .4.

Section 2:

1. Overall, we are concerned with the complexity of the client survey, especially considering the varying cognitive ability and literacy level of the clients. While using matrix question with Likert scales saves space on the form itself, the use of matrix formatting and Likert scale are cognitively difficult for short term memory processing and require a significant amount of learning and experience before they can be understood, processed, and answered correctly. This concern is especially high with the self administered form where the grantee is only able to minimally help a client that is experiencing difficulty

2. We are also concerned about your use of Scantron forms. Scantron form is a format that a typical respondent maybe able to use with little instruction or learning, however it may confuse or frustrate the respondents in this evaluation and may not yield the desired result. While using a scantron form will expedite the processing time for the evaluation, it will most likely increase the burden of form administration which will compromise the collected data.

We recommend simplifying the survey, eliminating complex matrix and Likert scales, and placing answers on the form itself and then transferring this information onto the scantron for processing. This removes the burden off the clients and the Grantees during the evaluation administration and onto the survey team during processing. A cognitive test of the evaluation with actual clients will help identify some areas for change and adjustment.

Response:

Question 1. The choice of Likert scales, time-based questions and areas of focus were based on 1) an extensive literature review (Broner et al., 2010) that indicated that these scales have been determined to be useful, sensitive, and appropriate for the population of interest here; and to yield accurate data in comparison with collateral information; 2) consultation with a panel of national experts that included leading homeless researchers, consumers, advocates, policy makers, Grantees, and government (e.g., VA, HUD, SAMHSA) representatives; 3) a review of the surveys with Grantees and GPRA interviewers; 4) a successful pilot by a “neutral interviewer” in two sites of the baseline and 6-month interviews with active clients; 5) review by two Grantees who collected information from an additional four GBHI clients on the baseline and 6-month interviews during the feedback/comment response period; and 6) A cognitive test of the evaluation with actual clients that indicated that these measures were acceptable, easily responded to with an interviewer and self-administered without more than hearing or reading the directions and that the questions were viewed as helpful. “The responses were equally positive for Likert questions and time-based questions as they were for simpler yes/no categorical questions.

There is an extensive literature on the ability of mentally ill and substance abusing populations—including criminal justice, veterans, family, and youth subpopulations--to accurately respond to time-based questions without the recall bias assumed for this population (for example, , Banks, McHugo, Williams, Drake, & Shinn, 2002; Broner et al., 2004; Brown et al., 1993; Burt, 2009; Carey, 1997; Clark & Rich, 2003; Dartmouth Psychiatric Research Center, 1997; Ehrman & Robbins, 1994; Fischer, Shinn, Shrout & Tsemberis, 2008; Peters & Wexler, 2005; Peters et al., 2008; Milby, Wallace, Ward, Schumacher, & Michael, 2005; North, Eyrich, Pollio, & Spitznagel, 2004; Sacks, Drake, Williams, Banks & Herrell, 2003; Smith, North, & Spitznagel, 1992; Sobell & Sobell 1992; Sobell & Sobell, 1995; Tsemberis, Gulcur, & Nakae, 2004; Tsemberis et al., 2007).

Likert-based measures have generally been viewed as very effective for this population and are the primary format of standardized testing whose psychometric properties are tested and published. For example, most standardized measures developed for community-based, institutionalized and vulnerable populations use a Likert-based format to assess psychiatric symptoms, substance use, trauma, attitudes, health, choice, satisfaction, quality of life including housing, homelessness, social support and so forth as described in the Tests and Measurement Yearbooks by Spies et al., 2007 or in empirical studies with the target population (For example, Bernstein et al., 1994; Bliese et al., 2008; Broner et al., 2002; Conrad et al., 2001; Davis et al., 2009; Derogatis & Melisaratos, 1983; Eisen et al., 2000, 2004; Gardner et al., 1993; Gulcur et al., 2007; Greenwood, Schaefer-McDanile, Winkel, & Tsemberis, 2005; Keane, Newman, & Orsillo, 1997; Koenig et al.,1993; Lang & Stein, 2005; Lehman, 1988; Lehman et al., 1991; McEvoy et al., 1989; Overall & Gorham, 1988; Peters & Wexler, 2005; Peters et al., 2008; Robbins et al., 2009; Rollnick et al., 1992; Ruggiero, Del Ben, Scotti, & Rabalais, 2003; Saltstone, Halliwell, & Hayslip, 1994; Shern et al., 1994; Skinner, 1982; Spies et al., 2007; Srebnik, Livingston, Gordon, & King, 1995; Storgaard, Nielson, & Gluud, 1994; Tsemberis, Moran, Shinn, Asmussen, & Shern, 2003; Weathers, Litz, Huska, & Keane, 1994; Zung, 1979).

While Likert scales have been criticized as promoting a response toward the middle if an even number of response categories is not collected (e.g.,Spies et al., 2007; Tansella & Thornicroft, 2001;  Weiner & Craighead, 2010), or that total sum scales may not always be sensitive to each items weight (Tansella & Thornicroft, 2001; Weiner & Craighead, 2010), the overwhelming evidence is that Likert scales are easily understood by respondents (children, seriously mentally ill, elderly,.; e.g., Broner et al., 2002; Hasson & Arnetz, 2005; Patterson, O’Sullivan & Spielberger, 1980; Sheilds et al., 2003; Spies et al., 2007; Tansella & Thornicroft, 2001; Van Laerhoven et al., 2004; Vickers, 1999). While visual aids alone are less reliable and more burdensome for respondents, they can be useful to add in response to Likert matrix questions if these questions are verbally administered (e.g., Hansson & Arnetz, 2005). We plan to use show cards and have added the instructions “You may use this card [HAND RESPONDANT SHOW CARD #] to indicate your response” before the likert scales to the client survey rather than relying on this instruction appearing only in the training materials per OMB’s recommendation.

Further, current and prior OMB approved evaluations include Likert scales, time-based questions, and domains of similar complexity to that proposed here [For example, Targeted Capacity Expansion Grants for Jail Diversion Programs, OMB control number 0930-0277; SAMHSA Homeless Families, OMB control number 0930-0223; HUD Life After Transitional Housing, 2528-0239; National Outcomes Performance Assessment of the Collaborative Initiative to Help End Chronic Homelessness, OMB control number 0930-0247; Access to Recovery, OMB control number 0930-0299; Safe Dates, OMB control number 0920-0783; School Violence, OMB control number 1850-0814] and client-level government reporting performance measurements [e.g., SAMHSA CMHS NOMs Adult Consumer Outcome Measures for Discretionary Programs, OMB control number 0930-0285, regarding functioning, symptom and satisfaction; SAMHSA CSAP Participant Outcome Measures for Discretionary Programs, OMB No. 0930-0208]. Further our pilot and cognitive testing of the client survey did not reveal any difficulties for the Part II client administration section which includes a matrix format of the likert scale satisfaction survey currently administered in both interviewer and self-administered formats in other OMB approved data collection efforts (CMHS Client Outcome Measures, OMB control number 0930-0285).

Question 2. As noted by the reviewers, cost, an important consideration for government funds, and reducing data entry errors helped drive the decision to use scannable forms. The scannable software that we plan to use for programming the client survey, Teleform, integrates the survey question text, response options, and fillable bubbles on the same page as opposed to other scannable technologies that have a separate sheet with fillable bubbles for respondent answers. With the proposed Teleform technology clients will have the written response options directly next to the bubble to fill in for that response option to make responding as straightforward as possible. Scannable forms such as this have been used in prior client satisfaction data collection as part of other OMB-approved cross-site evaluations that included self-report, self-administered client surveys (e.g., SAMHSA CSAT Access to Recovery, OMB control number 0930-0299). A sample screen shot of this design is shown below.

Questionnaire specifics:

In general, we would like to see a crosswalk of each question and their justification as part of your package. We would also like to see evidence that the questions have been vetted prior to inclusion in the evaluation since all questions need to be either cognitively tested or from an already existing well established study before they can be used in this evaluation.

Response: Below per your request in table format is the item number (noting if from baseline (B) or 6-month follow-up (FU)), domain, justification, a study citation, and OMB control number if previously included in OMB study or standing program performance measure. In addition to vetting the questions with the expert panel (see section A.2, page 4 and section A.8, pages 9-11 of the Supporting Statement), SAMHSA, each of the 25 GBHI Treatment for Homeless Grantees and their local evaluators during 2010-2011 site visits, and receiving information from two Grantees that tested the interviews with a few of their clients (during the response/comment period), we conducted a pilot study with cognitive testing of the GBHI client baseline and 6-month client measures. As described above, in preparation for possible OMB approval, and to ensure feasibility of procedures, and confirm that estimates of burden, ease of administration and response were consistent for active clients with those who had recovered, we completed a pilot study. The study included two sites with active SAMHSA clients, both the baseline survey and the 6-month survey were administered and administration was varied with only the GBHI client survey and with the GBHI and GPRA client surveys. The burden estimates were as proposed and there were no problems or issues with the 1) consent process, 2) administration of the surveys, 3) responses, 4) self-administration, and 5) formatting. Cognitive testing was included for both surveys as well as the combined survey assessment. One client suggested that the response order for H11 on the baseline (H8 on the 6-month follow-up) be changed to list the response option, “Not receiving mental health treatment” as the first option. In addition to making this change, a show card has been developed for this question to aid in client responding. All found the use of show cards helpful on piloting. This is consistent with the literature reviewed above, that Likert scales are relatively easy to answer with little training for this population and that Show Cards or other visual aids can augment, but take more learning if replace the traditional scale presentation (e.g., Hannson ARnetz, 2005). The lack of problems were not surprising as the majority of measures are derived from standardized measures (for example, Ditton, 1999; Dunlap et al., 2010; Ganju, 1999; Hser et al., 1998; Kean, Newman, & Orsillo, 1997; Lang & Stein, 2005; Robbins et al., 2009; Rollnick et al., 1992; Ruggiero, Del Ben, Scotti, & Rabalais, 2003; Salome et al., 2003; Schulper et al., 2000; Secker-Walker et al., 1999; Shireman et al., 2001; Srebnik et al’s (1995);Tsemberis et al., 2004; Tucker et al., 2004; Weathers et al., 1994; Yarbroff et al., 2005 ) and or OMB approved surveys that include several of these standardized measures (for example, OMB control numbers: 0930-0277, 0930-0223, 2528-0239, 0930-0247), implemented with similar or identical populations, were familiar to the cross-site evaluation team, and recommended by the expert panel and well received by Grantees. Below in table format we list the baseline (or follow-up if noted) item number, domain, justification, literature citations and the OMB control number if previously included in a OMB approved survey. A more in depth review of measures, their development and related peer review journal articles is also available in Broner et al., 2010, the literature review prepared and disseminated for the GBHI cross-site evaluation.

;

,

GBHI Cross-Site Evaluation Client Interview measures

Item number (Base-line)	Domain	Justification	Study citation	Population	OMB Control No.
M1-4 B	Military Service	Veterans are an important subset of homeless individuals and GBHI clients; data will be used to compare veterans to non-veterans on service receipt, substance use, and mental health. Analyses of PTSD and trauma data (see PTSD instrument below) will incorporate the combat era question.	Case et al., 2009; Naples et al., 2007.	SA, MI, COD, Veterans, CJ, homeless	0930-0277
E1	Employment	Increasing client employment is an important goal of GBHI Grantees; this single item will provide an efficient measure of employment in past 6 months	Broner et al., 2009; Case et al., 2009; Naples et al., 2007.	SA, MI, COD, Veterans, CJ, homeless	0930-0277
H1-H4	Homelessness	Given GBHI program’s focus, measuring homelessness is critical to understanding program implementation	Broner et al., 2009; Rog et al., 1995; Rog & Buckner, 2007; Rog et al., 1995.	Homeless, families, MI, SA, COD	0930-0223
H5 –H9	Housing/housing stability	Given GBHI program’s focus on supporting clients to obtain housing and increase stability, it is important to understand clients’ current and recent (past 6 mo.) housing	Burt, 2009; Burt, 2010.	Homeless youth, adults and families, MI, SA, COD	2528-0239
H10	Housing satisfaction and choice of housing	The dimensions measured in this item are related to the likelihood of clients remaining in the housing in question and are therefore important to understanding GBHI program successes and challenges in helping clients to obtain viable housing	Gulcur et al., 2007; Greenwood et al., , 2005; Srebnik et al., 1995; Tsemberis et al., 2003; Lehman, 1988; Lehman et al., 1991;	MI, COD, SA, homeless and housed elderly, youth, adults and families, Veterans	0930-0247; 2528-0239
H11	Housing coercion/choice	To understand the GBHI program and the housing models that Grantees implement, it is important to understand the roles of choice and coercion in clients’ housing situations, because choice has been shown to predict better housing outcomes	Robbins et al., 2009; Tsemberis et al., 2004; Pearson et al. 2007.	Housed in supportive housing and homeless, MI, COD, CJ, Veterans	0930-0247
Cj 1-3	Criminal Justice	Homelessness and criminal justice involvement are inter-related, as reflected on one hand in the criminalization of homelessness and on the other hand in the housing challenges faced by many offenders reentering the community. These data on arrests and incarceration will be used to identify GBHI clients with criminal justice involvement which will allow for comparison against non-CJ involved GBHI clients in terms of homelessness, housing, and service receipt	Broner et al., 2009; Broner et al., 2004; Broner et al. 2005; Case et al., 2009; Naples et al., 2007; Steadman & Naples, 2005.	SA, MI, COD, Veterans, CJ, homeless	0930-0277;
TXHI-TXH8	Treatment History	These questions provide important information on severity of mental health and substance use problems that will be used in subgroup group analyses. These types of questions have been used to estimate prevalence and severity in various populations and to better understand needs as well.	Brownsberger et al., 2004; Broner et al., 2010; Ditton, 1999; Lattimore et al., 2005; Winterfield et al., 2006	CJ, Homeless, MI. COD, SA	N/A
RTC 1-2	Stages of Change	A client’s status on “stages of change” is an important predictor of receptivity to treatment and likelihood of benefitting from treatment services. Many GBHI Grantees use the concept in tailoring services to clients. Therefore, measuring it is important to describe GBHI clients, for subgroup analysis, and as a proximal outcome that may explain subsequent grantee efforts.	Rollnick et al., 1992;	SA, MI, COD, youth and adults, Homeless, CJ	Peer review literature
PCL1	PTSD/Trauma symptoms	Many homeless individuals have suffered trauma, and many GBHI Grantees implement programs or practices to serve them. These measures of PTSD are important for describing GBHI clients and programming, and for subgroup analyses.	Broner et al., 2005; Broner et al; 2008; Bliese et al., 2008; Case et al., 2009;; Kean et al., 1997; Lang & Stein, 2005; Ruggiero et al., 2003; Naples et al., 2007; Weathers et al., 1994	Veterans, CJ, primary care, MI, SA, COD, youth and adults	0930-0277
SV1-14	Services, need and receipt	These items ask about services needed and received by clients. The data are important for describing service need and receipt, assessing the extent to which need is met, and for use in models to help understand the role of various services in improving outcomes in housing and other GBHI objectives.	Gulcur et al., 2007; Greenwood et al., 2005; Rog et al., 1995; Rog & Buckner, 2007; Rog et al., 1995, Srebnik, et al., 1995; Tsemberis et al., 2003.	MI, SA, COD, Homeless youth, adults and families, Veterans, CJ	0930-0223; 2528-0239;
CTB 1-2 (Follow-up)	Cost and client burden	To comprehensively assess the barriers and facilitators to clients accessing and using treatment, it is important to understand client burdens and costs. Economic evaluations from the societal perspective should include such measures. Burden refers to client time (transportation to, waiting for, and receiving the service) and perceived obstacles, and cost refers to monetary outlays (such as bus fare and other transportation costs, fees for service, medication). The study target population of homeless may not have ready access to transportation to get to appointments, and this population is poor and thus likely price sensitive to even small fees or costs. The proposed measures follow closely those used in the growing literature on patient burden in the health care literature, including the literature on care for substance use conditions.	Dunlap et al., 2010; Hser et al., 1998; Salome et al., 2003; Schulper et al., 2000; Secker-Walker et al., 1999; Shireman et al., 2001; Tucker et al., 2004; Yarbroff et al., 2005.	Treatment population for substance use, cancer	Peer review literature
PC1-7 (B); PC1-14 (FU)	Program Satisfaction	These items tap into constructs that have been shown to relate to client participation in, and benefit from, services. The data are useful to describe Grantees, to suggest areas for grantee improvement, and to model client participation and program outputs.	Davis et al., 2009; Ganju, 1999; Case et al., 2009;	MI, COD, SA, Homeless, , Veterans, elderly, CJ, primary care	0930-0285; 0930-0277;
TCC1-9	Treatment and housing perceived; program satisfaction	Literature has consistently indicated that change is mediated by perceived choice in terms of treatment and housing; Programs all address issues of choice or lack thereof with clients and per program model; clients indicated they wanted opportunity to provide assessment—confirmed in piloting with positive response to these questions	Adapted from Srebnik et al., 1995 and Robbins et al., 2009; Greenwood et al., 2005; Tsembaris et al., 2004.	Homeless, MI, COD, SA	Peer review literature

MI =mentally ill, SA = substance abusers, COD = co-occurring mental and addictive disorders, CJ = criminal justice,

.

Also, since you are not using professional interviewers you should clearly indicate any instructions and write out all response options, especially where there are large matrix questions, directly on the form.

Response: The instructions included for questions on the baseline and 6-month GBHI client survey, see Supporting Statement Appendix 6, are the original instructions for the standardized measures or previously approved OMB surveys. We have added additional instructions in the survey to “use Show Cards” for each of the matrix questions. Also as noted in the response to Questions 1-2 in Section 10 above, the Grantee interviewers proposed for the GBHI client level data collection are trained interviewers, who have been trained by SAMHSA in face-to-face trainings and via on-line, augmented by Grantee sponsored training, on interviewing and tracking with this target population. Thus, although additional training will be offered for the GBHI cross-site evaluation client interviews and companion instructions, the interviewers for the 25 Grantees have all previously received in-person and on-line training for interviewing and tracking.

M4: You mention in the introduction the importance of knowing if a homeless individual is suffering from post traumatic stress disorder but then M4 only asks if the person has served in a combat zone. Are you planning to extrapolate that if a person was in an active military zone and is homeless that they have PTSD? Are you concerned that other data users might make this extrapolation?

Response: We do not plan to extrapolate from the question about combat to issues associated with PTSD and trauma. Information from this question will be combined with PTSD and trauma information from other questions and differentially analyzed by subpopulation. Given that several Grantees treat HHS’ priority population of veterans, and with the advice of the VA and GBHI Grantees, the combat question is included so that we can distinguish combat from non-combat veterans. PTSD is assessed through a question on the baseline, TXH5, that asks for history of a PTSD diagnosis (self-report diagnosis and treatment services is increasingly proposed as a reliable method for gathering this information from this and similar target populations; Brownsberger et al., 2004; Trestman et al, 2007; Broner et al., 2010) and through a standardized PTSD diagnostic screening and change measure, PCL1, that was developed on civilian and veterans populations (e.g., Lang & Stein, 2005; Ruggiero et al., 2003; Weathers, Litz, Huska, & Keane, 1994) as noted in section A.2, page 6 of the Supporting Statement.

H3: Are you concerned that homeless individuals may have issues with time-based retrieval? Groups without routines or set structure, like the elderly for example, often have problems with time-based recall and we would like to see some research addressing this concern in the ICR.

Response: The comments highlight a very important consideration. Due to the importance of this consideration, time-based retrieval was one of the primary issues upon which we focused in our review of the homeless literature that was disseminated by SAMHSA (Broner et al., 2010). In that review, we identified studies that indicated that time-based information could be reliably collected from chronic street, shelter and housed homeless individuals with mental illness and substance abuse (for example, Banks, McHugo, Williams, Drake, & Shinn, 2002; Clark & Rich, 2003; Milby, Wallace, Ward, Schumacher, & Michael, 2005; North, Eyrich, Pollio, & Spitznagel, 2004; Smith, North, & Spitznagel, 1992; Tsemberis, Gulcur, & Nakae, 2004; Tsemberis, McHugo, Williams, Hanrahan, & Stefancic, 2007;).

We also addressed this issue with our expert panel (members listed in section A.8, pages 9-11 of the Supporting Statement). Further, as noted above in our previous responses, follow-back (time-based retrieval) questions have been demonstrated, for self-report questionnaires, to be within the cognitive capacity of this population and to produce accurate information when compared to collateral collected information for psychiatric populations (for example, Carey, 1997; Clark & Rich, 2003; Dartmouth Psychiatric Research Center, 1997), criminal justice populations (Broner et al., 2002; Broner et al., 2004; Case et al., 2009; Lattimore et al., 2005; Napels et al., 200; Winterfield et al., 2006), substance abuse populations (Brown et al., 1993; Ehrman & Robbins, 1994; Sacks, Drake, Williams, Banks & Herrell, 2003; Sobell & Sobell 1992; Sobell & Sobell, 1995), those with co-occurring mental and addictive disorders (Broner et al., 2002; Broner, Lattimore, Cowell & Schlenger, 2004; Case et al., 2007; Clark & Rich, 2003; Naples et al., 200; Sacks et al., 2003), and homeless populations with serious mental illness and or substance use disorders (Banks, McHugo, Williams, Drake, & Shinn, 2002; Burt, 2009; Clark & Rich, 2003; Fischer, Shinn, Shrout & Tsemberis, 2008; Milby, Wallace, Ward, Schumacher, & Michael, 2005; North, Eyrich, Pollio, & Spitznagel, 2004; Smith, North, & Spitznagel, 1992; Tsemberis, Gulcur, & Nakae, 2004; Tsemberis et al., 2007).

While not incorporated into the GBHI client interview due to the additional time it would take to administer, the majority of homeless studies incorporate a time-retrieval at minimum for 5 years prior to baseline interviewing as reviewed by Broner and colleagues (2010) and per the cites on homelessness above. Other studies have reduced this to three years and 6-months follow-back, such as in the OMB approved cross-site evaluation homeless studies, including Homeless Families (OMB contract number 0930-0223), Life After Transitional Housing (OMB contract number 2528-0239) and National Outcomes Performance Assessment of the Collaborative Initiative to Help End Chronic Homelessness (OMB control number 0930-0247). Thus, we are confident about the utility of these type of data based on the voluminous research on this population, standardized measures, prior OMB approved studies using these types of time-based retrieval (and Likert matrix) questions, and the findings from our pilot study of the proposed GBHI client survey described previously.

H4: This question asks a client how long they were homeless three months ago and six months ago. So does that indicate that if a client says they were homeless twice in three months and five times in six months, that they were homeless a total of five times, twice between month 1-3 and three time between month 4-6? Also, it would make more cognitive sense to ask them about incidents that occur over six months and then ask of those incidents how many occurred in the last three months instead of the current order.

Response: Yes, that is correct. The question is three years (not three months) and precedes the 6-month question as suggested by reviewers. The questions, as noted in section A.2, page 5 of the Supporting Statement, are from standardized measures incorporated into the OMB-approved cross-site evaluation homeless studies Homeless Families (OMB contract number 0930-0223) and Life After Transitional Housing (OMB contract number 2528-0239). These questions reflect our intent to make use of standardized questions and measures previously approved by OMB, and from which data have been published in peer review journal articles (e.g., Broner et al., 2009; Burt, 2009; Burt, 2010 Rog et al., 1995; Rog & Buckner, 2007; Rog et al., 1995)..

H7: Likert scales are cognitively difficult to process especially in audile modes. We would suggest asking the question and then asking the client if they are satisfied or dissatisfied with the statement first and then in the next question ask if they were strong satisfied or dissatisfied with the statement. This kind of branching is often used for Likert scales in telephone interviews but could also be used in interviewer administered modes. That said, this question may still be too cognitively difficult for this population.

Response: As noted in our response above, we are proposing to use standardized items that have been successfully used with similar populations. We planned to use Show Cards and have added instructions to the GBHI client survey to use Show Cards for each Likert scale matrix question. While visual aids alone are less reliable and more burdensome for respondents, they can be useful to add in response to Likert matrix questions if these questions are verbally administered (e.g., Hansson & Arnetz, 2005). With regard to H7 (H10 on baseline), specifically, this is a standardized measure that was also included in the following OMB approved studies with client-level data collection, with data collected depending on site by either Grantee staff, GPRA interviewers or local evaluation staff: National Outcomes Performance Assessment of the Collaborative Initiative to Help End Chronic Homelessness (OMB control number 0930-0247) and Life After Transitional Housing (OMB control number 2528-0239). H8 (H11 on 6-month) about coercion and housing was developed for the chronically homeless, chronically homeless and recently housed youth, adult and family populations with serious mental illness and co-occurring substance use (Robbins et al., 2009). RTC 1-2, described just above, is from a standardized measure assessing stage of change. The measure was standardized and tested on alcohol, drug and co-occurring mentally ill populations (e.g., Rollnick et al., 1992). PCL1, assessing trauma symptoms is a standardized widely used measure with civilian and veteran populations and has been OMB approved for SAMHSA Targeted Capacity Expansion Grants for Jail Diversion Programs, OMB control number 0930-0277.

The Likert scale questions, included in Part II, client satisfaction, and previously discussed are from the SAMHSA CMHS NOMs Consumer Outcome Measures for Discretionary Programs (OMB control number 0930-0285), also used in SAMHSA Targeted Capacity Expansion Grants for Jail Diversion Programs, OMB control number 0930-0277, and originally from a standardized measure developed by a consumer led development committee. This measure is self or interviewer administered with effectiveness for other method with adults, youth, criminal justice involved, elderly, veterans, homeless, families, and so forth (Ganju, 1999). The use of self-administerd completed surveys with Likert type matriz questions is also common and previously approved by OMB, for instance for Access to Recovery, OMB control number 0930-0299.

Of note, standardized measures for this population, along with such vulnerable populations as children, elderly, pregnant women, criminal justice offenders, veterans are almost exclusively developed with Likert scales and provide more flexibility to the respondent and for analytic methods, as noted in a response above (for example, Bernstein et al., 1994; Bliese et al., 2008; Broner et al., 2002; Conrad et al., 2001; Davis et al., 2009; Derogatis & Melisaratos, 1983; Eisen et al., 2000, 2004; Gardner et al., 1993; Gulcur et al., 2007; Greenwood, Schaefer-McDanile, Winkel, & Tsemberis, 2005; Keane, Newman, & Orsillo, 1997; Koenig et al.,1993; Lang & Stein, 2005; Lehman, 1988; Lehman et al., 1991; McEvoy et al., 1989; Overall & Gorham, 1988; Peters & Wexler, 2005; Peters et al., 2008; Robbins et al., 2009; Rollnick et al., 1992; Ruggiero, Del Ben, Scotti, & Rabalais, 2003; Saltstone, Halliwell, & Hayslip, 1994; Shern et al., 1994; Skinner, 1982; Spies et al., 2007; Srebnik, Livingston, Gordon, & King, 1995; Storgaard, Nielson, & Gluud, 1994; Tsemberis, Moran, Shinn, Asmussen, & Shern, 2003; Weathers, Litz, Huska, & Keane, 1994; Zung, 1979) and are found to be easily understood by respondents (children, seriously mentally ill, elderly, homeless) with little to no practice (e.g., Broner et al., 2002; Hasson & Arnetz, 2005; Patterson, O’Sullivan & Spielberger, 1980; Sheilds et al., 2003; Spies et al., 2007; Tansella & Thornicroft, 2001; Van Laerhoven et al., 2004; Vickers, 1999; Weiner & Craighead, 2010).

Further as previously noted, the evidence for time-based questions is also that these types of questions have been demonstrated to not be overly burdensome or leading to significant recall bias when implemented with criminal justice, veterans, and adult, family, and youth homeless populations. For example see, Banks, McHugo, Williams, Drake, & Shinn, 2002; Broner et al., 2004; Brown et al., 1993; Burt, 2009; Carey, 1997; Clark & Rich, 2003; Dartmouth Psychiatric Research Center, 1997; Ehrman & Robbins, 1994; Fischer, Shinn, Shrout & Tsemberis, 2008; Peters & Wexler, 2005; Peters et al., 2008; Milby, Wallace, Ward, Schumacher, & Michael, 2005; North, Eyrich, Pollio, & Spitznagel, 2004; Sacks, Drake, Williams, Banks & Herrell, 2003; Smith, North, & Spitznagel, 1992; Sobell & Sobell 1992; Sobell & Sobell, 1995; Tsemberis, Gulcur, & Nakae, 2004; Tsemberis et al., 2007).

In sum, prior OMB approved SAMHSA program wide performance measurement and cross-site evaluations have used a combination of Likert matrix questions and time-based questions focused on housing satisfaction, psychiatric symptoms and functioning (including trauma), drug use and attitudes, service choice, burden and satisfaction, quality of life, social supports, homeless and residential history, military history, education and employment, and so forth, have been included in current and prior OMB approved evaluations and SAMHSA-wide client-level program performance data collection (For example, , SAMHSA Targeted Capacity Expansion Grants for Jail Diversion Programs, OMB control number 0930-0277; SAMHSA Homeless Families, OMB control number 0930-0223; HUD Life After Transitional Housing, OMB control number 2528-0239; National Outcomes Performance Assessment of the Collaborative Initiative to Help End Chronic Homelessness  , OMB control number 0930-0247; Access to Recovery, OMB control number 0930-0299; Safe Dates, OMB control number 0920-0783; School Violence, OMB control number 1850-0814; SAMHSA CMHS NOMs Adult Consumer Outcome Measures for Discretionary Programs, OMB control number 0930-0285; SAMHSA CSAP Participant Outcome Measures for Discretionary Programs, OMB control number 0930-0208).

RTC1: In general, you should not change time periods. Typical respondents when reporting their behaviors often fail to hear the change in time period or are unable to hold that much information in their short term memory. Keeping a consistent time period will improve data quality and reduce cognitive burden for the client.

Response: With regard to the RTC item we inadvertently made a typo for the gate question using 12-months rather than 6-months and this has been corrected to ensure consistency for these types of questions in the GBHI client surveys. Further, we thank the interviewers for pointing this out and we have added to and strengthened the instructions to emphasize the time- frame as a note to the interviewers. First, although shifting time frames may create cognitive burden for clients, the RTC items are from a standardized measure and changing the time frame could change the meaning of the measure as its focus is measuring the current stage of change from the client’s perspective. Second, on cognitive testing with current GBHI Grantee clients, for the pilot study, clients had no difficulty shifting time frames. Third, the burden estimates included the time to answer these questions. Fourth, the interviewer did not find delay on these questions and found that clients understand the time frame and answered accordingly. They did not find this problematic when specifically asked during post-administration cognitive testing.

REFERENCES

Banks, S., McHugo, G. J., Williams, V., Drake, R. E., & Shinn, M. (2002). A prospective meta-analytic approach in a multisite study of homelessness prevention. New Directions for Evaluation, 94, 45–59.

Bassuk, E. L., Weinreb, L. F., Buckner, J. C., Browne, A., Salomon, A., & Bassuk, S. S. (1996). The characteristics and needs of sheltered homeless and low-income housed mothers. Journal of the American Medical Association, 276, 640–646.

Bernstein, D. P., Fink, L., Handelsman, L., Foote, J., Lovejoy, M., Wenzel, K., et al. (1994). Initial reliability and validity of a new retrospective measure of child abuse and neglect. American Journal of Psychiatry, 151, 1132-1136.

Bliese, P.D., Wright, K.M., Adler, A.B., Cabrera, O., Castro, C.A., & Hoge, C.W. (2008). Validating the Primary Care Posttraumatic Stress Disorder Screen and the Posttraumatic Stress Disorder Checklist with soldiers returning from combat. Journal of Consulting and Clinical Psychology, 76 (2), 272–281.

Broner, N., Borum, R., Whitmire, L., & Gawley, K. (2002). A review of screening instruments for co-occurring mental illness and substance use in criminal justice programs. In G. Landsberg, M. Rock, L. Berg, & A. Smiley (Eds.), Serving mentally ill offenders and their victims: Challenges and opportunities for mental health professionals (pp. 289–337). New York: Springer.

Broner et al. (2010, June). Homelessness: A review of prevalence, predictors, correlates, policy and interventions. New York, NY: RTI International.

Broner, N., Lang, M., & Behler, S (2009). The effects of homelessness, housing type, functioning and community reintegration supports on mental health court completion and recidivism. Journal of Dual Diagnosis, 5(3/4), 323–356.

Broner, N., Lattimore, P. K., Cowell, A. J., & Schlenger, W. E. (2004). Effects of diversion on adults with co-occurring mental illness and substance use: Outcomes from a national multi-site study. Behavioral Sciences and the Law, 22, 519-541.

Broner, N. Lattimore, P.K., & Steffey, D.M (2010). Mental health needs and services receipt of reentering offenders: A multi-site study of men, women and male youth. In H.A. Dlugacz (Ed.) Re-entry planning for offenders with mental disorders: Policy and practice (pp. 12-1-12-41). New Jersey: Civic Research Institute, Inc.

Brown, L. S., Alterman, A. I., Rutherford, M. J., & Cacciola, J. S. (1993). Addiction Severity Index scores of four racial/ethnic and gender groups of methadone maintenance patients. Journal of Substance Abuse, 5(3), 269-279.

Browne, A., & Bassuk, S. S. (1997). Intimate violence in the lives of homeless and poor housed women: prevalence and patterns in an ethnically diverse sample. American Journal of Orthopsychiatry, 72, 261–277.

Brownsberger, W. N., Love, C. T., Doherty, P. L., & Shaffer, H. J. (2004). Potential demand for substance abuse treatment in the criminal justice system. Criminal Justice Policy Review . 15 (1), 37-60.

Burt, M. R. (2009). Life after transitional housing: Tracking homeless families after they leave HUD-assisted transitional housing. Urban Institute.

Burt, M. R. (2010). Life after transitional housing for homeless families. Urban Institute.

Burt, M. R., Aron, L. Y., Douglas, T., Valente, J., Lee, E., & Iwen, B. (1999). Homelessness: Programs and the people they serve. Washington, DC: Interagency Council on the Homeless.

Carey, K.B. (1997). Reliability and validity of the time-line follow-back interview among psychiatric outpatients: A preliminary report. Psychology of Addictive Behaviors, 11(1), 26-33.

Case, B., Steadman, H. J., Dupuis, S. A., & Morris, L. S. (2009). Who succeeds in jail diversion programs for persons with mental illness? A multi-site study. Behavioral Sciences and the Law, 27(5), 661-674.

Clark, C., & Rich, A.R. (2003). Outcomes of homeless adults with mental illness in a housing program and in case management only. Psychiatric Services, 54(1), 78–83.

Conrad, K. J., Yagelka, J.R., Matters, M.D., Rich, A.R., Williams, V., & Buchanan, M. (2001). Reliability and validity of a Modified Colorado Symptom Index in a national homeless sample. Mental Health Services Research, 3 (3), 1573-1636.

Cottler, L. B., Compton, W. M., Ben-Abdallah, A., & Horne, M. (1996). Achieving a 96.6 percent follow-up rate in a longitudinal study of drug abusers. Drug and Alcohol Dependence, 41(3), 209-217.

Dartmouth Psychiatric Research Center (1997). Drug/Alcohol 6-Month Follow-Back Calendar. Hanover, NH: Author.

Davis, L., Foster, S., & Zerger, S. (2009). Services in Supportive Housing 2009 Annual Report. Newton Centre, MA: Center for Social Innovation, Services in Supportive Housing Technical Assistance Center.

Derogatis, L.R., & Melisaratos, N. (1983). The Brief Symptom Inventory: An introductory report. Psychological Medicine, 13, 595-605.

Dunlap, L. J., Zarkin, G. A., Bray, J. W., et al. (2010). Revisiting the cost-effectiveness of the COMBINE study for alcohol dependent patients: the patient perspective. Med Care, 48, 306 –331.

Ehrman, R. N., & Robbins, S. J. (1994). Reliability and validity of 6-month timeline reports of cocaine and heroin use in a methadone population. Journal of Consulting and Clinical Psychology, 62(4), 843-850.

Eisen, S. V., Dickey, B., & Sederer, L. I. (2000). A self-report symptom and problem rating scale to increase inpatients’ involvement in treatment. Psychiatric Services, 51, 349-353.

Eisen, S. V., Normand, S. L., Belanger, A. J., Spiro, A., & Esch, D. (2004). The Revised Behavior and Symptom Identification Scale (BASIS-R): Reliability and validity. Medical Care, 42 (12), 1230-1241.

Festinger, D. S., Marlowe, D. B., Dugosh, K. L., Croft, J. R., & Arabia, P. L. (2008). Higher magnitude cash payments improve research follow-up rates without increasing drug use or perceived coercion. Drug and Alcohol Dependence, 96(1-2), 128-135.

Fischer, S., Shinn, M., Shrout, P., & Tsemberis, S. (2008). Homelessness, mental illness, and criminal activity: examining patterns over time. American Journal Of Community Psychology, 42(3-4), 251-265.

Ganju, V. (1999). The MHSIP Consumer Survey: History, development, revisions, applications, commonly-asked questions. Washington, DC: National Institute of Mental Health, Mental Health Statistics Improvement Program. Planning, Research & Evaluation, Texas Department of MHMR. Retrieved February 2, 2010 from http://www.mhsip.org/mhsiptest/documents/MHSIPConsumerSurvey.pdf.

Gardner, W., Hoge, S. K., Bennett, N., Roth, L. H., Lidz, D. W., Monahan, J., et al. (1993). Two scales for measuring patients’ perceptions of coercion during mental hospital admission. Behavioral Sciences and the Law, 11 (3), 307-321.

Greenwood, R.M., Schaefer-McDanile, N.J., Winkel, G., & Tsemberis, S.J. (2005). Decreasing psychiatric symptoms by increasing choice in services for adults with histories of homelessness. American Journal of Community Psychology, 36, 223-238.

Goodman, L., Saxe, L., & Harvey, M. (1991). Homelessness as psychological trauma. American Psychologist, 46, 1219-1225.

Gulcur, L., Tsemberis, S., Stefancic, A., & Greenwood, R. M. (2007). Community integration of adults with psychiatric disabilities and histories of homelessness. Community Mental Health Journal, 43 (3), 211-228.

Hasson, D. & Arnetz, B. B. (2005). Validation and findings comparing VAS vs. Likert scales for psychosocial measurements. International Electronic Journal of Health Education, 8, 178-192.

Keane, T., Newman, E., & Orsillo, S. M. (1997). Assessment of military-related posttraumatic stress disorder. In J. Wilson, T. Keane, J. Wilson, T. Keane (Eds.) , Assessing psychological trauma and PTSD (pp. 267-290). New York, NY US: Guilford Press.

Kline, A., Callahan, L., Butler, M., St. Hill, L., Losonczy, M. F., & Smelson, D. A. (2009). The relationship between military service era and psychosocial treatment needs among homeless veterans with a co-occurring substance abuse and mental health disorder. Journal of Dual Diagnosis, 5, 357–374.

Koenig, H. G., Westlund, R. E., George, L. K., & Hughes, D. C. (1993). Abbreviating the Duke Social Support Index for use in chronically ill elderly individuals. Psychosomatics: Journal of Consultation Liaison Psychiatry, 34(1), 61-69.

Lang, A. J., & Stein, M. B. (2005). An abbreviated PTSD checklist for use as a screening instrument in primary care. Behaviour Research and Therapy, 43, 585–594.

Lattimore, P. K., Visher, C. A., Winterfield, L., Lindquist, C., & Brumbaugh, S. (2005). Implementation of prisoner reentry programs: Findings from the Serious and Violent Offender Reentry Initiative multi-site evaluation. Justice Research and Policy, 7(2), 87–109.

Le Fauve, C. (2009, Feb.). Lessons learned from the Center for Substance Abuse Treatment homeless portfolio. Presented at the CADCA Forum XIX, National Harbor, MD.

Lehman, A. F. (1988). A Quality of Life Interview for the chronically mentally ill. Evaluation and Program Planning, 11, 51‑62.

Lehman, A. F., Slaughter, J. G., & Myers, C. (1991). Quality of life in alternative residential settings. Psychiatric Quarterly, 62(1), 35-49.

McEvoy, J. P., Apperson, L., Appelbaum, P. S., & Ortlip, P. (1989). Insight in schizophrenia: Its relationship to acute psychopathology. Journal of Nervous and Mental Disease, 177(1), 43-47.

Milby, J. B., Wallace, D., Ward, C. L., Schumacher, J.E., & Michael, M. (2005). Towards a more sensitive assessment of homelessness: The Homelessness Severity Scale. Journal of Social Distress and Homelessness, 14(3/4), 151–169.

Naples, M., Morris, L.S. & Henry J. Steadman, H.J. (2007). Factors in disproportionate representation among persons recommended by programs and accepted by courts for jail diversion. Psychiatric Services, 58, 1095-1101.

North, C. S., Eyrich, K. M., Pollio, D. E., & Spitznagel, E. L. (2004). Are rates of psychiatric disorders in the homeless population changing? American Journal of Public Health, 94, 103–108.

Overall, J.E., & Gorham, D.R. (1988). The Brief Psychiatric Rating Scale (BPRS): Recent developments in ascertainment and scaling. Psychopharmacology Bulletin, 24, 97-99.

Patterson, R. L., O’Sullivan, M. J., & Spielberger, C. D. (1980). Measurement of state and trait anxiety in elderly mental health clients. Journal of Psychopathology and Behavioral Assessment, 2 (2), 89-97.

Peters, R.H., Bartoi, M.G., & Sherman, P.B. (2008). Screening and assessment of co-occurring disorders in the justice system. Delmar, NY: CMHS National GAINS Center.

Peters, R.H., & Wexler, H.K. (Editors) (2005). Substance abuse treatment for adults in the criminal Justice system. Treatment Improvement Protocol (TIP) 44. Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment.

Robbins, P. C., Callahan, L., & Monahan, J. (2009). Perceived coercion to treatment and housing satisfaction in housing-first and supportive housing programs. Psychiatric Services, 60, 1251–1253.

Rog, D. J., & Buckner, J. C. (2007). Homeless Families and Children. In D. Dennis, G. Locke, & J. Khadduri (Eds.), 2007 Symposium on Homelessness Research (pp. 5-1 through 5-33). Washington, DC: U.S. Department of Health and Human Services and U.S. Department of Housing and Urban Development. Retrieved on October 26, 2009, from http://aspe. hhs. gov/hsp/homelessness/symposium07/rog/

Rog, D. J., Holupka, C., & McCombs-Thornton, K. L. (1995). Implementation of the Homeless Families Program: 1. Service models and preliminary outcomes. American Journal of Orthopsychiatry, 65(4), 502-513. doi:10.1037/h0085057

Rog, D. J., McCombs-Thornton, K. L., Gilbert-Mongelli, A. M., Brito, M. C., & Holupka, C. S. (1995). Implementation of the Homeless Families Program: 2. Characteristics, Strengths, and Needs of Participant Families. American Journal of Orthopsychiatry, 65, 514–528. doi: 10.1037/h0079675

Rollnick, S., Heather, N., & Bell, A. (1992). Negotiating behaviour change in medical settings: The development of brief motivational interviewing. Journal of Mental Health, 1, 25–37.

Ruggiero, K., Del Ben, K., Scotti, J., & Rabalais, A. (2003). Psychometric properties of the PTSD Checklist-Civilian Version. Journal of Traumatic Stress, 16(5), 495-502.

Sacks, J.A., Drake, R., Williams, V.F., Banks. S.M., & Herrell, J. M. (2003). Utility of the Time-Line Follow-Back to assess substance use among homeless adults. Journal of Nervous & Mental Disease. 191(3), 145-153.

Saltstone, R., Halliwell, S., & Hayslip, M. (1994). A multivariate evaluation of the Michigan Alcoholism Screening Test and the Drug Abuse Screening Test in a female offender population. Addictive Behaviors, 19(5), 455-462.

Shern, D.L., & Wilson, N.Z. (1994). Client outcomes II: Longitudinal client data from the Colorado Treatment Outcome Study. Milbank Quarterly, 72, 123-148.

Shelton, K., Taylor, P., Bonner, A., & van den Bree, M. (2009). Risk factors for homelessness: Evidence from a population-based study. Psychiatric Services, 60, 465–472. doi:10.1176/appi.ps.60.4.465

Shields, B. J., Cohen, D. M., Harbeck-Weber, C., Powers, J. D., & Smith, G. A. (2003). Pediatric pain measurement using a visual analogue scale: A comparison of two teaching methods. Clinical Pediatrics, 42(3), 227-234.

Skinner, H.A. (1982). Drug abuse screening test. Addictive Behavior, 7, 263-371.

Smith, E. M., North, C. S., & Spitznagel, E. L. (1992). A systematic study of mental illness, substance abuse, and treatment in 600 men. Annals of Clinical Psychiatry, 4, 111–120.

Sobell, L. C., & Sobell, M. B. (1992). Timeline follow-back: A technique for assessing self-reported alcohol consumption. In R. Z. Litten, J. P. Allen, R. Z. Litten, J. P. Allen (Eds.) , Measuring alcohol consumption: Psychosocial and biochemical methods (pp. 41-72). Totowa, NJ US: Humana Press.

Sobell, L. C., & Sobell, M. B. (1995). Alcohol consumption measures. In J. P. Allen, M. Columbus (Eds.), Assessing alcohol problems: A guide for clinicians and researchers. (pp. 55-73). NIH Pub. No. 95-3745, Bethesda, Maryland: U.S. Department of Health and Human Services.

Spies, R.A ., Plake, B.S., Geisinger, K.F., & Carlson, J.F. (Eds) (2007). The Seventeenth Mental Measurements Yearbook. Buros Institute of Mental Measurements.

Srebnik, D., Livingston, J., Gordon, L., & King, D. (1995). Housing choice and community success for individuals with serious and persistent mental illness. Community Mental Health Journal, 31(2), 139-152.

Steadman, H. J. & Naples, M. (2005). Assessing the effectiveness of jail diversion programs for persons with serious mental illness and co-occurring substance use disorders. Behavioral Sciences and the Law, 23(2), 163-170.

Storgaard, H., Nielsen, S.D., & Gluud, C. (1994). The validity of the Michigan Alcoholism Screening Test (MAST). Alcohol & Alcoholism, 29(5), 493-502.

Tansella, M. & Thornicroft, G. (Eds.). (2001). Mental Health Outcome Measures (2^nd ed.). London: Royal College of Psychiatrists, Gaskell.

Trestman, R. L., Ford, J., Zhang, W., & Wiesbrock, V. (2007). Current and lifetime psychiatric illness among inmates not identified as acutely mentally ill at intake in Connecticut’s jails. Journal of the American Academy of Psychiatry and Law, 35(4), 490-500.

Tsemberis, S., Gulcur, L., & Nakae, M. (2004). Housing First, consumer choice, and harm reduction for homeless individuals with a dual diagnosis. American Journal of Public Health, 94, 651–656.

Tsemberis, S., Moran, L., Shinn, M., Asmussen, S.M., & Shern, D. L. (2003). Consumer preference programs for individuals who are homeless and have psychiatric disabilities: A drop-in center and a supported housing program. American Journal of Community Psychology, 32, 305-317.

Tsemberis, S., McHugo, G., Williams, V., Hanrahan, P., & Stefancic, A. (2007) Measuring homelessness and residential stability: The residential time-ling follow-back inventory. Journal of Community Psychology, 35, (29-42).

U.S. Department of Housing and Urban Development, Office of Community Planning and Development. (2009). The 2008 annual homeless assessment report to Congress. Washington, DC: Author.

van Laerhoven, H., van der Zaag-Loonen, H., & Derkx, B. (2004). A comparison of Likert scale and visual analogue scales as response options in children's questionnaires. Acta Paediatrica, 93, 830–835. doi: 10.1111/j.1651-2227.2004.tb03026.x

Vickers, A. J. (1999). Comparison of an ordinal and a continuous outcome measure of muscle soreness. International Journal of Technology Assessment in Health Care, 15, 709-716.

Weathers, F. W., Litz, B. T., Huska, J. A., & Keane, T. M. (1994). PCL Checklist-Civilian Version. Boston: National Center for PTSD, Behavioral Science Division.

Weiner, I. (Ed.), & Craighead, W. (Eds.). (2010). The Corsini encyclopedia of psychology, Vol 2 D–L (4th ed.). Hoboken, NJ: John Wiley & Sons Inc.

Winterfield, L., Lattimore, P. K., Steffey, D. M., Brumbaugh, S. M., & Lindquist, C. H. (2006). The serious and violent offender reentry initiative: Measuring the effects on service delivery. Western Criminology Review, 7(2), 3–19.

Zung, B.J. (1979). Psychometric properties of the MAST and two briefer versions. Journal of Studies on Alcohol, 40(9), 845-859.

Attachments:

IRB Approvals

Baseline and 6-Month Follow-up Client Survey