CSE SS Part B FINAL_090816

Community Support Evaluation

Supporting Statement

2. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Exhibit 1 displays the expected number of respondents to each data collection activity.

Exhibit 1. Number of Respondents by Data Collection Activity

Instrument	Participating Grantees	Respondents per Grantee	Total Respondents
BHTCC Study
BPI–BHTCC	17	1	17
SLA KIIS	17	5 per administration	170 (85 in Y2; 85 in Y4)
Concept Mapping Exercise 1 (Brainstorming & Sorting/Rating)	17	20	340
Concept Mapping Exercise 2 (Sorting/Rating)	17	20	340
Concept Mapping Exercises 3 & 4 (Brainstorming & Sorting/Rating)	10 (5 each)	20 per exercise	200 (100 per exercise)
18-Month Tool	17	1	17
Comparison Study Tool (BL & 6-Month)	2	1	2
18-Month Tool (Comparison Study)	2	1	2
SE Study
BPI–SE	7	1	7
SSA KIIs	7	5 per administration	70 (35 in Y2; 35 in Y4)
ENFG – Employer	7	6 per administration	84 (42 in Y2; 42 in Y4)
ENFG – Employment Specialist	7	6 per administration	84 (42 in Y2; 42 in Y4)

Using the Census of Grantees

For the Behavioral Health Treatment Court Collaborative (BHTCC) Evaluation a census of grantees (17) is being used for the following reasons:

• Key Informant Interviews (KIIs) – KIIs are being conducted for a system level assessment to understand enhancements and expansions at the grantee-level. Data gathered from each site will be used to assess the service infrastructure, capacity, and delivery of the BHTCC program. The data will be gathered two times and allow for description of change over time both within and across BHTCC sites. Finally, the KII system level assessment data can be transformed and incorporated into the client-level data as a mediating outcome variable for analysis of consumer outcomes. Given the site (grantee) specific analysis and use of the data, it is necessary to gather information from all grantees.

• Concept Mapping (CM) – CM will be conducted at the site level to assess the components of the BHTCC collaborative that are most important in supporting program participants in program adherence and recovery. Site specific (and cross-site) concept maps will be generated to depict consensus and variance in program stakeholder’s perspectives of the most important supports for program participants. Grantees can use this information to understand where there is agreement about key program components and where there is areas for refinement or change in the program supports to address stakeholder/consumer/needs. Given the site-specific analysis and maps to be provided back to grantees, it is necessary to conduct this activity with all grantees.

For the Supported Employment (SE) Evaluation a census of grantees (7) is being used for the following reasons:

• Key Informant Interviews (KIIs) – KIIs are being conducted to understand the scalability and sustainability of SE programs. Given that laws and state-level policies related to access and use of a range of health and social services will be site-specific, it is necessary to collect data from each grantee. Grantee-level contexts (laws, policies, funding mechanisms) are important aspects to assess when determining whether and how the SE program may be sustainable and scalable and the needs and resources of both site and cross-site programs to continue in these efforts.

The respondent universe and sampling methods are described below for the following data collection activities: SLA KIIs, Concept Mapping, 18-Month Abstraction Tool, Comparison Study Abstraction Tool, SSA KIIs, and ENFGs.

• BHTCC SLA KIIs: up to 170 stakeholders total will participate in the SLA KIIs across BHTCC grantees (i.e., 5 from each grantee) in Years 2 and 4 using purposive sampling to ensure a range of perspectives from BHTCC program stakeholders. The Contractor will work with SAMHSA to determine appropriate stakeholders for the KIIs. Participation will be based on the roles and responsibilities within each grantee. To identify participants, grantee program staff (e.g., project directors, local evaluators) will obtain written consent to contact from potential respondents and forward the forms to the evaluation team. Respondents will include: (1) court personnel (e.g., BHTCC administrators, coordinators, judges, and attorneys), (2) service providers (e.g., case managers and BHTCC peers), and (3) consumers (e.g., clients and family members). It is important that that KII respondents have sufficient experience and knowledge of the program activities to share during the interviews. Thus, it is necessary for grantees to share a pool of respondents from each group and respondents will be arbitrarily selected by the contractor and recruited until the maximum number of interviews has been conducted within type and grantee (i.e., up to five interviews per grantee and at least one respondent per type).

• Concept Mapping: up to 880 BHTCC grantee stakeholders total will participate in concept mapping through the use of purposive sampling. In Years 2–3, up to 340 stakeholders will participate in Exercise 1 (local concept map). In Year 4, up to 340 stakeholders will participate in Exercise 2 (KTR Map 1), 100 in Exercise 3 (KTR Map 2), and 100 in Exercise 4 (KTR Map 3). Respondents will include: court personnel, BHTCC peers, and consumers (clients and their families). Similar to the KIIs, the CM activity will request BHTCC stakeholder participation across court, service, and consumer (participants, peers, and families) groups. The purpose of CM is not to generalize findings to all grantees; rather, CM is intended to uncover critical aspects of court supports (across a range of stakeholders) that helps and hinders recovery and program adherence.

• SE SSA KIIs: up to 70 stakeholders total will participate in the SLA KIIs across grantees (i.e., 5 from each BHTCC grantee) in Years 2 and 4 using purposive selection to ensure State- and local-level perspectives on issues related to scalability and sustainability of SE programs. The Contractor will work in collaboration with SAMHSA, grantee program directors, and local project coordinators to create a list of potential respondents for each State, acknowledging that the relevant stakeholders may differ from state to state. KII interviews will be conducted to assess the scalability and sustainability of SE programs. The KIIs will be conducted with two respondent types: administrators of the program and service providers. Potential participants include program directors; State-level agency directors (e.g., Medicaid); members of the SECC; and service providers from each local implementation site, who can give insight into the experiences of SE clients and how the project functions on the ground. Once the list is determined, the evaluation team will contact participants directly for recruitment and scheduling purposes.

KIIs will be used to gather information about both state and local level resources, infrastructure, program achievements, and barriers and facilitators of program activities. It will be important that potential respondents are knowledgeable about SE program operations as well as the policies and regulations that support or hinder program implementation. Thus, it will be necessary to purposively recruit persons who have sufficient experience to answer the questions in the interviews. Grantees will provide a pool of respondents to the contractor of persons that both fit the respondent category type and who have sufficient experience in the role to provide information during the KIIs. The contractor will recruit from this list until the maximum number of interviews have been conducted within respondent type and within site. Findings are not intended to be generalized to all SE programs; rather, this information will be used to understand the aspects of grantee and cross-grantee program operations that support and hinder scaling and sustaining programs beyond the grant funded period.

• SE ENFGs: up to 84 employment specialists and 84 employers (e.g., hiring managers and supervisors) total will participate in the ENFG employment specialist and employer versions, respectively, across SE grantees in Years 2 and 4. Six of each respondent type will participate from each grantee; respondents will be recruited from two implementation sites at each grantee. Convenience sampling will be used to identify and recruit employment specialists. Purposive sampling will be used to identify and recruit employers who are knowledgeable of supported employees’ status as program participants. Most program participants have not disclosed program participation; thus, employment cases where consumers have disclosed participation will be targeted for participation.

The following data collection activities are reports on grant activities or existing data abstractions required from every grantee, so no sampling is required. Respondents to these activities will be grantee program staff and/or project evaluators and court clerks (BHTCC Comparison Substudy).

• BPI–BHTCC / BPI–SE: the BPI will be administered on a biannual over the course of the grant period, in the month following the end of the FY quarters 2 and 4. Each grantee designates a program staff respondent. Sampling is not required.

• 18-Month Abstraction Tool (for BHTCC participants): the 18-Month Tool will be completed for the census of BHTCC participants at each grantee (e.g., 100 participants per grantee). In addition, the 18-Month Tool will be completed for comparison cases at each participating grantee until data on 260 offenders has been abstracted.

• Comparison Study Abstraction Tool (BL and 6-Month): the Comparison Study Tool will be completed for the census of comparison study cases at each participating grantee until data on 260 offenders has been abstracted.

2. Information Collection Procedures

   1. BHTCC Study

BHTCC System Change Study

Biannual Program Inventory–BHTCC

Each BHTCC grantee will assign a respondent to complete the BPI. One month after the end of FY quarters 2 and 4 (April and October), respondents will receive a BPI password via e-mail and use the password to log in to the survey on the CSEDS. The respondent must finalize the submission by the end of the administration period, which lasts for 15 business days. All BPI entries will be reviewed to ensure data quality. BPI respondents will be provided with technical assistance via e-mail (e.g., help email) for any questions.

System Level Assessment KIIs

SLA KIIs will consist of one-on-one interviews with stakeholders from each of the 17 BHTCC programs. The Contractor will conduct original, qualitative data collection in Years 2 and 4 by conducting one-on-one KIIs with stakeholders from each of the 17 BHTCC programs. In Year 2, KIIs will focus on gathering information and feedback on implementation processes and outcomes related to adherence to the model; service infrastructure, capacity, and delivery processes, the service array; management structure and oversight, reward and sanction models; trauma-informed practices; collaboration among BHTCC participants; and facilitators and barriers to collaboration. In Year 4, the KIIs will focus on similar indicators as assessed in Year 2, in addition to changes over time and specific plans for future implementation.

The Contractor will work with SAHMSA to identify appropriate stakeholders for participation in the KIIs. Recruitment of KII participants will be initiated through contact with the BHTCC project directors for each grantee, and participation will be based on the roles and responsibilities within each grantee site. For each site, interviews will be conducted with court personnel, service providers, and participants (and their families) in BHTCC. Recruitment of KII participants will be managed by selected grantee program staff (e.g., project directors, local evaluators) who will obtain written consent to be contacted to participate in a KII and will send the forms to the evaluation team. Each interview will be conducted by one evaluation team member remotely, either via telephone or Skype. Interviews will last approximately 60 minutes each and be audio-recorded for analysis. The interviewer will use a semi-structured interview guide developed and tailored for different respondents at each site to gather the most pertinent information from interview participants.

At the start of the interview, the interviewer will obtain verbal consent by reading aloud the informed consent statement and obtain verbal consent from each respondent. The consent statement describes the purpose of the study, how the information will be used, and the steps that will be taken to protect participant privacy. The respondent also will be asked to provide verbal consent to be audio-recorded. Interviews will be semi-structured and conversational, but will follow a specific set of questions developed to collect data on identified evaluation questions and indicators of particular relevance and interest. Following the interviews, audio recordings will be transcribed to facilitate qualitative analysis.

Concept Mapping

Concept mapping will include the following steps:

1. A planning phase, anticipated to take 1–3 months: The planning phase is anticipated to be longer for the first concept mapping activity (closer to 3 months), and shorter for the subsequent concept mapping activities (approximately 1 month). In this phase, the focus prompt will be determined on the basis of input from SAMHSA, the BHTCC grantees, and the evaluation team; a lead contact at each BHTCC site will be identified; the potential inclusion of pre-existing brainstormed ideas that incorporate community support principles, trauma-informed care principles, and other clearly understood principles associated with BHTCC will be determined. The purpose of potentially incorporating these key foundational principles would be to avoid the need for BHTCC grantees to “reinvent the wheel” during brainstorming. Instead, participants would brainstorm on more nuanced items specific to their sites rather than spending time on core foundational principles. Participants would be able to “see” the list of already identified core ideas and add their own brainstormed ideas.

2. Training of BHTCC grantees and coordination of data collection activities, anticipated to take 1–3 months: Training and coordination of activities is anticipated to take longer for the first concept mapping activity (approximately 3 months), and shorter for the subsequent concept maps (approximately 1 month). Training is anticipated to occur during the quarterly training webinars. The BHTCC study team will determine whether one training for all 17 BHTCC grantees or smaller group training will best facilitate implementation of concept mapping. As part of the training process, a list of all expected BHTCC participants will be created by the lead contact at each site. Coordination of data collection activities at each site also will be carried out with the lead contacts and the evaluation team. The Concept System software supports carrying out brainstorming, sorting/rating, and interpretation via a web-based program on personal computers. The feasibility of web-based participation with all BHTCC stakeholders will be assessed and coordinated during this phase; specifically, how to accommodate involvement of consumers and their families (or any other stakeholders) who may not have computer access will be addressed including a plan for how and where concept map activities will be carried out and troubleshooting any potential computer access issues. Note that concept mapping activities can also be done manually, including by phone or with paper-and-pencil formats as a potential back-up to computer access issues.

3. Concept mapping data collection activities, anticipated to take 12–18 months for the 17 BHTCC local site-maps and up to 3 months for each cross-site map: All concept mapping activities will be carried out remotely, with the evaluation team available via phone or webinar. At each site, brainstorming is anticipated to take approximately one month; sorting and rating is anticipated to take an additional month; and interpretation of initial maps will occur remotely via webinar for each site.

BHTCC Consumer Outcome and Comparison Substudies

18-Month Abstraction Tool

At 18 months after baseline/intake for any client enrolled in the BHTCC program, grant staff will extract risk assessment and recidivism data from existing databases, including all data from the period between baseline and 18 months. The evaluation team will provide an Excel tool to record the data in a predefined format. All risk assessment and recidivism records will be identified by the performance monitoring system client ID to enable linkage across the two client-level data sources. No identifying information will be provided (i.e., just the client ID). Beginning in year 2, grantees will upload all extracted data on a quarterly basis. In their final upload (last month of grant activity), grantees will include data for all clients not currently submitted including those enrolled less than 18 months.

Comparison Study Client Level Abstraction Tool and 18-Month Tool

SAMHSA has proposed the matched comparison data collection as an enhancement to the BHTCC Consumer Outcome Study, to be administered in no more than two comparison sites (it is recognized that challenges will be involved for identifying matched comparison samples within grantee programs [i.e., wait listed participants] or with recruiting non-BHTCC courts [for participation as a comparison site]). Ideally, comparison samples will be drawn from the grantee site (i.e., the BHTCC program court) with individuals who are eligible but wait-listed for services as controls, because they will be more similar to program participants than individuals who opt out of services. However, preliminary reviews of grant program plans suggest that limited BHTCC programs have a sufficient number of clients waiting for services. For the matched comparison samples, 130 matched comparison cases per comparison site, distributed across 3 years of data collection will result in adequate power to detect differences in consumer outcomes such as recidivism gathered through the 18 month data abstraction tool. Further, it is yet unknown whether the wait-listed individuals will be waiting for services long enough to participate in 6 months of data collection. These are considerations that will be taken into account when selecting the two comparison study samples/sites (BHTCC wait list or non-BHTCC comparison site) for inclusion in the comparison study. If sufficient control cases exist using the wait-list approach, comparison sites will be recruited for a matched sample of individuals who are eligible but who opt out of services or go through an alternative court in a neighboring jurisdiction. To use this approach, control measures (e.g., prior criminal history, prior jail time, and presence of eligibility criteria) will need to be gathered for the control sample through a non-BHTCC program comparison site. This approach has been used in other research studies comparing participants in mental health and traditional courts (Moore, & Hiday, 2006).

In addition, client-level information on comparable offenders who are not being provided services will be gathered. The evaluation team will work with local program staff on strategies for recruitment of a non-BHTCC court for a matched control group sample, gaining buy-in, and emphasizing the value of their perspective and contributions. The evaluation team also will work with local BHTCC program staff on strategies to abstract data from matched control cases (i.e., wait-listed cases). For example, staff can schedule data abstraction for times and locations when individuals are already participating in regularly scheduled court appearances.

Power Analysis

SAMHSA estimates the minimum detectable difference in the proportion of clients with rearrests, recommitments or revocations at 18 months between two groups of clients defined by intake characteristics (e.g. gender, race-ethnic identification) or program participation (e.g. clients successfully engaged in the program during six months) with 80% power at a 5% significance level. Given the high prevalence of recidivism, the maximum variance for a proportion (i.e., 0.5) will be used for the power analysis. Two scenarios regarding the relative size of the groups that will be compared are presented: in the first one, the two groups have equal size; in the second one, one of the groups represents only 10% of the sample.

The 18-Month Data Abstraction Tool will use a census rather than a sampling approach. Because this data collection activity does not rely on interviews to clients, a significant amount of attrition is not anticipated. Missing data could arise, however, in the case of grantees unable to access this information. By the end of the program, it is anticipated that follow up information on approximately 100 clients per site will be collected (i.e. clients enrolled since program inception through the first half of the third year).

The estimated minimum detectable difference across sites is 7% (for groups of equal size) 12% (if one group represents only 10% of the sample) assuming 15 out of the 17 grantees are able to participate. On the other hand, the minimum detectable difference using the sample for a single site would be 28% (for groups of equal size) to 47% (if one groups represents only 10% of the sample). As a reference, the ATCC evaluation found differences of 14% in the prevalence of one or more mental health problems at 6 months, and as high as 16% in the proportion of respondents who reported spending 1 or more nights in jail or in the 30 days prior to the interview. Thus, while the 18-Month Data Abstraction Tool will generally not support precise estimations within each site, it will achieve adequate power to detect differences across sites. Furthermore, the framework for the power analysis is a highly simplified version of the proposed analytical approach (involving survival analysis) and provides, therefore, conservative estimates of the power. Exhibit 2 contains a power analysis.

Exhibit 2. Power Analysis

Source/Data Collection Activity	Target Sample Size		Effective Sample Size		Minimum Detectable Difference
	Per Site	Across Sites	Per Site	Across Sites	Per Site	Across Sites
Client-level TRAC (CPD) data	130	1,820	144	2,022	12%	3%
18-Month Tool (BHTCC Clients)	20	280	22	311	30%	8%
Matched comparison client-level data (Comparison Study)	130	260	144	289	23%	16%

Given the high prevalence of recidivism (assuming 50% chance of being rearrested), the maximum variance for a proportion (i.e., 0.5) was used in the power analysis. We used the minimum detectable difference (MDD) that defines the difference between the means of a treatment and the comparison group that must exist to detect a statistically significant effect as a parameter in the power analysis. We assumed that the estimated minimum detectable difference across sites is 7% (for groups of equal size) 12% (if one group represents only 10% of the sample) assuming 15 out of the 17 grantees are able to participate. Under this assumption, two scenarios regarding the relative size of the groups that will be compared are presented: in the first one, the two groups have equal size; in the second one, one of the groups represents only 10% of the sample.

Concerning design effect, the calculation of the effective sample size involves design effect (DEFF), thus effective sample size = actual (target) sample size/DEFF. Note that, in the proposed study, sites are considered as strata and not as clusters. Our power analysis shows that simple random sampling (SRS) is more efficient than cluster sampling (note that DEFF < 1).

In sum, the use of client-level data for the Consumer Outcome Study will allow us to detect differences of interest with adequate power both within each site and across sites. The primary data collection activities, on the other hand, will generally not support precise estimations by site, but will achieve adequate power to detect difference across sites. In the case of the Enhanced Outcome Study, the power will be adequate to detect differences in outcomes with rather larger changes such as those registered for recidivism indicators in the ATCC evaluation.

2. SE Study

SE System Change Study

Biannual Program Inventory–SE

Each SE grantee will assign a respondent to complete the BPI. One month after the end of FY quarters 2 and 4 (April and October), respondents will receive a BPI password via e-mail and use the password to log in to the survey on the CSEDS. The respondent must finalize the submission by the end of the administration period, which lasts for 15 business days. All BPI entries will be reviewed to ensure data quality. BPI respondents will be provided with technical assistance via e-mail (e.g., help email) for any questions.

Scalability/Sustainability Assessment KIIs

The evaluation team will conduct five KIIs with stakeholders from each SE grantee in Years 2 and 4. The Contractor will work in collaboration with SAMHSA, PRA, and state program directors to create a final sample for each state, acknowledging that the relevant stakeholders may differ from state to state. Selected grantee program staff (e.g., project directors, local evaluators) will assist with the recruitment of KII participants by obtaining written consent to be contacted to participate in a KII and sending the forms to the evaluation team. Once the sample is determined, the evaluation team will contact participants directly to schedule purposes. KIIs are expected to last approximately 60 minutes and will be conducted via Skype) when possible or by telephone. KIIs will be recorded with permission from participants and transcribed verbatim to capture responses in their entirety.

The KII instrument is a semi-structured guide with a basic foundation for all States and respondents that can be tailored to the specific state context and respondent’s role within the SE program by selecting specific questions and probes. These guides can be modified to include specific probes on the basis of the state context or information gleaned from BPI data and document review. For example, the sustainability section of each guide might include probes addressing information found through review of the states’ sustainability plans. This instrument design ensures that critical topics will be covered with all participants while allowing them to address issues specific to their context and role. KIIs in Year 2 focus on infrastructure and motivations for scale up and sustainability, as well as steps already in place to achieve those goals. Data collection in Year 4 will rely on a similar sample, most likely returning to the same participants or others who occupy the same role, and will use the same KII framework. While covering the same topics, Year 4 KIIs will follow up on how the infrastructure and activities taking place in Year 2 come to fruition, and will emphasize outcomes and intentions for the program beyond the grant cycle.

At the start of the interview, the interviewer will obtain verbal consent by reading aloud the informed consent statement and obtain verbal consent from each respondent. The consent statement describes the purpose of the study, how the information will be used, and the steps that will be taken to protect participant privacy. The respondent also will be asked to provide verbal consent to be audio-recorded. Interviews will be semi-structured and conversational, but will follow a specific set of questions developed to collect data on identified evaluation questions and indicators of particular relevance and interest. Following the interviews, audio recordings will be transcribed to facilitate qualitative analysis.

Employment Needs Focus Groups

The evaluation team will conduct two ENFGs within each of the 7 SE grantee States (i.e., one with employment specialists, one with employers) in Years 2 and 4. Each ENFG will consist of approximately 6 participants. Grantees will assist with recruitment by asking local implementation sites for the contact information of employment specialists and employers that are currently engaged in the program. Target participants in the Employer focus groups will include hiring managers or supervisors from employers currently engaged in the program whose employees have disclosed their status. Potential participants will receive e-mails and/or telephone to request their participation in the focus groups. The Employment Specialist focus groups will take approximately 90 minutes, while the Employer focus groups are designed to last approximately 60 minutes. Respondents will be asked to complete an online consent form and share times that they are available within a specific window for completing the focus group. Once the evaluation team has identified a time that works for a sufficient number of respondents, participants will receive a calendar invitation with information and a link to log into the online meeting.

The ENFG instruments are designed to elicit a variety of perspectives from participants around the aforementioned topics, and to promote interaction between group members. ENFGs are virtual focus groups hosted by the Contractor through Adobe Connect or a comparable online meeting platform; the use of such a platform raises distinct considerations for how to ensure the aforementioned design goals are achieved. In addition to the moderator, who will guide discussion and ensure active participation from all participants, each ENFG will also use two additional support staff. A note taker will be present to capture broad themes in the discussion and note the ordering of participants’ contributions to aid in matching transcript text to participants. Further, a third technical support person will attend all focus groups to troubleshoot issues that may occur with the online meeting platform and minimize the effects of technical difficulties on the overall flow of the discussion.

3. Methods to Maximize Response Rates

Participation in the CSE is a requirement for BHTCC and SE Program grantees. Therefore, completion of the BPI–BHTCC, BPI–SE, and 18-Month Tool by program staff will be required. However, the CSE team has taken a number of steps to minimize the burden on local programs to ensure that completion in data collection is timely. These steps include (1) developing a Web-based data collection system; (2) using updated technology to conduct activities; (3) scheduling data collection administration windows to allow for maximum participation; and (4) providing training and technical assistance to grantees, including hosting web trainings, assigning a one-to-one technical assistance liaison, and identifying specific procedures to improve participation in all aspects of the evaluation. Additional steps are described below.

• SLA/SSA KIIs: Methods that will be used to maximize response rates for qualitative interviews include obtaining buy-in from key program stakeholders, providing flexibility in scheduling, and conducting follow-up phone calls and e-mails to non-responders. In addition, local program staff will be utilized to obtain contact information for respondents, which will result in more accurate information, thus increasing response rates. The evaluation team will contact respondents at least three times to invite them to participate in the interviews. If any identified respondents for the qualitative interviews are nonresponsive, the CSE team will request that local program staff identify replacement respondents.

• Concept Mapping: To maximize participation by a large number of respondents, the evaluation team will assist each grantee in developing a local recruitment strategy for concept mapping. To the extent possible, participants will be recruited at already existing meetings (via flyers given to them by program staff), in small groups, or directly through grant staff.

Exhibit 3. Concept Mapping Recruitment Strategy by Respondent Type

Respondent Type	Recruitment Strategy Options
Court Personnel	Project director shares recruitment flyer directly with BHTCC team. CSE staff to share recruitment flyer with any team members that project director cannot locate or for whom it would be better for CSE staff to contact directly.
Service Providers	Lead contact talks to peer specialists individually or in a small group, shares recruitment flyer, and solicits involvement in a similar manner as Strategy A for Consumers (below).
Consumers–Clients	Lead contact recruits from already existing meetings by providing a flyer with the relevant information, verbally going over that information, and answering any questions. Participants will be asked to indicate if they wish to participate via a check-box on a half-sheet sheet in which they’ve written their contact information, if they wish to be contacted to participate. All participants can turn in their half-sheets in a “voting box” approach (or in a manner in which it is easy for people to refuse to participate without anyone knowing). Participants can still refuse to participate later, but the point is that the Site Contact does both recruitment and leaves with a list of consumers who have expressed an interest to participate. Invitation to participate is given to treatment staff (or other BHTCC staff) who meet with consumers individually. Recruitment flyer is shared with a contact number for lead contact to provide further information and get actual consent at that later point (and not with treatment or BHTCC staff).
Consumers– Family Members	Same as Strategy A for Consumers–Clients. Conduct snowball sampling in which consumers share the flyer with their family members, and families contact Site Contact (or TAL?) via telephone if they wish to inquire further about participation. Any treatment staff or BHTCC court staff who have contact with families to share recruitment flyer. Family members who are interested contact Site Contact (or TAL?) via telephone if they wish to inquire further about participation.

• ENFGs: Methods that will be used to maximize response rates for focus groups include obtaining buy-in from key program stakeholders, providing flexibility in scheduling, and utilizing technology that allows respondents to remain anonymous to the group if they so choose. Grantees will assist with recruitment by asking local implementation sites for the contact information of employment specialists and employers that are currently engaged in the program. Potential participants will be contacted via e-mail and/or telephone to request their participation in the focus groups.

Comparison Substudy

The Comparison Substudy will use nonparticipants as a control group. Exhibit 4 describes the plan to identify and recruit comparison sites to complete the Comparison Study Tool and 18-Month Tool.

Exhibit 4. Comparison Study Implementation Plan

Step	Description
Establish Eligibility Criteria	The evaluation team will establish eligibility criteria that will detail the minimal requirements for grantee participation in the Comparison Study and review BHTCC grant applications to narrow the list of potential sites that meet the criteria. Grantees proposing service enhancement were required to report the number of additional clients to be served for each year of the proposed grant. Similarly, grantees proposing service expansion were required to report on how the expansion would be achieved (e.g., reduction in waiting lists, partnering with a new agency to provide the specific services enhancement). This information will be used to identify areas where eligibility is currently exceeding service capacity and where recruitment from a waiting list is most achievable.
Select Grantees for Comparison Study	The evaluation team will follow up with grantees meeting the eligibility criteria to conduct a feasibility assessment to identify two BHTCC comparison samples (either BHTCC or non-BHTCC sites) where access and abstraction of control cases is most practicable. Working with local BHTCC program staff, the evaluation team will seek to understand the screening and eligibility determination process for identifying program participants; the process for placing individuals on a wait list; the average length of time eligible individuals spend on a wait list; and whether the capacity issues are great enough to yield a sufficient number of individuals to serve as control cases for the Comparison Study. If possible, one site implementing service expansion and one site implementing service enhancement will be selected to participate. In the event that BHTCC grantees do not offer sufficient samples for control cases, the Contractor will work with SAMHSA and BHTCC grantees to identify two comparison courts for comparison study samples. Courts that are in close geographic proximity and who have similar program/court eligibility criteria will be considered. The Contractor will develop MOU with the comparison study court site to abstract data on a matched comparison sample of eligible offenders. Offender priors, criminogenic risk, and age at first offender are abstracted from all BHTCC grantees through the 18-Month Tool and will be abstracted for the matched comparison samples. These variables will be used to match the control and BHTCC offender outcome data from the two groups.
Train Staff on Recruitment	The evaluation team will train BHTCC program staff on strategies for recruiting a comparison court for participation or for abstracting comparison study data from within a wait-listed approach, including how to respond to common barriers to participation. Our training will include various scenarios that local data collectors may encounter with data abstraction.
Train Staff on Data Collection	The evaluation team will train BHTCC program staff or comparison site court staff on abstracting data for the baseline and 6-month Comparison Study Tool and 18-Month Tool.

4. Tests of Procedures

As new measures were developed, standard instrument development procedures including review of the literature; item development; and content review by individuals from SAMHSA, the grantees, and the BHTCC and SE steering committees (see Section A.6.b for information on organizations and individuals that reviewed the data collection activities). A pilot of the proposed BPI was conducted with 5 grantees (3 BHTCC and 2 SE) to solicit feedback (e.g., item clarity, relevance to program, burden). Additional changes were made to these protocols to minimize respondent burden. Further, the 18-Month Abstraction Tool was developed with input from the steering committee and grantees to ensure no additional burden and direct data collection with program participants.

web-based instruments will undergo usability testing prior to fielding. Usability testing refers to pilot testing of the interface for administering questionnaires to determine the most efficient and understandable presentation. Typically, this is completed with a prototype and modifications are made before final fielding.

5. Statistical Consultants

The Contractor has full responsibility for the development of the overall statistical design, and assumes oversight responsibility for data collection and analysis. Training, TA, and monitoring of data collection will be provided by the NOE team. The individuals responsible for overseeing data collection and analysis are:

Robin Davis, PhD

ICF International

3 Corporate Square, Suite 370

Atlanta, GA 30329

Phone: (404) 592-2188

E-mail: [email protected]

The following individuals will serve as statistical consultants to this project:

Robin Davis, PhD

ICF International

3 Corporate Square, Suite 370

Atlanta, GA 30329

Phone: (404) 592-2188

E-mail: [email protected]

Lucas Godoy Garraza, PhD, Statistician

ICF International

Teleworks—Home Office

E-mail: [email protected]

Megan Brooks, MA

ICF International

Teleworks—Home Office

Email: [email protected]

Christine M. Walrath, PhD

ICF International

40 Wall Street, 34^th Floor

New York, NY 10005

Phone: (212) 941-5555

E-mail: [email protected]

The agency staff responsible for receiving and approving contract deliverables is:

Marian Scheinholtz, MS, OT

Center for Mental Health Services, SAMHSA

5600 Fisher Lane

Rockville, MD 20857

Phone: 240-276-1911

E-mail: [email protected]

Alyson Essex, PhD, MHS, Alternate Contracting Officer Representative

Center for Behavioral Health Statistics and Quality

5600 Fishers Lane–Room 15E61D

Rockville, MD 20857

Phone: 240-276-0529

E-mail: Alyson.[email protected]

References

Berman, G., & Feinblatt, J. (2005). Good courts: The case for problem-solving justice. New York: The New Press.

Bureau of Justice Assistance. (n.d.-a). What are problem-solving courts? Retrieved January 26, 2015, from https://www.bja.gov/evaluation/program-adjudication/problem-solving-courts.htm

Department of Health and Human Services, Office of Inspector General. (1993). Revitalizing the Community Support Program. Retrieved October 3, 2014, from https://oig.hhs.gov/oei/reports/oei-05-92-00120.pdf

Lurigio, A. J., & Snowden, J. (2009). Putting therapeutic jurisprudence into practice: The growth, operations, and effectiveness of mental health court. The Justice System Journal, 30(2), 196–218.

National Association of Drug Court Professionals. (n.d.). Problem solving courts: Addressing a spectrum of issues. Retrieved January 28, 2015, from http://www.nadcp.org/learn/what-are-drug-courts/models/problem-solving-courts

Pennsylvania Department of Public Welfare. (2013). Pennsylvania Community Support Program. Retrieved October 3, 2014, from http://www.dpw.state.pa.us/communitypartners/pennsylvaniacommunitysupportprogramcsp/index.htm

Substance Abuse and Mental Health Services Administration. (2001). Overcoming barriers to community integration for people with mental illness. Rockville, MD: Author. Retrieved January 26, 2015, from http://www.ahpnet.com/
communications/documents/OvercomingBarriers.aspx

———. (2014a). Leading change 2.0: Advancing the behavioral health of the nation 2015–2018. HHS Publication No. (PEP) 14-LEADCHANGE2. Rockville, MD: Substance Abuse and Mental Health Services Administration.

———. (2014c). Transforming lives through supported employment. (Request for Applications No. SM-14-011). Rockville, MD: Author. Retrieved January 26, 2015, from http://www.samhsa.gov/grants/grant-announcements/sm-14-011

U.S. Bureau of Labor Statistics. (2014). National Compensation Survey Statistics. Retrieved April 2015 from http://www.bls.gov/ncs/ocs/

Attachments

1. Biannual Program Inventory—BHTCC Version

2. SLA KII—Court Personnel Version (Year 2, Year 4)

3. SLA KII—Service Provider Version (Year 2, Year 4)

4. SLA KII—Consumer Version

5. Concept Mapping—Brainstorming Activity

6. Concept Mapping—Sorting/Rating Activity

7. 18-Month Client-Level Data Abstraction Tool

8. Consumer Study Tool—Baseline

9. Consumer Study Tool— 6-Month

10. Comparison Study Tool—18-Month

11. Biannual Program Inventory–SE

12. SSA KII—Administrator Version (Year 2, Year 4)

13. SSA KII—Service Provider Version (Year 2, Year 4)

14. ENFG—Employer Version (Year 2, Year 4)

15. ENFG—Employment Specialist Version (Year 2, Year 4)

16. SLA KII Verbal Consent Script

17. SSA KII Verbal Consent Script

18. ENFG Employer Informed Consent

19. ENFG Employment Specialist Informed Consent

9