Targeted Capacity Expansion Grants for Jail Diversion Programs
1) Circumstances of Information Collection
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) is requesting a revision from the Office of Management and Budget (OMB) for the data collection on the Targeted Capacity Expansion (TCE) Grants for Jail Diversion Program (OMB No. 0930-0277), which expires on July 31, 2010. Since 2002, CMHS has funded seven cohorts grants under Targeted Capacity Expansion (TCE) Grants for Jail Diversion program. These grants fund the diversion of individuals with mental illness from the criminal justice system to mental health treatment and appropriate support services.
The Jail Diversion grants awarded in FY2008 includes a special focus on trauma and veterans. The 2008 cohort of grantees differ from previous cohorts of grantees through the specific focus on trauma informed services and the identification of veterans as a priority population. The programs and services implemented by FY2008 Grantees have a decidedly different evaluation focus that requires modification of the existing data collection tools currently approved under OMB No. 0930-0277. CMHS is requesting OMB approval for these two closely-related, though separate data, collection efforts as follows:
1) CMHS is requesting OMB approval for the continuation of data collection with the instruments approved under OMB No. 0930-0277 for data collection of the FY2006 and FY 2007 grantees.
2) CHMS is also requesting approval for revisions to the data collection instruments approved under OMB No. 0930-0277, for data collection by the jail diversion programs funded in FY2008 and thereafter (with funding through 2015). The following data collection forms are included under this request: Baseline Interview (Attachment A), 6-Month Interview (Attachment B), 12-Month Interview (Attachment C), Arrest Data Collection Form (Attachment D), and Mental Health and Substance Abuse Service Use Data Collection Form (Attachment E).
The requested revisions only affect the three interviews to be administered by the most recent round of TCE Grantees (those awarded in FY2008 and any subsequent awards) and include: 1) Substituting CMHS NOMS items for GPRA items. At the time of the previous OMB submission, the NOMS measures were not finalized. 2) Replacing the DC trauma Screen with a new set of traumatic event questions. The new trauma questions better reflect the experiences of the target population. 3) Replacing the Colorado Symptom Index with the BASIS 24. 4) Adding questions related to military service experience at the baseline. These items will be added to capture characteristics of the target population of the new grantee cohort, veterans. 5) Adding questions on military combat experience at the six month interview only. These items will capture the types of traumatic experiences among clients with a combat history. 6) Adding questions on lifetime mental health/substance use and service use and the CAGE to the baseline. These questions will be added to capture client’s history of involvement with mental health and substance abuse systems, and the four CAGE items assess alcohol dependence. 7) Adding several lifetime criminal justice questions. These questions will assess client’s lifetime involvement with the criminal justice system.8) Adding the Recovery Enhancing Environment (REE) instrument to all interviews. The REE is a consumer oriented measure of recovery, a new and important program outcome. 9) Removing the MacArthur Perceived Coercion Scale from all instruments. 10) Removing the Mental Health Statistics Improvement Program questions from follow-up interviews. (These are replaced by a similar, but shorter, NOMS scale.)
All grantees must evaluate the process of planning and implementing the program and participate in a cross-site evaluation of the impact of the program described later in this document. Data collection for Grantees of the Targeted Capacity Expansion Grants for Jail Diversion Programs is mandated under the program’s legislation: Public Health Service Act, Section 520G, 42 USC Sec. 290bb-38 “Grants for Jail Diversion Programs”.
Historical Background
The TCE Jail Diversion Program is congressionally mandated under the Public Health Service Act, section 520G. It is coordinated with the Department of Justice's solicitation "Mental Health Court Grants Program," authorized in P.L. 106-515, Part V, Section 2201. The overall goal of this collaboration is to improve policy and practice for addressing the needs of persons with a mental illness or co-occurring mental health and substance abuse disorders who become involved with the criminal justice system. The term “jail diversion” refers to programs that divert individuals with mental illness and often co-occurring disorders in contact with the justice system from jail and provide linkages to community-based treatment and support services. The individual thus avoids or spends a significantly reduced time in jail and/or lockups on the current charge or on violations of probation resulting from previous charges. Over the past decades, jail diversion programs have been offered as a viable and humane solution to the criminalization and inappropriate criminal detention of individuals with mental disorders. Diverting certain individuals from jail to community-based mental health treatment has been heralded for its potential benefits to the criminal justice system, the community, and the diverted individual.
Grantees build service capacity by developing and implementing a strategic plan for creating a service delivery system for jail diverted persons, building the infrastructure to support the service delivery system, and providing treatment services directly or by arranging for them to be provided. Treatment services must be based on the best known practices and include case management, Assertive Community Treatment, medication management, integrated mental health and substance abuse treatment, psychiatric rehabilitation, and gender based trauma services. Grantees coordinate with social service agencies to ensure that life skills training, housing placement, vocational training, job placement, and health care are available to diverted persons.
The goals for the TCE Grants for Jail Diversion are to:
Divert persons with mental illness and/or co-occurring substance abuse disorders from jails to community based mental health;
Provide either directly or indirectly, treatment services that are based on best known practices; and
Promote the development of a comprehensive service delivery system.
The program also aims to improve access to and quality of treatment to persons from racial/ethnic minorities and rural settings and to foster cultural competence. The three main goals are to:
a) Create service linkages between individuals and groups that serve the targeted population (e.g., mental health and substance abuse service providers and criminal justice system personnel). This includes:
developing partnerships and coalitions among mental health, substance abuse and criminal justice systems to increase systems integration
developing specific linkages among key personnel in each system.
b) Undertake community outreach to communicate to the larger community the importance of mental health and the capacity of the jail diversion program to serve people with mental illness. Required activities include:
building consensus among stakeholders and potential stakeholders for the adoption, implementation, and evaluation of the jail diversion program
ensuring that services are available for the target population
ensuring that the community accepts the use of the services as beneficial.
c) Engage in program evaluation and dissemination to demonstrate program outcomes and the quality and completeness of services implementation. This includes:
collecting required Government Performance and Results Act (GPRA) data
obtaining, at minimum, an 80 percent response rate at each data collection point
disseminating program findings, including relevant materials directed to consumers, service providers, administrators, and community, state, and federal policy makers who need this type of knowledge.
New Cohort of Grantees
For the most recent cohort of grantees, those funded in FY2008, both the program focus and structure of the program is slightly different. Whereas prior Grantees focused on developing local programs, the purpose of the FY2008 program is to support local implementation and Statewide expansion of trauma-integrated jail diversion programs to reach the growing number of individuals with post traumatic stress disorder (PTSD) and trauma related disorders involved in the justice system. This two-pronged approach is necessary because diversion occurs locally in cities and counties where jails are operated and mental health services are delivered. However, policies and funding at the State level dramatically impact those local operations and services. The State approach will bring together government officials who develop and define State policies and funding with stakeholders who are impacted by the untreated trauma of veterans to develop knowledge dissemination and application strategies for the entire State.
The focus on trauma services is also a point of difference between FY2008 and prior cohorts. Unlike previous cohorts, FY2008 Grantees are required to screen, assess, and treat clients for trauma, whereas in the past grantees might conduct these activities voluntarily. In recognition of the dramatically higher prevalence of trauma related illnesses among veterans and the increasing number of combat veterans returning from duty PTSD and other mental illnesses, this program prioritizes eligibility for veterans.
Working with local communities, the State Mental Health Agencies (SMHAs) will determine the most suitable site or sites within the State to implement pilot projects to divert veterans and others with PTSD from jail to community based trauma-integrated services. Project participants must be adults involved in the criminal justice system with a priority given to veterans. Participants may be recruited from single or multiple intercept points along the justice continuum including first contact with law enforcement, initial detention, court hearings and community corrections. For those who are incarcerated, funds may be used only to prepare inmates for reentry prior to release.
Grantees must convene State Advisory Committees that should include representatives from State departments of corrections, mental health, rehabilitation, parole and probation, State National Guard, Veterans Affairs, State judiciary, State Medicaid, pilot sites, veterans’ organizations, their families, provider organizations and universities interested in the study and treatment of trauma. The State Advisory Committee must maintain contact with pilot communities, oversee the projects’ evaluations and develop and implement plans for Statewide dissemination and implementation.
Grantees must screen and assess clients for the presence of co-occurring substance use (abuse and dependence) and mental disorders and use the information obtained from the screening and assessment to develop appropriate treatment approaches for the persons identified as having such co-occurring disorders.
Since the program’s inception, 34 grants have been awarded to local programs (FY2002-2007) and six grants have been awarded to states (FY2008). Six additional awards to States are anticipated in FY2009.
Advocates for Human Potential’s (AHP) role with respect to the SAMHSA/CMHS TCE jail diversion Grantees
AHP was funded to conduct the cross-site evaluation of the newest grantee cohort (FY2008) in September 2008. This evaluation has two overarching goals: (1) to determine the extent to which trauma-informed treatment, services and supports implemented through the CMHS Jail Diversion program result in improved client outcomes, particularly for veterans; and (2) to document grantee implementation of screening and treatment strategies in pilot projects statewide.
The evaluation will include two discrete but interrelated pieces: a client outcome evaluation focusing on outcomes across pilot program sites, and a process evaluation documenting the strategies and practices used in the pilot programs and the adoption and expansion of these strategies in the Statewide infrastructure. There are two key sources of data for the client outcomes evaluation: information gathered from clients through individual in-person interviews and information collected through secondary sources on arrests, services, and events. The key sources of data for the process evaluation include semi-annual grantee reports; two face-to-face site visits; and a review of documents and records produced in the state-level change process.
As the cross-site evaluator, AHP is responsible for developing the data collection instruments, distributing them to grantees, and training grantees on administration. The local evaluators in each State are responsible for gathering the information. AHP is responsible for receiving the all client-level and secondary data from the grantees, cleaning and managing the data, and conducting statistical analysis. Data from the Baseline Interview (Attachment A), 6-Month Interview (Attachment B), and 12-Month Interview (Attachment C) instruments; the Arrest Data Collection Form (Attachment D); and the Mental Health and Substance Abuse Service Use Data Collection Forms (Attachment E) are reported to AHP.
Policy Research Associates (PRA) TAPA Center has been funded as the coordinating center for technical assistance for Jail Diversion grantees since 2002. The responsibilities for this task include: conducting site visits; providing technical assistance in all areas of program implementation and evaluation; fostering and facilitating consumer involvement; fostering and facilitating the development of gender-specific and trauma services; and organizing and planning a one-day technical assistance meeting. The TAPA Center is also responsible for the coordinating the cross-site evaluation for grantees prior to 2008. Currently, FY2006 and FY2007 cohorts are collecting data under the direction of the TAPA center, and these processes and instruments will remain the same as the prior OMB submission.
Purpose and Use of Information
The purpose and use of the tracking and outcome data collected under this program is to meet the congressional mandate to evaluate the program. TCE Grantees must submit tracking data bi-monthly and outcome data as collected (at least monthly). The tracking data will be used to measure eligibility screening activity and to monitor evaluation progress. Both of these uses allow for oversight of Grantee funding utilization. Outcome data, including interview and record review data, will be used to measure the success of the jail diversion programs through changes in mental health, substance use and criminal justice involvement measures.
SAMHSA/CMHS and the AHP will use the TCE Initiative’s data collection information to report findings on:
The breadth and volume of activities (e.g. screening, assessment, evaluations) necessary to identify and enroll people for diversion.
The determination of those who are eligible/ineligible for diversion and their characteristics.
The biases in determining who gets diverted.
Services, including evidence based practices that are most effective in which settings among which populations.
Improvements expected over time as a result of services received through jail diversion programs include:
reduced arrests/less time spent in jail,
reduced substance use,
higher functioning/improved mental health and/or
improved physical health
reduced trauma symptoms
and expansion of trauma informed services.
It is expected that the information collected through this evaluation will be of particular value to the Grantees, as well as to all levels of government and the private sector.
Measures Collected Through the TCE Initiative
There are three primary data sources utilized in this evaluation.
Interview Data – The baseline, 6- and 12- month interviews are administered by Grantee staff to consenting jail diversion program enrollees. The following discussion of client interview instruments is separated by FY2006-2007 Grantees and FY2008 and future Grantees.
1) FY2006-2007 Grantees
The Government Performance and Results Act (GPRA) measures - Because these measures are approved under OMB No. 0930-0208, the program is not requesting approval of the burden for this instrument. However, the GPRA Client Outcome Measures constitute the main components of all three of the interviews, These measures are required by CMHS include questions related to the following areas:
Demographics (baseline interview only)
Education, employment, and income
Drug and Alcohol use
Family and living conditions
Crime and criminal justice status
Mental and physical health problems and treatment
The DC Trauma Collaboration Study Violence and Trauma Screening –This screen inquires about events that have been upsetting or stressful in the respondent’s life including the witnessing of violence and the experience of physical and/or sexual abuse.
Posttraumatic Stress Checklist – This scale measures the prevalence and severity of posttraumatic stress symptoms based on DSM-IV criteria.
Perceived Coercion Scale (from MacArthur Mandated Community Treatment Survey) – This scale contains questions that ask about how one felt about entering the jail diversion program at baseline and about how one felt about receiving outpatient mental health services at 12 months.
Mental Health Statistics Improvement Program (MHSIP) – This section includes a subset of items from the full MHSIP Consumer Survey to assess access to treatment, quality/appropriateness of treatment, general satisfaction, consumer perceptions of the effectiveness of services, and the cultural sensitivity of care.
Colorado Symptom Index 1991 – This scale asks a series of questions about the occurrence (in the past month) of symptoms related to anxiety, depression, disturbed thought processes and perceived interference with normal activities in order to gauge the individual’s psychiatric symptoms status.
Services Used – This section was designed to obtain very basic service use information to be used as a starting point by the data collection staff in conducting the Service Use record review (described below). Grantee staff collects this information at 6 and 12 months only.
Interviewer Observation Questions – Asks the interviewers to report on the respondent’s understanding, cooperativeness and accurateness during the interview process and is collected after each of the three interviews. These questions provide some information regarding the validity of the data collected.
2) FY2008 and anticipated future Grantees
For the FY2008 and future grantees, the interviews are composed of the following sections, and separated by existing sections and revised sections.
EXISTING SECTIONS
The Government Performance and Results Act (GPRA) measures:
Demographics (baseline interview only)
Education, employment, and income
Drug and Alcohol use
Family and living conditions
Crime and criminal justice status
Mental and physical health problems and treatment
Posttraumatic Stress Checklist – This scale measures the prevalence and severity of posttraumatic stress symptoms based on DSM-IV criteria. The addition of this scale to each of the three interviews will provide information about the extent of trauma symptoms, as well as the incidence rates of trauma and re-trauma.
Services Used – This section was designed to obtain very basic service use information to be used as a starting point by the data collection staff in conducting the Service Use record review (described below). Grantee staff collects this information at 6 and 12 months only.
Interviewer Observation Questions – Asks the interviewers to report on the respondent’s understanding, cooperativeness and accurateness during the interview process and is collected after each of the three interviews. These questions provide some information regarding the validity of the data collected.
REVISED SECTIONS
The CMHS National Outcome Measures (NOMS) measures: Because these measures are approved under OMB No. 0930-0285, the program is not requesting approval of the burden for this instrument. These measures, as well as those embedded in the GPRA items above, are required by CMHS.
Functioning
Social Connectedness
Questions on lifetime mental health/substance use treatment services. These brief questions were added to assed client’s involvement with the mental health and/or substance abuse service system. Collected at Baseline only.
The CAGE – This four item instrument was added to assess alcohol dependence. Collected at Baseline Only.
Revised questions on lifetime and recent traumatic events will replace the DC Trauma Collaboration Study Violence and Trauma Screening. The new items were developed to better reflect the experiences of the target population. These questions inquire about eight traumatic events that may have cover the range of trauma client are likely to have experienced, including: witnessing or experiencing physical and sexual abuse. These questions are based on the Post-Traumatic Distress Disorder Scale (PDS)1. In the evaluation its purpose is non-clinical and intended to provide a sense of the level and recentness of trauma in the jail diversion population, and whether experiences may be related to outcomes. Lifetime and recent questions will be asked at baseline, and only questions about recent experiences will be asked at the follow-up interviews.
Questions on military service experience. These new questions were added to understand the characteristics of the target population of the priority population for the new cohort. This information is collected for descriptive purposes and is only collected at baseline.
Questions on military combat experience. These questions were developed to capture the types of experiences veterans may have encountered while on active duty which may contribute to their mental health symptoms. These questions were adapted from Hoge et al. 20042 and will be asked at the 6-month only.
Behavior and Symptom Identification Scale (BASIS 24) will replace the Colorado Symptom Index.- This scale is designed to measure symptoms and functional difficulties experienced by consumers seeking mental health services. The scale gives an overall score and scores for six subscales for the following domains of psychiatric and substance abuse symptoms and functioning: Depression and Functioning, Relationships, Self-Harm, Emotional Lability, Psychosis, and Substance Abuse. These questions are included in all three interviews.
Recovery Enhancing Environment (REE)- This instrument is designed to collect information on the client’s assessment of their personal recovery by focusing on the quality of their life and their relationships, and not on symptoms. This will be included in all interviews.
Benefits of Interview Additions/Revisions (FY2008 Grantees and subsequent Grantees only) – The benefits of including the proposed new items above include the following:
The lifetime Questions on Mental Health/Substance Treatment and Criminal Justice Involvement- will provide some descriptive information on program clients that may potentially discriminate between client outcomes and program completion. These items are not collected elsewhere.
Questions on military service and military combat experiences will provide descriptive information that applies to the priority population for grantees. Further, information on military combat will help us understand the breadth of experiences and whether they differentiate outcomes. These items are not collected elsewhere.
The REE instrument will provide information on the client’s recovery from his/her own perspective and how this relates to other symptom based outcomes. Consumer oriented measures of recovery are a new area and important area of focus, that are not otherwise measured by the TCE evaluation.
The revised lifetime trauma questions provides more detailed definitions of traumatic experiences, and includes a component to assess if an event occurred before the age of 18. This will provide important descriptive information on the history of trauma, and how this may relate to client outcomes.
The BASIS 24 is simpler to administer than the Colorado Symptom Index and offers broad range of mental health diagnoses.
Tracking Data
Events Tracking Data (Attachment D) – The Events Tracking Program is designed to help the funded Jail Diversion TCE sites and AHP count the number of people considered for jail diversion program eligibility and to measure the assessments conducted to screen those potential enrollees. In addition, information on individuals (e.g. charges, sex, race, age) involved in each event is entered into the program (this information is not linked to individuals and is thus not identifiable). Many programs have multiple layers of assessment before enrolling a client in a jail diversion program (though not all sites have each layer), including:
Initial screening, to determine mental illness and overall potential eligibility for the diversion program. This is usually done by a police officer, jail nurse, booking officer, pretrial services worker or other jail diversion staff person.
Subsequent assessment, usually done by jail diversion case manager, boundary spanner, or non-Ph.D./M.D. clinician, and often focuses on clinical issues.
Subsequent Evaluation, usually an intensive psychiatric evaluation done by a Ph.D. clinical psychologist or psychiatrist.
Court Decision (in post-booking program), where the court accepts the plan negotiated between client, legal defense and prosecutor (or other relevant parties).
The same for may be used for all four assessment points. All of the data collected through Events Tracking form is collected and entered by Grantee staff using information obtained either directly or indirectly from the potential diversion program enrollees.
REVISION for FY2008 and future grantees- The only revisions to the form is the addition of the military service question and modification to response codes for the ‘Most Serious Charge’ item, both of which are relevant to the specific target population for these grantees.
Person Tracking Data (Attachment E)- The Person Tracking program helps funded Jail Diversion TCE sites and AHP to keep track of individuals’ Baseline, 6-Month and 12-Month interview dates and to record basic information on all individuals who are diverted. The basic information gathered on each diverted individual includes:
Demographics
Diagnosis
Charge level and category
Point of diversion (e.g. pre-booking, post-booking, probation violation)
Condition of diversion (e.g. deferred prosecution, condition of bail, deferred sentencing, etc.)
Target arrest/incident date, release date, program enrollment date
All of the data collected through the Person Tracking Program is collected and entered by Grantee staff using information obtained either directly or indirectly from the diversion program enrollees.
REVISION for FY2008 and future grantees- The only revision to the form is the addition of the military service question, which is relevant to the target population for this program.
Record Review Data
Service Use data – The Mental Health and Substance Abuse Service Use Data Collection form (Attachment F) collects information from official sources such as statewide/agency management information systems or other agency records about the types of services received following diversion if available. Types of treatment include: emergency room services, other crisis services (e.g. mobile crisis services), psychiatric inpatient/hospital services, outpatient services, case management services, medication management/monitoring, residential treatment/community living arrangements, detoxification services, vocational/rehabilitation services, community support services (e.g. homeless outreach services, representative payee services), and jail services. Service use data is collected 6 months post-baseline for all major service providers and 1 year post-baseline for all hospitalizations and emergency room usage.
REVISIONS for FY2008 and Future Grantees- There are no changes to data collection form itself. However, several coding categories were added to capture trauma specific and other services relevant to the program.
Arrest and Jail Days data – The Arrest History Data Collection Form collects pre- and post-diversion arrest information from official sources (such as a statewide criminal justice database). Information collected includes dates of arrests, charges, and jail days occurring one year prior and one year following diversion.
All record review data is collected by Grantee staff through official sources (i.e., directly from service providers or criminal justice databases). The completed forms entered in the QDS Data Entry program. The data encrypted files are submitted to AHP on an established schedule.
REVISIONS for FY2008 and Future Grantees- There are two minor additions to this form: the addition of two items, one asking the staff to note the inclusion of any drug/alcohol related charge and one to note the inclusion of crime against person/ domestic violence charge. Further, the charge codes have been modified.
All the revisions to data collection instruments noted above are minor, and are not expected to lengthen or alter the time to collect these data.
Data collected through this evaluation will have an impact on municipalities in adopting jail diversion strategies through the extension of ideas, concepts, and program models learned from this and other SAMSHA/CMHS funded programs. Information coming out of this evaluation will also be relevant to the 40 Grantees, all levels of government, and the private sector. For the FY2008 cohort, the data collected from the program site will help States determine the effectiveness of strategies to divert veterans and others with PTSD from jail to the community based services, develop policies and approaches to trauma training, screening, treatment and recovery statewide. SAMHSA/CMHS and AHP Center are planning to disseminate findings to the field through national conference presentations, papers, and journal articles. This dissemination plan will provide municipalities with information regarding development, implementation, and outcomes of jail diversion programs.
3) Use of Information Technology
FY2006-2007 Grantees: Grantees collect this data from official sources or self-report data from their programs and submit it to the TAPA Center. This data is reported through an electronic database system or through paper copies. Resulting compiled data is used to provide information of interest to policy makers, researchers, and communities engaged in developing jail diversion programs
FY2008 and future Grantees: Grantees collect this data from official sources or self-report data from their programs using Questionnaire Development System (QDS) software. For client interviews, an automated computer assisted interview program will be developed by AHP and distributed to grantees. QDS software has many advantages for data collection, including: security features (User ID and password protection; and data encryption) and features that reduce data entry time and improve data quality (skip patterns; prohibits out-of-range values; defines and codes missing values consistently across all interviews). Data collected from official sources will be data entered into QDS instrument programs. Encrypted data will be transmitted by grantees on a monthly basis to AHP via email. This data will be compiled in a database, and used to provide information of interest to policy makers, researchers, and communities engaged in developing jail diversion programs.
4) Efforts to Identify Duplication
This data collection is significant only to this program and is not collected anywhere else.
5) Involvement of Small Entities
There is no significant impact on small entities or small businesses.
6) Consequences if Information Collected Less Frequently
Participation in the cross site evaluation by program participants is voluntary. Each jail diversion program participant is approached to request their consent for participation in the evaluation. Participants who grant consent are interviewed at Baseline (within 7 days of enrollment into the jail diversion program), at 6 months from the Baseline interview (within a 60-day window), and at 12 months (within a 60-day window). This follows the GPRA/NOMS timeframes. The arrest data collection is captured beginning 1 year after diversion. The service use data are collected beginning 6 months after diversion. Client interview data is submitted electronically to the AHP monthly with completed record review forms submitted at least once annually.
7) Consistency with the Guidelines in 5 CFR 1320.5(d)(2)
This information collection fully complies with 5 CFR 1320.5(d)(2).
8) Consultation Outside the Agency
The Federal Register notice required by 5 CFR 1320.8(d) soliciting comments on the information was published on April 20, 2009, Vol. 74, No. 74, p. 17960-17962. There were no public comments received.
An Evaluation Advisory Committee was established for this cross-site evaluation protocol for the FY2008 grantees. The Advisory Committee included SAMHSA/CMHS, AHP, the TAPA Center, and 2008 Grantees (including evaluators, program directors/staff, and consumer representatives). A series of conference calls were scheduled and all data collection instruments were presented to the committee for review and comment. Based in the feedback from the Evaluation Advisory Committee, modifications were made to the instruments and final revisions were shared before publishing in the Federal Register. AHP also will provide grantees with trainings and supportive materials on each evaluation component as well as participant protections. Annual meetings between Grantees, federal project officers and AHP project staff review evaluation components and collection efforts.
Name/Title |
Address |
Contact Information |
John Hornik, Ph.D. Evaluation Director
|
Advocates for Human Potential 324 Broadway Two E-Comm Sq. Albany, NY 12207 |
Phone: (518) 729-1242
|
David Morrissette, Ph.D. Government Project Officer |
Center for Mental Health Services, SAMHSA |
Phone: (240) 276-1912 |
Kristin Stainbrook, Ph.D. Quantitative Co-Director
|
Advocates for Human Potential 324 Broadway Two E-Comm Sq. Albany, NY 12207 |
Phone: (518) 729-1241
|
Amy Salomon, Ph.D. Process Evaluation Co-Director |
Advocates for Human Potential 324 Broadway Two E-Comm Sq. Albany, NY 12207 |
Phone: (978) 261-1409
|
Linda Frisman, Ph.D. Connecticut Evaluator |
CT. Dept. of Mental Health & Addiction Services 410 Capital Ave., MS # 14 RSD Hartford, CT 06106 |
Phone: 860-418-6788 |
William Fischer, Ph.D. Massachusetts Evaluator |
University of Mass Medical School 55 Lake Avenue North Worcester, MA 01655 |
Phone: 508-856-8711 |
Susan Pickett-Schenk, Ph.D. Illinois Evaluator |
University of Illinois At Chicago Dept. of Psychiatry 1601 W. Taylor St. PI578A Chicago, IL 60612 |
Phone: 312-355-1503
|
Organizations and individuals that reviewed the additional sections include the following:
9) Payment to Respondents
Individual Grantees may provide incentives to program enrollees to participate in the data collection efforts. Without providing these small incentives, clients would be significantly less likely to participate in the initial interview and/or subsequent follow-up interviews. Examples of incentives used include bus passes, grocery store and restaurant vouchers, phone calling cards, and clothing. Often incentive items are received as donations from local businesses. In cases where cash incentives are used, the amount provided does not exceed $20. Further, each Grantee must undergo human subjects review by an Institutional Review Board (IRB) with prisoner certification, consistent with Protection of Human Subjects Regulations (45 CFR 46) to receive approval for use of any incentives proposed to be given to respondents in exchange for their participation. These incentives are only provided for individual respondents, and do not apply to data collected from other agencies, programs or service providers affiliated with the Grantee site.
10) Assurance of Confidentiality
SAMHSA will not receive any individual level data. The activities for each cohort of Grantees are as follows:
FY2006-2007 Grantees: The TAPA Center provides trainings via conference call to all Grantees regarding appropriate procedures for meeting and maintaining privacy protection. These trainings include information on obtaining IRB approval, obtaining appropriate releases from participants that comply with state and federal law, including the Health Insurance Portability and Accountability Act (HIPAA), and insuring privacy protection from interviewers and other Grantee staff. In addition to the trainings, the TAPA Center provides the Grantees with materials that cover all of the information presented in the trainings with the intention that they be used by current Grantee staff for their own reference and to train any new staff regarding privacy protection procedures.
FY2008 and future Grantees: AHP will conduct telephone and in-person trainings on maintaining privacy protection of client information. AHP will conduct in-person training on the client interview, QDS administration, and database management, including appropriate procedures for meeting and maintaining privacy protection. AHP provides teleconference training on Institutional Review Board procedures, obtaining appropriate releases from participants that comply with state and federal law, including the Health Insurance Portability and Accountability Act (HIPAA), and insuring privacy protection from interviewers and other Grantee staff. In addition to the trainings, AHP will provide the Grantees with materials that cover all of the information presented in the trainings with the intention that they be used by current Grantee staff for their own reference and to train any new staff regarding privacy protection procedures.
11) Questions of a Sensitive Nature
There are questions of a sensitive nature such as drug and alcohol use that has been approved as GPRA measures under the OMB approval No. 0930-0208, and the trauma questions have been approved under OMB No. 0903-0277. The purpose of data collection is to evaluate the effectiveness of jail diversion programs, and a focus of the new cohort of grantees is developing trauma informed services. Program clients are approached to request their consent to participate in the interview. Each Grantee must develop procedures to obtain informed consent and seek IRB approval for this process. AHP will provide interviewer training that addresses how to handle questions of a sensitive nature and how to develop safety procedures for that outline how interviewers should handle potentially dangerous or unsafe situations in order to ensure their own physical safety. Additionally, questions about traumatic experiences, in particular military traumatic experiences may be difficult for clients to handle. AHP will provide training to help grantees develop a safety protocol, to monitor evaluation participants for signs of distress and help connect them to services. An example of a Consent Form currently in use by a Jail Diversion Grantee may be found in attachment M.
12) Estimates of the Annualized Hour Burden
The following tables present the information on the annualized burden for grantees. The estimates of burden for the client interviews are separated by the FY2006-2007 Grantees and the FY2008 and future Grantees. The burden for Record Management Data is combined as the burden is the same for all Grantees. However, there no Event Tracking or Person Tracking Data anticipated for FY2006 Grantees because their client enrollment has been completed.
The total amount of time that is estimated for completion of the client interview, record management by Grantee staff, and data submissions by the Grantees is 1,290 hours in CY2009, 2,999 hours in CY2010, 4,327 hours in CY2011, and 3,440 hours in CY2011. The annualized hourly costs to respondents are estimated be $4,970 in CY2009, $31,854 in CY2010, $37,957 in CY2011, and $28,530 in CY2012. The increase in the in the annualized burden estimates for the TCE Initiative is due to fact that a new cohort of Grantees is expected to begin data collect in FY2010 and FY2011. The number of grantees increases to 14 in FY2010, and another 6 are anticipated in FY2011. The annualized burden decreases slightly in FY2011, because the FY2008 cohort of grantees will be tapering data collection that year and the FY2007 cohort will have completed their programs. The burden estimates, summarized in the following table, are based on the reported experience of previous SAMSHA/CMHS Grantees and contractors in compiling, completing, and reporting this same data for previous funding cohorts of this grant program. More senior Grantee staff is expected to handle the data extraction and submission at an average salary of $25/hour (as estimated for SAMHSA’s Treatment Episode Data Set (TEDS), OMB No. 0930-0106). The minimum wage rate of $7.25 was used to calculate the hourly burden for client interviews. The estimated wage rate of $15 for Grantee staff who conducts record management is based on the experience of the GPRA Services (OMB No. 0930-0208).
CY 2009 Annual Reporting Burden |
|
||||||||||||||||||||||
Data Collection Activity |
Number of Respondents |
Responses per Respondent |
Total Responses
|
Average |
Total |
Hourly |
Total Hour Cost |
|
|||||||||||||||
Hours per |
Hour |
Rate |
|
||||||||||||||||||||
Response |
Burden |
|
|
||||||||||||||||||||
Client Interviews for FY2008: Revised Instrument |
|
||||||||||||||||||||||
Baseline (at enrollment) 1 |
510 |
1 |
510 |
0.95 |
485 |
$7.25 |
$0 |
|
|||||||||||||||
6 months |
408 |
1 |
408 |
0.92 |
375 |
$7.25 |
$93 |
|
|||||||||||||||
12 months |
102 |
1 |
102 |
0.92 |
94 |
$7.25 |
$72 |
|
|||||||||||||||
Sub Total |
1,020 |
|
1,020 |
|
954 |
|
$165 |
|
|||||||||||||||
Client Interviews for FY2006-2007 Grantees: Current Instrument |
|
||||||||||||||||||||||
Baseline (at enrollment) |
5 |
1 |
5 |
0.83 |
4 |
$7.25 |
$421 |
|
|||||||||||||||
6 months |
10 |
1 |
10 |
0.92 |
9 |
$7.25 |
$467 |
|
|||||||||||||||
12 months |
20 |
1 |
20 |
0.92 |
19 |
$7.25 |
$387 |
|
|||||||||||||||
Sub Total |
35 |
|
35 |
|
32 |
|
$1,275 |
|
|||||||||||||||
Record Management by FY2007 and FY2008 Grantee Staff5: |
|
||||||||||||||||||||||
Events Tracking2 |
8 |
800 |
6,400 |
0.03 |
192 |
$15 |
$2,880 |
|
|||||||||||||||
Person Tracking3 |
8 |
70 |
560 |
0.1 |
36 |
$15 |
$540 |
|
|||||||||||||||
Service Use 4 |
8 |
25 |
200 |
0.17 |
34 |
$15 |
$510 |
|
|||||||||||||||
Arrest History 4 |
8 |
25 |
200 |
0.17 |
34 |
$15 |
$510 |
|
|||||||||||||||
Sub Total |
32 |
|
7,360 |
|
296 |
|
$4,440 |
|
|||||||||||||||
FY2006 Grantees: |
|
||||||||||||||||||||||
Interview and |
8 |
12 |
48 |
0.17 |
8 |
$25 |
$200 |
|
|||||||||||||||
Tracking data |
|
||||||||||||||||||||||
submission |
|
||||||||||||||||||||||
OVERALL |
1,095 |
|
8,415 |
|
1,290 |
|
$6,080 |
|
|||||||||||||||
TOTAL: |
|
||||||||||||||||||||||
1 –Only program enrollees who agree to participate in the evaluation receive a Baseline interview |
|
||||||||||||||||||||||
2 – The number of responses per respondent for the Events Tracking depends on the design of the jail diversion program and can range from a single screening for eligibility to four separate screenings; here 800 responses represents the average number of responses per respondent for the period based on the experience of the previous Grantees. |
|
||||||||||||||||||||||
3– This estimate is an added burden proportion which is an adjustment reflecting the extent to which programs typically already collect the data items. The formula for calculating the proportion of added burden is: total number of items in the standard instrument, minus the number of core items currently included, divided by the total number of items in the standard instrument. For the Person Tracking program the burden estimate was calculated as follows: 56 times 0.65 (the proportion of added burden) = 36. |
|
||||||||||||||||||||||
4 – Record management forms (Service Use and Arrest) are only completed for those evaluation participants who receive both a Baseline interview and at least one follow-up (6 and/or 12 month) interview. |
|
||||||||||||||||||||||
5- Assumes 1 respondent at grantee site is responsible for compiling the information. |
|
|
|||||||||||||||||||||
CY 2010 Annual Reporting Burden |
|||||||||||||||||||||||
Data Collection Activity |
Number of Respondents |
Responses per Respondent |
Total Responses
|
Average |
Total Hour Burden |
Hourly |
Total |
||||||||||||||||
Hours per |
Rate |
Hour |
|||||||||||||||||||||
Response |
|
Cost |
|||||||||||||||||||||
Client Interviews for FY2008 and anticipated FY2009: Revised Instrument |
|||||||||||||||||||||||
Baseline |
1,110 |
1 |
1,110 |
0.95 |
1,055 |
$7.25 |
$7,645 |
||||||||||||||||
(at enrollment) 1 |
|||||||||||||||||||||||
6 months |
888 |
1 |
888 |
0.92 |
817 |
$7.25 |
$5,923 |
||||||||||||||||
12 months |
491 |
1 |
490.6 |
0.92 |
451 |
$7.25 |
$3,272 |
||||||||||||||||
Sub Total |
2,489 |
|
2,489 |
|
2,323 |
|
$16,840 |
||||||||||||||||
Client Interviews for FY2007 Grantees: Current Instrument |
|||||||||||||||||||||||
Baseline |
0 |
1 |
0 |
0.83 |
0 |
$7.25 |
$0 |
||||||||||||||||
(at enrollment) |
|||||||||||||||||||||||
6 months |
20 |
1 |
20 |
0.92 |
18 |
$7.25 |
$133 |
||||||||||||||||
12 months |
15 |
1 |
15 |
0.92 |
14 |
$7.25 |
$100 |
||||||||||||||||
Sub Total |
35 |
|
35 |
|
32 |
|
$233 |
||||||||||||||||
Record Management by FY2007, FY2008 FY2009 Grantee Staff5: |
|||||||||||||||||||||||
Events Tracking2 |
14 |
800 |
11,200 |
0.03 |
336 |
$15 |
$5,040 |
||||||||||||||||
Person Tracking3 |
14 |
80 |
1,120 |
0.1 |
62 |
$15 |
$930 |
||||||||||||||||
Service Use 4 |
14 |
50 |
700 |
0.17 |
119 |
$15 |
$1,785 |
||||||||||||||||
Arrest History 4 |
14 |
50 |
700 |
0.17 |
119 |
$15 |
$1,785 |
||||||||||||||||
Sub Total |
56 |
|
13,720 |
|
636 |
|
$9,540 |
||||||||||||||||
FY2008 and FY2009 Grantees: |
|||||||||||||||||||||||
Interview and |
12 |
12 |
48 |
0.17 |
8 |
$25 |
$200 |
||||||||||||||||
Tracking data |
|||||||||||||||||||||||
submission |
|||||||||||||||||||||||
OVERALL |
2,592 |
|
16,292 |
|
2,999 |
|
$31,854 |
||||||||||||||||
TOTAL: |
|||||||||||||||||||||||
1 –Since enrollment is anticipated to have ended for these Grantees by the end of CY2009 there is no Baseline burden in CY2010. |
|||||||||||||||||||||||
2 – The number of responses per respondent for the Events Tracking depends on the design of the jail diversion program and can range from a single screening for eligibility to four separate screenings; here 800 responses represents the average number of respondents. |
|||||||||||||||||||||||
3– This estimate is an added burden proportion which is an adjustment reflecting the extent to which programs typically already collect the data items. The formula for calculating the proportion of added burden is: total number of items in the standard instrument, minus the number of core items currently included, divided by the total number of items in the standard instrument. For the Person Tracking program the burden estimate was calculated as follows: 96 times 0.65 (the proportion of added burden) = 62 |
|||||||||||||||||||||||
4 – Record management forms (Service Use and Arrest) are only completed for those evaluation participants who receive both a Baseline interview and at least one follow-up (6 and/or 12 month) interview. |
|||||||||||||||||||||||
5- Assumes 1 respondent at grantee site is responsible for compiling the information. |
|
||||||||||||||||||||||
CY 2011 Annual Reporting Burden |
|||||||
Data Collection Activity |
Number of Respondents |
Responses per Respondent |
Total Responses |
Average Hours per Response |
Total Hour Burden |
Hourly Rate
|
Total Hour Cost |
|
|||||||
Client Interviews for FY2008 and anticipated FY2009 and 2010: Revised Instrument |
|||||||
Baseline |
1710 |
1 |
1710 |
0.83 |
1419.3 |
$7.25 |
$10,290 |
(at enrollment) 1 |
|||||||
6 months |
1368 |
1 |
1368 |
0.92 |
1258.56 |
$7.25 |
$9,125 |
12 months |
879 |
1 |
879 |
0.92 |
808.68 |
$7.25 |
$5,863 |
Sub Total |
3,957 |
|
3,957 |
|
3,487 |
|
$25,277 |
Record Management by FY2008 and anticipated FY2009 and FY2010 Grantee Staff5: |
|||||||
Events Tracking2 |
18 |
800 |
14,400 |
0.03 |
432 |
$15 |
$6,480 |
Person Tracking3 |
18 |
80 |
1,440 |
0.1 |
94 |
$15 |
$1,410 |
Service Use 4 |
18 |
50 |
900 |
0.17 |
153 |
$15 |
$2,295 |
Arrest History 4 |
18 |
50 |
900 |
0.17 |
153 |
$15 |
$2,295 |
Sub Total |
72 |
|
17,640 |
|
832 |
|
$12,480 |
FY2008 and FY2009 Grantees: |
|||||||
Interview and |
18 |
12 |
48 |
0.17 |
8 |
$25 |
$200 |
Tracking data |
|||||||
submission |
|||||||
OVERALL |
4,047 |
|
21,645 |
|
4,327 |
|
$37,957 |
TOTAL: |
|||||||
1 –Since enrollment is anticipated to have ended for these Grantees by the end of CY2009 there is no Baseline burden in CY2010. |
|||||||
2 – The number of responses per respondent for the Events Tracking depends on the design of the jail diversion program and can range from a single screening for eligibility to four separate screenings; here 800 responses represents the average number of respondents. |
|||||||
3– This estimate is an added burden proportion which is an adjustment reflecting the extent to which programs typically already collect the data items. The formula for calculating the proportion of added burden is: total number of items in the standard instrument, minus the number of core items currently included, divided by the total number of items in the standard instrument. For the Person Tracking program the burden estimate was calculated as follows: 144 times 0.65 (the proportion of added burden) = 94 |
|||||||
4 – Record management forms (Service Use and Arrest) are only completed for those evaluation participants who receive both a Baseline interview and at least one follow-up (6 and/or 12 month) interview. |
|||||||
5- Assumes 1 respondent at grantee site is responsible for compiling the information. |
|
||||||
CY 2012 Annual Reporting Burden |
|||||||
Data Collection Activity |
Number of Respondents |
Responses per Respondent |
Total Responses
|
Average |
Total Hour Burden |
Hourly Rate |
Total Hour Cost |
Hours per |
|||||||
Response |
|
||||||
Client Interviews for anticipated FY2009 and FY2010: Revised Instrument |
|||||||
Baseline |
1,200 |
1 |
1,200 |
0.83 |
996 |
$7.25 |
$7,221 |
(at enrollment) 1 |
|||||||
6 months |
1,080 |
1 |
1,080 |
0.92 |
993.6 |
$7.25 |
$7,204 |
12 months |
1,084 |
1 |
1,084 |
0.92 |
997.28 |
$7.25 |
$7,230 |
Sub Total |
3,364 |
|
3,364 |
|
2,987 |
|
$21,655 |
Record Management by anticipated FY2009 and FY2010 Grantee Staff5: |
|||||||
Events Tracking2 |
12 |
800 |
9,600 |
0.03 |
288 |
$15 |
$4,320 |
Person Tracking3 |
12 |
70 |
840 |
0.1 |
55 |
$15 |
$825 |
Service Use 4 |
12 |
25 |
300 |
0.17 |
51 |
$15 |
$765 |
Arrest History 4 |
12 |
25 |
300 |
0.17 |
51 |
$15 |
$765 |
Sub Total |
48 |
|
11,040 |
|
445 |
|
$6,675 |
FY2008 and FY2009 Grantees: |
|||||||
Interview and |
12 |
12 |
48 |
0.17 |
8 |
$25 |
$200 |
Tracking data |
|||||||
submission |
|||||||
OVERALL |
3,424 |
|
14,452 |
|
3,440 |
|
$28,530 |
TOTAL: |
|||||||
1 –Since enrollment is anticipated to have ended for these Grantees by the end of CY2009 there is no Baseline burden in CY2010. |
|||||||
2 – The number of responses per respondent for the Events Tracking depends on the design of the jail diversion program and can range from a single screening for eligibility to four separate screenings; here 800 responses represents the average number of respondents. |
|||||||
3– This estimate is an added burden proportion which is an adjustment reflecting the extent to which programs typically already collect the data items. The formula for calculating the proportion of added burden is: total number of items in the standard instrument, minus the number of core items currently included, divided by the total number of items in the standard instrument. For the Person Tracking program the burden estimate was calculated as follows: 84 times 0.65 (the proportion of added burden) = 55 |
|||||||
4 – Record management forms (Service Use and Arrest) are only completed for those evaluation participants who receive both a Baseline interview and at least one follow-up (6 and/or 12 month) interview. |
|||||||
5- Assumes 1 respondent at grantee site is responsible for compiling the information. |
|
||||||
Total Annualized Reporting Burden
Data Collection Activity
|
Annualized Number of Respondents
|
Annualized Total Responses
|
Annualized Total Hour Burden
|
Baseline (at enrollment) |
1,133 |
1,133 |
988 |
6 months |
943 |
943 |
858 |
12 months |
648 |
648 |
586 |
Events Tracking |
13 |
10,397 |
312 |
Person Tracking |
13 |
986 |
62 |
Service Use |
13 |
523 |
88 |
Arrest History |
13 |
523 |
88 |
Interview and Tracking Data Submission |
13 |
48 |
32 |
|
|||
TOTAL ANNUALIZED |
2,789 |
15,201 |
3,014 |
FY2006-2007 Grantees: For the forms currently approved under OMB No. 0930-0277, there are 54 baseline interview questions and 66 follow-up interview questions (including record management items and excluding GPRA). The baseline will take approximately 50 minutes per enrollee/participant to administer and the follow-up interviews will take approximately 55 minutes each to administer.
FY2008 and future grantees: For the revised forms (to be used by the FY2008 and subsequent Grantees only), there are 117 baseline interview questions and 116 follow-up interview questions (including record management items and excluding GPRA), though there are many skip patterns and questions that may not be applicable to respondents (e.g. military questions)- therefore it is anticipated that the average grantee administration will take 58 minutes for the baseline and 55 minutes for the follow-up per enrollee/participant to administer.
Record Management by Grantee staff -
There are four management burdens placed on the Grantee staff, these are the same for all Grantees:
Events Tracking data are managed and entered by the Grantee staff. These data provide counts of the number of people considered for jail diversion program eligibility and measures the number of assessments conducted to screen those potential enrollees. In addition, information on individuals (e.g. charges, sex, race, age) involved in each event is entered into the tracking program (this information is not linked to individuals and is thus not identifiable). All of this information is generally collected through the normal course of business so that the only burden on the staff is data entry into the tracking program. It is estimated that it should take approximately 2 minutes to enter these forms for each participant.
Person Tracking data are also managed and entered by the Grantee staff. The Person Tracking data helps Grantees to keep track of individuals’ interview dates and to record basic information on all individuals who are diverted. Collecting and entering this information should take approximately 6 minutes for each participant. Of the 17 items, however, 6 are commonly collected by the providers. The resulting Added Burden Proportion is the total number of items in the standard instrument, minus the number of core items currently included, divided by the total number of items in the standard instrument which calculates as follows: (17-6)/17, or 0.65.
Service Use forms collect information about the types and number of services received following diversion and is obtained directly from provider records. These forms are estimated to take approximately 10 minutes to complete for each participant.
Arrest History forms collect pre- and post-diversion arrest information from provider records and include dates of arrests, charges and jail days occurring one year prior and one year following diversion. These forms are estimated to take approximately 10 minutes to complete for each participant.
Grantee Data Submissions –
FY2006-2007 Grantees: Grantees are responsible for submitting data to the TAPA Center regularly. For Events and Person Tracking data this involves extracting and emailing their data bi-monthly. For Interview and Record Review data this involves mailing the completed forms at least monthly. It is estimated that these submissions take approximately 10 minutes monthly.
FY2008 and future Grantees: Grantees are responsible for submitting data AHP regularly. To reduce data entry time and cost, computer assisted interviewing will be utilized for client interviews. Data collected via computer assisted software will be securely transferred to AHP by email. For any interviews conducted on paper, grantees will enter into the software using the data entry module and send to AHP electronically. For the Arrest, Service, and Events Tracking- these will entered into the QDS software and electronically submitted to AHP monthly. The Person Tracking data this involves extracting and emailing data from the Tracking database bi-monthly. It is estimated that these submissions take approximately 10 minutes monthly.
13) Estimates of Annualized Cost Burden to Respondents
There are no startup or capital costs, nor are there maintenance costs to the respondents.
14) Estimates of Annualized Cost to the Government
AHP, with its subcontractors Cloudburst Consulting Group, coordinates, monitors, collects, reports and analyzes the data provided by the Grantees and submits a monthly report to the Government project officer. AHP also provides Grantee sites with training and supportive materials on each evaluation component as well as participant protections. The TAPA Center’s total budget for its evaluation activities is $371,610. The Federal Government employee (GS-14, $77,793) expends 20% of time overseeing the Jail Diversion Trauma Recovery Contract, equaling $15,558.
15) Changes in Burden
Currently the annualized burden is 1,225 hours/year in the OMB inventory. The program is requesting an average of 3,014 hours/year. This program change is mainly due to the new cohort of grantees that is expected in FY2009 and FY2010. While there is some minor added burden with the changes to the client interview instrument, the main source of burden in the increasing number of grantees that are expected to be collecting data.
16) Time Schedule, Publication and Analysis Plans
16.a. Time Schedule
Tasks Dates
OMB Approval: Pending
Data Collection: Immediately upon OMB approval
Data Collection Ends: June 2015
Analysis of Data: July 2015
16.b. Publication Plans
Tracking and outcome data will be collected through AHP from each JDTR Grantee. Data will be reported to Congress regarding program performance as specified in the SAMHSA Budget Justification report. In addition, data summaries will be presented at annual Grantee meetings in order to provide a performance overview of the entire group of attending Grantees. Furthermore, AHP plans to disseminate information related to the development, implementation and outcomes of this initiative’s jail diversion programs through journal articles, monographs/fact sheets and national conferences.
16.c. Analysis Plans
The primary purpose of the cross-site evaluation data collection effort is to provide program monitoring and oversight. There are three categories of evaluation questions that this program expects to address through its data analysis: Descriptive, Individual Change and Overall Outcomes.
Descriptive– This type of analyses will consist of counts, frequency distributions and basic aggregate tests (such as t-tests and Chi-square tests) using Tracking, Interview and Service Use data. These analyses will address questions such as:
What volume of activities (e.g. screening, assessment, evaluations) goes into identifying people for diversion? (Events Tracking)
How many people are determined eligible/ineligible for jail diversion and what are their characteristics (demographic and criminal justice)? (Events Tracking)
How many people are enrolled in the diversion programs and what are their characteristics (demographic, criminal justice and mental health)? (Person Tracking)
How many program enrollees agree to participate in the evaluation and what are their characteristics (demographic, education, employment, drug/alcohol use, criminal justice and mental/physical health)? (Interviews)
What services do people who are diverted receive? (Service Use)
As required, SAMSHA/CMHS reports on the characteristics of the participants seen in its grant portfolios to the Department of Health and Human Services and the Office of Management and Budget. Also CMHS will provide SAMHSA with program monitoring reports to use for performance review, improvement and oversight.
Individual Change – Analysis of these data will consist of a pre/post measurement methodology so that the individual acts as his own control. This methodology will be implemented in the following manner:
For interview data, consenting participants will receive a baseline interview within seven days of enrollment in the jail diversion program followed by 6 and 12 month follow-up interviews. Change within each participant across the time period will then be determined by examining the difference in scores on the mental health, substance use and other outcome measures between the baseline and 6 and 12 month interviews.
For Arrest History, criminal justice data will be collected both one year prior and one year post enrollment in the diversion program. Change within each participant across time will then be determined by comparing the number and type of criminal justice events and the amount of jail time occurring before and after enrollment.
Once individual difference scores are calculated, central tendency measures and frequency distributions will be utilized to examine how individuals have changed over the 6 and 12 months on specific measures. Tables will be constructed to summarize participant outcome changes across 6 and 12 months.
Overall Outcomes – The analyses discussed above will provide insight into the process behind jail diversion programs and how these programs are affecting individuals on specific measures. The Cross-site evaluation is, however, also interested in examining the following broader evaluation question that considers how the many different levels work together:
Do people who are diverted improve over time as a result of services received through the jail diversion program as measured by:
reduced arrests/less time spent in jail
reduced substance use
obtained housing
higher psychological functioning
decreased trauma symptoms
improved mental health and/or
improved physical health
decreased trauma symptoms
client’s assessment of personal recovery.
In order to answer this broad-based question, AHP expects to utilize regression techniques in an attempt to predict which factors (demographics, treatment history, trauma history, service use, arrest history etc.) are related to and/or contribute to positive mental health, substance use and criminal justice outcomes in both the short-term (6 months) and the long-term (12 months).
The new sections of the client interview will also provide information on:
Lifetime Mental Health and Substance Use Treatment questions- will provide descriptive information on the treatment history of clients, factors which may discriminate between client outcomes and program participation.
Military Service and Military Combat Experiences- will provide descriptive information on program clients, and may identify factors that discriminate between client outcomes and program participation. Further, veterans are a priority population for the new cohorts of grantees, and it will therefore be important to be able to identify the extent to which programs were able to serve this population, strategies for service provision, and characteristics of program that successfully engage veterans. The information on military combat experiences will help the program describe the types and extent of trauma experienced by veterans.
Recovery Markers (REE)- will provide information on the client’s assessment of their personal recovery, including the quality of life, social relationships, sense of meaning and sense of well-being, This instrument will provide a way to measure changes in the client’s life that focuses on the quality of their life and not on symptoms.
Analysis collected through the gathered through the client interview will provide insight into the following areas:
1) Identification of traumatic experiences and symptoms:
The incident rates of trauma and re-trauma; military trauma; and relationship to childhood trauma;
The extent (severity)of specific trauma symptoms;
Which factors are (and are not) related to trauma and re- trauma (e.g., substance use, homelessness
How targeting individual-specific symptom profiles can inform the use of appropriate evidence-based interventions.
2) Examination of individual change across the participation period –
The changing patterns of trauma incidence, symptoms and symptom severity during enrollment in a jail diversion program;
The relationship of trauma symptom severity and trauma with other mental health and criminal justice outcomes during the participation period;
The relationship between trauma experience and PTSD symptoms in the jail diversion population;
The role of evidence-based interventions in mitigating trauma symptoms and re-trauma across time.
The extent to which trauma symptoms influence the treatment needs and responses of individuals relative to other mental health symptoms assessed in the evaluation (e.g., BASIS 24);
The relationship between client assessment of Recovery (REE) and improvements in symptoms (e.g. BASIS 24; PCL).
3) Informing treatment plans in jail diversion programs –
Traumatic Events – Research has shown that earlier trauma, especially childhood sexual abuse, increases risk of later trauma and symptoms of posttraumatic stress over the lifetime (3, 4, 5).
Clients’ assessment of Recovery as a new marker for program feedback.
17) Display of Expiration Date
The expiration date will be displayed.
18) Exceptions to Certification Statement
This collection complies with the requirements in 5 CFR 1320.9. The certifications are included in this submission.
1) Respondent Universe and Sampling Methods
The TCE jail-diversion Grantees will be attempting to collect data about all individuals screened for diversion in addition to all jail diversion program enrollees. The starting point for data collection is everyone that the programs screen for possible entry into the jail diversion program. Based on data received from the previous cohorts of TCE Initiative Grantees, on average approximately 1,200 individuals per year, per Grantee are screened at least once for possible jail diversion enrollment. (This number is variable across Grantees, however, due to differences in the size and types of programs.) Although this is a large number, the evaluation requires only basic information on these individuals in order for SAMSHA/CMHS to obtain an accurate representation of the population that the enrollees are part of and to credit the programs with the extra level of planning and work required in order to enroll individuals into the jail diversion programs.
Based on data received from the previous cohorts of TCE Initiative Grantees, on average, 50 individuals are enrolled in jail diversion by each Grantee per year. (Again this number is variable across Grantees due to differences in the size and types of programs.) Hence, compared to the census of everyone screened, the number of individuals deemed eligible and enrolled in the jail-diversion programs is quite small. Also program enrollees may decline to participate in the evaluation and evaluation participants may decline to participate in one or both follow-up interviews. Furthermore, it should be noted that prior to the addition of the grant mandated evaluation, no formal sample size was formulated. The goal for the program evaluation is to collect data on as many jail diversion program enrollees as possible at each site. For these reasons a universal sampling frame will be required to obtain the largest n possible.
Based on Grantees already collecting data, there is an expectation that the newer Grantees will meet or exceed the required minimum evaluation participation rate of 80 percent which should insure a sufficient sample size for the purposes of this evaluation. FY 2005 Grantees achieved a participation rate of 93% and FY2006 Grantees achieved a participation rate of 87%. We expect a participation rate of 90% based on the experience of these previous cohorts.
Once the evaluation is completed, AHP will perform basic demographic comparisons between those enrolled in the evaluation and those who refused participation to determine whether any selection bias occurred due to refusals. AHP will also examine whether 6 and 12 month attrition resulted in any bias. If bias has occurred, adjustments will be made in analysis in order to account for this.
2) Information Collection Procedures
As discussed earlier, there are three primary data sources that comprise all of the data collected through this evaluation:
Interview Data composed of the following primary measures
FY2006-2007 cohort
GPRA measures
DC Trauma Collaboration Study Violence and Trauma Screening,
Posttraumatic Stress Checklist (FY2006 and subsequent Grantees only)
Colorado Symptom Index 1991
Perceived Coercion Scale (from the Mac Arthur Mandated Community Treatment Survey)
Mental Health Statistics Improvement Program (MHSIP)
FY2008 and subsequent cohorts:
GPRA/NOMS measures
Military Service Questions/Combat Experiences
Lifetime Mental Health/Substance Use Treatment Services
Drug and Alcohol Use,
Criminal Justice Questions
Traumatic Events
Posttraumatic Stress Checklist
BASIS-24
REE Recovery markers
Tracking Data
Events Tracking (demographic and eligibility information)
Person Tracking (demographics, diagnosis, charges, diversion point and condition and target arrest/incident and enrollment information)
Record Review Data
Service Use (type and number of services received)
Arrest History (type and jail days of arrests)
The starting point for data collection is everyone that the programs screen for possible entry into the jail diversion program. Once a person is deemed eligible for enrollment, and the court accepts the diversion plan (if applicable), then the person is enrolled in the jail diversion program. All of the data gathered about those individuals screened for possible diversion are collected and entered by Grantee staff using information obtained from the potential diversion program enrollees.
All diversion program enrollees are eligible to participate in the evaluation and unless there are extenuating circumstances (such as difficulties locating the person or hospitalization) all enrollees are approached for consent to participate. Attachment F provides an example of a consent form. Of those enrollees agreeing to participate in the evaluation, all are expected to receive a baseline interview within seven days of enrollment which can be conducted by Grantee staff. The participants are then called back for 6 and 12 month follow-up interviews, which must be administered within 30 days on either side of the due date and are conducted by Grantee staff only. Attachment H provides an example of a follow-up interview reminder letter. Note that while program staff may administer baseline interviews, only Grantee staff who is not in any way involved in providing services to program participants administers follow-up interviews. This is done to protect the privacy of the participant and to ensure that no adverse effects result from a refusal to participate in the evaluation or from any responses given.
Participants completing a baseline interview as well as a 6 and/or 12 month interview will receive service use and arrest history record reviews, which are conducted by Grantee staff. Person Tracking data are collected on all enrollees regardless of enrollment in the evaluation and contain information obtained either directly or indirectly from the potential diversion program enrollees. For those enrollees participating in the evaluation, the Person Tracking data also include information about interview completion statuses (when due and whether/when completed).
Most Grantee staff will record participant information through a pencil and paper method. This evaluation will not interfere with ongoing program operations. Most Grantee staff will collect client interviews through computer assisted interview programs. Grantee staff will submit electronic interviews forms monthly, electronic tracking data extracts bimonthly and electronic records forms at least once annually. All data, except the non-identifiable Events Tracking data, are matched using a unique client identifier created by the Person Tracking program.
3) Methods to Maximize Response Rates
The response rate for enrollments in the evaluation is 90% as described above under 1) Sampling Universe. We expect the Grantees to obtain a follow-up response rate of 80% as required by CMHS. To support Grantees in achieving this rate, each jail diversion program has an Access-based software database distributed by AHP to assist in collecting required demographic and background information to assist in tracking participants and providing demographic information for the evaluation. The tracking software program has the capability to provide information on the current addresses for the participants and when they are due for a follow-up interview. The Baseline Person Tracking Program Information Form (Attachment E) is part of the tracking software program. It is intended to help interviewers contact participants for follow-up interviews.
The participants are contacted in several ways in order to assure that they return for the 6 and 12 month interviews. These methods vary from site to site and may mailing a reminder letter, contacting friends, family, case managers, and/or therapists (with permission), dropping in on commonly visited locations (e.g., soup kitchens, shelters, AA meetings), and coordinating follow-up interviews with other scheduled appointments at the program. AHP will provide technical assistance to Grantees to achieve the 80% follow-up response rate.
4) Tests of Procedures
The measures included in the TCE Initiative’s interviews are a combination of mandated NOMS/GPRA items and additional non-GPRA mental health scales.
FY2006-2007 Grantees: For FY2007 and earlier cohorts of TCE Grantees, data collection of all of the measures have been implemented and successfully administered.
FY2008 and future Grantees: As part of the Evaluation Advisory Committee Process, FY2008 TCE Grantees had the opportunity to review and comment on the revised instruments, agreed to the inclusion of all items approved by the OMB. All of the non-GPRA measures contained in the instruments have been pilot tested and/or are well established data collection tools tested for validity and reliability. The four main non-GPRA measures along with their developmental background are as follows:
Traumatic Events – These screening questions for trauma are adapted from the Posttraumatic Diagnostic Scale (PDS) developed by Edna Foa and the DC Trauma Screening developed by Community Connections in Washington, DC. Both of these instruments are used by clinicians to determine an individual's trauma history, including the recentness of the trauma. The screen is a descriptive tool only and, as such, has no psychometric properties. For the TCE Initiative, its inclusion is intended to provide basic descriptive information about individual trauma levels.
Posttraumatic Stress Checklist - The PCL is a 17-item self-report measure of Posttraumatic Stress Disorder (PTSD) symptoms based on DSM-IV criteria, with a 5-point Likert scale response format that rates the severity of each symptom over the past month. Continuous scores are used to assess symptom severity and a cut-point of 3 (moderate severity) is used per each PTSD symptom to derive a PTSD diagnosis. The PCL has good psychometric properties. It has been found to be highly correlated with the Clinician Administered PTSD Scale (r = .929), the “gold standard” measure of PTSD, has good diagnostic efficiency (> .70), and robust psychometric properties with a variety of trauma populations (6, 7). Among individuals with serious mental illness, high internal consistency of the PCL was reported (.94 coefficient alpha), along with moderate test-retest reliability (.66) and moderate convergent validity with the CAPS (κ=.67) (8). Based on the brevity of the scale, along with its validity and reliability, the AHP Center agreed that this would be a useful measure of PTSD symptoms to potentially be included in the TCE Initiative’s evaluation.
Behavior and Symptom Identification Scale (BASIS 24) – The revised 24-item self-report tool is used to assess change in mental health symptoms and behavioral distress following treatment. The instrument covers six domains: depression/functioning, difficulty in interpersonal relationships, self-harm, emotional lability, psychotic symptoms, and substance abuse- and an overall mental health score computed. Its predecessor (BASIS 32) is a widely used and tested behavioral health tool. It addition to being shorter, the BASIS 24 is more comprehensive, cutting across diagnoses by identifying a wide range of symptoms and problems that occur across the diagnostic spectrum. Validated and found reliable in inpatient, residential, and outpatient settings, BASIS-24 assesses treatment outcomes from the patient perspective9. The instrument, along with other options, was reviewed by the Evaluation Advisory Committee and approved for inclusion in the proposed revised instrument.
Military Service and Combat Experience Questions – These questions were adapted from the several sources, and reviewed by Evaluation Advisory Committee members with experience and knowledge in this area. The military service questions were developed from the RAND the survey items used in the RAND Monograph Invisible Wounds of War10 and the questions from the Office of Justice Program, Bureau of Justice Statistics veteran questions11. The RAND monograph presents the results of a comprehensive study conducted between April 2007 and January 2008 on the post-deployment related health needs associated with PTDS, major depression, and traumatic brain injury of Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) veterans. This BJS Report presents data on the military and criminal backgrounds of incarcerated veterans. The combat experience questions were adapted from Hoge et al (2004)12 study of military combat duty in Iraq and Afghanistan and the associated mental health problems. These questions have been used in other studies of returning military personnel because of the broad range of types of experiences, which include not only not being hurt or hurting someone, but also include the aftermath of combat. A version of the combat experiences was also used in the RAND monograph, and questions drafted for inclusion were reviewed for relevance and appropriateness by consumer veterans and a representative from the Veteran’s Administration.
CAGE- The Cut down, Annoyed, Guilty, Eye-Opener (CAGE)13 is a four item screening tool used to detect alcohol abuse or dependence. It is structured in a “have you ever” format that is applicable to the interviewee’s past or present. Answering Yes to two questions provides strong indication for substance abuse or dependency on alcohol. The tools validity has been demonstrated with substance abusing populations and it is commonly used in criminal justice settings. It is short, easy to remember, and easy to incorporate into an interview (NIAAA, 2002)14.
Lifetime Mental Health and Substance Use Questions-These are eight questions were developed to assess client’s participation in treatment during their lifetime. There are four areas covered each for mental health and for substance abuse services: outpatient services, inpatient services, peer supported services, and age at first contact. These questions were based on previous instruments, including the CSAT/CMHS Homeless Families Study and the CMHS Support Housing Study. This information is primarily for descriptive purposes.
Lifetime Criminal Justice Questions- There are six criminal justice questions that were included to assess the client’s previous involvement (longer than the past year) with the criminal justice system. These include age at first arrest, prior experience on probation, number of times incarcerated, and history retraining/protection order. While the purpose of these items is descriptive, their inclusion may distinguish between successful and unsuccessful program clients. These questions were based on previous SAMHSA cross-site instruments, including the CSAT/CMHS Homeless Families Study and the CMHS Support Housing Study.
Recovery Markers from the Recovery Enhancing Environment Measure (REE)- The REE Recovery Markers is a 23 item self-assessment of personal recovery. The instrument examines personal recovery by focusing markers of recovery (immediate outcomes) they currently experience, including: motivation, goals, social role reclamation, basic needs such as housing and income, symptom self-management, physical health, quality of life, and personal strengths and positive relationships. Two formal pilot tests have been conducted on the REE15 and preliminary analyses indicate that the instrument is psychometrically sound; the coefficient alphas indicate for subscales range from (.77-.98). Additionally, the REE provides an important new element to client outcomes that focuses on clients’ well-being instead of symptoms.
The non-interview forms (for events tracking, person tracking, service use and arrest history data collection) collect commonly used descriptive and/or publicly available information. As with the interview forms, the Grantees on the Evaluation Advisory Committee provided feedback about the information in, and format of, the tracking and record review forms. These forms collect the same information as the previously approved forms, however the formatting has changes. These forms were implemented and successfully completed by all prior cohorts of the Grantees.
As described above, the FY2008 Grantees had the opportunity to review the revised interviews the new items described above and agreed to include all items previously approved by OMB. These revised interviews will be administered by all Grantees awarded in FY2008 and in subsequent years.
5) Statistical Consultants
Contractors/Statistical Consultants:
Name/Title |
Address |
Contact Information |
John Hornik, Ph.D. Director Advocates for Human Potential |
Advocates for Human Potential 324 Broadway Two E-Comm Sq. Albany, NY 12207 |
Phone: (518) 729-1244
|
Kristin Stainbrook, Ph.D. Assistant Director of Research Advocates for Human Potential |
Advocates for Human Potential 324 Broadway Two E-Comm Sq. Albany, NY 12207 |
Phone: (518) 729-1241 |
Jenneth Carpenter, Ph.D. Senior Research Associate Advocates for Human Potential |
Advocates for Human Potential 324 Broadway Two E-Comm Sq. Albany, NY 12207 |
Phone: (518) 729-1221
|
Nick Huntington Analyst Advocates for Human Potential |
Advocates for Human Potential 490-B Boston Post Road Sudbury, MA 01776 |
Phone: (978) 261-1453
|
Federal Project Officers/Statistical Consultants
Name/Title |
Address |
Contact Information |
David Morrissette, Ph.D. Government Project Officer
|
Center for Mental Health Services, SAMHSA |
Phone: (240) 276-1912
|
List of Attachments for FY2008 and future Grantees:
Baseline Interview
6-Month Interview
12-Month Interview
Event Tracking Screen
Baseline Person Tracking Information Form
Service Use Data Collection Form
Arrest Data Collection Form
List of Attachments for Continuation Approval for FY2006-2007 Grantees:
Baseline Interview
6-Month Interview
12-Month Interview
Arrest Data Collection Form
Mental Health and Substance Abuse Service Use Data Collection Form
Example of a Consent Form
Baseline Person Tracking Information Form
Follow-up Interview Reminder Letter
1 Foa, E. B., Cashman, L., Jaycox, L. H., & Perry, K. (1997). The validation of a self-report measure of PTSD: The Posttraumatic Diagnostic Scale (PDS). Psychological Assessment, 9, 445–451
2 Hoge et al. 2004. Combat Duty in Iraq and Afghanistan, Mental Health Problems, and Barriers to Care. The New England Journal of Medicine, 351 (1): 13-22.
3 Burnam, M.A., Stein, J.A., Golding, J.M., & Seigel, J.M. (1988). Sexual assault and mental disorders in a community population. Journal of Consulting and Clinical Psychology, 56(6), 843-850.
4 Nishith, P., Mechanic, M.B., & Resick, P.A. (2000). Prior interpersonal trauma: The contribution to current PTSD symptoms in female rape victims. Journal of Abnormal Psychology, 109(1), 20-25.
5 Polusny, M.A. & Folette, V.M. (1995). Long term correlates of child sexual abuse. Applied & Preventive Psychology, 4(3), 143-166.
6 Andrykowski, M.A., Cordova, M.J., Studts, J.L., & Miller, T.W. (1998). Posttraumatic stress disorder after treatment for breast cancer: Prevalence of diagnosis and use of the PTSD Checklist–Civilian Version (PCL–C) as a screening instrument. Journal of Consulting and Clinical Psychology, 6, 586–590.
7 Blanchard, E.B., Jones-Alexander, J., Buckley, T.C., & Forneris, C.A. (1996). Psychometric properties of the PTSD Checklist (PCL). Behaviour Research and Therapy, 34, 669-673.
8 Mueser, K.T., Rosenberg, S.D., Fox, L., Salyers, M.P., Ford, J.D., & Carty, P. (2001). Psychometric evaluation of trauma and posttraumatic stress disorder assessments in persons with severe mental illness. Psychological Assessment, 13(1), 110-117.
9 Eisen, S.V. Normand, G.R., Belanger et al. The Revised Behavior and Symptom Identification Scale: Reliability and validity. Medical Care, 2004, 42: 1230-1241.
Eisen, S.V., Ranganathan, G, Seal, P, and Spiro, A. 2007. Measuring Clinically Meaningful Change following Mental Health Treatment. Journal of Behavioral Health Sciences Research, 43 (3): 272-289.
10 Tanielian, T. & Jaycox, L.A., Eds. (2008). Invisible wounds of war: Psychological and cognitive injuries, their consequences, and services to assist recovery. Santa Monica, CA: RAND Center for Military Health Policy Research.
11 Noonan, Margaret E.; Mumola, Christopher J., "Veterans in State and Federal Prisons, 2004 ." Special Report. NCJ 217199, Washington, DC: United States Department of Justice, Bureau of Justice Statistics, May 2007.
12 Hoge, C.W., Castro, C.A., Messer, S.C., McGurk, D., Cotting, D.I., & Koffman, R.L. (2004). Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. New England Journal of Medicine, 351, 13-22.
13 Isaacson JH, Schorling JB. Screening for alcohol problems in primary care. Med Clin North Am. 1999 Nov;83(6):1547-63, viii. PubMed Entry
14 National Institute on Alcohol Abuse and Alcoholism (NIAAA). Screening for alcohol problems—an update [Internet]. Bethesda, MD: National Institute on Alcohol Abuse and Alcoholism; 2002 Apr [accessed May 20, 2009]. (Alcohol Alert; 56). Available from: http://pubs.niaaa.nih.gov/publications/aa56.htm
15 Ridgeway, P, Press, A. Ratzlaff, S, Davidson, L, and Rapp, C (2003). Reports on the field testing of the Recovery Enhancing Environment Measure. Lawrence, KS. School of Social Welfare, Office of Mental Health Research and Training.
Ridgeway, P, Press, A., Anderson, D., and Deegan, P.E. (in preparation). Pilot testing the Recovery Enhancing Environment Measure: The Massachusetts experience. Byflield, MA: Pat Deegan and Associates.
| File Type | application/msword |
| File Title | THE SUPPORTING STATEMENT |
| Author | Sarah Fleury |
| Last Modified By | kstainbrook |
| File Modified | 2009-09-04 |
| File Created | 2009-09-04 |