TCE-HIV OMB Supporting Statement 10-19-10 FINAL

TARGETED CAPACITY EXPANSION PROGRAM FOR SUBSTANCE ABUSE TREATMENT AND HIV/AIDS (TCE-HIV) SERVICES

MULTI-SITE EVALUATION

SUPPORTING STATEMENT

A. JUSTIFICATION

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) is requesting approval from the Office of Management and Budget (OMB) for the data collection activities for the Targeted Capacity Expansion Program for Substance Abuse Treatment and HIV/AIDS (TCE-HIV) Services Multi-Site Evaluation (hereafter referred to as the Multi-Site Evaluation). In September 2008, SAMHSA contracted with JBS International, Inc. (JBS) to conduct the Multi-Site Evaluation and with approval from SAMHSA, JBS developed and executed subcontract agreements with its partners Alliances for Quality Education, Inc. (AQE), Battelle Memorial Institute, and Oregon Health & Science University (OHSU) to form a multidisciplinary team of methodologists, statisticians, and evaluation experts in the fields of substance abuse and HIV/AIDS (hereafter referred to as the Multi-Site Evaluation Team).

The Multi-Site Evaluation is designed to monitor implementation activities of 48 SAMHSA TCE-HIV funded Grantees including the evidence based practices (EBPs), service delivery, and contextual factors that facilitate or impede program success. The Evaluation is also intended to obtain information on the risk behaviors, HIV testing/serostatus, social support, medical and mental health, and motivation for treatment from clients of the TCE-HIV Grantees and monitor their satisfaction with services received.

There are a total of five data collection instruments for the Multi-Site Evaluation: administrative staff semi-structured interview guide, Grantee direct services staff semi-structured interview guide, partner/collaborator semi-structured interview guide, client focus group guide, and client level survey to be administered at client intake, discharge, and 6-months post baseline. The Multi-Site Evaluation activities will be conducted with 48 Grantees, 40 of which are treatment Grantees and 8 of which are outreach/pretreatment Grantees. Evaluation activities will include administering baseline, discharge, and follow-up surveys to clients receiving TCE-HIV treatment services; conducting focus groups with a subset of these clients; and conducting semi-structured interviews with TCE-HIV program staff and program partners/collaborators who are overseeing or delivering TCE-HIV services.

Minority AIDS Initiative (MAI). SAMHSA recognized the association between substance abuse and HIV risk and developed the TCE-HIV services grant program to improve the delivery of services to racial and ethnic minority communities disproportionately impacted by substance abuse and HIV across the United States (Centers for Disease Control and Prevention, 2006). The TCE-HIV grants are authorized under Section 509 of the Public Health Service Act (42 U.S.C.290bb-2), as amended. The program also addresses Healthy People 2020 Focus Area 2020-6 (Substance Abuse) (U.S. Department of Health and Human Services, 2009). The TCE-HIV Grantees are tasked with enhancing and expanding substance abuse treatment and/or outreach and pretreatment services in conjunction with HIV/AIDS services in high-risk, underserved communities highly affected by both substance abuse and HIV/AIDS. SAMHSA’s TCE-HIV program is funded by Minority AIDS Initiative (Minority AIDs Initiative, 1998) appropriated dollars. The MAI was established in October 1998 in response to the growing concern about the impact of HIV/AIDS on racial and ethnic minorities in the United States. The primary goals of the MAI are to improve HIV-related health outcomes in communities disproportionately affected by HIV/AIDS and reduce HIV-related disparities for racial and ethnic minority groups. The MAI was designed to focus attention on solving a growing public health problem and to develop and improve the capacity of minority community-based organizations to more effectively serve their communities. This approach was tailored to yield innovative and successful strategies specifically targeted to the highest risk and hardest to serve populations which for the past two decades have eluded more traditional HIV/AIDS prevention, treatment, and education efforts. (U. S. House of Representatives Committee on Appropriations, 2001)

SAMHSA provides the MAI funds for grants to expand treatment through increasing access and availability of services to a larger number of clients and/or to enhance services by improving the quality and/or the intensity of services in conjunction with HIV/AIDS services in African-American, Hispanic, and/or other racial and ethnic communities affected by the twin epidemics of substance abuse and HIV/AIDS. As described in the Request for Applications (RFA), enhanced and expanded services will have an influence on client substance use/abuse, HIV/AIDS prevention, criminal activities and criminal justice involvement, and risk behaviors for HIV infection. The primary goal of SAMHSA’s TCE-HIV program is to improve access to substance abuse treatment services through increasing capacity and outreach to racial and ethnic minority populations. In SAMHSA’s effort to increase capacity of the TCE-HIV funded programs, Grantees are expected to develop memoranda of agreement with community-based organizations with experience in providing recovery support and other services in the targeted communities. Examples of possible community linkages include, but are not limited to:

• primary health care;

• mental health and medical services for those who are HIV positive, have AIDS, or are at high risk of HIV infection;

• community-focused educational and preventive efforts;

• private industry-supported work placements for recovering persons;

• faith-based organizational support;

• support for the homeless;

• HIV/AIDS community-based outreach projects;

• HIV counseling and testing services;

• opioid treatment programs; and

• health education and risk reduction information.

Although SAMHSA has provided TCE-HIV funding to Grantees for more than a decade, this Multi-Site Evaluation effort represents the first multi-site evaluation designed to monitor the implementation activities of the MAI-funded TCE-HIV Grantees. SAMHSA intends to use the successful conceptualization, planning, and implementation of the multi-site evaluation to independently evaluate the MAI-funded TCE-HIV grant programs in a manner that is comprehensive, will establish performance goals, and will describe the EBPs and service delivery of the TCE-HIV program as well as obtain information about the substance abuse and sexual risk behaviors of the clients they serve. Further, this first-of-a-kind Multi-Site Evaluation will allow SAMHSA to fully describe the programs, their providers, the clients they serve, and the community in which the programs operate.

Centers for Disease Control and Prevention (CDC) Program Evaluation Framework. The Multi-Site Evaluation Team is informed by the CDC Program Evaluation Framework (Centers for Disease Control and Prevention, 1999) (see Exhibit 1 below). This framework includes a set of standards that address utility, feasibility, propriety, and accuracy and aid in determining the potential effectiveness of the evaluation. The framework also includes six steps that can serve as the basis for an evaluation. Although each of these steps builds on the previous one, the steps may be completed out of sequence. The activities conducted by the TCE-HIV Multi-Site Evaluation Team to address these steps are as follows:

• Engage Stakeholders: Held a series of Expert Panel Meetings and Project Working Group Meetings, obtained input from all Grantees through preliminary site visits, and met with SAMHSA Government Project Officer to get information regarding lessons learned from previous TCE-HIV cohorts.

• Describe the Program: Conducted data extraction to gather information from each grant application to explore commonalities and differences among the Grantees and conducted a preliminary visit to each grantee site to learn about their proposed program implementation.

• Focus the Evaluation Design: Gathered information from stakeholders to help focus the evaluation design in order to ensure that it is rigorous yet realistic and promotes efficient use of time and project resources.

• Gather Credible Evidence: The TCE-HIV Multi-Site Evaluation Team’s goal is to collect information that will convey a well-rounded picture of the TCE-HIV program and be relevant for answering the evaluation questions.

• Justify Conclusions. TCE-HIV Multi-Site Evaluation Team plans to justify its conclusions based on the data collected so stakeholders can use the evaluation results with confidence.

• Ensure Use and Share Lessons Learned. TCE-HIV Multi-Site Evaluation Team will make every effort to ensure that the evaluation processes and findings are disseminated and used to benefit current and future TCE-HIV cohorts.

Exhibit 1. CDC Program Evaluation Framework

Source: Centers for Disease Control and Prevention, 1999

The Multi-Site Evaluation contract was awarded September 1, 2008. The activities conducted in the first 2 years of the 5-year contract have (1) systematically followed the CDC evaluation framework; (2) developed the Multi-Site Evaluation plan through meetings with multidisciplinary groups of experts to focus the evaluation design; (3) reviewed each Grantee application to extract descriptive program information; and (4) conducted preliminary site visits to engage stakeholders and build relationships. As stipulated in the RFA, Grantees are required to participate in the Multi-Site Evaluation of the TCE-HIV grant program to allow Grantees and SAMHSA to assess the progress of individual projects as well as measure the effectiveness of the grant program overall. In addition, all grantees are required to provide performance data for outreach/pretreatment and clinical treatment to SAMHSA. Grantees will be required to comply with any changes in data collection requirements that are a result of the Multi-Site Evaluation.

Description of the Fiscal Year 2008 TCE-HIV Grantees. According to the RFA, all Grantees must ensure that they use methods to reach populations of high-risk substance abusers in racial/ethnic minority communities (e.g., African American, Hispanic/Latino) and design programs that are effective in reaching those populations. Grantees could target one or more of the high-risk substance abusing populations listed in Exhibit 2 below. This table demonstrates that the 48 Grantees of the FY08 TCE-HIV program are a diverse group of programs, providing a wide variety of services to varied target populations.

Exhibit 2. Grantee Self-Reported Target Populations

High-Risk Target Population	Frequency*
Injecting and at-risk non-injecting drug (including alcohol) users and their partners	44
Women, including women and their children	36
Individuals who have been released from prisons and jails within the past 2 years	31
Men who have sex with men (MSMs)	26
Adolescents (ages 12–17) and/or young adults (ages 18–24)	15
Returning veterans	7

* Grantees could target multiple high-risk populations, thus these numbers do not add to 48 (i.e., the number of Grantees).

As specified in the RFA, Grantees could expand substance abuse treatment and/or outreach/pretreatment services in conjunction with HIV/AIDS services, enhance substance abuse treatment and/or outreach and pretreatment services, or do both. Additionally, Grantees could implement outreach/pretreatment services, treatment services, or both outreach/pretreatment services and treatment services as part of their grant funding (see Exhibit 3 below for program model types).

Exhibit 3. Grantee Program Model Type

Program Models	Frequency
Outreach/Pretreatment Grantees	9
Treatment Grantees	20
Both Outreach/Pretreatment and Treatment Grantees	19

Based on a review of their grant applications, it is notable that the 48 Grantees proposed a vast number of different Evidence-Based Practices (EBPs) (See Exhibit 4). Some Grantee agencies are implementing multiple EBPs and vary widely with regard to mechanisms for tracking fidelity of EBP implementation. Further, informal modification of EBPs appears relatively common across the Grantees, which presents a need for further clarification regarding the degree to which implementation of the same EBPs is consistent across the 48 Grantees.

Exhibit 4. Commonly Seen Practices Across the FY09 TCE-HIV Cohort
Evidence-Based Practice	Frequency*
Case Management (including comprehensive case management and medical case management)**	48
Matrix Model***	15
Motivational Interviewing (MI) / Motivational Enhancement Therapy (MET)****	12
Cognitive Behavioral Therapy	8
Seeking Safety***	7
NIDA Outreach Model	5
Integrated Dual Diagnosis Treatment (IDDT)	4
Trauma Recovery and Empowerment Model (TREM)/Men's Trauma Recovery and Empowerment Model (MTREM)	3
Intensive Outpatient Treatment	3
*Grantees could implement multiple EBPs, thus these numbers do not add to 48 (i.e., the number of Grantees). Exhibit 4 reflects only commonly seen EBPs across the cohort (i.e., those reported 3+ or more times across the cohort). ** Case management is considered an EBP by some Grantees and not by others (and some Grantees utilize subtypes of case management, such as strength-based or comprehensive case management). To prevent confusion associated with whether a given Grantee was utilizing some form of case management, the frequency tally represents all Grantees reporting the use of any form of case management. *** These manualized practices are not implemented in their entirety at all Grantee agencies using the EBP. **** MI/MET may be underrepresented here because a number of Grantees do not include it as an EBP in their program, despite reporting receiving training and using the practice in site visits and/or their Grantee applications.

Based on the complexities of the 48 Grantees due to multiple target populations, varied forms of EBPs (or lack thereof), and multiple programmatic approaches (i.e., outreach/pretreatment, treatment, both), there is a need to better understand who they are serving, what programs they are implementing, and what services are being delivered and by whom.

The Multi-Site Evaluation will be able to address specific goals and objectives and provide SAMHSA with valuable information regarding how the TCE-HIV program is meeting the overall goals of the MAI. Specifically, the overarching goals and associated objectives of this formative evaluation are as follows:

Goal #1: To monitor the implementation activities of the TCE-HIV funded grantees including their EBPs, service delivery, and contextual factors that facilitate or impede program success.

Objective #1: Conduct semi-structured interviews with each of the grantees’ administrative staff within 3 months of receiving OMB clearance and in the fourth year of program implementation

Objective #2: Conduct semi-structured interviews with the direct service staff who are engaged in service delivery with clients within 3 months of receiving OMB clearance and in the fourth year of program implementation

Objective #3: Conduct semi-structured interviews with no more than four grantee partner agencies of each grantee that are engaged in service delivery to clients within 3 months of receiving OMB clearance and within the fourth year of program implementation

Goal #2: Obtain information on risk behaviors, HIV testing/serostatus, social support, medical and mental health, and motivation for treatment from clients of the TCE-HIV grantees and monitor their satisfaction with services received.

Objective #1: Administer a client questionnaire to each client at the same three points (i.e., intake, discharge, 6 months post baseline) that grantees are currently required to submit data to SAMHSA and explore if grantees are achieving program results

Objective #2: Conduct one focus group with no more than 9 clients from each of the 40 treatment grantees to obtain contextual information on client substance abuse treatment experience and satisfaction within 3 months of receiving OMB clearance and in the fourth year of program implementation

This Multi-Site Evaluation is the first TCE-HIV evaluation and represents the most comprehensive assessment of SAMHSA’s TCE-HIV program ever undertaken. This evaluation will allow SAMHSA to determine the extent to which the TCE-HIV program has met its objective of expanding and enhancing substance abuse outreach/pretreatment and treatment services in conjunction with HIV/AIDS services to racial and ethnic minority communities.

2. Purpose and Use of Information

The purpose of this evaluation is to provide information about the implementation activities of substance abuse outreach/pretreatment and treatment programs offered through SAMHSA TCE-HIV grant funding, and to obtain information on the HIV risk behaviors and psychosocial factors of clients in racial and ethnic minority communities. The data collected will be utilized to address the purpose of the Multi-Site Evaluation, which is informed by the CDC Framework for Program Evaluation in Public Health (1999) and described in Section A1, Circumstances of Information Collection. Based on this program evaluation framework, a formative evaluation effort with two primary goals is supported:

• Evaluation Goal #1: To monitor the implementation activities of the TCE-HIV funded Grantees including their EBPs, service delivery, and contextual factors that facilitate or impede program success.

• Evaluation Goal #2: To obtain information on risk behaviors, HIV testing/serostatus, social support, medical and mental health, and motivation for treatment from clients of the TCE-HIV grantees and monitor their satisfaction with services received.

Data collected related to the first evaluation goal will describe the content of Grantees’ interventions and their underlying EBPs as well as provide information about the TCE-HIV services delivered, who delivered them, how they were delivered, to whom they were delivered, and whether these services modified the Grantee’s overall substance abuse treatment service delivery system. Consolidating the information on program components across Grantees provides the basis for identifying commonalities among the diverse programs and populations being served. Information related to the second evaluation goal will provide an understanding of the clients being served with regard to their risk behaviors and other conditions and will allow for monitoring of these factors over time.

To address the two formative evaluation goals, data associated with client risk behaviors will be collected continuously via a counselor-administered client survey, and program implementation data collection will occur at two distinct points in time through site visits. The purpose of the site visits is to observe program processes related to project implementation, to conduct semi-structured interviews with project staff and partners/collaborators, and to conduct client focus groups. The site visit for the first data collection point (Year 3) is designed to capture information on and changes to the specific activities associated with implementation, the staff involved in program implementation activities, and the amount of time required for a client to complete treatment. The second site visit (Year 4) will collect a repeat measure of all data collected during Year 3. Comparisons of the data collected during the Year 3 and Year 4 site visits will yield information on community and program changes as measured by site visit instruments. Simultaneously, counselor-administered client questionnaires will be collected continuously to allow for monitoring of the key constructs of interest (i.e., HIV risk behaviors, HIV testing/serostatus, social connectedness, quality of life) over time. TCE-HIV counselors will facilitate the completion of the client questionnaire for each treatment client at baseline, discharge, and 6 months post baseline. The client questionnaire will be administered to treatment clients (versus outreach/pretreatment clients) in order to maximize response rates at each of the three data collection points and reduce burden on Grantees staff responsible for collecting the data (see Section B3, Methods to Maximize Response Rates for additional information).

Evaluation Goal #1: Program Implementation Activity Data Collection. The evaluation of the program implementation activities of the 48 TCE-HIV Grantees will focus on questions related to the service delivery process to determine activities required of Grantees to implement and sustain TCE-HIV services. Specifically, these questions will assess:

• Community/contextual conditions

• Community partnerships

• Program characteristics

• System resources

• Grantee activities and operations

• Staffing and training

• Client conditions (e.g., barriers to receiving services)

The semi-structured interviews with project directors, Grantee staff, and community collaborators/partners, along with client focus groups will provide the data necessary to assess Grantees’ implementation activities. The following sections present a description of the implementation activity data collection methodologies.

Administrative Staff Semi-Structured Interview: (See Attachment 3 - Document 1.) The goal of this interview is to collect information regarding the development and changes in TCE-HIV program operations, staffing, training, and services (e.g., outreach-pretreatment and/or treatment activities); to provide improved understanding of program, agency, and community capacity changes that result from TCE-HIV activities; and to gather documentation of changes in the number or nature of partnerships and collaborations both internal and external to the TCE-HIV program agency.

Individuals who perform administrative tasks related to the TCE-HIV program (e.g., Project Director, Program Manager, and Executive Director) are eligible to participate in the semi-structured interview. It is estimated that one administrative staff member per Grantee site will be interviewed during a given interview session. The administrative staff interviews will be conducted in two parts. The first part will be conducted with an executive staff member of the agency; and the second part with the Project Director or Program Manager of the TCE-HIV project. It is likely that one person from the agency may fulfill both roles/positions. If this is the case, the full interview will be conducted with that person.

Direct Services Staff Semi-Structured Interview: (See Attachment 5 - Document 1.) The goal of this interview will be to collect information about the development of outreach/pretreatment and treatment operations/activities and to understand program changes that result from TCE-HIV activities. Individuals from the Grantee organization who have direct contact with clients to perform outreach/pretreatment and treatment-related tasks will be eligible to participate in the semi-structured interview. Administrative staff members and direct services staff members perform different functions in Grantee organizations and can each provide different perspectives on treatment operations/activities within an organization. To obtain a balanced set of data, information will be collected from both. Examples of those performing direct service activities include outreach workers, treatment counselors, and case managers. It is estimated that more than one direct services staff member will participate in a given interview session.

Partner/Collaborator Semi-Structured Interview: (See Attachment 7 - Document 1.) The goal of this interview will be to collect information regarding the relationship between community partners/collaborators and the TCE-HIV Grantee agency; to document the types of activities and services the partners/collaborators provide in cooperation with the Grantee; and to assess achievement of goals for clients related to collaboration between the partner/collaborator and Grantee agency. A partner/collaborator is an agency or organization that provides services and activities related to the TCE-HIV program. It is estimated that up to two individuals per community partner/collaborator agency will participate in a given interview session.

Client Focus Groups: (See Attachment 1a - Document 1.) The goal of the client focus group conducted during TCE-HIV Multi-Site Evaluation site visits will be to collect data from a subset of TCE-HIV treatment clients regarding their experiences with the treatment program, including barriers and facilitators of treatment services, and client-level behaviors (i.e., substance use/abuse, risk behavior, quality of life). The Multi-Site Evaluation staff will work collaboratively with Grantee staff to identify client focus group participants. Only those clients who have been administered the Government Performance and Results Act (GPRA) Tool and who are receiving TCE-HIV funded treatment services will be eligible to participate in the client focus group. Several inclusion and exclusion criteria will be used to ensure client focus group participants are representative of the target audience served by the Grantee. For example, clients who have been receiving services for fewer than 14 days may be unable to share information based on their limited time in the program. Additionally, the recruitment of clients for the focus groups will strive for a balance in participants with regard to gender and age. A specified list of focus group inclusion and exclusion criteria will be developed and shared with the Grantees to aid in client identification. It is estimated that up to nine clients will participate in each focus group session.

Evaluation Goal #2: Client Behavior-Related Data Collection. While the program implementation data collection activity will produce critically useful descriptive information at a community and grantee level about the TCE-HIV Grantees, how their programs are implemented, and changes through the course of the 5-year funding cycle, the evaluation of client behaviors for the 40 treatment Grantees will provide an opportunity to identify key risk behaviors for this population and monitor client risk behaviors and other client conditions over time.

Specifically, client behavior-related data collection activities will provide information about:

• Client characteristics

• HIV testing and status

• Criminal activity (the sole measure is the GPRA Tool)

• Education and housing (the sole measure of education is the GPRA Tool)

• Substance abuse (the sole measure is the GPRA Tool)

• HIV risk behavior

• Exposure to services (i.e., dosage; the measure is the GPRA Tool)

The following paragraphs present a description of the client survey.

Client Survey: (See Attachment 1a - Documents 2 and 3, Attachment 1b - Document 1.) The goals of the client survey are to collect information that will document client HIV risk behavior and client conditions.

The targeted audience for the TCE-HIV Multi-Site Evaluation client surveys is clients in a treatment program administered by the 40 Grantee TCE-HIV treatment programs. The client surveys will be administered by Grantee counselors. All clients who receive the initial baseline survey (Attachment 1a - Document 2) will be asked to complete follow-up surveys at discharge (Attachment 1a - Document 3), and at 6 months post baseline (Attachment 1b - Document 1). Data collection at the two follow-up points is necessary to monitor client risk behaviors and other conditions over time.

The client survey was developed from a combination of new questions (i.e., Sections A–C, items E13–E25) and from subscales of existing measures (i.e., Section D, items E1–E12, and Section F). Each subscale was carefully selected to assess the construct of interest yet maintain its ability to be a stand-alone subscale that can be scored separately from the entire measure.

Section D (items D1–D9) on the client survey is from the Social Support subscale questions from the Texas Christian University Client Evaluation of Self and Treatment (TCU CEST). Questions for items E1–E12 are the depression and anxiety symptom dimensions from the Brief Symptom Inventory (BSI). Section F contains the Taking Steps subscale questions from the Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES). A general description of these existing measures and the reliability and validity information for each subscale can be found below.

• Texas Christian University Client Evaluation of Self and Treatment (TCU CEST; Texas Christian University, 2005). The TCU CEST was developed as part of NIDA Grant R37 DA13093, Transferring Drug Abuse Treatment and Assessment Resources. The TCU CEST includes a set of assessments that target specific client needs and the status of clients in different stages of change during treatment. The coefficient alpha for the Social Support Scale is .84.

• Brief Symptom Inventory (BSI; Derogatis, 1993). The purpose of this measure is to identify self-reported, clinically relevant psychological symptoms in adults. Internal consistency estimates for the two subscales are .85 (depression) and .81 (anxiety). Good internal consistency reliability is supported by several other independent studies (Croog, Levine, Testa, & Brown, 1986; Derogatis & Coons, 1993).

• Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES; Miller & Tonigan, 1997). Psychometric analyses reveal the following psychometric characteristics of the Taking Steps scale questions:

• Cronbach Alpha = .83 – .96

• Intraclass Test-Retest Reliability = .91

• Pearson Test-Retest Reliability = .93

Data collected via semi-structured interviews, client focus groups, and surveys will be used to enable SAMHSA’s TCE-HIV program to increase its effectiveness in providing substance abuse treatment and HIV services for minority populations. Additionally, the Multi-Site Evaluation will help SAMHSA achieve the goals of its TCE-HIV program and the Minority AIDS Initiative (MAI).

3. Use of Information Technology

Data collection for the Multi-Site Evaluation requires the use of information technology to develop an online data collection system. This online system will provide the client survey (i.e., baseline, discharge, and 6 months post baseline) for data entry by Grantee staff. Technology will be used to manage, secure, and store the data to ensure data management control.

Client Survey. Grantee staff will administer the client survey during a face-to-face encounter at baseline, discharge, and 6 months post baseline. Client survey data will be entered at the Grantee site into an online system that will be password protected. Each day, the TCE-HIV evaluation staff will upload the data over a secure network connection directly to a server at JBS headquarters where the data will also be encrypted and password protected. Details about JBS’ network security procedures and security protocols are presented in Attachment 2a - Document 1. Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost by the survey staff or of data being lost in transit or delivered to an incorrect location. However, not all of the Grantees may be equipped to enter this data into the online system, in which case paper copies of the completed surveys from these Grantees will be submitted to JBS only with unique alphanumeric identifiers and the data will be entered into the online system at JBS. Paper copies will be stored in a locked file cabinet, with no name or identifying information attached.

4. Effort to Identify Duplication

The Multi-Site Evaluation Team has identified items answered through GPRA data collection that could be used to monitor some of the implementation activities of the TCE-HIV grantees. It should be noted that the burden for the GPRA Tool is approved under OMB No. 0930-0208. To ensure minimal duplication and burden on Grantee staff and clients, the Multi-Site Evaluation Team did not include items collected by the GPRA Tool (e.g., demographic information, substance use information) in the client survey. However, because of the nature of the work the TCE-HIV grantees perform in regards to HIV risk prevention, the GPRA Tool does not have the measures that allow for monitoring the grantees and their clients on sexual risk behaviors, HIV testing and HIV serostatus, and motivation for treatment. Therefore, the client survey was developed to monitor grantee program results and capture information regarding client risk behaviors (i.e., sexual risk, HIV testing/serostatus), motivation for treatment). In this evaluation effort, the GPRA Tool and the client survey will provide stakeholders a better understanding of how to enhance services, build program capacity, and strengthen practice. Thus, it is important to note that the Multi-Site Evaluation Team will utilize GPRA measures in conjunction with the client survey to help monitor program results.

Because the GPRA data are the key and only measure of client demographics and substance use/abuse data, SAMHSA will provide the Multi-Site Evaluation Team with all cleaned individual-level TCE-HIV Grantee GPRA data at the three data collection points (i.e., baseline, discharge, and 6 months post baseline). The GPRA data items included in this evaluation are:

• Client ID numbers (needed to link GPRA data to TCE-HIV client surveys)

• Client demographics (i.e., gender, ethnicity, race, date of birth)

• Client substance use/abuse

• Family living conditions

• Education, employment, and income

• Crime and criminal justice status

• Mental and physical health

• Social connectedness.

Grantees will be asked to use the client GPRA identification number on the client survey at the three data collection points so the client surveys can be linked to the GPRA data, which contains demographics, substance use/substance abuse, and other constructs (e.g., social connectedness, family living conditions, employment) that are needed to fully answer the TCE-HIV Multi-Site Evaluation questions.

5. Involvement of Small Entities

As part of their TCE-HIV grant requirements, Grantees are required to participate in the data collection activities of the Multi-Site Evaluation. Client participation in data collection activities is voluntary and access to treatment services is not based on participation in the Multi-Site Evaluation. It is likely that many of the project’s respondents will be from relatively small treatment programs. Information collection for this study is not anticipated to have a significant impact on the individuals or on the programs or practices with which respondents may be affiliated.

The information to be obtained from respondents is the minimum necessary to achieve the objectives of the evaluation; however, completion of survey instruments, participation in focus groups, and/or participation in semi-structured interviews will likely induce burden. To reduce this burden, every attempt will be made to move respondents quickly through questions. For example, a screener question will be used in the client survey that asks about previous testing with HIV positive results. If the respondent indicates that he/she has not previously tested HIV positive, the interviewer will skip to the next category.

6. Consequences if Information Is Collected Less Frequently

During this 5-year Multi-Site Evaluation, the frequency of data collection from the Grantees and the clients they serve is held to the minimum necessary to meet the needs of the evaluation goals and objectives. Data will be collected in year 3 and year 4 to assess changes in the community- and program-level activities that will be monitored. Similarly, the information collection points for the client survey (i.e., baseline, discharge, 6 months post baseline) are designed to coincide with GPRA data collection.

Client Survey. Client surveys will be administered to each client who have been administered a GPRA Tool and is receiving treatment from the 40 treatment Grantees. All clients completing the survey at baseline will also complete the survey at discharge from treatment, and 6 months following baseline to mirror the GPRA Tool administration. Data collection at discharge and 6 months following baseline will be used to monitor change in client HIV risk behaviors, social support, and quality of life indicators. Waiting until 6 months after baseline from treatment services allows ample time for monitoring changes in risk behaviors and/or quality of life conditions. Alternatively, waiting more than 6 months jeopardizes the validity of the information obtained because self-reported data become less accurate as time passes. Moreover, follow-up response rates, especially among many of the populations to which TCE-HIV is being delivered, decrease over time.

Semi-Structured Interviews. The semi-structured interviews will be administered to staff and partners/collaborators from each of the 48 TCE-HIV Grantees during site visits conducted in Year 3. To monitor changes in implementation activities of TCE-HIV programs, evaluation staff will compare baseline measures with the same indicators approximately 16 months (Year 4) after the initial measurement. Comparisons between these time points will yield information on community, program, and staff changes as measured at Year 3 and Year 4. Collecting follow-up data at 16 months after the Year 3 site visit is optimal for producing useful comparison data, as community and programmatic level changes may be manifested within the 16 month cycle. Alternatively, if the information is collected after more than 16 months, the resultant shortened period for data analysis may result in a lack of important information for SAMHSA on how best to understand the community and contextual conditions in which the TCE-HIV programs exist and provide services.

Client Focus Groups. Focus groups will be conducted during site visits in Year 3 and Year 4 with a subset of treatment clients at each of the 40 treatment Grantees. The goals of the client focus groups are to gather information about client substance abuse treatment history, client behavior (i.e., substance use/abuse, risk behavior, quality of life), client perception of social support, and client satisfaction with the TCE-HIV program.

No technical or legal barriers to reduce burden exist if information is collected less frequently.

7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The notice required by 5 CFR1320.8(d) was published in the Federal Register on October 23, 2009 (74 FR 54830-54831). No comments were received in response to this notice.

SAMHSA has made extensive use of experts in the area of substance abuse and HIV/AIDS research to provide guidance on the design of the Multi-Site Evaluation. An expert panel meeting was held November 17–18, 2008, to review various aspects of the Multi-Site Evaluation, including the evaluation plan, data collection procedures, methods, and literature review. The list of experts who participated is provided in Exhibit 5.

Exhibit 5. Expert Panel Members

Expert	Affiliation	Contact Information
Faye Belgrave, PhD	Director, Center for Cultural Experiences in Prevention Virginia Commonwealth University College of Humanities and Sciences Department of Psychology 806 West Franklin Street Richmond, VA 23284-2018	Phone: 804-827-3908 E-mail: [email protected]
Benjamin Bowser, PhD	Interim Dean, California State University, East Bay 3103 Meiklejohn Hall Hayward, CA 94542	Phone: 510-885-3173 Fax: 510-885-2390 E-mail: [email protected]
Karen Corsi, ScD	University of Colorado Medical School––Project Safe 1741 Vine St. Denver, CO 80206	Phone: 303-315-0951 Fax: 303-316-7697 E-mail: [email protected]
Chinazo Cunningham, MD	Clinical Faculty, Residency Program in Social Internal Medicine and Primary Care Montefiore Medical Center 111 East 210th Street Bronx, NY 10467	E-mail: [email protected]
Agatha Eke, PhD	Centers for Disease Control and Prevention 1600 Clifton Rd., NE Mailstop E-37 Atlanta, GA 30333	Phone: 404-639-1906 Fax: 404-639-1950 E-mail: [email protected]
Jeffrey Friedman, MA	Long Island Association for AIDS Care 60 Adams Ave. Hauppauge, NY 11788	Phone: 631-385-2451 E-mail: [email protected]
Robert Fullilove, EdD	Associate Dean, Community and Minority Affairs Professor, Clinical Sociomedical Sciences Co-Director, Community Research Group Columbia University Mailman School of Public Health 722 West 168th Street New York, NY 10032	Phone: 212-305-4734 E-mail: [email protected]
Vincent Guilamo-Ramos, PhD	Columbia University, Associate Professor, School of Social Work 1255 Amsterdam Avenue New York, NY 10027	Phone: 212-851-1659 Fax: 212-851-2206 E-mail: [email protected]
Lena Lundgren, PhD	Boston University Center for Addictions Research and Services 232 Bay State Road, 4th Floor Boston, Massachusetts 02215	Phone: 617-353-7222 Fax: 617-358-2368 E-mail: [email protected]
Jacques Normand, PhD	Director, AIDS Research Program, National Institute on Drug Abuse (NIDA) 6001 Executive Boulevard Bethesda, MD 20892	Phone: 301-402-1919 E-mail: [email protected]
Sybil Ward, AA	Jefferson Comprehensive Care System 2020 W. 3rd Street, Suite 201 Little Rock, AR 72205	Phone: 501-663-7166 E-mail: [email protected]
Chyvette Williams, PhD	Assistant Professor Health and Policy Administration School of Public Health (MC 923) University of Illinois at Chicago 1603 W. Taylor Street Chicago, IL 60612	Phone: 312-355-5299 E-mail: [email protected]
Nickolas Zaller, PhD	The Miriam Hospital Immunology Center 164 Summit Ave RISE/CFAR Building Providence, RI 02096	Phone: 401-793-4875 E-mail: [email protected]

A smaller Project Working Group (PWG) was convened to advise and provide recommendations to SAMHSA and the Multi-Site Evaluation Team on all phases of the TCE-HIV Multi-Site Evaluation including the design, instrumentation development, implementation, and analysis. As a result of PWG consultation and feedback, data collection instruments were revised. See Attachment 2a - Document 2 for a summary of PWG feedback and revisions made to instruments. The scope and purpose of the PWG will evolve in response to the changing needs of the project as the evaluation is implemented and the Multi-Site Evaluation Team engages in data collection, management, and analysis. The PWG will meet frequently (i.e., quarterly) during the initial phase of the evaluation project and will meet as needed during the implementation, analysis, and reporting phases.

The PWG is an interdisciplinary group of Federal staff, the Multi-Site Evaluation Team, and external consultants (i.e., evaluators and researchers) who can provide a broad perspective on various evaluation-related tasks. The PWG members have expertise in substance abuse treatment and prevention issues, HIV testing and prevention, program implementation, minority populations, and program evaluation. The list of PWG members is provided in Exhibit 6.

Exhibit 6. Project Working Group Members

Member	Affiliation	Contact Information
Ravinia Hayes-Cozier	National Minority AIDS Council Director of Government Relations and Public Policy 1931 13th St. NW Washington, DC 20009-4432	Phone: 202-483-6622 ext. 308 Fax: 202-483-1135 E-mail: [email protected]
Jim Derzon, PhD	Battelle Memorial Institute Task Lead for Quantitative Analysis 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-248-1640 Fax: 703-527-5640 E-mail: [email protected]
Colin Flynn	Department of Health and Mental Hygiene Chief, Center for HIV Surveillance and Epidemiology 500 North Calvert St. Shillman Building, 5th Floor Baltimore, MD 21202	Phone: 410-767-5050 Fax: 410-767-6489 E-mail: [email protected]
Vincent Guilamo-Ramos, PhD	Columbia University, Associate Professor, School of Social Work 1255 Amsterdam Avenue New York, NY 10027	Phone: 212-851-1659 Fax: 212-851-2206 E-mail: [email protected]
Susan Hayashi, PhD	JBS International, Inc. Vice President 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 301-495-1080 ext. 4588 Fax: 301-587-4352 E-mail: [email protected]
Warren Hewitt, MS	SAMHSA/CSAT AIDS Coordinator 1 Choke Cherry Rd Rockville, MD 20857	Phone: 240-276-1616 Fax: 240-276-1670 E-mail: [email protected]
Kevin Hylton, PhD	Alliances for Quality Education, Inc. TCE-HIV Multi-Site Evaluation Deputy Project Director 8181 Professional Place, Suite 110 Landover, Maryland 20785	Phone: 301-583-8424 Fax: 301-583-8422 E-mail: [email protected]
Jennifer Kasten, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Lead 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4145 Fax: 301-495-1080 E-mail: [email protected]
Resa Matthew, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Project Director 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4608 Fax: 301-495-1080 E-mail: [email protected]
Naomi Tomoyasu, PhD	SAMHSA/ CSAT Branch Chief 1 Choke Cherry Rd Rockville, MD 20857	Phone: 240-276-1613 E-mail: [email protected]
Willie Tompkins, PhD	SAMHSA/CSAT Government Project Officer 1 Choke Cherry Rd Rockville, MD 20857	Phone: 240-276-2899 E-mail: [email protected]
Ping Yu, PhD	Battelle Memorial Institute TCE-HIV Multi-Site Evaluation Subcontract Director/Co-PI 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-875-2981 Fax: 703-527-5640 E-mail: [email protected]

Additional consultation was sought from TCE-HIV Grantees and representative respondents during pilot site visits conducted October–December 2009. The visits provided the Multi-Site Evaluation Team with feedback and comments regarding data collection instruments and processes. As a result of the consultation and feedback, data collection instruments were revised. See Attachment 4 - Document 1 for a summary of pilot site visit feedback and revisions made to instruments. The list of TCE-HIV Grantee consultants is provided in Exhibit 7.

Exhibit 7. TCE-HIV Grantee Consultants

Consultant	Affiliation	Contact Information
Carla Hewitt	Safe Haven 1140 North Capital St, NW Washington, DC 20018	Phone: 202-589-1505 E-mail: [email protected]
Basha Silverman, MA	Brandywine Counseling 2713 Lancaster Avenue Wilmington, DE 19805	Phone: 302-656-2348 Fax: 302-656-0746 E-mail: [email protected]
Jim May, PhD	Richmond Behavioral Health Authority 107 South Street Richmond, VA 23219	Phone: 804-819-4202 Fax: 804-819-8783 E-mail: [email protected]

9. Payment to Respondents

In keeping with the Office for Human Research Protections (OHRP) guidelines, compensating research subjects in exchange for their participation is a common and, in general, acceptable practice. Further, compensation for participation in research should be just or fair.

Although participation in the focus groups is voluntary, participants are likely to perceive a time cost and burden associated with their participation. Thus, for the client focus groups, participants will be compensated for an hour of their time. Cash equivalent compensation will be offered in lieu of cash payments. Gift cards ($20) from major stores (e.g., Walmart™, Target™) will be used for remuneration for treatment clients to participate in the focus groups.

In studies of considerable duration or that involve multiple participant interactions or interventions, such as this formative evaluation, OHRP recommends that payment be prorated for the time of participation in the study rather than delayed until study completion, because the latter could unduly influence a participant’s decision to exercise his or her right to withdraw at any time. For example, if the study is conducted over a period of 6 months, there might be a monthly or bimonthly payment. Similarly, compensation (as part of this formative evaluation) for the client surveys will be provided to clients in a tiered structure with a $10 gift card provided at baseline, $15 gift card at discharge, and $20 gift card at 6 months post baseline. As with the focus groups, cash equivalent compensation will be offered in lieu of cash payments. Gift cards from major stores (e.g., Walmart™, Target™) will be used for compensation for treatment clients to complete the client survey at baseline, discharge, and 6 months post baseline.

The literature suggests compensation to substance abuse treatment clients is more likely to improve their engagement in the informed consent process and increase their follow-up rates. In support, empirical studies of substance abuse treatment clients tested the effects of varying levels of compensation on research retention, participant perceptions of coercion, and self-reported substance use behavior. In the first study, participants enrolled at intake to substance abuse treatment were randomized to receive $10, $40, or $70 via cash or gift card (Festinger, Marlowe, Croft, Dugosh, Mastro et al., 2005). In the second study, participants were randomized to receive $70, $100, $130, or $160 (Festinger, Marlowe, Dugosh, Croft, & Arabia, 2008). At 6 months follow-up, participants in both studies were more likely to return for follow-up when offered cash incentives from the higher end of the scale. There were no significant differences among any of the groups in terms of new substance use, perceived levels of coercion (which was generally low), or satisfaction with research (which was generally high). Client compensation also improves participant engagement in the informed consent process. In a study of 31 drug court participants, Festinger and colleagues (2009) randomized participants to either an incentivized or standard consent process. Participants were notified that they would be quizzed on details of the consent information within 2 weeks of enrollment. Following quiz administration, members of the incentivized group ($5 per correct answer, up to $75) recalled 65 percent of the consent information versus 42 percent recall in the standard group. The authors posit that monetary incentives may motivate participants to pay better attention to the consent process, and is possibly a more effective mechanism for improving recall.

SAMHSA and the Multi-Site Evaluation Team are committed to maintaining the highest standards for this Multi-Site Evaluation, and reliance on empiric evidence suggests that our proposed client compensation strategy for this evaluation is effective in improving client engagement in the consent and follow-up processes. Removing client compensation from the proposed evaluation could potentially jeopardize the process to fully engage clients in the consent process and yield high response rates.

10. Assurance of Confidentiality

JBS’ Institutional Review Board (IRB) application has been approved (JBS IRB # RM10-001) (Attachment 2b – Document 5), which ensures the Multi-Site Evaluation Team meet corporate, industry, and society standards to protect study participants. This approval ensures compliance with the spirit and the letter of regulations from the Department of Health and Human Services (DHHS) governing such projects. JBS’ systems and procedures for collecting and processing data are designed to help ensure the privacy, to the extent of the law, of study participants and the data they provide. Documents with data about Grantees or individual clients will be identified by an assigned study alpha numeric identification number.

Grantees have been approved to administer the GPRA Tool, thus informed consent is in place for Grantee staff to collect GPRA Tool data. The collection of GPRA data was cleared under OMB No. 0930-0208. After completing GPRA data collection, Grantee staff who deliver TCE-HIV services will then briefly explain to the client the reason for an additional survey, describe the survey length, and explain the process. Grantee staff members will tell the clients that it is not mandatory that they be administered the client survey and that their treatment will not be affected if they decline to complete the survey. If clients agree to participate, the Grantee staff will provide an Information Sheet which they will read/present to the client as part of the survey administration. The Information Sheet provides pertinent information about the administration, risks, and benefits of the client survey. However, it is advised that Grantees also follow local and/or State requirements related to client informed consent. The process for administering the client survey is designed to protect privacy, reduce client discomfort and burden, and ensure the collection of quality data.

The client survey will include the OMB approval expiration date, the statement of survey burden, and the statement that the evaluation is federally sponsored. Grantee staff will administer the survey to the client in a private location (e.g., an office) to ensure privacy. Staff will read each question and the list of responses for questions to clients and record their answers.

Client survey data will be entered at the Grantee site into a password-protected online system. Each day, the TCE-HIV evaluation staff will upload the data over a secure network connection directly to a server at JBS headquarters where the data will also be encrypted and password protected. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the surveys with only alpha numeric identifiers (i.e., GPRA identification numbers) from these Grantees will be provided and entered onsite at JBS. Any paper copies will be stored in a locked file cabinet, with no name/identifying information attached.

The TCE-HIV Multi-Site Evaluation staff will use passwords to safeguard project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. Multi-Site Evaluation Team staff also will be trained on handling sensitive data and the importance of privacy. As a further precautionary measure, the data being collected will have no identifying information that can be linked back to the client. In keeping with 45 CFR 46, Protection of Human Subjects (Attachment 2a - Document 3), the TCE-HIV procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected.

Data from the TCE-HIV client surveys will be kept strictly private to the extent of the law in compliance with the Privacy Act of 1974 (5 U.S.C. 552a; Attachment 2b - Document 1.) The privacy of data records will be explained to all respondents during the consent process. Limits to privacy (e.g., notifying authorities to protect study participants or someone else from serious harm, including child abuse/neglect) will be explained.

For clients providing information in the client focus groups, the responses will be kept private; that is, no identifying information will be linked with the information provided and the information will be reported in aggregate. This de-identification process will extend to semi-structured interviews completed by project directors, Grantee staff, and partners/collaborators. Any direct quotes that are used in reporting will not be attributed directly to the speaker, but will be credited only as an interview or focus group participant comment.

11. Questions of a Sensitive Nature

Client Surveys and Client Focus Groups. The TCE-HIV client surveys and focus groups, by necessity, will collect sensitive information (e.g., substance use, mental health, and other health and social risk factors) because this information is of interest to SAMHSA. Sensitive information of this nature is always regarded as highly private and clients’ privacy in federally assisted treatment programs is assured through strict adherence to the statute (42 U.S.C. §290dd.2) and regulation (42 C.F.R. Part 2) regarding confidentiality of Alcohol and Drug Abuse Patient Records. A Certificate of Confidentiality has been issued by SAMHSA (Attachment 2b – Document 6). The Certificate is designed to protect identifiable research information from forced or compelled exposure. The Certificate of Confidentiality is designed to protect the investigators from being forced, even under a court order or subpoena, to release information that could identify survey or focus group participants. However, the TCE-HIV Multi-Site Evaluation Team may release identifying information in some circumstances. For example, the team may disclose medical information in cases of medical necessity, or take steps (including notifying authorities) to protect participants or someone else from serious harm, including child abuse/neglect. The Multi-Site Evaluation Team will also ensure that additional appropriate mechanisms and procedures are in place to protect the privacy of the identifiable information to be obtained in the evaluation.

Respondents will be informed about the purpose of the data collection and that responding to all questions is voluntary. They will be assured that they may stop taking the survey at any time or discontinue participation in the client focus group. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

Administrative Staff Semi-Structured Interview. No sensitive information will be collected from those individuals who perform administrative tasks related to the TCE-HIV program (e.g., project director, program manager, and executive director). The interview staff of the Multi-Site Evaluation Team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

Direct Services Staff Semi-Structured Interview. No sensitive information will be collected from direct services staff members (e.g., outreach workers, counselors). The interview staff of the Multi-Site Evaluation Team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

Partner/Collaborator Semi-Structured Interviews. No sensitive information will be collected from partner/collaborator (e.g., agencies or organizations that provide services and activities related to the TCE-HIV program). The interview staff of the Multi-Site Evaluation Team will obtain signed consent for participation in the interview data collection. Respondents will be informed about the purpose of the data collection. In addition, specific assurances will be provided to respondents concerning the safety and protection of data collected from them.

12. Estimates of Total Hour Burden

Estimate of the Total Hour Burden of the Collection of Information from Clients. The total client sample size for the TCE Multi-site data collection effort is estimated to be a maximum of 2,400 adult respondents (i.e., ages 18 and over). The baseline survey is expected to have a response rate of 100 percent, resulting in 2,400 respondents completing the baseline survey. The discharge survey is also expected to have a response rate of 100 percent resulting in 2,400 respondents completing the discharge survey. The set goal response rate for the 6-months post baseline survey is 80 percent of the baseline sample, resulting in 1,920 respondents completing the 6 months post baseline survey. The Year 3 and Year 4 treatment focus groups are each expected to have a response rate of 80 percent (i.e., not every Grantee will be able to provide nine clients who are willing to participate), resulting in a total of 720 respondents participating in focus groups. Based on these response rates, it is expected that client total responses will be 7,440. Exhibit 8 presents estimates of total burden and Exhibit 9 presents estimates of annualized burden based on pilot testing.

The hour burden for the client survey (i.e., baseline, discharge, 6-months post baseline) is calculated using the average completion time based on survey pilot testing (see Attachment 4 - Document 1). The time required to complete the surveys varies with client characteristics, in particular, substance use behaviors. Based on pilot testing, the total time to complete the client survey––including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information––was found to be 25 minutes. Exhibit 8 presents estimates of total burden based on pilot testing.

Exhibit 8. Multi-site Data Collection Burden for Clients, Grantee Staff, and Collaborators

Instrument/ Activity	Number of Respondents	Responses per Respondent	Total Responses	Hours per Response	Total Burden Hours	Hourly Wage	Total Respondent Cost*
Baseline data collection (clients)	2,400	1	2,400	.42	1,008	$20.32	$20,482.56
Discharge data (clients)		1	2,400	.42	1,008	$20.32	$20,482.56
6-months post baseline data collection (clients)		1	1,920	.42	806.4	$20.32	$16,386.05
Treatment focus group Year 3 (client)		1	360	1.0	360	$20.32	$7,315.20
Treatment focus group Year 4 (client)		1	360	1.0	360	$20.32	$7,315.20
Client subtotal	2,400		7,440		3,542.4		$71,981.57
Annualized Client Total	800	--	2,480	--	1,180.8
Semi-structured interviews (project director/program manager )	96	2	192	.75	144	$29.12	$4,193.28
Annualized PD/PM Total	32	--	64	--	48
Semi-structured interviews (Grantee direct services staff)	432	2	864	1.0	864	$19.05	$16,459.20
Annualized Service Staff Total	144	--	288	--	288
Semi-structured interviews (Community collaborators)	240	2	480	1.0	480	$19.21	$9,220.80
Annualized Collaborators Total	80	--	160	--	160
TOTAL	3,168		8,976		5,030.4		$101,854.87
Annualized Totals (3-year clearance for project)	1,056	--	2,992	--	1,676.8	--	--

^*Total respondent cost is calculated as hourly wage × time spent on survey × number of respondents.

Estimate of the Total Hour Burden of the Collection of Information from Project Directors/Program Managers. The total project director/program manager sample size for the TCE-HIV multi-site data collection effort is estimated to be a maximum of 96 respondents (48 sites, 2 respondents at each site). Exhibit 8 presents estimates of total burden and Exhibit 9 presents estimates of annualized burden based on pilot testing.

Estimate of the Total hour Burden of the Collection of Information from Grantee Direct Services Staff. The total grantee staff sample size for the TCE-HIV Multi-Site Evaluation data collection effort is estimated to be a maximum of 432 respondents (48 sites, 9 respondents in each site). Exhibit 8 presents estimates of total burden and Exhibit 9 presents estimates of annualized burden based on pilot testing.

Estimate of the Total Hour Burden of the Collection of Information from Community Collaborators. The total community collaborator sample size for the TCE-HIV Multi-Site Evaluation data collection effort is estimated to be a maximum of 240 respondents (48 sites, 5 respondents in each site). Exhibit 8 presents estimates of total burden and Exhibit 9 presents estimates of annualized burden based on pilot testing.

Estimate of the Total Cost Burden to the Respondents for the Collection of Information from Clients. There are no direct costs to respondents other than their time to participate in the study. The total cost for the evaluation of the time respondents spend completing the surveys and participating in focus groups is $71,981.57 (number of total baseline client respondent hours, plus discharge and 6 months post baseline hours, plus treatment focus group client respondent hours × $20.32, the estimated average hourly wages for adults as published by the U. S. Bureau of Labor Statistics, 2008).

Estimate of the Total Cost Burden to the Respondents for the Collection of Information from Project Directors/Program Managers. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend completing these interviews is $4,193.28 (number of project director respondent hours × $29.12, the estimated average hourly wages for individuals working in health-related occupations as published by the U.S. Bureau of Labor Statistics, 2008).

Estimate the Total Cost Burden to the Respondents for the Collection of Information from Grantee Direct Services Staff. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend participating in the semi-structured interviews and completing client treatment dosage forms is $16,459.20 (number of grantee staff respondent hours × $19.05, the estimated average hourly wages for individuals working in health-related staff occupations as published by the U. S. Bureau of Labor Statistics, 2008).

Estimate the Total Cost Burden to the Respondents for the Collection of Information from Community Collaborators. There are no direct costs to respondents other than their time to participate in the study. The total cost of the time respondents spend participating in these interviews is $9,220.80 (number of practitioner respondent hours × $19.21, the estimated average hourly wages for individuals working in health services-related occupations as published by the U. S. Bureau of Labor Statistics, 2008).

Exhibit 9. Annualized Summary Table

Respondents	Number of Respondents	Responses/Respondent	Total Responses	Hours per Response	Total Burden Hours
CLIENT DATA COLLECTION INSTRUMENTS
Clients––baseline, discharge, 6-months post baseline data collection	2400	3	6,720*	.42	2,822.4
Annualized Client Survey Total	800	--	2,240	--	940.8
Client Focus Groups (FG) Year 3 and Year 4	720**	1	720	1.0	720
Annualized Client FG Total	240	--	240	--	240
ADMINISTRATOR INTERVIEW INSTRUMENTS
Project Director/Program Manager (Semi-Structured Interviews)	96	2	192	.75	144
Annualized PD/PM Total	32	--	64	--	48
DIRECT SERVICE STAFF INSTRUMENTS
Grantee Direct Services Staff (Semi-Structured Interviews)	432	2	864	1.0	864
Annualized Service Staff Total	144	--	288	--	288
COLLABORATORS/PARTNERS INTERVIEW INSTRUMENTS
Community Collaborators (Semi-Structured Interviews)	240	2	480	1.0	480
Annualized Collaborators Total	80	--	160	--	160
Annualized Totals (3-year clearance for project)	1,296	--	2,992	--	1,676.8

*This number is derived from 2400+2400+1920 = 6,720 for 100% response rate at two data collection time points and 80% at the third data collection time point.

**These respondents are a subset of the 2,400 clients so they are not included in the total number of respondents.

13. Estimates of Annualized Cost Burden to Respondents

There are no respondent costs for capital or start-up or for operation or maintenance.

14. Estimates of Annualized Cost to the Government

The annualized cost for the project per year is $2,105,178. These costs cover all aspects of meetings and logistics; evaluation design development and finalization; instrumentation development; pilot testing; data collection; site visits; and use of information technology for online data collection, data management, analysis, and reporting. In addition, it is estimated that one full-time SAMHSA staff member will spend 25 percent of his or her time (520 hours) to manage and administer the project. Assuming an annual salary of $100,000, government personnel costs will be $25,000 each year and $125,000 over a 5-year period. Thus, total annualized project costs are $2,130,178.

15. Changes in Burden

This is a new data collection.

16. Time Schedule, Publications, and Analysis Plan

Time Schedule. Exhibit 10 outlines the key time points for the study and for the collection of information. The proposed period also allows for training and preparation activities associated with the systematic process required for data collection.

Exhibit 10. Time Schedule for Entire Project

Activity	Time Schedule
Obtaining OMB approval for data collection	December 2010
Site visit data collection (Year 3)	3 months post OMB approval for 4 months
Baseline client-level survey data collection	3 months post OMB approval for 18 months
Discharge client-level survey data collection	4–8 months after OMB approval at client discharge (discharge will vary based on scheduled treatment duration)
Six-months post baseline client-level survey data collection	10–14 months after OMB approval
Site visit data collection (Year 4)	16 months after OMB approval for 4 months
Data analysis	Beginning one year post OMB approval
Dissemination of findings Interim reports, manuscripts, final report	Beginning 18 months post OMB approval through 2013

Publications. The TCE-HIV Multi-Site Evaluation is a formative evaluation designed to monitor the implementation activities of the TCE-HIV programs, identify the extent to which the programs have achieved the goals of the MAI, and obtain information on the risk behaviors and other related information from clients of the TCE-HIV programs. It is therefore important to prepare and disseminate reports, concept papers, journal articles, and oral presentations that clearly and concisely present evaluation results so that they can be appreciated by both technical and nontechnical audiences. The TCE-HIV Multi-Site Evaluation Team will:

• Produce rapid-turnaround analysis papers, briefs, and reports

• Prepare and submit monthly progress reports and a final Multi-site Evaluation report

• Prepare a final multi-site findings report, including an executive summary

• Deliver presentations at professional and federally sponsored conventions and meetings

• Prepare and submit articles for publication in peer-reviewed journals

• Disseminate reports and materials to entities inside and outside SAMHSA.

Analysis Plan. This formative evaluation requires the use of multiple measurements, collection of information, and analytical strategies. Both qualitative and quantitative data will be collected and analyzed to assess adaptation made to EBPs, barriers and facilitators of program implementation, activities associated with service delivery for the TCE-HIV programs, and primary risk behaviors for TCE-HIV clients. The planned approach is to use state-of-the-art statistical methods to analyze the qualitative and quantitative data collected. Qualitative data analysis will utilize a thematic approach to uncover underlying themes. The three types of quantitative analyses that will be used are descriptive statistics, multivariate analysis, and multilevel analysis, when appropriate. Finally, triangulation of methods (e.g., qualitative and quantitative data), when feasible, will be used to examine additional aspects of program achievements that cannot be accomplished with individual methods. The qualitative and quantitative data analysis plan is explained in more detail in the remainder of this section.

Qualitative Data Analysis: Qualitative data for this project will come from semi-structured interviews and focus group discussions. The information will be utilized to address the first evaluation goal, which is to monitor the implementation activities of the TCE-HIV funded grantees, including their EBPs, service delivery, and contextual factors that facilitate or impede program success. To facilitate the systematic analysis of the interview data, which will be collected during Grantee staff and community partner/collaborator interviews as well as client focus groups, ATLAS.ti (Version 6.0), a computer-assisted qualitative data analysis software package, will be utilized. Before the analyses begin, TCE-HIV Multi-Site Evaluation Team staff will take part in a training workshop to improve their facility with the capabilities and updated features of the software program.

Semi-Structured Interviews: Interview data will be analyzed to identify themes extracted from material collected from Grantee administrators, direct service staff, and partners/collaborators about their respective programs and services. As a first step in the data cleaning process, audiotapes of the interviews will be transcribed and cleaned to remove any respondent identifying information and all transcription mistakes; these transcriptions will be the qualitative data used for the study.

Client Focus Groups: Focus group data will be analyzed to determine common themes expressed by clients about their substance abuse treatment history, risk behaviors (e.g., substance use/abuse risk behaviors, sexual risk behaviors), perception of social support, and satisfaction with the TCE-HIV program. As noted for the semi-structured interview data analysis plan, audiotapes of the focus groups will be transcribed and cleaned to remove any respondent identifying information and all transcription mistakes. These transcriptions will be the qualitative data used for the study. After the data is prepared for analysis, an inductive content analysis will be conducted on qualitative data.

The initial step in the analysis process will be reading the raw data (i.e., cleaned and coded interview and focus group transcripts) to discover underlying raw data themes. Raw data themes will then be grouped according to salient responses that correspond to the first goal of the evaluation:

• What are the community and contextual conditions in which the TCE-HIV programs exist and provide services?

• How do TCE-HIV Grantee characteristics facilitate or impede program activities, such as comprehensive services, service coordination/integration, and expanded organizational/ programmatic capacity?

• What adaptations do the TCE-HIV Grantees make, if any, to their EPBs?

• What are the processes used by Grantees to monitor fidelity to their EBPs?

The raw data themes will then be grouped into lower order themes based on common topics. Next, following the same coding procedures for grouping raw data themes, lower order themes will be grouped into higher order themes. Finally, higher order themes will be grouped into major categories. Consensus among TCE-HIV Multi Site Evaluation Team members conducting the analyses will be reached at each step of the analytical process (i.e., raw data themes, lower order themes, higher order themes, and major categories) before proceeding to the next step to achieve inter-coder reliability. This process ensures a consistent understanding and interpretation of the data.

Quantitative Data Analysis: TCE-HIV programs are complex with multiple levels and multiple influences such as programmatic characteristics and client characteristics. This evaluation therefore calls for multivariate and multilevel analyses to understand the complex relationships among treatment program operations, individual risk characteristics, and broader community contextual and socio-cultural environments. These analyses will address the second overarching evaluation goal (i.e., obtain information on the risk behaviors, HIV testing/serostatus, social support, medical and mental health, and motivation for treatment from clients of the TCE-HIV Grantees and monitor their satisfaction with services received).

Depending on the availability of the data and the number of waves of data available (data points) for each data element, the approach will be to apply the most appropriate statistical model in support of examining the evaluation’s second goal. To the extent possible, the analytic approach may include multivariate or multilevel analyses referred to as “cross-site” or “main effect” analyses, as they may be based on pooled data from a portion of 40 Grantees providing treatment services.

The Multi-Site Evaluation Team will use quantitative data analysis to examine whether TCE-HIV programs are achieving their goals for clients they serve. The analysis of these goals is explained in more detail below.

Descriptive Statistics: The Multi-Site Evaluation Team will use descriptive statistics to describe the main features of Grantees, their clients, and the communities in which the Grantees provide services. Descriptive statistics will allow the reporting of community, Grantee, and client characteristics that will provide a detailed picture of the TCE-HIV program and its participants. The purpose of descriptive analysis will be to understand the distribution of variables of interest. Frequencies will be run or the means and standard deviations of each variable will be calculated to examine the central tendency and distribution of variables. Knowledge of the distribution of data will inform the use of proper statistical techniques to conduct further analyses. The quantitative analysis team will also conduct cross tabulations to examine the relationship between the variables. The degree and statistical significance of association between variables is not only important for reporting relationships of interest, but also in supporting higher-level analyses.

Grantee Focused Analysis/Multivariate Analysis: Grantee characteristics and their relationship to program service administration and service delivery can be examined through multivariate analyses. The Grantee administration and service delivery variables will aid in identifying the services most likely to be affected by Grantee characteristics. Quantitative analyses of the Grantee characteristics will provide critical information about the services being provided and how these services may evolve over time. In addition to quantitative data gathered, some Grantee qualitative data will be quantified (e.g., staff training, core treatment services) and will come from the administrative staff semi-structured interviews (including the Grantee profile update), direct services staff semi-structured interviews, and partners/collaborators semi-structured interviews.

Client Focused Analysis/Multilevel Analysis: Client-focused quantitative analyses will be primarily model-based and will be performed for both individual Grantee client populations and the total client population pooled across Grantees, as appropriate. Within individual Grantee client populations, models will be created to predict influences on client substance abuse and sexual risk behaviors while controlling for different client characteristics (e.g., age, sex, race, ethnicity, substance abuse treatment history, and HIV status). Across the total client population, hierarchical linear modeling (HLM) will be used to account for the nested characteristics of the data. In any analysis where the population is grouped into larger units (here the Grantees), the performance of the individuals within the same group will be correlated. These correlations must be accounted for in the analysis to draw the correct inferences. For example, HLM models will be able to test the proposition that clients’ risk behaviors would vary while in treatment depending on the levels of social support they receive.

17. Display of Expiration Date

OMB approval expiration dates will be displayed on the client surveys, participant data sheets, and consent forms.

18. Exceptions to Certification for Statement

There are no exceptions to the certification statement. The certifications are included in this submission.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

This section presents information about the collection of data for the Multi-Site Evaluation. As noted previously, the Multi-Site Evaluation has two major goals and related objectives. Specifically:

Goal #1: To monitor the implementation activities of the TCE-HIV funded grantees including their EBPs, service delivery, and contextual factors that facilitate or impede program success.

Objective #1: Conduct semi-structured interviews with each of the grantees’ administrative staff within 3 months of receiving OMB clearance and in the fourth year of program implementation

Objective #2: Conduct semi-structured interviews with the direct service staff who are engaged in service delivery with clients within 3 months of receiving OMB clearance and in the fourth year of program implementation

Objective #3: Conduct semi-structured interviews with no more than four grantee partner agencies of each grantee that are engaged in service delivery to clients within 3 months of receiving OMB clearance and within the fourth year of program implementation

Goal #2: Obtain information on the risk behaviors, HIV testing/serostatus, social support, medical and mental health, and motivation for treatment from clients of the TCE-HIV grantees and monitor their satisfaction with services received.

Objective #1: Administer a client questionnaire to each client at the same three points (i.e., intake, discharge, 6-months post baseline) that grantees are currently required to submit data to SAMHSA and explore if grantees are achieving program results

Objective #2: Conduct one focus group with no more than 9 clients from each of the 40 treatment grantees to obtain contextual information on client substance abuse treatment experience and satisfaction within 3 months of receiving OMB clearance and in the fourth year of program implementation

A comprehensive multi-method data collection effort is proposed to better understand the 48 Grantees, who they are serving, what programs they are implementing, and what services are being delivered and by whom. A multi-method data collection approach will be used to learn more about the results of service delivery for the clients they serve and how grantees achieve their goals.

1. Respondent Universe and Sampling Methods

Attachment 4 - Document 2 presents the subset and estimated universe of respondents for evaluation data collection activities. Specifically, the semi-structured interviews will consist of a subset of program staff (e.g., program administrator, treatment counselor) and a subset of partners/collaborators. Client focus groups will consist of a representative subset of clients that are targeted by each of the 40 treatment programs.

Data collection activities using the client survey will occur with the estimated universe of respondents (i.e., individuals entering substance abuse treatment programs) across the 40 substance abuse treatment programs, which is estimated to be 2,400 at baseline, 2,400 at discharge, and 1,920 at 6-months post baseline. It is expected that recruitment methods and intensive tracking mechanisms will result in response rates of 100 percent at baseline and discharge, and 80 percent at 6 months post baseline for the client survey administration. The following sections present a description of data collection methods.

Administrative Staff Semi-Structured Interviews. Individuals from each of the 48 Grantee sites who perform administrative tasks related to the TCE-HIV program (e.g., project director, program manager, and executive director) are eligible to be interviewed and their input will help document community and contextual factors. It is estimated that two administrative staff members per Grantee site will be interviewed in Year 3 and Year 4 of the Multi-Site Evaluation. (See Attachment 3 - Document 1).

It is estimated that two administrative staff members per Grantee site will be interviewed in a two-part interview session. The first part of the interview will be conducted with an executive staff member of the agency; and the second part with the project director, program manager, or program coordinator of the TCE-HIV project. The determination was made to conduct a two-part interview with these two categories of staff because sometimes these staff members have different perspectives on some of the key issues to be explored. In some instances, one person from the agency may fulfill both roles/positions. In this situation, the full interview will be conducted with that person.

Direct Services Staff Semi-Structured Interviews. Individuals from the 48 Grantee organizations who have direct contact with clients to perform outreach/pretreatment and/or treatment-related services will be eligible to be interviewed. Examples of people performing direct services include outreach workers, counselors, and case managers. It is estimated that up to nine direct service staff members per Grantee site will participate in the interview session conducted during site visits in Year 3 and Year 4 (See Attachment 5 - Document 1).

Community Partners/Collaborators Semi-Structured Interviews. Grantee community partners/collaborators are those agencies or organizations that provide services and activities related to the TCE-HIV program. It is estimated that up to five community partners/collaborators per Grantee may participate in the interview sessions conducted during site visits in Year 3 and Year 4. Grantees may have more than five community partners, thus selection criteria has been developed to prioritize partners recruited to be interviewed. Selection criteria include whether the partner/collaborator provides the following services: treatment, housing, employment, re-entry criminal justice services, and medical services. These priorities may shift based on the nature of the Grantee program (e.g., for programs targeting HIV positive clients, interviews with partners providing medical care would be a main priority). (See Attachment 7 - Document 1).

Client Focus Groups. The targeted universe for the client focus groups are clients who have been in substance abuse treatment for at least 14 days. The Multi-Site Evaluation Team will work collaboratively with Grantee staff to identify client focus group participants. A member of the Grantee staff assigned to help with focus group recruitment will be asked for assistance in the distribution of flyers advertising the focus group, announcing the focus group to clients when appropriate, and serving as a primary point of contact regarding the client focus group. See Attachment 2b - Document 2 for an example of a focus group recruitment flyer and Attachment 2b – Document 3 for the client focus group sign-up roster. The focus groups will strive for a balance in participants with regard to gender, age, length and number of times in treatment, and serostatus. It is estimated that up to 9 clients from each of the 40 treatment Grantees will participate in focus groups conducted during site visits in Year 3 and Year 4. (See Attachment 1a - Document 1.)

Client Survey. The targeted universe for the Multi-Site Evaluation client surveys are clients from the 40 treatment Grantee programs. Clients who enter treatment following OMB clearance will be eligible to be surveyed. The first 60 clients from each of the 40 treatment Grantees who are administered the GPRA Tool will be approached to be surveyed. The client surveys will be administered by trained Grantee program counselors when clients enter the treatment program. All of those clients receiving the baseline survey will be asked to complete a follow-up survey at discharge, and at 6 months post baseline. It is expected that the use of tracking mechanisms will result in an 80 percent response rate at 6 months post baseline. (See Attachment 1a - Documents 2 and 3, Attachment 1b - Document 1).

2. Information Collection Procedures

To conduct the Multi-Site Evaluation, data will be collected using different methods for each of the two data collection points (i.e., during Year 3 and Year 4 site visits), as well as the ongoing client survey data collection. Each data collection method proposed supports the evaluation goals and objectives. The data collection process will be a careful and systematic mixed-method data collection approach, in order to gather high quality data from each of the 48 TCE-HIV Grantees. Attachment 4 - Document 2 presents the proposed estimated universe and selection methods for the project. Semi-structured interviews, focus groups, and client surveys will be used to collect data from the target population (e.g., Grantee staff, partners/collaborators, and clients). A general description of key data collection procedures is provided below. A description of the site visit and data collection instruments is also provided.

Site Visits. To plan the Year 3 site visits, a Multi-Site Evaluation team member will call the Grantee Project Director. See Attachment 4 - Document 3 for the site visit call script. The initial phone contact will be followed by an e-mail confirmation letter that provides additional details about the site visit. See Attachment 4-Document 4 for a sample site visit confirmation letter.

The Multi-Site Evaluation Team will conduct site visits at two data collection time points with each of the 48 Grantees. For the Year 3 site visits, a Multi-Site Evaluation team member will contact the Grantees to schedule the site visit. The Year 4 visit will be conducted approximately 16 months following Year 3 site visit. Each site visit will be conducted by experienced members of the Multi-Site Evaluation Team. The site visits are anticipated to last 2 days, in order to gather all required data. The purpose of the site visits are to observe program processes, to conduct semi-structured interviews with project staff and community partners/collaborators, to conduct client focus groups, and to provide technical assistance to Grantees on client survey administration.

Semi-Structured Interviews. During each site visit, semi-structured interviews will be conducted with the following respondents: 1) program administrator, 2) Grantee direct services staff, and 3) community partners/collaborators. The semi-structured interview process will be led by a senior Multi-Site Evaluation Team member. The interviews may take place with a single respondent or a group of respondents (e.g., more than one treatment staff member may be interviewed in the same session). The lead interviewer will describe the purpose of the interview and have respondent(s) review and sign the informed consent form (Attachment 4 - Document 5, Attachments 6 and 8). The lead interviewer will use the appropriate interview guide (e.g., administrator, direct services staff, partners/collaborators) for conducting the interviews (see Attachment 3 - Document 1, Attachment 5 - Document 1, Attachment 7 - Document 1). The lead interviewer will guide the discussion and respondents will complete a participant demographic information sheet following the discussion (Attachment 3 - Document 2, Attachment 5 – Document 2, Attachment 7 - Document 2). A note taker will record in detail the respondents’ statements. To provide additional documentation, and for quality assurance, interviews will also be digitally recorded with the permission of the interviewees. At the conclusion of the interview session, the lead interviewer will summarize the discussion. Debriefing sessions will be conducted with Grantee personnel at the conclusion of the 2-day site visit to summarize the visit and address any questions posed by Grantee staff about the visit. Follow-up telephone calls to Grantee staff may be conducted when necessary to further clarify information obtained during the visit.

Focus Groups. Clients from the 40 treatment Grantee sites will be recruited to participate in the in-person focus groups. A small group of clients (i.e., up to nine clients per focus group) will be invited to the focus groups discussion and guided by a moderator to address specific questions and discuss their experience in treatment. The Multi-Site Evaluation staff will work collaboratively with Grantee staff to identify and recruit client participants based on criteria including age, gender, length and number of times in treatment, and HIV serostatus.

The client focus groups will be led by a moderator who is a senior Multi-Site Evaluation Team member. An additional evaluation staff person will assist by attending to recording and note taking. The moderator will describe the purpose of the focus group, identify SAMHSA as the sponsoring agency, explain all focus group procedures, ensure privacy to the client participants, and request participation. In an effort to ensure that clients understand what is being asked of them, Grantee staff will read aloud the consent form to clients (see Attachment 2b - Document 4). After clients have agreed to participate, they will be asked to sign the consent form and will be given a copy for their records.

A scheduled time and place for the focus group will be established prior to the site visit. Focus groups will be scheduled at a time when a licensed clinician is on duty, in the event that disclosure of sensitive information by focus group participants causes them discomfort. After respondents agree to participate, the focus group will begin. The moderator will use a written guide (see Attachment 1a - Document 1) to conduct the focus group discussion. With the permission of the respondents, all focus groups will be audiotaped for later transcription and analysis. Following the discussion, the moderator will read aloud the client participant demographic sheet and will assist clients as they complete the sheet (see Attachment 1b - Document 2). Finally, the moderator will summarize the activities and comments at the end of the discussion. Incentives will be distributed after the focus group is completed.

Client Surveys. A client survey will be completed as part of the project evaluation activities. Upon OMB approval, the Multi-Site Evaluation Team will train Grantee staff via Web/teleconference how to administer client surveys. The team will develop a comprehensive question-by-question training session for Grantee staff with information about how to complete the survey along with the context for the collection of these data. The training will be developed as a PowerPoint presentation and will provide a step-by-step process for administration of the survey (see Attachment 2c.).

As described in Section A.6, the client survey will be used to collect data from clients at baseline, discharge, and 6 months post baseline. Data collection at discharge and 6 months post baseline is necessary to monitor short-term and long-term change in client substance abuse and HIV risk behaviors, social support, and quality of life indicators.

Grantee staff at each of the 40 Grantee treatment sites will administer the client survey to 60 clients admitted into treatment post OMB approval. The client survey will be administered in a private location (e.g., an office) to provide privacy. It is expected that the client survey will be administered to 2,400 clients at baseline, 2,400 clients at discharge, and 1,920 at 6 months post baseline.

Client survey data will be entered into a password protected online system at the Grantee sites. Each day, the TCE-HIV evaluation staff will upload the data over a secure network connection directly to a server at the contractor’s headquarters where the data will also be encrypted and password protected. Details on the network security procedures and security protocols are presented in Attachment 2a – Document 1. Using protected electronic data is the most secure form of data management because it eliminates the possibility of paper documents being lost by the survey staff or of data being lost in transit or delivered to an incorrect location. However, not all the Grantees may be equipped to enter this data into the online system, in which case paper copies of the completed surveys from these Grantees will be submitted and these data entered at JBS by the Multi-Site Evaluation staff. No name/identifying information will be collected at any time and any paper copies will be stored in a locked file cabinet.

3. Methods to Maximize Response Rates

The ability to gain the cooperation of potential respondents is important to the success of this Multi-Site Evaluation. A discussion of methods to maximize response rates for the different data collection methods is presented below.

Semi-Structured Interviews. The TCE-HIV Multi-Site Evaluation Team anticipates that the semi-structured interviews will be completed for 100 percent of the Grantee staff and community partners/collaborators who are asked to participate. To maximize participation rates, the TCE-HIV Evaluation Team interview staff will follow protocols that will reduce the burden on Grantee staff and community partners. Planning and preparation in advance of the interview is crucial to these protocols, which will include proper timing, scheduling, and location of the interviews to accommodate the Grantee staff and their partners/collaborators. The Grantee staff and their community partners/collaborators will be informed, in advance, of the purpose and significance of the interview.

Focus Groups. The TCE-HIV Multi-Site Evaluation Team anticipates over recruiting (n=12 clients) for each focus group, but it is estimated that no more than 75 percent (n=9 clients) will participate in each focus group. Clients will be recruited based on specific criteria such as clients who have been administered the GPRA, have been in substance abuse treatment for at least 14 days, and are willing to participate in the focus group.

The TCE-HIV Multi-Site Evaluation Team will work collaboratively with Grantee staff to recruit a representative group of clients for the focus group utilizing inclusion and exclusion criteria, to accomplish a balance of gender, age, length and number of times in treatment, and HIV serostatus across focus groups. Each focus group will consist of seven to nine treatment clients who have been administered the GPRA and who are receiving TCE-HIV services. The TCE-HIV Multi-Site Evaluation Team will then work with project staff to provide them with information about the focus groups they can post and/or distribute to potential participants. As an added strategy to maximize response rates, gift cards ($20) will be given to participating clients as compensation for their time, and the cards will be distributed after the focus group is completed.

Client Survey. The TCE-HIV Multi-Site Evaluation Team anticipates a 100 percent response rate for the baseline and discharge client surveys, and an 80 percent response rate for the 6 months post baseline survey. After OMB approval, trained Grantee staff will administer the survey to the first 60 clients admitted into treatment who have been administered the GPRA Tool and have an assigned GPRA identification number. The client survey will be administered to treatment clients (not outreach/pretreatment clients) in order to maximize response rates at each of the three data collection points and to reduce burden on Grantee staff members responsible for collecting the data. As an additional measure, cash equivalent compensation will be offered in lieu of cash payments. Gift cards from Walmart™ or Target™ will be used as compensation for treatment clients to complete the client survey at baseline, discharge, and 6 months post baseline. Compensation for their time will be provided to clients in a tiered structure with a $10 gift card provided at baseline, $15 gift card at discharge, and $20 gift card at 6 months post baseline. In addition, the evaluation team and Grantee staff will employ several other strategies to maintain high response rates at discharge and 6 months post baseline:

• Stress the importance of the project as well as the evaluation team’s commitment to respondent privacy

• Train survey staff for handling sensitive information collection in a respectful manner

• Develop surveys in English and Spanish

• Ensure that Grantees have comprehensive tracking forms and procedures in place

To improve follow-up response rates, the Grantee staff will collect detailed contact information, including alternate addresses and phone numbers and contact information of secondary sources who may know the respondents’ contact information at follow-up.

4. Tests of Procedures

Pilot tests of semi-structured interviews, client focus groups, and client surveys, and data collection procedures to be used in the formative evaluation were conducted with representative subsamples of the target populations. All pilot tests were conducted with nine or fewer individuals. Attachment 4 - Document 1 provides a summary of pilot test feedback for each of the measurement instruments tested and outlines the changes that were made to the data collection instruments based on this feedback.

Semi-Structured Interviews. The pilot study participants for the semi-structured interview guides and procedures consisted of representatives from prior cohorts of the TCE-HIV initiative. The decision was made not to pilot test the semi-structured interviews with any Grantee staff and partners/collaborators from the current cohort as this may have primed them for actual data collection. Grantee staff and partners/collaborators who were interviewed for the pilot test were asked to respond to the interview questions and to provide feedback on the clarity of the questions and guidelines. They were also asked to comment on the appropriateness of the questions and procedures for the intended audience.

Client Focus Groups. Pilot tests of the focus group guides were conducted with substance abuse treatment clients from a prior TCE-HIV grantee. The pilot study participants took part in a focus group discussion and after participating in the focus group, participants were asked to comment on the clarity of the questions and identify problems or concerns with the questions. Participants were also asked to provide feedback on the appropriateness of the questions.

Grantee staff from a prior TCE-HIV cohort assisted with the recruitment of focus group participants for the pilot test. Case managers and counselors shared with their clients that participation in the focus group would give them an opportunity to talk about their experiences in the TCE-HIV program. Interestingly, during the focus group discussion, one client stated that participation in the discussion was an opportunity to “give back” (i.e., an opportunity to share experiences that might help others). Grantee staff provided recommendations for how to recruit focus group participants for the Multi-Site Evaluation including placing flyers and sign-up sheets on walls or having case managers recommend clients for participation in the focus group.

Client Survey. Program staff members from the current cohort of TCE-HIV Grantees were trained in how to administer the client survey. Pilot tests of the client survey were conducted with a subsample of nine clients from the target population. Specifically, trained Grantee staff administered the client-level survey to three clients at baseline, three clients at discharge, and three clients at 6 months post baseline. Grantee staff members were then asked to comment on the clarity of the questions and identify problems or concerns with the questions and format of the surveys. Grantee staff provided feedback on the appropriateness of the questions for the intended audience. Finally, Grantee staff also provided their clients’ feedback about the client-level survey.

5. Statistical Consultants

As noted in Section A.8, the TCE-HIV Multi-Site Evaluation Team has consulted with an expert panel and a project working group that have reviewed all data collection and analysis methodologies outlined in this package. They will also continue to provide expert advice throughout the course of the project. In addition, several in-house experts will be consulted throughout the 5-year project on various statistical aspects of the design, methodological issues, implementation issues, database management, and data analysis. Exhibit 11 provides information about the in-house advisors.

Exhibit 11. Senior Advisors

Jim Derzon, PhD	Battelle Memorial Institute TCE-HIV Multi-Site Evaluation Task Lead for Quantitative Analysis 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-248-1640 Fax: 703-527-5640 E-mail: [email protected]
Susan Hayashi, PhD	JBS International, Inc. Vice President 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 301-495-1080 ext. 4588 Fax: 301-587-4352 E-mail: [email protected]
Kevin Hylton, PhD	Alliances for Quality Education, Inc. TCE-HIV Multi-Site Evaluation Deputy Project Director TCE-HIV Evaluation Project 8181 Professional Place, Suite 110 Landover, Maryland 20785	Phone: 301-583-8424 Fax: 301-583-8422 E-mail: [email protected]
Jennifer Kasten, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Lead 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4145 Fax: 301-587-4352 E-mail: [email protected]
Resa Matthew, PhD	JBS International, Inc. TCE-HIV Multi-Site Evaluation Project Director 5515 Security Lane, Suite 800 North Bethesda, MD 20852-5007	Phone: 240-645-4608 Fax: 301-587-4352 E-mail: [email protected]
Ping Yu, PhD	Battelle Memorial Institute TCE-HIV Multi-Site Evaluation Subcontract Director/Co-PI 2101 Wilson Boulevard Suite 800 Arlington, VA 22201-3008	Phone: 703-875-2981 Fax: 703-527-5640 E-mail: [email protected]

REFERENCES

Centers for Disease Control and Prevention. (1999). Framework for Program Evaluation in Public Health. MMWR 1999; 48 (No. RR-11).

Centers for Disease Control and Prevention. (2006). Racial/ethnic disparities in diagnoses of HIV/AIDS--33 states, 2001–2004. Morbidity and Mortality Weekly Report, 55(5), 121–125.

Croog, S. H., Levine, S., Testa, M. A., & Brown, B. (1986). The effect of antihypertensive therapy on the quality of life. The New England Journal of Medicine, 314(26), 1657–1664.

Derogatis, L. R. (1993). SCL-90-R: Symptom checklist-90-R: administration, scoring & procedures manual (3rd ed.). Minneapolis, MN: National Computer Systems, Inc.

Derogatis, L. R. & Coons, H. L. (1993). Self-report measures of stress, In L. Goldberger, S. Breznitz (Eds.), Handbook of Stress: Theoretical and Clinical Aspects (2nd ed.). New York: Simon & Shuster.

Festinger, D. S., Marlowe, D. B., Croft, J. S., Dugosh, K., Mastro, N., Lee, P., DeMatteo, D. S.,

& Patapis, N. S. (2005). Do research payments precipitate drug use or coerce participation? Drug and Alcohol Dependence. 78, 275–281.

Festinger, D. S., Marlowe D. B., Dugosh K. L., Croft J. R., & Arabia P. L. (2008). Higher

magnitude cash payments improve research follow-up rates without increasing drug use or perceived coercion. Drug & Alcohol Dependence, 96, 128–135.

Festinger, D. S., Marlowe, D. B., Croft, J. R., Dugosh, K. L., Arabia, P. L., & Benasutti, K. M. (2009). Monetary incentives improve recall of research consent information: It pays to remember. Experimental and Clinical Psychopharmacology, 17, 99–104.

Miller, W. R. & Tonigan, J. S. (1997). Assessing the drinkers’ motivation for change: the Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES), In A. Marlatt, G. VandenBos (Eds.), Addictive behaviors: Readings on etiology, prevention, and treatment. Washington, DC: American Psychological Association.

Minority AIDs Initiative. (1998). U.S. Congress, Omnibus Consolidated and Emergency Supplemental Appropriations Act, 1999 (PL 105-277), October 21, 1998. Retrieved on February 12, 2010 at http://minorityhealth.hhs.gov/templates/browse.aspx?lvl=2&lvlID=36.

Texas Christian University. (2005). Client Evaluation of Self and Treatment, Intake Version (TCU CEST-Intake). Retrieved February 12, 2010 from http://www.ibr.tcu.edu/pubs/datacoll/commtrt.html#Form-CEST.

U.S. Bureau of Labor Statistics (2008). Retrieved on February 12, 2010 at http://www.bls.gov/oes/2008/may/oes_nat.htm#b21-0000).

U.S. Department of Health and Human Services, Office of Disease Prevention and Health

Promotion. (2009). Healthy People 2020: The Road Ahead. Retrieved on February 9, 2010, at http://www.healthypeople.gov/HP2020/objectives/ViewObjective.aspx?Id=595&TopicArea=Substance+Abuse&Objective=SA+HP2020%e2%80%936&TopicAreaId=46.U.S. House of Representatives Committee on Appropriations. (October 9, 2001). House Report 107-229, Departments of Labor, Health and Human Services and Education and Related Agencies Appropriations Bill.

LIST OF ATTACHMENTS

Attachment 1a – Client Instruments

• Document 1 - Client Focus Group Discussion Guide

• Document 2 - Intake/Baseline Client-Level Survey

• Document 3 - Discharge Client-Level Survey

Attachment 1b – Client Instruments Continued

• Document 1 – Six Month Follow-Up Client-Level Survey

• Document 2 - Client Data Sheet

Attachment 2a – Client Other

• Document 1 - JBS International Network Security

• Document 2 - Project Working Group Feedback Results

• Document 3 - 45 CFR 46

Attachment 2b – Client Other Continued

• Document 1 - The Privacy Act of 1974

• Document 2 - Client Focus Group Recruitment Flyer

• Document 3 - Client Focus Group Sign Up Roster

• Document 4 - Client Consent Form

• Document 5 – Institutional Review Board (IRB) Approval Letter

• Document 6 - Certificate of Confidentiality Approval Letter

Attachment 2c – Client Other Continued

• Document 1 - Client-Level Survey Grantee Training Presentation

Attachment 3 - Administrative Staff Instruments

• Document 1 - Administrative Staff Semi-Structured Interview Guide

• Document 2 - Administrative Staff Data Sheet

Attachment 4 – Administrative Staff Other

• Document 1 - Pilot Test Results

• Document 2 - Proposed Sample and Methods by Stage

• Document 3 - Site Visit Call Script

• Document 4 - Site Visit Confirmation Letter

• Document 5 - Administrative Staff Consent Form

Attachment 5 – Direct Services Staff Instruments

• Document 1 - Direct Services Staff Semi-Structured Interview Guide

• Document 2 - Direct Services Staff Data Sheet

Attachment 6 – Direct Services Other

• Document 1 - Direct Services Staff Consent Form

Attachment 7 – Partner/Collaborator Instruments

• Document 1 - Partner/Collaborator Semi-Structured Interview Guide

• Document 2 - Partner/Collaborator Data Sheet

Attachment 8 – Partner/Collaborator Other

• Document 1 - Partner/Collaborator Consent Form