Attachment 4 - Other

Attachment 4: Document 1 - Pilot Test Results

Pilot Test Results
The instruments included in this OMB package have been revised and incorporate the pilot test
results.

CLIENT LEVEL SURVEY (CLS)

Average time to complete CLS = 25 minutes
Feedback
Client does not read well and could not read
response cards.

Changes Made to Instrument
Instructions were added for the administrator of the
instrument to read questions aloud for those who
request it.

DOSAGE FORM

Average Time to Complete Dosage Form = 16 minutes
Feedback
Form was easily completed. No problems were
encountered.

Changes Made to Instrument
No proposed changes to the dosage form based on
pilot testing.

Form is simple to complete.

CLIENT FOCUS GROUP INTERVIEW GUIDE

Average time to complete consent process = 5 minutes
Average time to complete the focus group discussion = 60 minutes
Average time to complete participant data sheet (following discussion) = 10 minutes
Feedback
Confidentiality: need to make it clear their
comments won’t get back to the program staff (or
CSAT).

Changes Made to Instrument
Consent form revised to clearly state that only deidentified or aggregate data will be disclosed to those
other than the research staff.

Clients did not clearly understand the concept of
community when we asked about “drug use in the
community.”
It would have been helpful to discuss some
medical issues.

Revised/added the term neighborhood when
necessary.
No changes/addition related to this feedback were
made. Because medical issues are not the information
being sought in this survey, these questions were not
added to the instrument.

1

Attachment 4: Document 1 - Pilot Test Results

ADMINISTRATOR INTERVIEW GUIDE

Average time to complete consent process = 5 minutes
Average time to compete interview = 90 minutes (in both pilots two administrators took part in at
least a portion of the interview (e.g., Executive Director and Program Coordinator)
Feedback
Confidentiality: Make it clear it’s aggregated
presentation of interview data and that “no
identifiers would get back to CSAT.”
Section III: Program Description, add a
Question: ask if the PDs felt the program
was doing what it had originally been
intended to do, and if they could change or
tweak it, what they would do.
Need to re-word questions to capture
variability of programa. What does
participation look like in your program
recognizing the variability for each client?
Missing future plans…or what’s on drawing
board.
Advisory board question could be reframed
to ask about an oversight group. “In this
project, how does this group function and
what do they do for you?”
Nothing related to the frequency of
contact/meeting with collaborators/partners.

Changes Made to Instrument
Consent form revised to clearly state that only de-identified
or aggregate data will be disclosed to those other than the
research staff.
The proposed question was not added to the instrument.

The question asked as written in the guide provides for a
client variability discussion, therefore no revisions were
made.
This can be viewed as a sustainability question, and it is
addressed in Section V: Probe 1.
No changes were made to the guide regarding the
reframing of the term “advisory board.” The term
“Advisory Board” is in the language of the RFP for the
grant program.
We added a question that captures this in the
administrative guide and in the partner guide.

DIRECT STAFF INTERVIEW GUIDE

Average time to complete consent process = 10 minutes
Average time to complete interview = 60 minutes
Feedback
Confidentiality: need to make it clear the
information won’t get back to PD (or to
CSAT).

Changes Made to Instrument
Consent form revised to clearly state that only de-identified
or aggregate data will be disclosed to those other than the
research staff.

Section IV Program Description – it seemed
like you just wanted to ask “how do you
know what you’re doing.” Ask about
training opportunities rather than specific
split of EBPs and training. Ask about how
you implement/practice what you learn.

No changes were made to the guide based on this feedback.
The questions as asked address the concern.

It would be nice to ask about what changes

No questions were added to the guide based on this

2

Attachment 4: Document 1 - Pilot Test Results

PARTNERS INTERVIEW GUIDE

Average time to complete consent process = 10 minutes
Average time to complete interview = 90 minutes

Feedback
Changes to Instrument
Participants thought the questions were
No changes made based on participant feedback.
appropriate for the purpose of the interview.
we have seen in our clients.
feedback. The client survey (completed at baseline,
discharge, and 6-month post-baseline) and client focus
group discussion guide capture this information. Inclusion
of additional items to the guide would result in increased
burden.

3

Attachment 4: Document 2 - Proposed Sample and Methods by Stage

Proposed Sample and Methods
By Stage

Stage

Target Audience(s)

Administrative Staff
Process
Evaluation
Grantee Staff*

Estimated Sample Size Per Methodology

Semi-Structured Interviews: 768
Administrative Staff: 96
Grantee Direct Services Staff: 432
Community Partners/Collaborators: 240

Selection Criteria by Data Collection Methods

Total sample Semi-Structured Interviews:
size for process
evaluation: 6,288 Administrative Staff: Individuals from each of the 48 Grantee sites
who perform administrative tasks related to the TCE-HIV program
(e.g., Project Director, Program Manager, and Executive Director)
are eligible to be interviewed.

Community
Partners/Collaborators*

Clients

Total Estimated
Sample Size

Focus Groups: 720
(9 respondents from each of the 40 grantees during
Year 2 and Year 4)

Grantee Staff: Individuals from the 48 Grantee organizations who
have direct contact with clients to perform pretreatment/outreach,
and/or treatment-related tasks will be eligible to be interviewed
(e.g., outreach workers, counselors, and case managers).

Dosage Form: 4,800
(120 clients from each of the 40 treatment grantees)

Community Partners/Collaborators: Grantee community
partners/collaborators are those agencies or organizations that
provide services and activities related to the TCE-HIV program.
Focus Groups:
In-person Focus Groups (Year 2 and Year 4): The targeted
audience for the client focus groups are a selection of the first 120
clients from the 40 treatment Grantee programs who meet specific
criteria (i.e., have been administered the GPRA, and have been in
substance abuse treatment for at least 14 days and are willing to
participate in the focus group).
Dosage Form:
The targeted audience for the TCE-HIV Multi-Site Evaluation
client dosage forms are the first 120 clients from the 40 treatment
Grantee programs who have been administered the GPRA and
client survey.

1

Attachment 4: Document 2 - Proposed Sample and Methods by Stage

*Proposed Sample and Methods
By Stage

Stage

Outcome
Evaluation

Target Audience(s)

Clients

Estimated Sample Size Per Methodology

Total Estimated
Sample Size

Selection Criteria by Data Collection Methods

Total sample size Surveys:
for outcome
evaluation: 4,800 The targeted universe for the TCE-HIV Multi-Site Evaluation client
surveys are clients from the 40 treatment Grantee programs. The
first 120 clients at each 40 treatment Grantees who are administered
the GPRA will be surveyed.

Surveys:
Client Surveys: 4,800
(120 respondents from each of the 40 treatment
grantees)
Client Survey (Baseline)
Client Survey (Discharge)
Client Survey (6-month post baselin )

2

Attachment 4: Document 3 - Site Visit Call Script

Template of Site Visit Call Script
All correspondence, including phone calls and e-mails, between TCE-HIV evaluation team members and
the project sites must be recorded on the communication log. The following is the suggested script for
calling sites to recruit them for visits.
Good (morning/afternoon/ evening), my name is ________________ and I am calling on behalf
of JBS International and the Center for Substance Abuse Treatment TCE-HIV multi-site
evaluation.
May I please speak with Project Director Name

?

If Project Director is unavailable:
Do you know a better time to reach him/her?
If a time is suggested record it in the communication log.
May I please leave a message/voicemail?
Message left with another person:
This is ________________ calling on behalf of JBS and the Center for Substance Abuse
Treatment TCE-HIV multi-site evaluation. I am calling to speak with Project Director’s name to
schedule a visit to Agency Name. If you would please have him/her contact me at phone number,
I would appreciate it.
Request the name and title of the person with whom you leave the message. Record his or her
name and title in the communication log.
Voicemail:
Hello this is__________ calling on behalf of JBS International, the independent evaluator
for the Center for Substance Abuse Treatment TCE-HIV multi-site evaluation. At your
earliest convenience, if you would please return my call at _______________to schedule
a visit to your program, I would appreciate it. Again this is _____________ calling on
behalf of JBS International and my number is _________________. Thank you and I
look forward to speaking with you soon.
Record date and time of voicemail message in the communication log.
If Project Director is available:
(Dr. /Ms. / Mr.)__________________, I am calling on behalf of JBS International to schedule a
visit to your TCE-HIV program. JBS is an independent evaluator for the TCE-HIV multi-site

1

Attachment 4: Document 3 - Site Visit Call Script

evaluation conducted on behalf of the Center for Substance Abuse Treatment. Last year, JBS
conducted a preliminary visit to your program, and we are planning a second site visit to your
agency this year.
Sometime between __month__ and __month__, three members of the TCE-HIV evaluation team
would like to visit your site to meet with you and your staff. The primary purpose of the visit is to
learn about any changes in your program implementation over the past year, and for you to
provide us with updated information regarding your TCE-HIV program goals and activities.
The visits will take 1–2 days and we would like to speak with you and as many key staff
members and collaborators as possible through a series of one-on-one or group interviews. For
instance, we’d like to meet with the executive director, the program manager, your TCE-HIV
direct services program staff (including outreach workers, treatment counselors, case managers),
and representatives from your TCE-HIV program’s key partners and collaborators. We are also
planning to conduct a focus group with a small group of your current TCE-HIV program clients.
Currently, we have the following dates available to visit your program: ____________.
Would the staff members and partners I mentioned be available for a 60–90 minute meeting on
one of the days of our site visit?
We would like to confirm these dates as soon as possible in order to accommodate all of the
grantees’ availability.
Are you able to confirm one of these dates now?
If able to confirm date:
Which date would you like?
Great, we will schedule a team to come to your site on ______________at_______(e.g.,
Wednesday, February 10, at 8 a.m.)
Record the details regarding the date and time of the visit on the communication log.
Is there a hotel in the area that you would recommend?
If Yes: Record the hotel name and address on the communication log.
I will send you an e-mail in the next day or two with the confirmed date and contact information
of the persons who will be visiting you. My e-mail will also include a draft agenda and additional
details on how best to prepare for the meeting. At your convenience, please review the agenda
and e-mail any suggested changes to me. I will also send you an e-mail about client recruitment
for the focus group we will be conducting during our visit.

2

Attachment 4: Document 3 - Site Visit Call Script

After getting your feedback about the site visit agenda and scheduling, I will send a final agenda
and a copy of the grantee profile that we updated during last year’s visit to your program. If
possible, please review the grantee profile prior to our visit because we will spend some time
during our visit updating the document to reflect any changes that have occurred over the past
year.
If you have any additional questions or if you need to change the dates for the visit, please contact
me at any time by phone or e-mail. My phone number is ________________and my e-mail
address is ____________.
Also, I would like to confirm that the number where I reached you is the best contact number. I
dialed_____________. Is this the best way to reach you? I also wanted to confirm that I have the
proper e-mail address for you: ____________________. Is that correct?
We appreciate the opportunity to visit your site to learn more about your program. Thank you
very much for talking with me today. I appreciate it.
If unable to confirm date:
We would like to confirm a time slot and accommodate all of our grantees’ schedules, so please
let us know your availability as soon as possible. If for some reason none of these dates are
suitable, we will work with you and your staff to come up with an alternate date that is more
convenient.
How much time do you need to provide me with your availability? Okay, I will follow up with
(hours/ days/ weeks) to confirm a date and time for the visit. I just wanted to
you in #
remind you that we would also like to meet with your TCE-HIV program staff members, the
executive director, and representatives from your key partner organizations. So please confirm
their availability to meet with us on at least one of the two days of our visit, as well. In the
meantime, if you need any additional information, or if you would like to confirm a visit date
sooner, you can reach me by phone or e-mail.
My phone number is ________________and my e-mail address is ____________.

Also, I would like to verify that the number where we reached you is the best contact number. I
dialed_____________. Is that the best number to reach you? I also wanted to confirm that we
have the proper e-mail address for you ____________________. Is that correct?
Okay, thank you very much for your time. I look forward to talking with you soon.

3

Attachment 4: Document 4 - Site Visit Confirmation Letter

TCE-HIV Multi-Site Evaluation
Template of Confirmation Letter
Address of site (example below):
La Clinica del Pueblo
2831 15th Street, NW
Washington, DC 20009

Date, 2010
Dear_________________: (e.g., name of Project Director, such as Dear Dr. Jones)
This letter is to confirm that we will be visiting your site as part of the TCE-HIV Multi-Site
Evaluation Project on Dates, 2010. (e.g., March 11–12, 2010)
The visit will be conducted by members of the TCE-HIV evaluation team, the Center for
Substance Abuse Treatment’s (CSAT’s) contractor for the TCE-HIV evaluation project. An
evaluation team member will work with you prior to the visit to ensure that time spent on site is
efficient and well organized.
The visit will include interviews with the administrative staff, direct-service staff, and
partners/collaborators of your TCE-HIV program. The interviews conducted with key staff at
your agency will gather information about successes and challenges of implementing GRANTEE
NAME’s TCE-HIV program. The goals of the interviews are to better understand how your
TCE-HIV program has developed; how it assesses client outcomes; and how agency and
community partnerships, linkages, and capacity have developed through the course of your
project’s operations.
In addition to staff and partner interviews, a focus group with a small number of your TCE-HIV
clients will also be conducted to gather information regarding their perspectives about your
program. To effectively recruit clients to participate in the focus group, we ask that a designated
program staff member act as a point of contact to assist with recruitment of focus group clients.
The remainder of this letter outlines the visit schedule and requested feedback about the
proposed agenda. The visit will begin with a welcome and introductions on the morning of
DATE, 2010. Immediately following the introductions and a brief overview, TCE-HIV
evaluation team members will meet individually with key staff members from your program and
from your partners/collaborators, as appropriate.
Attachment A contains a list of key personnel typically associated with the administration of
substance abuse and HIV/AIDS service delivery at most agencies. It is essential that we meet
with many of these individuals during the site visit, and we would appreciate it if you worked
1

Attachment 4: Document 4 - Site Visit Confirmation Letter

with our team to develop the meeting schedule. Attachment B is a proposed site visit schedule.
Please fill in the names and titles of the staff members we will interview in the time slots of the
proposed agenda. Feel free to change the times listed for the client focus group and interview
with the project director if other times are more convenient for your program schedule. If there
are other individuals at NAME OF SITE you believe would be useful for us to meet, please add
them to the agenda in Attachment B.
In general, interviews take approximately 60 to 90 minutes to complete.
Please send Attachment B with the names and titles of key staff via e-mail to me at e-mail
address by (day of week), Date , 2010. Please contact me at phone number (e.g., 240-645-1234)
with any questions you may have about the upcoming visit or the proposed agenda. We are
looking forward to meeting with you and your staff in Month of visit.
Sincerely,

Your name and credentials
Your title
TCE-HIV Evaluation Project
cc: Government Project Officer Name (e.g., Stephen Carrington, MA, CSAT)
Attachments

2

Attachment 4: Document 4 - Site Visit Confirmation Letter

ATTACHMENT A
TCE-HIV EVALUATION PROJECT VISIT
AGENCY STAFF TO BE INTERVIEWED

The following is a list of staff that the members of the Multi-Site Evaluation Team may need to
meet with during the CSAT TCE-HIV evaluation project site visit. The actual position title and
location of staff will vary depending on the size and structure of the agency. This list is intended
as a guide. We may not need to meet with each individual, but staff members in these positions
are typical of those who usually provide information. Please use this information as a guide for
filling out the table in Attachment B.
1.

Executive Director

2.

Project or Clinical Director and Other Management Staff
• Project director
• Clinical director
• Other managers

3.

Direct Service Staff
• Outreach workers
• Counselors
• Treatment providers
• Case managers

4.

Partners/Collaborators

3

Attachment 4: Document 4 - Site Visit Confirmation Letter

ATTACHMENT B
TCE-HIV EVALUATION PROJECT SITE VISIT
NAME OF SITE (e.g. LA CLINICA DEL PUEBLO)

SCHEDULE

TIME

Date, 2010

Date, 2010

9:00 – 9:15 a.m.

Welcome and introductions

9:15 – 9:30 a.m.

Overview and purpose of the visit

9:30 – 11:00 a.m.

NAME
Project Director

12:00 – 1:00 p.m.

NAME
Partners/Collaborators

NAME
Case manager

2:00 – 3:00 p.m.
Client focus group
3:00 – 4:00 p.m.

4

Attachment 4: Document 5 - Administrative Staff Consent Form

Form Approved
OMB No. ####-####
Expiration Date: ##/##/####

Consent Form for Participation in the Targeted Capacity Expansion Program for Substance Abuse
Treatment and HIV/AIDS Services Multi-site Evaluation

ADMINISTRATIVE STAFF SEMI-STRUCTURED INTERVIEWS

A.
BACKGROUND AND PURPOSE
JBS International, the Alliance for Quality Education, Battelle Memorial Institute, and the Oregon Health & Science
University (JBS team) have a contract with the Substance Abuse Mental Health Services Administration
(SAMHSA), Center for Substance Abuse Treatment (CSAT) to conduct a Multi-Site Evaluation of the TCE-HIV
program. The JBS team is not part of CSAT or any other federal agency.
The purpose of this study is to learn more about the effect of outreach/pretreatment and substance abuse treatment
programs supported by TCE-HIV funding. The goal of the study is to improve outreach/pretreatment and substance
abuse treatment for racial and ethnic minorities at risk for HIV/AIDS. Information from your participation will help
the JBS team understand how programs can better reduce substance abuse and HIV risk behaviors.
You are being asked to participate in this study because you are an administrator of a TCE-HIV program.
B.
PROCEDURES
If you agree to participate, the following will occur:
• You will complete a form providing background information (e.g., age, gender, and years at current staff
position).
• You will take part in an interview. The interview will be about your role, activities, and experiences as an
administrator of this program.
• The interview will last approximately 90 minutes.
• The interview will take place at a time and place convenient to you.
• The interview discussion will be audio taped to ensure accuracy in reporting your statements.
• Neither your name nor identity will be used in any published reports.
All information you provide is anonymous. Input you provide during the interview will be combined with
information from other interviews from across the United States. The combined information will be analyzed. Only
combined results will be presented in reports.
C.
RISKS
The risks for participating in the study are expected to be minimal. Responding to the questions does not involve
great risk, but this activity might be tiring. The JBS team does not have a program to pay you or provide medical
care if you are hurt by participating in this research project.
Notice to Respondents
Public reporting time for this collection is estimated to average 90 minutes, including the time for reviewing instructions, searching existing data
sources, gathering and maintaining the data needed and completing and reviewing the collection of information. Send comments regarding this
burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to SAMHSA OMB Officer, 1
Choke Cherry Road Room 7-1044, Rockville, MD 20850. An agency may not conduct or sponsor, and a person is not required to respond to, a
collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is XXXX-XXXXX.

1

Attachment 4: Document 5 - Administrative Staff Consent Form
D.
PRIVACY
The privacy of the information we collect about you will be very carefully protected. The evaluation staff will be
trained on handling sensitive data and the importance of privacy. All of the data will be kept in locked files at JBS
International, and only the official project staff will have access to these files. At the end of the project, all data will
be given to CSAT. The data that is given to CSAT will not include names or participant identification.
E.
BENEFITS
There is no direct benefit to you for participating in this research project. However, the information you share might
benefit the TCE-HIV program and similar programs targeting persons who abuse substances and are at risk for HIVAIDS.
F.
RIGHT TO REFUSE OR WITHDRAW
Your participation in this interview is completely voluntary. You may end your participation in the interview at any
time. If you refuse to participate, there will be no penalty or loss of benefits to you.

G.
PERSONS TO CONTACT
If you have any questions about this study, please contact Dr. Resa Matthew at JBS International.
Resa Matthew, Ph.D.
JBS International
5515 Security Lane, Suite 800
Bethesda, MD 20852
(301) 495-1080
[email protected]
If you have any concerns about your rights in this study or any questions about injuries related to the research
project, please contact Dr. Amanda Gmyrek of the JBS International Institutional Review Board.
Amanda Gmyrek, Ph.D.
JBS International
5515 Security Lane, Suite 800
Bethesda, MD 20852
(301) 495-1080
[email protected]

YOUR CONSENT
You have read this consent form. You have been given a chance to ask questions, and you feel that all of your
questions have been answered. You know that you are free to participate in the interview or not. You know that after
choosing to participate in the interview, you may stop at any time without penalty. You are signing below to indicate
that you agree to participate in this interview and give permission for your responses to be audio recorded.

___________________________________

__________________

Participant Name (Print)

Date

_____________________________________
Participant Signature

2

Attachment 4: Document 5 - Administrative Staff Consent Form
I acknowledge that I witnessed the participant sign this consent form.

_____________________________________

__________________

Witness’ Name (Print)

Date

_____________________________________
Witness’ Signature

_____________________________________

__________________

Researcher Signature

Date

3