Overview of Instruments Guide

Substance Abuse and Mental Health Services Administration

Center for Substance Abuse Prevention

Data Analysis Coordination and Consolidation Center

National Minority SA/HIV Prevention Initiative

Overview of Instruments and

Data Collection Guide

Overview of CSAP Data Collection Instruments

The Substance Abuse and Mental Health Services Administration (SAMHSA) within the Center for Substance Abuse Prevention’s (CSAP) National Minority Substance Abuse and HIV/AIDS Prevention Initiative supports an array of activities to assist grantees in building a solid foundation for delivering and sustaining effective substance abuse prevention and related services. While grantees have substantial flexibility in designing their grant projects, all are required to base their project on the five steps of SAMHSA’s Strategic Prevention Framework (SPF). Grantees must also conduct on-going monitoring and evaluations of their projects to meet CSAP’s reporting requirements.

This guide provides a detailed framework for planning and preparing for the data collection process. The actual arrangements must be worked out by each local program to maintain consistency within the coding and completion of the instruments. Major issues and solutions concerning the administration of the instruments (e.g., which sections of the questionnaire to administer based on intervention duration, targeted group size, the use of translated versions) will be determined by your local data collection team in consultation with your assigned CSAP Project Officer. For more detailed information on how to administer or proctor the instruments, administration guides have been provided for the Youth Questionnaire, Adult Questionnaire, Group Dosage Form, and Individual Dosage Form in addition to this document. CSAP hopes that grantees will find the data collection requirement useful to their own evaluation processes and future program planning.

General Administration Guidelines

Develop Storage System

Over the life of this initiative, each grantee will be collecting information that must be documented and organized. Each local evaluation team or the person responsible for data management will be required to store:

• Individual questionnaires and forms (Youth & Adult Questionnaires and Individual & Group Dosage Forms) until they are entered or uploaded into PMRTS

• Consent forms

• Tracking forms for each participant (intervention and control/comparison) in the study

Before initiating data collection, set up a filing and storage system that will accommodate these needs. Any form containing a participant’s name and unique identification number should be kept in a locked location. Completed questionnaires and dosage forms should not be stored in the same location as forms which contain the unique participant identification number and the respondent’s name.

Make Arrangements for Data Collection

For administration of the questionnaire, develop a clear understanding with program staff concerning the time and place of administration well ahead of time. The room where administration takes place should have adequate seating to accommodate the respondents, adequate lighting and ventilation, and seating should be spaced to ensure privacy. The administrative portion of the questionnaires should be completed by the program staff responsible for coordinating data collection before questionnaires are distributed to respondents.

A system should be implemented for the collection of dosage data. The program staff responsible for coordinating data collection should determine which individuals are responsible for the collection of dosage data during a session or encounter. Ideally, dosage forms should be prepared for the program staff by the program staff who are coordinating data collection.

Familiarity with the Instruments

It is critical that the program staff responsible for data collection be familiar with questionnaire and dosage form format and content prior to administering the instrument. Individuals proctoring the questionnaire should read it carefully until an understanding of the wording of all items is established. The questionnaire item structure (outlined in the corresponding administration guide) should also be reviewed before the survey is administered. The individual proctoring the questionnaire should be prepared to answer questions from respondents regarding the questionnaire items. For dosage forms, the program staff responsible for data collection should ensure that all staff who fill out dosage forms understand each service category and the appropriate duration code.

Possible Issues during the Survey Process

The program staff responsible for data collection should train those administering the questionnaires on how to deal with any issues of emotional distress that may arise as a result. Although this is not expected, some questionnaire items ask about personal issues such as partner abuse or accepting money or other goods in exchange for sex. Participants should be instructed that their answers are private and will be very helpful to program evaluators, but that completing the questionnaire is optional. Services or referrals to appropriate services should be provided in the event of emotional distress.

Incentive Materials

Incentives for participating in the program or data collection process, if offered, will be determined locally. Any incentives offered should be approved by your CSAP Project Officer. Procedures for documenting the distribution of incentives are the responsibility of the grantees.

Assign Individual Identification Numbers

Assign a unique identification number to each respondent, whether they are in the intervention or control/comparison group. A 5-digit unique identification number (ID) is required on each form in order to track the responses of an individual and the corresponding dosage data over time. On the questionnaire, each participant’s name and unique 5‑digit ID should be written on the face (cover) sheet of the questionnaire and the same 5 digit ID should be entered on page 2 of the instrument. This unique ID should also be used on every dosage form with which the participant is associated. All record management information should be entered on the questionnaires by the program staff responsible for data collection prior to distributing the questionnaire to the respondent. For dosage forms, a system should be created by the grantee to designate what the program staff responsible for coordinating data collection fills out and what the individual recording the dosage data completes on the forms. Participant names must not be written on any other page but the face sheet on the questionnaire and should never appear on dosage forms. Record management information has the following components:

• Grant Identification Number: Each grantee will use its assigned grant identification number provided by CSAP.

• Study Design Group: Record whether the respondent is receiving an HIV funded intervention (coded as “1” on dosage record forms) or is a control/comparison group participant. As discussed in more detail below, the use of control/comparison groups is optional. However, even if your site is not using a control/comparison group and all of your participants are receiving services, it is important to complete this field (by entering a “1” for all of your participants) for data management purposes. A data record that has the Study Design Group left blank will not be included in the multisite analysis.

• Unique Participant Identification Number: Unique identification numbers should be assigned to each program participant and used for all data records associated with the participant. Numbers can range from 00001 to 99999. Programs with multiple service locations may want to consider assigning a range of individual identifiers to each location to allow for easy identification of a participant’s service location. For example, one location could be assigned numbers 1000 to 1999, numbers 2000 to 2999 to a second location, and so forth.

• Administration Date: The month (2-digit), day (2-digit), and year (4-digit) of the data collection should be entered.

• Interview Type (Questionnaires Only): Record whether this is a baseline, exit, or followup survey.

• Intervention Duration (Questionnaires Only): Record whether the intervention the participant is receiving is Single Session (lasting one day or less), Multiple Session Brief Intervention (less than 30 days duration), or Multiple Session Long Intervention (30 days or longer duration) (See Table 1 below).

• Intervention Name(s) (Questionnaires Only): The name(s) of any intervention used by the grantee to construct the sessions delivered to the participant. The intervention name is not the name of a specific service (i.e. HIV education or case management), but rather the name of the curriculum or evidence-based program used by the grantee to guide its intervention. Some examples of intervention names are: Guiding Good Choices, Healthy Workplace, Keepin’ It REAL, Life Skills Training, Lions Quest Skills for Adolescence, Parenting Wisely, Positive Action, and Voices/Voces.

Overview of CSAP Data Collection Instruments

Youth and Adult Questionnaires

1. Questionnaires

CSAP has created two instruments for HIV Grant participants: the Youth Questionnaire and the Adult Questionnaire. The questionnaires provide CSAP and grantees with a tool that can be used to compare participants across grant sites. These instruments should be filled out by program participants and are meant to document basic characteristics about the participants and/or changes in their behavior and attitudes relevant to substance abuse and HIV prevention.

Some of the items in the Youth and Adult Questionnaires are identical and have been shown to be appropriate for both youth and adults. However, the Youth Questionnaire was created to be cognitively appropriate for youth and contains construct measurements that are specific to this age group. The Adult Questionnaire contains items that are cognitively more advanced and asks about behaviors and beliefs that may not be appropriate for youth.

The Youth and Adult Questionnaire have been divided into three sections. These sections group items by common theme. The three sections are:

• Section One: Facts about You -- demographic items as well as information relevant to CSAP’s target population

• Section Two: Attitudes & Knowledge -- attitudes and knowledge about drug use, sexual behavior, HIV/AIDS, and other relevant topics

• Section Three: Behavior & Relationships – behavioral items about drug use, sexual behavior, HIV/AIDS, and other relevant topics

Sections will be administered to program participants based on the type of intervention the participant receives. CSAP has identified three mutually exclusive intervention durations: Single Session Interventions, Multiple Session Brief Interventions (30 days or less duration), and Multiple Session Long Interventions (30 days or longer duration) (Table 1). The three sections in the Youth and Adult Questionnaire correspond to these intervention durations:

Table 1: Sections of the Questionnaire

Intervention Duration	Length	Definition	Sections of Survey To be Administered
Single Session Intervention	1 day or less	A direct service intervention that lasts one day or less. Participants may receive multiple services during the session, but do not continue in a CSAP HIV grant funded activity for more than one day.	Section One: Facts about You 3 to 5 questions from Section Two
Multiple Session Brief Intervention	Less than 30 days	The participant should receive at least two HIV Grant funded sessions or service encounters. The period of time between the first session or encounter and the last session or encounter should be two to 29 days.	Section One: Facts about You Section Two: Attitudes & Knowledge
Multiple Session Long Intervention	30 days or more	The participant should receive at least two HIV Grant funded sessions or service encounters. The period of time between the first session/encounter and the last session/encounter should be 30 days or more.	Section One: Facts about You Section Two: Attitudes & Knowledge Section Three: Behavior & Relationships

The staff responsible for data collection, in consultation with its CSAP Project Officer, should determine how to identify what type of intervention a participant will receive. This is done by counting the number of days the program plans to have with the participant during the course of the intervention. Only sessions or encounters that are funded by the HIV grant should be included in the count. Once the duration of the intervention has been determined, the staff responsible for data collection should prepare the Youth or Adult Questionnaire for the participant. Sections of the questionnaire that are not applicable for a participant based on the duration of intervention he or she receives should be removed and discarded prior to administration.

Even though CSAP requires data collected from all individuals served by the funded program, please note that only participants who were served in interventions lasting 30 days or more will count towards grantees’ declared targets.

The same sections should be given at every time point the survey is administered. The appropriate sections of the questionnaire to administer should be determined before baseline (1^st data collection point). For example, if the participant is receiving a Multiple Session Brief Intervention and is being given the Youth Questionnaire, the participant should receive Section One: Facts About You and Section Two: Attitudes & Knowledge at all the appropriate time points. Additional local evaluation instruments may be administered to participants in conjunction with, but not instead of, the CSAP questionnaires.

Occasionally, a participant who took the baseline survey, completed the intervention, and was given an exit survey, later comes back for further services. In such cases, the participant should be given the same Participant ID that was assigned during the first intervention but a new set of baseline and exit surveys.

2. Single Session Intervention Protocol

Grantees choosing to implement single session interventions are required to collect demographic and outcome data from participants. Data collection should occur once, immediately following the intervention, at exit. To assess outcomes that are directly relevant to the intervention’s content, grantees are instructed to choose 3 to 5 survey items from Section Two of the Youth or Adult Questionnaire to be administered in addition to Section One. Participants who are only receiving HIV testing and testing counseling services should not be given any section of the questionnaire. Aggregate testing data will be collected through the PRMT.

Questions from Section Two must be chosen by the staff responsible for data collection prior to delivery of the intervention. Complete these steps to prepare questionnaires for participants:

1. Carefully fill out the Record Management section, making sure to fill in ‘Single Session Intervention’ under Intervention Duration.

2. For Interview Type in the Record Management section, indicate ‘Exit’.

3. Highlight or circle the selected items and corresponding instructions from Section Two, making sure they are clearly marked.

4. Tear off and discard Section Three of the questionnaire.

Following delivery of the single session intervention, instruct participants to complete the entire Section One and only the questions selected from Section Two. Dosage forms should be completed by the appropriate project staff.

The 3 to 5 selected items should be directly relevant to the intervention’s content, that is, to the specific attitudes and/or knowledge the intervention is targeting. For example, if you are implementing VOICES/VOCES, which aims to prevent HIV transmission by encouraging condom use, appropriate question items from the Adult Questionnaire might be:

• Intention to practice safe sex (item 36)

• Confidence in ability to negotiate condom use (item 30)

• Refusal to have sex if partner will not use a condom (item 31)

• General knowledge of HIV (items 37 and 39)

Participants should not be given a mix of items from the Youth and Adult Questionnaires. For example, if Section One of the Adult Questionnaire is being administered to participants, please choose the additional Section Two items from the Adult Questionnaire. For more detailed guidance on how to choose survey items that are appropriate for your single session intervention, please refer to the ‘Single Session Intervention Data Collection Guide’.

The 3 to 5 items from Section Two must be selected by the staff responsible for data collection prior to delivery of the single session intervention.

nce the appropriateipant shouldns or encounters the program plans to have with the participant.ts across grant sites. The ites
. For example, if someone partnsidered to be mutually exclusive. For example, if someone participants in a one session int

3. When to Administer the Questionnaires

Once the intervention duration has been determined, the same sections should be given at every time point the survey is administered. Three possible data collection time points exist in which to collect data:

Table 2: When to Administer Questionnaires

Instrument	Baseline (1st Data Collection Point)	Exit (2nd Data Collection Point)	Followup (3rd Data Collection Point)
Single Session Intervention	N/A	Immediately following the single session intervention	N/A
Multiple Session Brief Intervention	Within 30 days before intake and before program exposure or beginning of the sessions/encounters	Within 10 days after program exposure has ended	Not required
Multiple Session Long Intervention	Within 30 days before intake and before program exposure or beginning of the sessions/encounters	Within 10 days after program exposure has ended	Within 30 days of the planned followup which happens 3-6 months after program exit

¹ No baseline data collection is required for single-session interventions.

² This is the only data collection point for single-session interventions.

³ No followup data collection is required for single-session interventions.

4. Select Questionnaire Proctors

CSAP strongly recommends that program staff who provide direct services to those taking the survey, such as case managers, should not be involved in administering questionnaires. Ideally, a staff member who is responsible for data collection but not directly involved in delivering services will be selected to proctor administration of the questionnaire. Staff who serve as proctors should be fully trained according to the administration procedures set out in this guide.

At the beginning of a questionnaire administration session, a program staff member should be onsite to introduce the proctor. The program staff member may then leave the room and return later to talk to participants and conduct the debriefing. The proctor should be familiar with the questionnaire’s corresponding administration guide and have it accessible during the session. Proctors should report to the staff responsible for data collection if respondents have serious problems understanding the questionnaire. If this occurs, please contact your assigned CSAP Project Officer.

5. Administration Time

The amount of time required to complete the questionnaire depends on the number of sections that are included.

Table 3: Administration Time

Intervention Duration	Sections of Survey To be Administered	Approximate Amount of Time To Complete*
Single Session Intervention	Section One: Facts about You 3 to 5 questions from Section Two: Attitudes & Knowledge	5 minutes
Multiple Session Brief Intervention	Section One: Facts about You Section Two: Attitudes & Knowledge	30 minutes
Multiple Session Long Intervention	Section One: Facts about You Section Two: Attitudes & Knowledge Section Three: Behavior & Knowledge	50 minutes

*These are estimates; respondents with limited reading and/or English language ability may require more time.

Please allow extra time for the distribution of the questionnaires, reading of the instructions, collection of completed questionnaires, and any local administration activities (e.g., distributing incentives, collecting tracking information). Administering the questionnaire to respondents with limited reading abilities (those with a reading level less than 8^th grade) may require full proctoring (reading the entire questionnaire aloud). Full proctoring requires more time than self-administration. If the questionnaire is read aloud, the proctor should practice reading the questions aloud several times beforehand. At the beginning of the sessions, instruct the respondents on how the questionnaire will be read to them. It is important to tell the respondents not to answer the questions out loud, but to simply mark their answers in the questionnaire.

6. Makeup Administrations

If there are missing respondents when a questionnaire is being administered, take the following steps:

• Arrange to attend the next program session to speak to the respondent(s) who did not attend the questionnaire administration session.

• Arrange a makeup session that is convenient for respondents; this can be done in person, by phone, or in writing.

• If a respondent misses a makeup session, or if a group makeup cannot be arranged, make reasonable efforts to administer the questionnaire individually.

7. Translated Instruments

The Youth and Adult Questionnaires are available in English only. Grantees that serve participants with other language needs should plan to translate the questionnaires and instructions locally after obtaining authorization from the Project Officer. A copy of the translated instruments should be sent to CSAP as an e-mail attachment addressed to your CSAP Project Officer.

Dosage Data: Individual and Group Dosage Forms

1. Dosage Forms

Documenting participant exposure to program services is an important part of the data collection process. Dosage data are designed to reflect the actual prevention practices of each grantee. Dosage is a measure of the type and amount of contact that a participant has in each direct encounter with the program. Programs are required to collect dosage (services) data for each individual participating in the intervention using the Individual and Group Dosage Forms.

2. Collection of Dosage Data

Individual Service dosage data should only be collected for individual service encounters in which the participant is directly involved with staff in a one-on-one basis. Group Service dosage data should be collected for services delivered to multiple participants during a single encounter, such as a group education session attended by more than one participant. CSAP recognizes that Health Services are typically not delivered in a group format; however, Health Services can be recorded on either the Individual Dosage Form or the Group Dosage Form. Dosage information is only collected on services that are partially or fully funded by the HIV grant.

Dosage data are collected over the course of participants’ engagement in the intervention. Once the participant has exited the intervention, the collection of dosage data should cease for this participant. If the individual participates in a Single Session Intervention, a dosage form should be completed for that service encounter. If the participant is involved in a Multiple Session Intervention, each service encounter should be recorded on a separate form, using the appropriate dosage form (Group or Individual).

Dosage data collection is performed on an ongoing basis throughout the intervention duration. Dosage forms are completed by the program staff responsible for data collection or the service provider. A standard dosage collection procedure should be implemented by the data collection team.

3. Collecting Dosage Data

Those who fill out dosage forms should have an understanding of the service definitions outlined in the corresponding dosage administration guide. If a grantee finds that recording duration for sessions or encounters places too much burden on staff, service providers should identify the services that are typically provided in these encounters or sessions and define a time estimate for each. These time allocations can be applied to all service encounters (i.e., all participants who complete a specific service would be given the same dosage codes, and the dosage codes would be the same across all program encounters). A participant should not receive a time estimate for a service that they never received. Although this approach would not be as accurate as determining the dosage codes for each encounter, over the course of the entire program the dosage data would reflect approximately how much exposure each participant received for each service. In either case, precautions should be made to maintain consistency in the definition of service types and duration code assignments.

If the grantee chooses to implement this system, the participant identification number on the Group Dosage Form should be filled out by the person conducting the session or encounter to indicate who was present at each session. If this is not possible, an attendance sheet should be kept by the person conducting the session or encounter, and the program staff responsible for data collection should enter the corresponding participant identification numbers on the Group Dosage Form using the attendance sheet.

Control/Comparison Groups

A comparison/control group can be used to examine the effect of an intervention. It is comprised of individuals from the same population as participants who receive the intervention; yet individuals in this group do not receive intervention services. While not required by CSAP, if your grant site is using a comparison/control group, these individuals should receive the same questionnaires as program participants (intervention group). Surveys for comparison/control participants should be administered within two weeks before or after the administration of the intervention group instruments. Dosage data should not be collected for these individuals as they do not receive intervention services.

Web-Based Data Entry and Upload System

The Prevention Management Reporting and Training System (PMRTS) website (The Portal) provides prevention information, data collection tools, documents, data entry functions, and access to reporting statistics and tracking. Questionnaires and dosage forms are available for individual grant sites to download and make copies for administration to clients or participants.

The program staff who are responsible for data collection are expected to enter participant responses to questionnaires via the Portal. The NOMs Data Entry Tools provide CSAP grantees with direct access to the management of their data entries as well as the ability to track the status of their submitted data. Grantees will also be able to upload response databases through the Portal provided that they use the appropriate variable naming and value coding conventions described in the Codebooks. Once data have been entered into the Portal, grantees may download their data into a standard format (i.e. Excel, csv, or xml). CSAP’s Data Analysis Coordination and Consolidation Center (DACCC) will then begin to clean the data and each grantee will have access to their cleaned downloadable, electronic data files.

Questions about the Instruments and Submission to CSAP

As previously mentioned, CSAP offers an online data entry system through PMRTS, where all instrumentation can be found and downloaded. Sites may enter the data online on a continual basis, or upload prepared data files if necessary. CSAP will extract data periodically for cleaning, analysis and reporting purposes; however the data will remain accessible for local evaluations.

Questions regarding the instruments or data submissions to CSAP should be addressed to the Technical Assistance Hotline for PMRTS, available Monday through Friday, from 9 am to 6 pm Eastern Standard Time via telephone to 1-888-DITIC-4-U, or via email to [email protected].

CSAP HIV Prevention Initiative

Updated 3/2011 Page 12