0196 Ss-a 6.22.16

Pretesting of Substance Abuse Prevention and Treatment

and Mental Health Services Communications Messages

Supporting Statement

A. Justification

1. Circumstances of Information Collection

Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting approval from the Office of Management and Budget (OMB) for an extension of the information, Pretesting of Substance Abuse Prevention and Treatment and Mental Health Services Communications Messages. The current approval under OMB No. 0930-0196 expires on September 30, 2016. There are no changes to this submission.

As the Federal agency responsible for developing and disseminating authoritative knowledge about substance abuse prevention, addiction treatment, and mental health services and for mobilizing consumer support and increasing public understanding to overcome the stigma attached to addiction and mental illness, SAMHSA is responsible for development and dissemination of a wide range of education and information materials for various consumer publics and the professional communities. Authorization for SAMHSA’s education and information dissemination activities is contained in 42 USC 290aa, Section 501(d)(3) of the Public Health Service Act.

Information programs within SAMHSA create and use a variety of media, including print (for example, brochures, posters, fact sheets, information kits), broadcast (for example, public service announcements and video news releases), electronic media (for example, SAMHSA’s web site) and direct response (the National Clearinghouse on Drug and Alcohol Information and the National Mental Health Information Center) to inform and educate the public and treatment professionals about substance abuse and mental health. Production of these materials is the major way that SAMHSA relays messages to the audiences it is mandated to reach.

To ensure that messages have the potential to be received, understood, and accepted by those for whom they are intended, SAMHSA employs formative evaluation. This type of evaluation involves (1) assessing audience knowledge, attitudes, behavior and other characteristics for the planning and development of messages, communication strategies and public information programs; and (2) testing these messages, strategies and program components in developmental form to assess audience comprehension, reactions and perceptions. Information obtained from testing can then be used to improve materials and strategies while revisions are still affordable and possible. This also maximizes the effectiveness of these messages and strategies, and reduces the frequency with which publications and programs need to be modified.

Since SAMHSA’s leadership responsibilities in the substance abuse and mental health communities, it has a responsibility to provide credible information messages and services. If it is unable to empirically predict whether its materials and messages have the potential of being understandable and useful to their respective targets, it is unable to carry out its mandate effectively.

Formative (pretest) evaluation is an activity that SAMHSA should uniformly perform on all print pieces, broadcast products, and informational services and messages to maximize their usefulness. These pretests should be conducted on a small scale and focus on potential effectiveness with specific target audiences.

This submission is for an extension of a generic approval for up to 30 pretests per year using methods described in Section B with audiences targeted by messages to be developed by SAMHSA. The previous approval was not widely used, but there are currently plans for several new information programs that will be making use of this generic approval. The previous approval had the following Terms of Clearance:

This request for a generic clearance of pretesting materials is approved. Consistent with other generic clearances, HHS/SAMHSA will submit each individual pretest for OMB review, along with a brief description of the purpose of the pretest, sample design if applicable, any respondent incentives planned, and individual burden. OMB will either approve or provide comments on the individual request within 10 working days of receipt.

2. Purpose and Use of Information

Assessment is an integral part of SAMHSA’s overall plan for its communications strategies. Pretesting of print and broadcast products and information services assesses the potential effectiveness of these products while they are still in the developmental stage. Later on, analysis of other SAMHSA data collection mechanisms (such as the National Survey on Drug Use and Health and the National Health Interview Survey (NHIS) mental health component) enables SAMHSA to evaluate the eventual effectiveness of its substance abuse and mental health education programs.

The formative evaluation process is used to determine whether a draft message is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages through either individual or group interviews. Respondents assess the materials in the following areas:

• Attention — Do the messages attract and/or hold the audience’s attention? For example, if they were to see a particular brochure in their doctor’s office, would they pick it up and look at it? That is, does it attract their attention?

• Comprehension — Is the message clearly understood, and are the main points conveyed? Does the main theme of the message get across to the audience? Is the language clear? Does the message convey the intended information in a manner that makes it effective for the target audience? For example, it is well‑known that marijuana use generally starts between ages 9‑14, a difficult demographic group to impact with conventional messages. Therefore, prior to finalizing the development of our communications messages for this group, we might convene focus groups of adolescents age 9‑14 to determine whether we are using a motif that is appealing, compelling, and effective.

• Personal Relevance and Self‑efficacy — Does the target audience perceive the message as personally relevant? For example, do the respondents see that the message in the public service announcement (PSA) applies to them ‑ that the information is important to them? Do the respondents see themselves as capable of acting on the message?

• Believability — Is the message and/or its source perceived as credible? For example, does the respondent believe the message in a PSA that talking to their children about not experimenting with drugs will decrease their chance of becoming a drug abuser? Is the person delivering the message credible?

• Acceptability — Is there anything in the message that may be offensive or unacceptable to the target audience? For example, does the respondent react negatively to a print message that warns of the dangers of using drugs? Is the piece culturally sensitive?

• Behavioral Intent — Do respondents think they will take action as a result of seeing/hearing the message? For example, does an educational piece containing signs or indications of depression motivate the target audience to think more about this as an explanation for the behavior of someone they know?

The information from respondents’ reactions in each of these areas provides insight into how the audiences for these messages may react and how the messages should be revised to help ensure the most favorable reaction from the audience.

Other information gathered on respondents regarding gender, age, socioeconomic level, race/ethnicity, and applicable family history with substance abuse or mental health provides a basis for evaluating whether the messages may be perceived differently by different segments of the audience. For example, selected age groups may find a particular brochure or message on substance abuse more relevant than other age groups.

Systematic formative evaluation has been widely adopted by health education program planners as an integral step in the development of messages and materials. Through pretesting SAMHSA will be able to:

• Understand characteristics of the target audience— its attitudes, beliefs and behaviors — and use these in the development of effective communications tools;

• Produce messages that have the greatest potential to favorably influence target audience attitudes and behavior; and

• Expend limited program resource dollars wisely and effectively.

3. Use of Information Technology

The information will be collected through the use of one‑to‑one interviews or self‑administered questionnaires, depending upon the target audience being questioned and the subject matter being addressed. Improved technology in the collection and processing of data will be used to reduce respondent burden and make processing maximally efficient. When telephone interviews are used, computer‑assisted telephone interviewing (CATI) will be employed whenever possible. For self‑administered questionnaires, closed‑ended questions (for example, multiple choice) and machine‑readable answer sheets will be used when feasible. When appropriate, data will also be collected via electronic mail or facsimile transmission.

As SAMHSA’s information sites on the World Wide Web continue to be developed and improved, SAMHSA may explore using either Website questionnaires or conducting on‑line focus groups with Internet users such as treatment providers or consumers with Internet access. Using computer‑assisted information technology to transmit data collection instruments and/or collect responses is expected to reduce the burden on respondents; for example, respondents may be able to access and respond to data collection requests at a time and place that is more convenient to them, eliminating the need to travel for in‑person or group interviews.

4. Efforts to Identify Duplication

The general areas in which information needs to be gathered (as described in A.2 above — attention, comprehension, etc.) to pretest effective substance abuse and mental health messages (brochures, PSAs, media campaigns, etc.) will be generally similar from pretest to pretest. However, the specific questions that will be asked of respondents will differ with the message content, audience targeted, and medium of the message.

As each new message, strategy, or product is developed, SAMHSA reviews existing literature and data bases, including pretesting reports on existing messages and materials, and consults with outside experts to evaluate available information in similar messages with comparable audiences. However, since the substance abuse and mental health fields are so diverse and complex, and each message is essentially different, new data collection instruments must be prepared for each pretest.

5. Involvement of Small Entities

Substance abuse and mental health care providers may sometimes be the target audience for SAMHSA information materials. When testing of these materials is required, SAMHSA will generally work through established professional organizations to gain access to the audience, and to obtain feedback on our instruments and data collection plans. As a result of this contact, SAMHSA will be able to minimize the placement of additional burden on treatment professionals.

6. Consequences If Information Collected Less Frequently

Information will be collected one time for each print or broadcast message, product or strategy tested. Respondents will not be re‑contacted.

7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)

SAMHSA recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, when SAMHSA requests responses to a self‑administered written questionnaire in a short time period, receipt of the questionnaire will generally be preceded by advance notification to respondents explaining the purpose of the questionnaire, the approximate length of time that the questionnaire will take, and the voluntary nature of participation. All efforts will be made to keep such questionnaires short and well focused.

Because SAMHSA’s pretesting activities will be primarily qualitative in nature, the results will not be generalizable to the population at large or to the particular target audience under study. However, the nature of pretesting is such that generalizability is not a critical feature; the emphasis is on obtaining timely, useful information that can be fed back into the development of new messages or materials or the revision of existing ones.

There are no other special circumstances.

8. Consultation Outside the Agency

The 60-day notice soliciting public comment published in the Federal Register on June 22, 2016 (81 FR 40715). SAMHSA received no comments from the Notice. 

11. Questions of a Sensitive Nature

As mentioned above, some studies require the inclusion of people who match selected characteristics of the target audience, which SAMHSA is trying to reach. This sometimes requires asking a question about race/ethnicity, income, education, and/or drug abuse or mental health problems on the initial screening questionnaire used for recruiting. Potential participants are informed that this is being done to make sure that SAMHSA speaks with the kinds of people for whom its messages are intended. Again, respondents will be assured that the information is voluntary and will be handled in a confidential manner.

Since SAMHSA communications are concerned with the treatment of mental health problems, some projects may involve asking questions about (or discussing) how one perceives his/her own personal risk for such problems. Fears of and experiences with mental illness may also be covered. This information is needed to gain a better understanding of the target audience so that the messages, strategies and materials designed will be appropriate and sensitive. Questions of this nature, while not as personal as those on sexual behavior or religious beliefs, still require some sensitivity in how they are worded and approached. In face‑to‑face data collections, questions of this kind will generally be asked later in the interview or group discussion when respondents are more comfortable with the interview situation and are more at ease with the interviewer/moderator. As noted in Section A.10., participants will already have been informed at the beginning about the nature of the activity and the voluntary nature of their participation. The interviewer/moderator makes it clear that they do not have to respond to any question that makes them uncomfortable.

Raw data from data collections that include sensitive information (for example, screening questionnaires and audio tapes) will not be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.

12. Estimates of Annualized Hour Burden

The number of respondents to be included in each pretest will vary, depending on the nature of the material or message being tested and the target audience. However, the Annual Response Burden table that follows provides a likely annual distribution of respondents and hours by type of data collection. Time to read, view, or listen to the message being tested is built into the "hours per response" figures.

Activity	No. of Respondents	Responses/ Respondent	Hours per Response	Total Hours	Hourly Wage Rate ($)1	Total Hour Cost ($)
Individual In-depth Interviews: General Public Service Providers	400 200	1 1	.75 .75	300 150	$15.50 25.00	$4,650 3,750
Focus Group Interviews: General Public Service Providers	3,000 1,500	1 1	1.5 1.5	4,500 2,250	15.50 25.00	69,750 56,250
Telephone Interviews: General Public Service Providers	335 165	1 1	.08 .08	27 13	15.50 25.00	419 325
Self-Administered Questionnaires: General Public Service Providers	2,680 1,320	1 1	.25 .25	670 330	15.50 25.00	10,385 8,250
Gatekeeper Reviews: General Public Service Providers	1,200 900	1 1	.50 .50	600 450	15.50 25.00	9,300 11,250
TOTAL	11,700	--	--	9,290	--	$174,329

¹ The hourly wage of $15.50 for the general public was calculated based on weighted data from the 2017 NSDUH respondents' personal annual income. The $25 hourly wage rate for providers is an average across counselors and other service provider staff.

13. Estimates of Annualized Cost Burden to Respondents

There are no capital or start‑up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.

14. Estimates of Annualized Cost to the Government

The total annual cost to the Federal Government will be approximately $222,750. This estimate is based on annual performance of up to: 225 focus groups (15 group sessions per each of 15 media campaigns at $550 each session); 7 telephone surveys at $3,000 each; 7 mail self‑administered surveys at $6,000 each, and 3 in‑depth interview studies at $12,000 each. These figures include the costs of study design, facility rental (e.g., for focus groups), data collection, analysis, and report/publication writing; the total cost of these activities is approximately $305,000.

In addition, there will be annual costs of approximately $31,410 associated with project officer monitoring, projected to be about 1,000 hours of effort a year at a SAMHSA personnel cost of $31.41 per hour.

15. Changes in Burden

There is no change in the burden.

16. Time Schedule, Publication and Analysis Plans

The data collection plan, schedule and analysis for each project will be determined by the objectives of the pretest, the messages being tested, and the audience for the message. Specifics of the analyses cannot be determined until the messages to be tested are prepared.

The process for developing the analytical plan for the pretest will be similar to that used in any formal evaluation. SAMHSA staff will review the material to be pretested, discuss the objectives with the individuals responsible for developing the materials, determine the analytic questions to be addressed in the pretest, and then after resolution of any problems, approve the pretest procedures, instruments, and data analysis plan.

Techniques will include primarily qualitative analyses (for example, content analysis for focus group interviews), although some results may be summarized quantitatively using descriptive statistics. No complex analytic techniques will be used.

While the primary purpose of a pretest is to provide information to the developers of the messages for the purpose of improving them, SAMHSA may make pretest results available to substance abuse and mental health program planners in the states, voluntary organizations and others. Moreover, the Center for Substance Abuse Prevention may include some information from findings in publications that they prepare for substance abuse professionals and community entities when appropriate.

While the pretesting period will vary somewhat depending on the complexity of the testing and number of respondents required the typical pretest will require approximately 14 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical pretest would be:

Activity Time Schedule

Initial review of materials Week 1

Preparation of pretest design Week 2

Review of design Weeks 3

Preparation of OMB submission Weeks 4-5

Submission for OMB review Weeks 6-9

Collection of data Weeks 10 and 11

Analysis of data Weeks 12 and 13

Report on pretest Week 14

17. Display of Expiration Date

All data collection instruments will display the OMB control number and expiration date in the upper right corner of the cover page.

18. Exceptions to Certification Statement

This activity will fully comply with the Certification for Paperwork Reduction Act Submissions.

9