0930-0196 Ssa 12-7-2021

Pretesting of Substance Abuse Prevention and Treatment

and Mental Health Services Communications Messages

Supporting Statement

A. Justification

1. Circumstances of Information Collection

Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting approval from the Office of Management and Budget (OMB) for reinstatement of the information collection, Pretesting of Substance Abuse Prevention and Treatment and Mental Health Services Communications Messages. This generic clearance allows SAMHSA to collect qualitative feedback and survey data on communication products (e.g. brochures, booklets, posters, toolkits, fact sheets, videos, public service announcements, websites, campaign materials, etc.). The previous approval under OMB No. 0930-0196 was discontinued on August 31, 2019.

As the Federal agency responsible for developing and disseminating authoritative knowledge about substance abuse prevention, addiction treatment, and mental health services and for mobilizing consumer support and increasing public understanding to overcome the stigma attached to addiction and mental illness, SAMHSA is responsible for development and dissemination of a wide range of education and information materials for various consumer public and the professional communities. Authorization for SAMHSA’s education and information dissemination activities is contained in 42 USC 290aa, Section 501(d) (3) of the Public Health Service Act.

Information programs within SAMHSA create and use a variety of media, including print (for example, brochures, posters, fact sheets, information kits), broadcast (for example, public service announcements and video news releases), electronic media (for example, SAMHSA’s web site and smartphone applications) and direct response (the National Clearinghouse on Drug and Alcohol Information and the National Mental Health Information Center) to inform and educate the public and treatment professionals about substance abuse and mental health. Production of these materials is the major way that SAMHSA relays messages to the audiences it is mandated to reach.

To ensure that messages have the potential to be received, understood, and accepted by those for whom they are intended, SAMHSA employs formative evaluation. This type of evaluation involves (1) assessing audience knowledge, attitudes, behavior and other characteristics for the planning and development of messages, communication strategies and public information programs; and (2) testing these messages, strategies and program components in developmental form to assess audience comprehension, reactions and perceptions. Information obtained from testing can then be used to improve materials and strategies while revisions are still affordable and possible. This also maximizes the effectiveness of these messages and strategies and reduces the frequency with which publications and programs need to be modified.

SAMHSA is responsible to be a federal leader for providing credible and effective messages and information on substance use and mental health. If it is unable to empirically predict whether its materials and messages have the potential of being understandable and useful to its target audiences, it is unable to appropriately carryout its mandate.

Formative (pretest) evaluation is an activity that SAMHSA should uniformly perform on all print pieces, broadcast products, and informational services and messages to maximize their usefulness. These pretests should be conducted on a small scale and focus on potential effectiveness with specific target audiences.

This submission is for reinstatement of a generic approval for up to 30 pretests per year using methods described in Section B with targeted messages for diverse audiences. The previous approval was not widely used when it was discontinued, but there are currently plans for information programs that will be making use of this generic approval. The previous approval had the following Terms of Clearance:

This request for a generic clearance of pretesting materials is approved. Consistent with other generic clearances, HHS/SAMHSA will submit each individual pretest for OMB review, along with a brief description of the purpose of the pretest, sample design if applicable, any respondent incentives planned, and individual burden. OMB will either approve or provide comments on the individual request within 10 working days of receipt.

2. Purpose and Use of Information

Assessment is an integral part of SAMHSA’s overall plan for its communication strategy. Pretesting of print and broadcast products and information services assesses the potential effectiveness of these products while they are still in the developmental stage. After implementation, analysis of other SAMHSA data collection mechanisms (such as the National Survey on Drug Use and Health) enables SAMHSA to evaluate the eventual effectiveness of its substance abuse and mental health education programs.

The formative evaluation process is used to determine whether a draft message is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages through either individual or group interviews. Respondents assess the materials in the following areas:

Attention — Do the messages attract and/or hold the audience’s attention? For example, if they were to see a particular brochure in their doctor’s office, would they pick it up and look at it? That is, does it attract their attention?

Comprehension — Is the message clearly understood, and are the main points conveyed? Does the main theme of the message get across to the audience? Is the language clear? Does the message convey the intended information in a manner that makes it effective for the target audience? For example, it is well‑known that marijuana use generally starts between ages 9‑14, a difficult demographic group to impact with conventional messages. Therefore, prior to finalizing the development of our communications messages for this group, we might convene focus groups of adolescents age 9‑14 to determine whether we are using a motif that is appealing, compelling, and effective.

Personal Relevance and Self‑efficacy — Does the target audience perceive the message as personally relevant? For example, do the respondents see that the message in the public service announcement (PSA) applies to them ‑ that the information is important to them? Do the respondents see themselves as capable of acting on the message?

Believability — Is the message and/or its source perceived as credible? For example, does the respondent believe the message in a PSA that talking to their children about not experimenting with drugs will decrease their chance of becoming a drug abuser? Is the person delivering the message credible?

Acceptability — Is there anything in the message that may be offensive or unacceptable to the target audience? For example, does the respondent react negatively to a print message that warns of the dangers of using drugs? Is the piece culturally sensitive?

Behavioral Intent — Do respondents think they will take action as a result of seeing/hearing the message? For example, does an educational piece containing signs or indications of depression motivate the target audience to think more about this as an explanation for the behavior of someone they know?

The information from respondents’ reactions in each of these areas provides insight into how the audiences for these messages may react and how the messages should be revised to help ensure the most favorable reaction from the audience.

Other information gathered on respondents regarding gender, age, socioeconomic level, race/ethnicity, and applicable family history with substance abuse or mental health provides a basis for evaluating whether the messages may be perceived differently by different segments of the audience. For example, selected age groups may find a particular brochure or message on substance abuse more relevant than other age groups.

Systematic formative evaluation has been widely adopted by health education program planners as an integral step in the development of messages and materials. Through pretesting SAMHSA will be able to:

• Understand characteristics of the target audience— its attitudes, beliefs and behaviors — and use these in the development of effective communications tools;

• Produce messages that have the greatest potential to favorably influence target audience attitudes and behavior; and

• Expend limited program resource dollars wisely and effectively.

To obtain approval for a collection that meets the conditions of this generic clearance, a standardized form will be submitted to OMB along with supporting documentation (e.g., a copy of the comment card). The submission will have automatic approval, unless OMB identifies issues within 10 business days.

The types of collections that this generic clearance covers include, but are not limited to:

• Customer comment cards/complaint forms

• Small discussion groups

• Focus Groups of customers, potential customers, delivery partners, or other stakeholders

• Qualitative customer satisfaction surveys (e.g., post-transaction surveys; web surveys)

• In-person observation testing (e.g., campaign usability tests)

• Telephone interviews

• Social media or web-based polls

3. Use of Information Technology

The information will be collected using one‑to‑one interviews, focus groups, or self‑administered questionnaires, depending upon the target audience and the subject matter. Improved technology in the collection and processing of data will be used to reduce respondent burden and make processing maximally efficient. When telephone interviews are used, computer‑assisted telephone interviewing (CATI) will be employed whenever possible. Self‑administered questionnaires will be collected via online surveys. SAMHSA may also explore conducting focus groups virtually with treatment providers or consumers if appropriate.

4. Efforts to Identify Duplication

The general areas in which information needs to be gathered (as described in A.2 above — attention, comprehension, etc.) to pretest effective substance abuse and mental health messages (brochures, PSAs, media campaigns, etc.) will be generally similar from pretest to pretest. However, the specific questions that will be asked of respondents will differ with the message content, target audience, and medium of the message.

As each new message, strategy, or product is developed, SAMHSA reviews existing literature and databases, including pretesting reports on existing messages and materials, and consults with outside experts to evaluate available information in similar messages with comparable audiences. However, since the substance abuse and mental health fields are so diverse and complex, and each message is essentially different, new data collection instruments must be prepared for each pretest.

5. Involvement of Small Entities

Substance abuse and mental health care providers may sometimes be the target audience for SAMHSA information materials. When testing of these materials is required, SAMHSA will generally work through established professional organizations to gain access to the audience, and to obtain feedback on our instruments and data collection plans. As a result of this contact, SAMHSA will be able to minimize the placement of additional burden on treatment professionals.

6. Consequences If Information Collected Less Frequently

It is possible that a single communication product will be surveyed several times. For example, feedback may be collected at the concept phase, in advance of product development to help determine product need, as well as throughout the development process – message testing, product prototype testing, and testing after dissemination. The number of times a single communication product may be surveyed will vary based on type of product and production schedule.

7. Consistency with the Guidelines in 5 CFR 1320.5(d) (2)

SAMHSA recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, when SAMHSA requests responses to a self‑administered written questionnaire in a short time period, receipt of the questionnaire will be preceded by advance notification to respondents explaining the purpose of the questionnaire, the approximate length of time that the questionnaire will take, and the voluntary nature of participation. All efforts will be made to keep such questionnaires short and well-focused.

Because SAMHSA’s pretesting activities will be primarily qualitative in nature, the results will not be generalizable to the population at large or to the particular target audience under study. However, the nature of pretesting is such that generalizability is not a critical feature; the emphasis is on obtaining timely, useful information that can be fed back into the development of new messages or materials or the revision of existing ones.

There are no other special circumstances.

8. Consultation Outside the Agency

The 60-day notice soliciting public comment published in the Federal Register on January 12, 2022 (87 FR 1770). SAMHSA received no comments from the Notice. 

9. Remuneration of Respondents

There will be a need for nominal remuneration to focus group participants who are asked to leave their usual location and travel to a central location to compensate them for the time and inconvenience required. The level of remuneration is not expected to exceed $50 for participation in a focus group and, will depend on the amount of respondent time and expense projected for each focus group.

10. Assurance of Confidentiality

The protection of respondents’ identification and information will be assured to the maximum extent allowed by law. Participation will be voluntary and, to the extent possible, responses will be anonymous. In instances where respondent identity is needed (e.g., for follow-up of non-respondents, or for a longitudinal design), the information collection will fully comply with all aspects of the Privacy Act (System of Records 09-30-0036, Alcohol, Drug Abuse and Mental Health Epidemiologic and Biometric Research Data). Respondents will be assured that neither their participation/non-participation nor any responses to items will have any effect on their eligibility for or receipt of services.

11. Questions of a Sensitive Nature

As mentioned above, some studies require the inclusion of people who match selected characteristics of the target audience, which SAMHSA is trying to reach. This sometimes requires asking a question about race/ethnicity, income, education, and/or drug abuse or mental health problems on the initial screening questionnaire used for recruiting. Potential participants are informed that this is being done to make sure that SAMHSA speaks with the kinds of people for whom its messages are intended. Again, respondents will be assured that the information is voluntary and will be handled in a confidential manner.

Since SAMHSA communications are concerned with the treatment of mental health problems, some projects may involve asking questions about (or discussing) how one perceives his/her own personal risk for such problems. Fears of and experiences with mental illness may be covered. This information is needed to gain a better understanding of the target audience so that the messages, strategies and materials designed will be appropriate and sensitive. Questions of this nature require some sensitivity in how they are worded and approached. In face‑to‑face data collections, questions of this kind will generally be asked later in the interview or group discussion when respondents are more comfortable with the interview situation and are more at ease with the interviewer/moderator. As noted in Section A.10., participants will already have been informed at the beginning about the nature of the activity and the voluntary nature of their participation. The interviewer/moderator makes it clear that they do not have to respond to any question that makes them uncomfortable.

Raw data from data collections that include sensitive information (for example, screening questionnaires and audio tapes) will not be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.

12. Estimates of Annualized Hour Burden

The number of respondents to be included in each pretest will vary, depending on the nature of the material or message being tested and the target audience. However, the Annual Response Burden table that follows provides a likely annual distribution of respondents and hours by type of data collection. Time to read, view, or listen to the message being tested is built into the "hours per response" figures.

Activity	No. of Respondents	Responses/ Respondent	Hours per Response	Total Hours	Hourly Wage Rate ($)1	Total Hour Cost ($)
Individual In-depth Interviews: General Public Service Providers	400 200	1 1	.75 .75	300 150	$25.00 35.00	7,500 5,250
Focus Group Interviews: General Public Service Providers	3,000 1,500	1 1	1.5 1.5	4,500 2,250	25.0 035.00	112,500 78,750
Telephone Interviews: General Public Service Providers	335 165	1 1	.08 .08	27 13	25.00 35.00	675 455
Self-Administered Questionnaires: General Public Service Providers	2,680 1,320	1 1	.25 .25	670 330	25.00 35.00	16,750 11,550
Gatekeeper Reviews: General Public Service Providers	1,200 900	1 1	.50 .50	600 450	25.00 35.00	15,000 15,750
TOTAL	11,700	--	--	9,290	--	$264,180

¹ The hourly wage of $25.00 for the general public was calculated based on weighted data from the 2019 NSDUH respondents' personal annual income. The $35 hourly wage rate for providers is an average across counselors and other service provider staff.

13. Estimates of Annualized Cost Burden to Respondents

There are no capital or start‑up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.

14. Estimates of Annualized Cost to the Government

The total annual cost of message testing to the Federal Government will be approximately $363,500. This estimate is based on annual performance of up to: 225 focus groups (15 group sessions per each of 15 media campaigns at $1,100 each session); 7 telephone surveys at $4,000 each; 7 online self‑administered surveys at $7,000 each, and 3 in‑depth interview studies at $13,000 each. These figures include the costs of study design, facility rental (e.g., for focus groups), data collection, analysis, and report/publication writing.

In addition, there will be annual costs of approximately $49,680 associated with a GS 13 project officer monitoring, projected to be about 1,000 hours of effort a year at a SAMHSA personnel cost of $49.68 per hour.

15. Changes in Burden

There is no change in the burden.

16. Time Schedule, Publication and Analysis Plans

The data collection plan, schedule and analysis for each project will be determined by the objectives of the pretest, the messages being tested, and the audience for the message. Specifics of the analyses cannot be determined until the messages to be tested are prepared.

The process for developing the analytical plan for the pretest will be similar to that used in any formal evaluation. SAMHSA staff will review the material to be pretested, discuss the objectives with the individuals responsible for developing the materials, determine the analytic questions to be addressed in the pretest, and then after resolution of any problems, approve the pretest procedures, instruments, and data analysis plan.

Techniques will include primarily qualitative analyses (for example, content analysis for focus group interviews), although some results may be summarized quantitatively using descriptive statistics. No complex analytic techniques will be used.

While the primary purpose of a pretest is to provide information to the developers of the messages for improving them, SAMHSA may make pretest results available to substance abuse and mental health program planners in the states, voluntary organizations and others. Moreover, the Center for Substance Abuse Prevention may include some information from findings in publications that they prepare for substance abuse professionals and community entities when appropriate.

While the pretesting period will vary somewhat depending on the complexity of the testing and number of respondents required the typical pretest will require approximately 12 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical pretest would be:

Activity Time Schedule

Initial review of materials Week 1

Preparation of pretest design Week 2

Review of design Weeks 3

Preparation of OMB submission Weeks 4-5

Submission for OMB review Weeks 6-7

Collection of data Weeks 8 and 9

Analysis of data Weeks 10 and 11

Report on pretest Week 12

17. Display of Expiration Date

All data collection instruments will display the OMB control number and expiration date in the upper right corner of the cover page.

18. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.

9