0196 Ss 4-9-07

Pretesting of Substance Abuse Prevention and Treatment

and Mental Health Services Communications Messages

(OMB No. 0930-0196)

Supporting Statement

A. Justification

1. Circumstances of Information Collection

As the Federal agency responsible for developing and disseminating authoritative knowledge about substance abuse prevention, addiction treatment, and mental health services and for mobilizing consumer support and increasing public understanding to overcome the stigma attached to addiction and mental illness, the Substance Abuse and Mental Health Services Administration (SAMHSA) is responsible for development and dissemination of a wide range of education and information materials for various consumer publics and the professional communities. Authorization for SAMHSA’s education and information dissemination activities is contained in 42 USC 290aa, Section 501(d)(3) of the Public Health Service Act.

SAMHSA is requesting approval from the Office of Management and Budget (OMB) for an extension of the information, Pretesting of Substance Abuse Prevention and Treatment

and Mental Health Services Communications Messages. The current approval under OMB No. 0930-0196 expires on June 30, 2007. There are no changes to this submission.

Information programs within SAMHSA create and use a variety of media, including print (for example, brochures, posters, fact sheets, information kits), broadcast (for example, public service announcements and video news releases), electronic media (for example, SAMHSA’s web site) and direct response (the National Clearinghouse on Drug and Alcohol Information and the National Mental Health Information Center) to inform and educate the public and treatment professionals about substance abuse and mental health. Production of these materials is the major way that SAMHSA relays messages to the audiences it is mandated to reach.

To ensure that messages have the potential to be received, understood, and accepted by those for whom they are intended, SAMHSA employs formative evaluation. This type of evaluation involves (1) assessing audience knowledge, attitudes, behavior and other characteristics for the planning and development of messages, communication strategies and public information programs; and (2) testing these messages, strategies and program components in developmental form to assess audience comprehension, reactions and perceptions. Information obtained from testing can then be used to improve materials and strategies while revisions are still affordable and possible. This also maximizes the effectiveness of these messages and strategies, and reduces the frequency with which publications and programs need to be modified.

Because of SAMHSA’s leadership responsibilities in the substance abuse and mental health communities, it has a responsibility to provide credible information messages and services. If it is unable to empirically predict whether its materials and messages have the potential of being understandable and useful to their respective targets, it is unable to carry out its mandate effectively.

Formative (pretest) evaluation is an activity that SAMHSA should uniformly perform on all print pieces, broadcast products, and informational services and messages to maximize their usefulness. These pretests should be conducted on a small scale and focus on potential effectiveness with specific target audiences.

This submission is for an extension of a generic approval for up to 30 pretests per year using methods described in Section B with audiences targeted by messages to be developed by SAMHSA. The previous approval was not widely used, but there are currently plans for several new information programs that will be making use of this generic approval. The previous approval had the following Terms of Clearance:

This request for a generic clearance of pretesting materials is approved. Consistent with other generic clearances, HHS/SAMHSA will submit each individual pretest for OMB review, along with a brief description of the purpose of the pretest, sample design if applicable, any respondent incentives planned, and individual burden. OMB will either approve or provide comments on the individual request within 10 working days of receipt.

2. Purpose and Use of Information

Assessment is an integral part of SAMHSA’s overall plan for its communications strategies. Pretesting of print and broadcast products and information services assesses the potential effectiveness of these products while they are still in the developmental stage. Later on, analysis of other SAMHSA data collection mechanisms (such as the National Survey on Drug Use and Health and the National Health Interview Survey (NHIS) mental health component) enables SAMHSA to evaluate the eventual effectiveness of its substance abuse and mental health education programs.

The formative evaluation process is used to determine whether a draft message is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. These respondents are asked to give their reaction to the messages through either individual or group interviews. Respondents assess the materials in the following areas:

• Attention — Do the messages attract and/or hold the audience’s attention? For example, if they were to see a particular brochure in their doctor’s office, would they pick it up and look at it? That is, does it attract their attention?

• Comprehension — Is the message clearly understood, and are the main points conveyed? Does the main theme of the message get across to the audience? Is the language clear? Does the message convey the intended information in a manner that makes it effective for the target audience? For example, it is well‑known that marijuana use generally starts between ages 9‑14, a difficult demographic group to impact with conventional messages. Therefore, prior to finalizing the development of our communications messages for this group, we might convene focus groups of adolescents age 9‑14 to determine whether we are using a motif that is appealing, compelling, and effective.

• Personal Relevance and Self‑efficacy — Does the target audience perceive the message as personally relevant? For example, do the respondents see that the message in the public service announcement (PSA) applies to them ‑ that the information is important to them? Do the respondents see themselves as capable of acting on the message?

• Believability — Is the message and/or its source perceived as credible? For example, does the respondent believe the message in a PSA that talking to their children about not experimenting with drugs will decrease their chance of becoming a drug abuser? Is the person delivering the message credible?

• Acceptability — Is there anything in the message that may be offensive or unacceptable to the target audience? For example, does the respondent react negatively to a print message that warns of the dangers of using drugs? Is the piece culturally sensitive?

• Behavioral Intent — Do respondents think they will take action as a result of seeing/hearing the message? For example, does an educational piece containing signs or indications of depression motivate the target audience to think more about this as an explanation for the behavior of someone they know?

The information from respondents’ reactions in each of these areas provides insight into how the audiences for these messages may react and how the messages should be revised to help ensure the most favorable reaction from the audience.

Other information gathered on respondents regarding gender, age, socioeconomic level, race/ethnicity, and applicable family history with substance abuse or mental health provides a basis for evaluating whether the messages may be perceived differently by different segments of the audience. For example, selected age groups may find a particular brochure or message on substance abuse more relevant than other age groups.

Systematic formative evaluation has been widely adopted by health education program planners as an integral step in the development of messages and materials. Through pretesting SAMHSA will be able to:

• Understand characteristics of the target audience— its attitudes, beliefs and behaviors — and use these in the development of effective communications tools;

• Produce messages that have the greatest potential to favorably influence target audience attitudes and behavior; and

• Expend limited program resource dollars wisely and effectively.

3. Use of Information Technology

The information will be collected through the use of one‑to‑one interviews or self‑administered questionnaires, depending upon the target audience being questioned and the subject matter being addressed. Improved technology in the collection and processing of data will be used to reduce respondent burden and make processing maximally efficient. When telephone interviews are used, computer‑assisted telephone interviewing (CATI) will be employed whenever possible. For self‑administered questionnaires, closed‑ended questions (for example, multiple choice) and machine‑readable answer sheets will be used when feasible. When appropriate, data will also be collected via electronic mail or facsimile transmission.

As SAMHSA’s information sites on the World Wide Web continue to be developed and improved, SAMHSA may explore using either Website questionnaires or conducting on‑line focus groups with Internet users such as treatment providers or consumers with Internet access. Using computer‑assisted information technology to transmit data collection instruments and/or collect responses is expected to reduce the burden on respondents; for example, respondents may be able to access and respond to data collection requests at a time and place that is more convenient to them, eliminating the need to travel for in‑person or group interviews.

4. Efforts to Identify Duplication

The general areas in which information needs to be gathered (as described in A.2 above — attention, comprehension, etc.) to pretest effective substance abuse and mental health messages (brochures, PSAs, media campaigns, etc.) will be generally similar from pretest to pretest. However, the specific questions that will be asked of respondents will differ with the message content, audience targeted, and medium of the message.

As each new message, strategy, or product is developed, SAMHSA reviews existing literature and data bases, including pretesting reports on existing messages and materials, and consults with outside experts to evaluate available information in similar messages with comparable audiences. However, since the substance abuse and mental health fields are so diverse and complex, and each message is essentially different, new data collection instruments must be prepared for each pretest.

5. Involvement of Small Entities

Substance abuse and mental health care providers may sometimes be the target audience for SAMHSA information materials. When testing of these materials is required, SAMHSA will generally work through established professional organizations to gain access to the audience, and to obtain feedback on our instruments and data collection plans. As a result of this contact, SAMHSA will be able to minimize the placement of additional burden on treatment professionals.

6. Consequences If Information Collected Less Frequently

Information will be collected one time for each print or broadcast message, product or strategy tested. Respondents will not be re‑contacted.

7. Consistency With the Guidelines in 5 CFR 1320.5(d)(2)

SAMHSA recognizes the need to collect information in a manner that places minimal burden on each respondent. Therefore, when SAMHSA requests responses to a self‑administered written questionnaire in a short time period, receipt of the questionnaire will generally be preceded by advance notification to respondents explaining the purpose of the questionnaire, the approximate length of time that the questionnaire will take, and the voluntary nature of participation. All efforts will be made to keep such questionnaires short and well focused.

Because SAMHSA’s pretesting activities will be primarily qualitative in nature, the results will not be generalizable to the population at large or to the particular target audience under study. However, the nature of pretesting is such that generalizability is not a critical feature; the emphasis is on obtaining timely, useful information that can be fed back into the development of new messages or materials or the revision of existing ones.

There are no other special circumstances.

8. Consultation Outside the Agency

The 60-day notice soliciting public comment which appeared in the Federal Register on March 22, 2007 (72 FR 13505). No public comments were received in response to that notice.

Each project may have an ad hoc expert panel of researchers and/or practitioners chosen for their background and knowledge in the content area of the specific project. These consultants from outside the agency would provide direction and feedback to the project in all phases of development, including pretesting.

9. Payment to Respondents

It is standard practice in commercial market research to offer recruited respondents an incentive to help assure their participation in qualitative research. A small fee not to exceed $50 for members of the general public and $100 for providers, a free meal or snack scheduled around the time of the pretest, or remuneration for parking and/or transportation are most often used in recruiting hard‑to‑reach and minority respondents from the public and will be used in this project.

Market research literature suggests that monetary incentives have a strong positive effect on the response rate and no known adverse effect on reliability. Circumstances, however, do not always require that remuneration be given. Service providers, for example, often participate gratis because of their interest or involvement in the topic, or as a professional courtesy.

10. Assurance of Confidentiality

Information provided by respondents will be treated in confidence and kept private, except as otherwise required by law. This will be communicated to respondents by means of introductory letters, explanatory texts on the cover pages of questionnaires, scripts read prior to focus groups or telephone interviews, and/or consent forms. Prior to any data collection, respondents will be advised of the following: the nature of the activity; the purpose and use of the data collected; SAMHSA sponsorship; and the fact that participation is voluntary at all times. Since responses are voluntary, respondents will be assured there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.

As a further guarantee of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals being preserved. Reports will only be used by the project staff for research purposes and for the development of specific communication messages and educational materials.

Although some personal information will be collected, data will not be retrieved by personal identifiers, and thus the Privacy Act does not apply to these activities.

11. Questions of a Sensitive Nature

As mentioned above, some studies require the inclusion of people who match selected characteristics of the target audience, which SAMHSA is trying to reach. This sometimes requires asking a question about race/ethnicity, income, education, and/or drug abuse or mental health problems on the initial screening questionnaire used for recruiting. Potential participants are informed that this is being done to make sure that SAMHSA speaks with the kinds of people for whom its messages are intended. Again, respondents will be assured that the information is voluntary and will be handled in a confidential manner.

Since SAMHSA communications are concerned with the treatment of mental health problems, some projects may involve asking questions about (or discussing) how one perceives his/her own personal risk for such problems. Fears of and experiences with mental illness may also be covered. This information is needed to gain a better understanding of the target audience so that the messages, strategies and materials designed will be appropriate and sensitive. Questions of this nature, while not as personal as those on sexual behavior or religious beliefs, still require some sensitivity in how they are worded and approached. In face‑to‑face data collections, questions of this kind will generally be asked later in the interview or group discussion when respondents are more comfortable with the interview situation and are more at ease with the interviewer/moderator. As noted in Section A.10., participants will already have been informed at the beginning about the nature of the activity and the voluntary nature of their participation. The interviewer/moderator makes it clear that they do not have to respond to any question that makes them uncomfortable.

Raw data from data collections that include sensitive information (for example, screening questionnaires and audio tapes) will not be retained once the data has been extracted and aggregated; nor will the information become part of a system of records containing permanent identifiers that can be used for retrieval.

12. Estimates of Annualized Hour Burden

The number of respondents to be included in each pretest will vary, depending on the nature of the material or message being tested and the target audience. However, the Annual Response Burden table that follows provides a likely annual distribution of respondents and hours by type of data collection. Time to read, view, or listen to the message being tested is built into the "hours per response" figures.

Activity	No. of Respondents	Responses/ Respondent	Hours per Response	Total Hours	Hourly Wage Rate ($)1	Total Hour Cost ($)
Individual In-depth Interviews: General Public Service Providers	400 200	1 1	.75 .75	300 150	$13.99 25.00	$4,197 3,750
Focus Group Interviews: General Public Service Providers	3,000 1,500	1 1	1.5 1.5	4,500 2,250	13.99 25.00	62,955 56,250
Telephone Interviews: General Public Service Providers	335 165	1 1	.08 .08	27 13	13.99 25.00	378 325
Self-Administered Questionnaires: General Public Service Providers	2,680 1,320	1 1	.25 .25	670 330	13.99 25.00	9,373 8,250
Gatekeeper Reviews: General Public Service Providers	1,200 900	1 1	.50 .50	600 450	13.99 25.00	8,394 11,250
TOTAL	11,700	--	--	9,290	--	$165,122

¹ The hourly wage of $13.99 for the general public was calculated based on weighted data from the 2005 NSDUH respondents' personal annual income. The $25 hourly wage rate for providers is an average across counselors and other service provider staff.

² Brief interviews with callers to SAMHSA’s public information centers to test message concepts and strategies following their call-in request to the 1-800-number.

13. Estimates of Annualized Cost Burden to Respondents

There are no capital or start‑up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.

14. Estimates of Annualized Cost to the Government

The total annual cost to the Federal Government will be approximately $222,750. This estimate is based on annual performance of up to: 225 focus groups (15 group sessions per each of 15 media campaigns at $550 each session); 7 telephone surveys at $3,000 each; 7 mail self‑administered surveys at $6,000 each, and 3 in‑depth interview studies at $12,000 each. These figures include the costs of study design, facility rental (e.g., for focus groups), data collection, analysis, and report/publication writing; the total cost of these activities is approximately $305,000.

In addition, there will be annual costs of approximately $31,410 associated with project officer monitoring, projected to be about 1,000 hours of effort a year at a SAMHSA personnel cost of $31.41 per hour.

15. Changes in Burden

There is no change in the burden.

16. Time Schedule, Publication and Analysis Plans

The data collection plan, schedule and analysis for each project will be determined by the objectives of the pretest, the messages being tested, and the audience for the message. Specifics of the analyses cannot be determined until the messages to be tested are prepared.

The process for developing the analytical plan for the pretest will be similar to that used in any formal evaluation. SAMHSA staff will review the material to be pretested, discuss the objectives with the individuals responsible for developing the materials, determine the analytic questions to be addressed in the pretest, and then after resolution of any problems, approve the pretest procedures, instruments, and data analysis plan.

Techniques will include primarily qualitative analyses (for example, content analysis for focus group interviews), although some results may be summarized quantitatively using descriptive statistics. No complex analytic techniques will be used.

While the primary purpose of a pretest is to provide information to the developers of the messages for the purpose of improving them, SAMHSA may make pretest results available to substance abuse and mental health program planners in the states, voluntary organizations and others. Moreover, the Center for Substance Abuse Prevention may include some information from findings in publications that they prepare for substance abuse professionals and community entities when appropriate.

While the pretesting period will vary somewhat depending on the complexity of the testing and number of respondents required the typical pretest will require approximately 14 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical pretest would be:

Activity Time Schedule

Initial review of materials Week 1

Preparation of pretest design Week 2

Review of design Weeks 3

Preparation of OMB submission Weeks 4-5

Submission for OMB review Weeks 6-9

Collection of data Weeks 10 and 11

Analysis of data Weeks 12 and 13

Report on pretest Week 14

17. Display of Expiration Date

All data collection instruments will display the OMB control number and expiration date in the upper right corner of the cover page.

18. Exceptions to Certification Statement

This activity will fully comply with the Certification for Paperwork Reduction Act Submissions.

B. Statistical Methods

1‑2 Respondent Universe and Sampling Methods/Information Collection Procedures

Pretesting includes a variety of methods and approaches. The methods chosen for use depend on the nature of the message or materials tested, as well as their intended target audience. A few samples of the types of materials to be pretested are included as Attachment C. Recommended methodologies and sample sizes will be based on a review of the relevant literature, consultation with experts in the field.

In general, pretesting relies on qualitative methods and is not intended to yield results that are statistically projectable. In qualitative studies, quota sampling is used to select a sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience. Response rate is not applicable to quota sampling because this type of sampling results in a non‑probability sample which is not representative of the population. In qualitative studies, all respondents will be initially contacted by telephone or through the mail; over‑recruiting is done to compensate for not following up with non‑respondents.

Questions in all pretesting methodologies will include the following:

• Standard measures of communications that are designed to assess to what degree the message was successful in communicating information. These questions include measures of main idea recall, comprehension, believability, personal relevance, and likes and dislikes.

• Questions tailored for the specific print piece/public service message to assess audience comprehension of specific information or to address any special concerns the producer of the message may have — for example, are the illustrations appropriate?

The methodologies planned for use in this pretesting submission represent the standard state‑of‑the art approaches adapted from marketing and communications research. In this context, the term pretesting refers to testing messages, strategies and communication materials, and should not be confused with "pre‑testing of questionnaires" prior to their full‑scale use. The following methodologies will be used:

Focus Groups. Focus groups, or group interviews, will be used to obtain insights into target audience perceptions beliefs and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials development). Focus groups will be usually composed of 8‑10 people who have characteristics similar to the intended target audience. The groups will be conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discussion unfolds and new topics emerge. Focus groups are valuable in exploring consumer reactions to message concepts before additional resources are put into their development.

Individual In‑depth Interviews. Individual in‑depth interviews will be used for pretesting PSA concepts, draft materials, and communication strategies when the information in question requires in‑depth probing or when individual rather than group responses are considered more appropriate. This methodology is appropriate for determining target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics. In‑depth interviews are also cost‑effective in eliciting comments on print materials. Respondents can be sent a copy of the material in advance, asked to read it, and told that someone will call to get their opinion. At the designated time, the recipient of the material is called and interviewed.

Self‑Administered Questionnaires. Self‑administered questionnaires will be used to pretest drafts of concepts and materials. They can either be mailed to respondents along with the draft being pretested or distributed to respondents gathered at a central location. When the latter method is employed, people frequenting a central location representative of the target audience may be randomly stopped and screened to determine whether they meet the pre‑determined selection criteria.

Gatekeeper Review. Because public and patient education materials are often distributed to their intended target audiences through health professionals or other intermediary organizations, the input of these groups to the concepts and materials is critical to the program’s success. As a result, these intermediaries, or gatekeepers, are often queried as part of the pretesting process. The information included in the questionnaire is complementary to that requested of target audience members.

3. Methods to Maximize Response Rates

Typically, quota sampling is used in the developmental activities which comprise the bulk of effort under this approval. With such non‑probability sampling, response rates per se are not as important as assuring that focus groups and one‑on‑one interviews reflect the input of representatives of key consumer or provider populations (e.g., age, race).

4. Tests of Procedures

The activities to be conducted under this approval are all pretests. Interview guides and questionnaires to be used in the pretests will all be carefully developed and given careful scrutiny and limited, informal testing to assure completeness and smooth flow.

5. Statistical Consultants

Activities conducted under this approval will be under the overall supervision of the Associate Administrator for Communications, Mark Weber (240‑276-2130). Most pretesting activities will be carried out by contractors. Following is a list of the current contractors and associated program staff who will be responsible for these activities. Each contract project director will have access to the services of the corporate statistician and will be responsible for statistical aspects as well as programmatic aspects (i.e., data collection and analysis) for each pretest.

Message & Material Development Support

11420 Rockville Pike

Rockville, Maryland 120852

Contract Officer: James Whitt – 240.276.1503

SAMHSA Project Officer: Mark Weber – 240.276.2130

Vanguard Communications

2121 “K” Street, NW

Suite 300

Washington, D.C. 20037

202-331-4323

Contract officer: James Whitt – 240.276.1503

SAMHSA Project Officer: Lisa Rubenstein 240.276.1927

List of Attachments

1. Samples of the types of materials to be pretested

Attachment A

Samples of the types of materials to be pretested