Appendix 7 Campaign Materials Testing Plan

Appendix 7:

SAMHSA Underage Drinking Prevention National Media Campaign Materials Testing Plan Summary

This document outlines the plan for testing the SAMHSA’s “Talk. They Hear You.” Underage Drinking Prevention National Media Campaign (the “Campaign”). We will assess the Campaign through process measures to monitor activities, as well as outcome measures to measure changes in parent knowledge, attitudes, and behaviors that result from the Campaign.

Process Measures

National Campaign Implementation

The exact components of the Campaign process measures will depend on the Campaign tactics, which we will determine more conclusively after completing the next phase of testing message and materials. However, the testing likely will include the process measures listed below. We will define measurable objectives for each of our process measures prior to launching the Campaign.

Materials disseminated Partnerships developed Estimated reach of partnerships Number of training/ technical assistance "engagements" Number of attendees at training events Media placements Media impressions Message penetration in media reports PSA placements

Attendance at Campaign events Mentions of Campaign on other Web sites, blogs, etc. Links from other sites to Campaign content on the underagedrinking.samhsa.gov Web site Underagedrinking.samhsa.gov Web site metrics (the same as those that currently exist on the Web site and others as available, e.g., time spent on each page) SAMHSA Web 2.0 metrics (e.g., clicks and comments on Campaign content on SAMHSA Facebook, Twitter, YouTube accounts and Blog)

Pilot Site Implementation

We will test the implementation of the Campaign through the five pilot sites. These sites are required to conduct various activities to test the Campaign’s public service announcements, messages, and collateral materials, prior to a national launch in the fall of 2012. We will gather the following data:

• Local process measures;

• Evaluative feedback collected by pilot sites about their activities;

• Interviews with pilot site administrators; and

• Process case studies.

Outcome Measures

We do not expect the Campaign messages to reach a large percentage of U.S. parents; thus, we cannot rely on national surveys of random samples of parents to accurately measure awareness of the Campaign and possible resulting knowledge, attitude, and behavior changes. Therefore, we propose assessing the Campaign's effect on groups of parents we know have been exposed to it as a means of determining the Campaign’s effectiveness among those who see it.

Gallup Panel™ Study (Pre-Launch)

The Gallup Panel™ is a national probability panel of the U.S. population composed of more than 50,000 households and 70,000 individuals. We propose using the Gallup Panel to assess awareness of the Campaign and effectiveness of Campaign messages and materials. The Gallup Panel is an existing mechanism easily accessed by Gallup and requires minimum effort for recruitment, incentives, and other typically resource-heavy activities. This study will have both longitudinal and cross-sectional aspects.

Quantitative Pre-launch Pre-test

• Recruit a small sample of panel members (e.g., n=250) to provide feedback on finalized Campaign material prior to the national launch. Respondents will be asked to review specific Campaign materials and provide reactions to content, layout, and delivery.

Forced Exposure Longitudinal (Group A)

• Recruit large sample (e.g., n=2,500) of Gallup Panel parents of children ages 9 to 15. This sample will be split in half to determine those who will receive the campaign materials (“stimuli group”) and a representative “control group” who will not get the campaign materials.

• Administer a Web survey to establish a baseline of familiarity with the Campaign and its messages, and their current knowledge, attitudes, and behaviors related to underage drinking.

• Expose the parents in the “stimuli group” to selected Campaign materials (e.g., PSAs, materials, etc.).

• Conduct an immediate post-test to see if their knowledge, attitudes, and intended behaviors changed after this exposure.

• Mail the materials they saw in the testing and other Campaign materials for use at home.

• Re-contact survey participants from both the stimuli and control groups without notice at the 3-month (or less) interval after initial exposure to assess how their knowledge, attitudes, and behaviors changed over time.

Pilot Site Study (Pre-launch)

We propose also using the pilot sites to assess awareness of the Campaign and effectiveness of Campaign messages and materials. Each of the five pilot sites (one for every National Prevention Network region) is required to host an event as part of their pilot work, where Campaign information and materials will be distributed. We will capitalize on this natural opportunity to interview participants by traveling to each site to test the effectiveness of the Campaign materials distributed as well as finding out their reactions on the topic of underage drinking, overall.

Natural Exposure/Event Evaluation

• Administer pencil and paper survey of parents attending pilot site (n=500) and partnership events throughout the Campaign.

• Respondents will be asked to assess the event and the Campaign materials, and provide information on current knowledge, attitudes, and behaviors related to underage drinking.

• Three-month (or less) follow-up with respondents via a Web survey to measure change in awareness, knowledge, attitude and behavior.

Existing Survey Data (Post-launch and Ongoing)

We will track the findings of existing national surveys that include questions relevant to underage drinking, parenting, or both to keep informed of relevant trends to which the Campaign may be contributing. Surveys that may have relevant questions include the following:

• Behavioral Risk Factor Surveillance System

• Monitoring the Future

• National Longitudinal Study of Adolescent Health

• National Survey on Drug Use and Health

• Partnership Attitude Tracking Study

• Youth Risk Behavior Survey

Timeline

The pre-launch activities will begin May 2012 and continue through September 2012. The post-launch activities will begin the fall of 2012 after the Campaign has been launched and will run throughout the duration of the Campaign pending future funding of our contract by SAMHSA on this project.

3