“Talk. They Hear You.”® Campaign Evaluation: Case Study
Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA) Center for Substance Abuse Prevention (CSAP) is requesting approval from the Office of Management and Budget (OMB) for an extension of the data collection for the “Talk. They Hear You.”® Campaign Evaluation: Case Study (the “case study”). This data collection is approved under OMB No. 0930-0373, expiring on May 31, 2020, and includes three instruments:
Parent/Caregiver Pre-Test/Post-Test Survey
Youth Pre-Test and Post-Test Survey
Parent/Caregiver Interview Guide
The case study data collection is part of a larger effort to evaluate the impact of the “Talk. They Hear You.” campaign. The overall evaluation includes a parent questionnaire that is currently being executed under a separate OMB control number.
Despite laws prohibiting the consumption of alcohol by individuals under 21 years of age in all 50 states, underage drinking in the United States remains a serious public health and safety problem, undermining the well-being of America’s youths. Alcohol continues to be the most widely misused substance among America’s youths, with an estimated 7.4 million people younger than the age of 21 drinking alcohol in the past month.
Alcohol use is responsible for approximately 4,300 deaths annually among youths under age 21 in the United States, shortening their lives by an average of 60 years.1 Underage drinking also contributes to a wide range of costly health and social problems, including motor vehicle crashes (the greatest single mortality risk for underage drinkers); suicide; interpersonal violence (including homicides and sexual and other assaults); unintentional injuries (such as burns, falls, and drownings); cognitive impairment; alcohol use disorders; risky sexual activity; poor school performance; and alcohol and other drug overdoses. Additionally, binge drinking (i.e., consuming four drinks in a row for a female or five for a male) exacerbates underage drinking’s harmful consequences and increases with age: by age 20, 29 percent of young people report binge drinking at least once in the past month (Center for Behavioral Health Statistics and Quality 2018a).
Parental attitudes toward drinking, as well as parental communication, have a significant influence on adolescent alcohol use, particularly among younger adolescents.2,3 Through a direct association with adolescent perceptions and cognition, parental beliefs and communication about the acceptability of underage drinking may positively impact use of alcohol by youths.4 This research suggests that parental interaction with youths regarding underage drinking may provide a unique opportunity for early intervention and prevention.
In December 2006, Congress passed the Sober Truth on Preventing Underage Drinking Act (STOP Act), the first comprehensive national legislation addressing the problem of underage drinking. The act mandates a coordinated, multifaceted approach to addressing the prevention and reduction of underage drinking through the modification in enforcement of drinking laws, reduction of alcohol availability to adolescents, increased research and resources for state and local efforts, and an adult-oriented media campaign. Under section 2(d), the STOP Act directs the Secretary of the U.S. Department of Health and Human Services to fund and oversee a national, adult-oriented media public service campaign and to report annually on the production, broadcasting, and evaluation of this campaign.
In response to the directives set forth in the STOP Act of 2006, SAMHSA CSAP created the Underage Drinking Prevention National Media Campaign titled “Talk. They Hear You.” The campaign was originally designed to educate and empower parents and caregivers to talk with children about alcohol. To prevent initiation of underage drinking, the campaign targeted parents and caregivers of children aged 9–15. However, through separate funding and strategic expansion, “Talk. They Hear You.” now targets parents and caregivers of children under the age of 21 and supports SAMHSA’s efforts to reduce both underage drinking and substance use by aiming to:
Increase parents’ awareness of the prevalence and risk of underage drinking and substance use;
Equip parents with the knowledge, skills, and confidence to prevent underage drinking and substance use; and
Increase parents’ actions to prevent underage drinking and substance use.
The “Talk. They Hear You.” campaign adds to parents’ and caregivers’ current knowledge base about underage drinking and substance use prevention, and empowers them to address these issues by increasing their level of comfort with these topics and encouraging open communication between them and their children. During campaign development, parents/caregivers, youths, and stakeholders provided feedback on all aspects of concept and message development. In 2012, SAMHSA conducted a national pilot program to test and refine campaign creative materials and objectives. Then, in 2013, SAMHSA launched the “Talk. They Hear You.” campaign designed to help parents and caregivers talk to children aged 9–15 about the consequences of underage drinking and introduce skills to help their children avoid situations that result in these behaviors. In 2017, the campaign received separate funding to expand its content to include information to help prepare parents and caregivers to talk to their children about alcohol and other drugs, including prescription pain medications and marijuana. Then, in 2018, the campaign expanded its age range beyond 9–15 to include resources for parents and caregivers of children under the age of 21. This is a national campaign with an ongoing rollout period.
The “Talk. They Hear You.” materials consist of public service announcements, infographics, and a mobile application. These materials show parents and caregivers using everyday opportunities to talk with their children about alcohol and other substances, and reinforce the importance of starting the prevention conversation at an early age. They were created and provided to partners to display and distribute to parents/caregivers and community members. The partners, in turn, generated feedback on campaign successes, such as reach and volume of social media and blog posts, newsletter articles, and direct distribution of campaign materials.
In addition to developing and disseminating the “Talk. They Hear You.” campaign components, SAMHSA is also charged with evaluating the campaign. While message testing during development indicated the potential impact of the materials, and media measures from partner distribution of materials indicate the reach of the campaign, there also needs to be an evaluation of the impact of the campaign as it has been implemented in the general population. SAMHSA’s evaluation design consists of a two-part evaluation mechanism—a parent questionnaire and the supplemental case study. The annual questionnaire of parents/caregivers of 9- to 20-year-old children will examine how campaign products are viewed by our target audience, as well as the potential impact they may have on parents’ and caregivers’ attitudes and/or behavior regarding talking to their children about underage drinking and other substance use after seeing the materials.
To supplement findings from the parent questionnaire effort, SAMHSA intends to conduct a forced-exposure case study in a selected school site that currently has limited campaign exposure. This is the current focus of this OMB application package. Because we intend to measure knowledge and attitudes before and after a focused campaign outreach effort in an area that has not had significant exposure to the campaign, this component of the evaluation requires a pre-/post-test study design. The case study will include baseline surveys of both parents/caregivers and children aged 9–15, followed by exposure to campaign materials, and post-exposure surveys of both parents/caregivers and children. SAMHSA will additionally conduct interviews with parents and caregivers following the post-exposure survey to obtain more detailed information about the impact of the campaign.
The unit of analysis for the case study will be a middle school community, which will involve students; parents/caregivers; school administrators; and partnering organization(s), such as parent/caregiver organizations and/or local prevention organizations. There will be two sites selected for the case study—one site will serve as the intervention group, and the other site will serve as the control group. The intervention group will be exposed to the “Talk. They Hear You.” campaign materials using standard campaign materials and dissemination strategies, which will be coordinated directly with the participating school site as well as any local partner organizations recruited for this purpose. The control group will not be intentionally exposed to the campaign materials.
In total, there will be three data collection instruments used in the case study:
Parent/Caregiver Pre-Test/Post-Test Survey—Pre- and post-test survey for use with parents and caregivers of children who are 9–15 years of age. This survey will include questions about respondents’ attitudes and behaviors regarding underage drinking and awareness of the “Talk. They Hear You.” campaign. This survey will be administered electronically and on paper.
Youth Pre-Test and Post-Test Survey—Pre- and post-test survey for use with youths who are in 6th through 8th grade. This survey will include questions about respondents’ attitudes and behaviors regarding underage drinking. This survey will be administered on paper.
Parent/Caregiver Interview Guide—Interview guide for use with parents and caregivers of children who are 9–15 years of age. Parents and caregivers who respond to the post-test survey, described above, will be invited to participate in an individual follow-up interview to discuss, in greater detail, the effect—if any—of the “Talk. They Hear You.” campaign on their attitudes and behaviors regarding underage drinking. The individual interviews will be conducted over the phone.
The purpose of the case study component of the campaign evaluation is twofold:
To identify evidence regarding the relationship between campaign exposure and changes in parents’ and caregivers’ attitudes and behavior regarding underage drinking in the selected sites; and
To collect information on whether changes in parent/caregiver attitudes and/or behaviors identified align with measurable changes in youth attitudes and/or behaviors on underage drinking during the same period.
Whereas the parent questionnaire data will provide an evaluation of overall campaign materials, this case study will explore details of if and how exposure to the “Talk. They Hear You.” campaign affects adult and student attitudes and behaviors in the selected district. Together, the results from both components of the evaluation—the parent questionnaire and the case study—will allow an estimation of overall campaign impact in reaching parents and caregivers of 9- to 20-year-old youths and, ultimately, in preventing and reducing underage drinking. This campaign evaluation will additionally satisfy section 2(d) of the STOP Act, which requires the national, adult-oriented media public service campaign to be evaluated.
The results of the campaign evaluation—as well as progress on implementation of the evaluation—will be reported in an annual report to Congress. Section 2(d) of the STOP Act requires the Secretary of the U.S. Department of Health and Human Services to annually report on the production, broadcasting, and evaluation of the “Talk. They Hear You.” campaign. Since the STOP Act went into effect, SAMHSA has reported annually on the progress of the campaign and campaign evaluation in fulfillment of this act. SAMHSA will continue to document the progress of the campaign evaluation—and the case study findings, in particular—in future reports to Congress in fulfillment of this law.
SAMHSA will disseminate one of the three case study instruments—the parent and caregiver pre- and post-test survey—electronically. The parent and caregiver pre-test and post-test survey will also be available on paper to accommodate the needs and preferences of the parent/caregiver populations at both the intervention site and the control site. The electronic surveys will be disseminated via email, and the paper surveys will be sent home with students by the case study middle school sites.
SAMHSA will use the Qualtrics© Survey Suite to develop and disseminate the electronic version of the survey. The use of the Qualtrics© Survey Suite will enable SAMHSA to use branching logic in the electronic survey that provides respondents with different questions based on their responses to previous questions (and effectively enables respondents to skip over the questions that do not apply to them). Such branching logic will reduce the average time of the survey for respondents, thereby reducing overall burden. The use of Qualtrics© also provides SAMHSA with the ability to randomize response options in order to mitigate response option bias. Qualtrics© also includes built-in Section 508 compliance systems to accommodate respondents with vision impairments and built-in mobile accessibility to accommodate respondents using a range of electronic devices. SAMHSA owns and controls all data collected using Qualtrics©, and the data maintained in the Qualtrics© database will be kept confidential. Qualtrics© uses industry best practices to maintain data security and privacy, and has devised proprietary methods to prevent disclosing data to the wrong requester due to programming errors.
The STOP Act required the development and dissemination of a new, adult-oriented media public service campaign and an evaluation of that campaign. The data collection efforts outlined in this document are specific to the campaign evaluation and are not duplicated elsewhere.
This data collection will involve the participation of up to two school districts. Involvement will be voluntary and will align with their respective missions to prevent the use of alcohol among youths. Further, the summary information from the data collected will be useful to the districts that will be involved in the data collection.
SAMHSA plans to conduct the case study every 3 years.
This information collection fully complies with 5 CFR 1320.5(d)(2).
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on November 29, 2019 (84 FR 65829). SAMHSA did not receive any comments.
The Interagency Coordinating Committee to Prevent Underage Drinking (ICCPUD) coordinates federal efforts to reduce underage drinking, guide policy and program development across the federal government as well as serve as a resource for the development of the Comprehensive Plan for Preventing Underage Drinking. ICCPUD has received input from experts and organizations representing a wide range of parties, including public health advocacy groups, the alcohol industry, ICCPUD member agencies, U.S. Congress, and subject matter experts, which has been instrumental in the development of the national media campaign—TTHY.
The ICCPUD will continue to guide the development process of the national media campaign, TTHY, over the period of this contract. The annual evaluation regarding the implementation and outcomes of “Talk. They Hear You.” is provided in Chapter 5, of the “Report to Congress on the Prevention and Reduction of Underage Drinking” and can be found at www.stopalcoholabuse.gov.
No payments will be made to respondents for completing the surveys.
Payments will be made to participants for completing the individual parent/caregiver interviews, in the form of gift cards, at a rate of $35.00 per hour.
Both email and paper distributions will be controlled by the schools, and no email or student contact information will be shared by the schools with SAMHSA. Only information provided voluntarily by the respondents will be collected by the system as outlined here.
Contact information from a subset of the adults participating in the survey is necessary to identify participants for the interview portion of the study. The parent/caregiver post-test survey includes a question prompting respondents to share contact information—first name and a phone number or email address—if they would be willing to be contacted for a follow-up interview. The survey requests this contact information with the following instructions:
Would you be willing to provide more information on your responses to this survey in a follow-up interview on the topic of underage drinking? Interviews will take no more than 1 hour and will be conducted by phone. For your participation, you will receive a $35 gift card from a local store. If you are interested in learning more about the interview or scheduling a time to participate, please share your first name and either a phone number or email address below so we may contact you.
Your name and contact information will NOT BE IDENTIFIED in any reports that result from this survey.
The sole purpose of requesting this contact information is to identify subjects willing to participate in the interview component of the case study; no names will appear in the Report to Congress on the Prevention and Reduction of Underage Drinking or any other report.
Survey data will be stored in password-protected, encrypted files. Access to these files will be limited to the data analyst and supervisor. Upon completion of data collection through the parent/caregiver post-test survey and the follow-up interviews, any identifying information for participants will be purged from the data files.
Our procedures ensure that students can opt out from the study, should they desire. Before administering the student survey, the teacher from each participating classroom will read the following script: “Shortly, I will be passing out a brief survey. This survey is part of a larger initiative to address underage drinking prevention. The questions are anonymous, so please do not put your name on this survey. Your participation is voluntary. Therefore, if for any reason, before, during, or upon survey completion, you choose to withdraw from the study, your answers will not be included. If you have any questions or concerns about the survey or find that the survey topic triggers feelings or concerns, please let me know and I will connect you with appropriate supports.”
Questions of a sensitive nature for each of the data collection instruments are detailed in the table below:
Table 1: Questions of a Sensitive Nature
Instrument |
Questions of a Sensitive Nature |
Pre- and post-test survey for use with parents and caregivers of children who are 9–15 years of age. |
Questions will include only those related to awareness of the “Talk. They Hear You.” campaign as well as parent/caregiver attitudes and behaviors regarding underage drinking and, specifically, speaking with their children about alcohol use. No questions relating to actual alcohol use, sexual behaviors, religious beliefs, or other matters commonly considered private or of a sensitive nature are asked. |
Pre- and post-test survey for use with youths who are in 6th through 8th grade. |
This anonymous survey includes three questions related to alcohol use and another three questions about attitudes regarding alcohol use. The questions about behaviors and attitudes associated with alcohol use are not specific. Instead, respondents are prompted to select categories of alcohol use behaviors and attitudes. These six questions have been used previously in the Monitoring the Future survey, which is funded by the National Institute on Drug Abuse, a part of the National Institutes of Health. The collection of this information about alcohol use behaviors and attitudes is necessary to investigate whether a relationship exists between changes in parent/caregiver attitudes and behaviors as a result of the “Talk. They Hear You.” campaign and changes in youth behavior regarding alcohol use attitudes and behaviors. Such a relationship will be one of the measures of the campaign evaluation. |
Interview guide for use with parents and caregivers of children who are 9–15 years of age. |
Interview questions will include topics related to awareness of the “Talk. They Hear You.” campaign as well as parent/caregiver attitudes and behaviors regarding underage drinking and, specifically, speaking with their children about alcohol use. There is one question on whether they believe their underage children have tried alcohol, which is designed to gauge awareness of the problem of underage drinking. Parents and caregivers often underestimate youth access to alcohol. No questions relating to specific alcohol use among the respondents or their children, sexual behaviors, religious beliefs, or other matters commonly considered private or of a sensitive nature are asked. |
Table 2 indicates the estimated total annual burden on each participant for each data collection activity associated with the “Talk. They Hear You.” campaign evaluation case study. Table 2 was calculated based on the following assumptions:
There are 575 students, on average, in U.S. middle schools.5 There will be two middle school sites included in the case study (one control site and one intervention site), so SAMHSA estimates approximately 1,150 middle school students to be respondents for the pre- and post-test surveys for youths. Since the survey for youths will be administered during class, SAMHSA estimates a 95 percent response rate (approximately 1,093 students). SAMHSA assumes that the pre-test survey and post-test survey will each take approximately 10 minutes to complete. This estimate includes time for distributing the survey, reviewing the instructions, taking the survey, and turning in the completed survey.
SAMHSA estimates one parent/caregiver per student as the base population for potential respondents for the parent/caregiver pre-test and post-test surveys—or 575 parents/caregivers per site. Across both sites—the control and intervention sites—SAMHSA estimates a total of 1,150 parents/caregivers as the base population. As the parent/caregiver pre-test and post-test surveys will be administered by the school administration, SAMHSA estimates a response rate of approximately 60 percent, which is in line with average response rates found in a search for recent surveys administered to parents and caregivers by schools,6,7,8 as well as a 2008 meta-analysis of response rates of individuals within organizations participating in evaluation.9 SAMHSA assumes that the pre-test survey and post-test survey will each take approximately 10 minutes to complete. This estimate includes time for distributing the survey, reviewing the instructions, taking the survey, and turning in the completed survey.
SAMHSA aims to conduct hour-long individual interviews with approximately 30 parents and caregivers following the post-test survey. Participants in the interview will be compensated approximately $35.00 per hour for their participation.
Table 2: Estimated Burden for Respondents
Instrument |
Total number of respondents |
Total responses/ respondent |
Total responses |
Hours per response |
Total hour burden |
Wage rate |
Total hour cost |
Pre-test survey for 6th–8th youths |
1,093 |
1 |
1,093 |
0.17 |
185.8 |
$4.25* |
$789.65 |
Post-test survey for 6th- through 8th-grade youths |
1,093 |
1 |
1,093 |
0.17 |
185.8 |
$4.25* |
$789.65 |
Pre-test survey for parents and caregivers |
690 |
1 |
690 |
0.17 |
117.3 |
$7.25** |
$850.42 |
Post-test survey for parents and caregivers |
690 |
1 |
690 |
0.17 |
117.3 |
$7.25** |
$850.42 |
Individual interviews with parents and caregivers |
30 |
1 |
30 |
1 |
30 |
$35.00 |
$1,050.00 |
Total |
1,783 |
|
3,596 |
|
636.2 |
|
$4,330.15 |
*Federal Youth Minimum Wage Program (http://webapps.dol.gov/elaws/whd/flsa/docs/ymwplink.asp)
**Federal minimum wage per Fair Minimum Wage Act of 2007 (http://webapps.dol.gov/elaws/faq/esa/flsa/001.htm)
There will be no capital, startup, operation, or maintenance of services costs to respondents.
The estimated cost to the government for the data collection is $67,212. This includes approximately $65,000 for a 1-year contract for data collection, data analysis, reports, etc., and approximately $2,212 per year represents SAMHSA costs to manage/administer the survey for 2 percent of one employee (GS-13). The total annualized cost is approximately $67,212.
There is no burden change.
Date |
Case Study Activity |
December 2019–March 2020 |
SAMHSA recruits sites and partner organizations following OMB approval. |
October 2020 |
SAMHSA conducts parent/caregiver and youth pre-tests. |
November 2020–March 2021 |
SAMHSA works with partner organizations to expose intervention site to campaign materials. |
April 2021 |
SAMHSA conducts parent/caregiver and youth post-tests. |
May 2021–June 2021 |
SAMHSA conducts parent/caregiver individual interviews. |
July 2021 |
SAMHSA writes internal report on case study findings. |
December 2021 |
SAMHSA reports findings in the 2021 Report to Congress on the Prevention and Reduction of Underage Drinking. |
The analysis plan for the case study is designed to address the following questions:
Is there a relationship between campaign exposure and changes in parent/caregiver attitudes and behavior about underage drinking at the selected sites?
What are the effects of campaign exposure on parents/caregivers in the selected sites?
Do individual interview data from the school sites suggest a cause–effect relationship between campaign exposure and changes in parent/caregiver attitudes and behaviors in the intervention school district?
Is there a relationship between parents’/caregivers’ exposure to campaign materials with youth behavior/attitudes about drinking in the intervention school district?
Do any observed changes in overall parent/caregiver behavior (between the pre-test and the post-test) align with any observed changes in overall youth behavior (between the pre-test and the post-test) in the selected school districts?
Data collected in the case study will be both numerical and descriptive. SAMHSA will utilize a variety of strategies to analyze these data and will produce a final case study report that integrates the results from each data collection instrument.
The findings obtained from the case study will be included in Chapter 5 of the annual Report to Congress on the Prevention and Reduction of Underage Drinking.
The expiration date will be displayed.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
1 Stahre, M., Roeber, J., Kanny, D., Brewer, R. D., & Zhang, X. (2014). Contribution of excessive alcohol consumption to deaths and years of potential life lost in the United States. Preventing Chronic Disease. June 26, 2014;11:E109. doi: 10.5888/pcd11.130293.
2 Ennet, S. T., Bauman, K. E., Foshee, V. A., Pemberton, M., & Hicks, K. A. (2001). Parent-child communication about adolescent tobacco and alcohol use: What do parents say and does it affect youth behavior? Journal of Marriage and Family 63, 48–62.
3 Wood, M. D., Read, J. P., Mitchell, R. E., & Brand, N. H. (2004). Do parents still matter? Parent and peer influences on alcohol involvement among recent high school graduates. Psychology of Addictive Behaviors, 18(1), 19–30.
4 Sieving, R. E., Maruyama, G., Williams, C. L., & Perry, C. L. (2000). Pathways to adolescent alcohol use: Potential mechanisms of parent influence. Journal of Research on Adolescence, 10(4), 489–514.
9https://www.researchgate.net/profile/Yehuda_Baruch2/publication/228079609_Survey_Response_Rate_Levels_and_Trends_in_Organizational_Research/links/09e415112cd948d7f6000000.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Samantha Spinney |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |