Ss_b_01 23 2013

Evaluation of the National Tobacco Prevention and Control Public Education Campaign

OMB No. 0920-0923

Revision

Revised Supporting Statement: Part B

Centers for Disease Control and Prevention

National Center for Chronic Disease Prevention and Health Promotion

Office of Smoking and Health

Health Communications Branch

January 23, 2013

Project Officer: Robert Alexander, Ph.D.

Health Communications Specialist

Phone: (770) 488-1212

Fax: (770) 488-5939

Email address: [email protected]

Centers for Disease Control and Prevention

4770 Buford Highway, Mailstop K-50

Atlanta GA 30341

Contents

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

B.2. Procedures for the Collection of Information

B.3. Methods to Maximize Response Rates and Deal with Non-response

B.4. Tests of Procedures or Methods to be Undertaken

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

List of Attachments

A-1. Public Health Service Act

A-2. Family Smoking Prevention and Tobacco Control Act

A-3. PPHF

B-1. Federal Register Notice

B-2. Summary of Public Comments and CDC Response

C-2a. Smoker Phase 2 Follow-Up Questionnaire

C-2b. Sample Screen Shots of the Smoker Phase 2 Follow-Up Questionnaire

D-2a. Non-Smoker Phase 2 Follow-Up Questionnaire

D-2b. Sample Screen Shots of the Non-Smoker Phase 2 Follow-Up Questionnaire

E-1. Screening and Consent Process (screen shots)

E-2. Email Notifications and Reminders

F-1. Knowledge Networks Panel Recruitment Procedures

F-2. Knowledge Networks Privacy and Confidentiality Procedures

G-1. RTI IRB Approval Notice

H-1 and H-2. Reference Articles on Knowledge Networks

1. Information Collection Instruments Used in the Phase 1 Evaluation (2012)

   1. Attachment C-1, Smoker Baseline Questionnaire

   2. Attachment C-2, Smoker Follow-Up Questionnaire

   3. Attachment D-1, Non-Smoker Baseline Questionnaire

   4. Attachment D-2, Non-Smoker Follow-Up Questionnaire

J. Primary and Secondary Outcomes, Survey Questions, and Expected Timeframe for Outcome Change

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

We will conduct an additional third wave of survey data among adult smokers and adult non-smokers in the United States after the implementation of Phase 2 of the “Tips From Former Smokers” Campaign. The Wave 3 survey will be fielded June/July 2013, soon after the conclusion of the Phase 2 Campaign. Building on the original baseline data (Wave 1) that has already been collected for this campaign in early 2012, this design facilitates longer-term analysis of relationships between individuals’ exposure to the combined Phase 1 and Phase 2 campaigns and pre-post changes in outcomes of interest. Because many participants in this new data collection (Wave 3) will have also participated in the original baseline survey (Wave 1), this design further enables longitudinal sub-analysis that will allow us to calculate long-term Wave 1 to Wave 3 changes in campaign-targeted outcomes for each study participant. We hypothesize that if the cumulative campaign is effective, the Wave 1 to Wave 3 changes in outcomes should be larger among individuals exposed to the campaign more frequently (i.e., dose-response effects).

The primary survey sample will consist of 14,250 additional interviews of smokers from the KN panel in addition to 3,286 additional interviews of non-smokers, also from the KN panel. (Table B.1.a). Evidence on the accuracy of self-reported data from the KN panel has been demonstrated in prior research, notably in two recent studies published in Public Opinion Quarterly (Chang & Krosnick, 2009 (Attachment H-1); Yeager, Krosnick, & Chang et al., 2011; Attachment H-2). These studies explicitly examined the comparison between KN panel survey results and results from RDD telephone and opt-in non-probability Web panels. Yeager et al. (2011) conducted an experiment by administering the same survey instrument to multiple samples which included seven non-probability Internet platforms and two probability-based survey platforms which included a RDD telephone survey and a probability-based Internet survey. Although it was not directly named, the Knowledge Networks KnowledgePanel was the probability-based Internet survey used in this study. This study showed that the KN panel was the most accurate in terms of primary demographics even compared to RDD telephone surveys. KN interview cases (unweighted) were on average 2.47 percentage points different than Census benchmarks compared to an average 3.43 percentage point difference for RDD telephone surveys. This may be due to improved coverage of cell phone only households in the KN address-based sampling frame which are usually excluded from RDD landline telephone surveys. The overall conclusion of these studies is that the foundations of statistical sampling were sustained in both types of probability samples (RDD telephone and KN panel data) and these data yield quite accurate results even when response rates are not especially high.

Table B.1.a. Sample Sources for Second Follow-Up Survey

	Smoker Sample	Nonsmoker Sample
Knowledge Network Sample	6,250 (a)	3,286 (b)
Survey Sampling International	8,000 (c)	-
Total Sample	14,250	3,286

1. 2,975 retained from baseline sample

2. 2,000 retained from baseline sample

3. 1,000 retained from baseline sample

Similar to the currently-approved ICR (OMB no. 0920-0923, exp. 2/28/2013)with the baseline and the first follow-up survey, we will again recruit a secondary sample of approximately 8,000 interviews of smokers to the KN web survey from Survey Sampling International (SSI), a leading provider of online sampling in the U.S. for the second follow-up survey. SSI is a global online sampling provider consisting of a large ongoing panel of participants as well as participants from online communities, social media, and other partners affiliated with SSI. The KN panel will be the primary source of sample and will be used to generate a singular estimate of this outcome for the Phase 2 Campaign across a large, geographically and socio-demographically diverse sample of smokers and nonsmokers in the U.S. However, the KN panel alone may not be sufficient to assess awareness at more granular levels, particularly within groups of smaller television markets that may be targeted to receive higher doses of the Phase 2 Campaign. In order enhance our ability to obtain estimates of campaign awareness within these areas, more sample is needed, particularly in groups of markets with relatively smaller population densities. The additional sample of smokers from the SSI panel will help address this gap. The combined KN and SSI sample will therefore be used to provide estimates of campaign awareness (and other outcomes) at more granular levels than the U.S. as a whole.

The SSI sampling frame is identified with pre-existing panel profile information, similar to the profile information that KN uses to pre-identify its own panelists for specific studies. We will also be able to pre-identify panelists from SSI who participated in the previous Phase 1 baseline data collection in February/March 2012. This will enable us to conduct longitudinal follow-up of approximately 1,000 original SSI participants in addition to 7,000 new SSI cross-sectional participants at the post-Phase 2 survey. It is important to note that survey procedures for the supplemental sample of smokers including screening, consent, and survey completion will all occur within the KN-administrated survey just as with the primary smoker and non-smoker samples from the existing KN panel.

All data collected for this Wave 3 study will be weighted for analysis. KN will weight all data to facilitate separate analysis of KN-only sample (for singular estimates of a large, geographically and socio-demographically diverse sample as described above) and for analysis of pooled KN and SSI sample (for estimates of smaller geographic regions). Weights for the KN sample are calculated using a standard post-stratification weighting procedure that adjusts for survey non-response as well as non-coverage. This weighting procedure also applies a standard post-stratification adjustment based on demographic distributions from the most recent October 2010 data from the Current Population Survey (CPS). Benchmark distributions for Internet access used in this weight are obtained from the most recent (October 2009) special CPS supplemental survey measuring Internet access. Weights for the pooled KN and SSI sample are generated by first adding the SSI cases to the independently weighted KN sample. This combined sample is then reweighted using the weighted KN sample as its benchmark, resulting in sample that is similar to the KN panel in terms of its weighted demographic profile. As noted elsewhere in this information collection request, limitations of the pooled SSI and KN data must be acknowledged. While this data will provide valuable additional information to CDC about campaign awareness in these smaller geographic areas, the opt-in nature of the SSI sample limits our ability to project those results to those areas generally.

Study sample sizes were determined through power analyses that were originally conducted to determine the necessary number of interviews to detect specific relationships between self-reported campaign awareness and outcomes of interest. For purposes of this study, we examined existing evaluation literature and research to determine the expected effect sizes on the outcome of making a quit attempt. Based on these analyses, we have powered the study to detect an underlying odds ratio of 1.20 between self-reported campaign awareness and the likelihood of a quit attempt. This power analysis is based on KN sample sizes only, given that overall U.S. estimates will be derived from the KN panel alone. Previous media evaluations of statewide campaigns have demonstrated relationships of this magnitude between self-reported campaign awareness and the likelihood of a quit attempt. We have conservatively powered the sample to detect this effect at 80% power among KN smokers in the sample. For non-smokers we have reserved sufficient sample to detect this same effect on other outcomes relevant to non-smokers at the standard 80% power level. This power analysis applies to models of quit attempts in the past year as a function of self-reported campaign awareness.

Based on this power analysis, we conservatively anticipate collecting a total of 14,250 new interviews of smokers (6,250 from KN, 8,000 from SSI) for the Wave 3 evaluation survey. Because this sample also includes recontacts of all participants in the original Wave 1 baseline study conducted in February/March 2012, we anticipate the newly proposed data collection will yield a sizeable longitudinal sample of participants who will have participated in both the pre-Phase 1 and post-Phase 2 surveys. Among the original Wave 1 baseline sample of KN participants, we anticipate an approximate 72% retention rate, yielding approximately 2,975 interviews of KN smokers who participated in the original Wave 1 survey. Because of the “opt-in” nature of the SSI panel, we anticipate a considerably lower long-term retention rate (approximately 13%) among the original baseline SSI smokers who are included in the Wave 3 survey sample. We therefore estimate that our Wave 3 survey will contain approximately 1,000 SSI panel smokers who also completed the original baseline survey. Combined, we expect a longitudinal subsample of approximately 3,975 (2,975 from KN, 1,000 from SSI) smokers in this survey who also completed the Wave 1 original baseline.

The newly proposed Wave 3 data collection will also contain approximately 3,286 new interviews of nonsmokers, sampled from the KN panel. This sample will also include a longitudinal subsample of nonsmokers who also participated in the original Wave 1 baseline survey, as we are re-contacting all previous nonsmoker participants from the baseline survey. In total, we anticipate the final nonsmoker sample for this survey will include approximately 2,000 nonsmokers who participated in the original Wave 1 baseline survey and 1,286 new cross-sectional smokers.

We anticipate that the initial cooperation rate from study invitations to complete the initial screenings will be approximately 70% among Knowledge Networks smokers and non-smokers and approximately 30% among the SSI sample. Based on this initial cooperation rate for screening, we anticipate that a total of 8,929 KN smokers, 4,694 KN non-smokers, and 30,114 SSI smokers for the supplemental sample will complete the introductory screenings for this study. Therefore, the total number of unique respondents in this information collection is 43,737.

All decisions about assumptions that guided our power analysis were intended to err in favor of a larger sample size to safeguard for the possibility of being able to detect smaller effect sizes from the campaign. These assumptions increased our confidence that smaller effects produced by The Campaign than those found by previous prevention programs would be reasonably detected using the sample sizes we identified. As noted earlier, our sample design is also based on conservative assumptions about survey response. Thus our estimates of longitudinal retention rates should be viewed as “worst case” scenarios that if hold true, would still ensure sufficient sample sizes to reasonably detect small campaign effects.

It should be noted that while the KN panel’s recruitment procedures are designed to approximate a nationally representative sample, the limitations associated with the panel decrease our capacity to draw nationally representative conclusions about either smoking-related knowledge and behavior or the impact of the campaign on long-term quit rates in sub-populations. Although KN panelists must be invited to participate and cannot volunteer on their own, there may be systematic differences between individuals who choose to join an ongoing internet panel and the type of individuals who do not wish to participate in either an internet panel or an ongoing committee. Furthermore, our estimates from the combined KN and SSI samples are more limited with respect to representativeness because the SSI response rate will be about 1/3 of that expected for the KN panel. Therefore, evaluation results must be interpreted with appropriate caution regarding our ability to generalize the findings to the national population of smokers and nonsmokers.

The KN-only sample will provide valuable information on the knowledge and behavior of a geographically and socio-demographically varied population of smokers and nonsmokers in the U.S., as well as any differences in knowledge and behavior after exposure to an intensive, communication campaign. More granular conclusions about the awareness (and change in awareness) of smokers in smaller groups of media markets will be made possible by the supplemental panel sample, although the conclusions drawn from the combined sample will be more limited with respect to population representativeness than the KN sample due to a variety of methodological limitations.

The evaluation design used allows CDC to estimate the potential for a longer-term national campaign to reach a large portion of the population, to gauge change in knowledge and immediate behaviors of smokers and nonsmokers, and to generate hypotheses about potential differences in responsiveness within groups of media markets that receive varied doses of the cumulative Campaign. Study design limitations decrease our capacity to draw nationally representative conclusions about either smoking-related knowledge and behavior or the impact of the campaign on long-term quit rates in sub-populations. However, the design is the best available solution to CDC’s evaluation objectives, within time, cost, and feasibility constraints.

Table B.1.b provides a summary of respondents, by type, panel (source), and information collection (form name) for the newly proposed, additional data collection.

Table B.1.b. Summary of Respondents by Type, Source, and Form Name

Type of Respondent	Information Collection/Form Name	Number of Respondents
Adult smokers, ages 18-54 (KN Panel)	Screening and Consent Process – Phase 2 Survey (Smokers)	8,929
	Smoker Phase 2 Questionnaire (Smokers)	6,250
Adult non- smokers, ages 18-54 (KN Panel)	Screening and Consent Process – Phase 2 Survey (Non-smokers)	4,694
	Nonsmoker Phase 2 Questionnaire (Non-smokers)	3,286
Adult smokers, ages 18-54 (Supplemental Off-Panel)	Screening and Consent Process – Phase 2 Survey (Smokers)	30,114
	Smoker Phase 2 Questionnaire (Smokers)	8,000
		*43,737

*Total number of unique respondents calculated as total respondents who complete introductory screenings.

B.2. Procedures for the Collection of Information

When the study is assigned to the sampled email addresses, individuals will receive email notification that the survey is available for completion. Nonrespondents will receive two e-mail reminders from Knowledge Networks requesting their participation in the survey. See Attachment E-2 for study email notifications and reminders. The surveys will be self-administered and accessible any time of day for a designated period. Participants can complete the survey only once. Study screeners will be used to determine study eligibility, including information on current smoking behavior. Eligible participants will include smokers and non-smokers in the U.S. and participants will be allowed to complete the survey in either English or Spanish. The Spanish language surveys will be identical in terms of items, question wording, and substantive meaning. The Spanish translations will be done in a culturally competent manner and all survey items will be cross-checked with Spanish-speaking adults. The Spanish language surveys will be provided upon OMB approval of the content of this information collection request. Informed consent will be sought from participants for participation in the Web survey. Participants will consent by selecting the appropriate link on the Web screen. A detailed description of Knowledge Networks’ panel recruitment methodology is provided with this submission (Attachment F-1).

B.3. Methods to Maximize Response Rates and Deal with Non-response

The following procedures will be used to maximize cooperation and participation in this study:

 Participants from the KN panel will be offered 15,000 bonus points (equivalent to $15 cash) for completion of the newly-proposed additional survey. SSI panelists will receive $3 for completion of this survey. This incentive structure is intended to recognize the time burden placed on participants, encourage their cooperation, and to convey appreciation for contributing to this important study.

• Email reminders (Attachment E-2) will be sent to all sampled participants who do not complete their assigned survey within a given period of time after it is assigned. A second round of email reminders will be sent to nonresponders who do not complete the survey once the initial email reminder is delivered.

 An attempt will be made to locate participants who leave the Knowledge Networks panel before the end of this study. Location efforts will include mailings of refusal conversion materials designed to persuade participants to complete the study. In addition to using mailed refusal conversion materials, Knowledge Networks may also conduct telephone-based refusal conversion, contacting each non-responders via telephone.

 Knowledge Networks will provide a toll-free telephone number to all sampled individuals and invite them to call with any questions or concerns about any aspect of the study.

 Knowledge Networks data collection staff will work with RTI project staff to address concerns that may arise.

B.4. Tests of Procedures or Methods to be Undertaken

Prior to launching the original Wave 1 baseline survey, we fielded an eight-case pretest of the survey instrument. This survey was very similar to the instrument that will be used in the newly-proposed Wave 3 data collection with the exception of a few additional questions to assess overall clarity of instrument questions and respondent’s opinions on any aspects of the survey that were not clear. The purpose of the pilot test was twofold: (1) to assess technical aspects and functionality of the survey instrument, and (2) to identify areas of the survey that were either unclear or difficult to understand. Once this pretest was completed, Knowledge Networks created a data file for analysis by RTI International. This data contained diagnostic data on average time of survey completion, survey completion patterns (e.g., are there any concentrations of missing data?), and other aspects related to the proper function of the survey. We also examined data on pilot test measures that will be used to assess the clarity of item wording and ease of understanding. This pretest analysis yielded no significant results to suggest that changes to the instruments were necessary.

In addition to the aforementioned eight-case pretest, RTI also conducted rigorous testing of the original online survey instrument prior to its fielding. RTI researchers will again have access to an online test version of the revised instrument that we will use to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and that all survey questions are worded correctly and in specification with instrument approved by OMB.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals inside the agency have been consulted on the design and statistical aspects of this information collection as well as plans for data analysis:

Tim McAfee, MD, MPH

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5709

Email: [email protected]

Terry Pechacek, PhD

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5592

Email: [email protected]

Diane Beistle

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5066

Email: [email protected]

Robert L. Alexander Jr., PhD, MPH, CHES

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-1212

Email: [email protected]

Jami L. Fraze, PhD, MEd, CHES

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5186

Email: [email protected]

Bob Rodes, MS, MBA, MEd

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5748

Email: [email protected]

Jeffrey McKenna, MS

National Center for Chronic Disease Prevention and Health Promotion
Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K40

Atlanta, GA 30341

Phone: (770) 488-5131

Email: [email protected]

Karen Debrot, DrPH, MNS, RD

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-1037

Email: [email protected]

The following individuals outside of the agency have been consulted on the questionnaire development, statistical aspects of the design, and plans for data analysis:

Kevin C. Davis, MA

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 541-5801

Email: [email protected]

Jennifer Duke, PhD

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 485-2269

Email: [email protected]

Donna Vallone, PhD

Legacy Foundation

1724 Massachusetts Avenue, NW

Washington, DC 20036

Phone: (202) 454-5783 
Email: [email protected]

April Brubach

FDA, Center for Tobacco Products

9200 Corporate Boulevard

Rockville, MD 20850

Phone: (301) 796-9214

Email: [email protected]

The following individuals will conduct data collection and analysis:

Kevin C. Davis, MA

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 541-5801

Email: [email protected]

Jennifer Duke, PhD

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 485-2269

Email: [email protected]

References

Abreu, D.A., & Winters, F. (1999). Using monetary incentives to reduce attrition in the survey of income and program participation. Proceedings of the Survey Research Methods Section of the American Statistical Association.

Chang L. & Krosnick J.A. (2009). National surveys via RDD telephone interviewing versus the Internet: comparing sample representativeness and response quality. Public Opinion Quarterly. 74(4):641-678

Shettle, C., & Mooney, G. (1999). Monetary incentives in U.S. government surveys. Journal of Official Statistics, 15, 231-250.

Southwell B, Barmada C, Hornik R, et al. (2002).Can we measure encoded exposure? Validation

evidence from a national campaign. J Health Commun, 7:445-453.

Yeager D.S., Krosnick J.A., Chang L., et al. (2011). Comparing the accuracy of RDD telephone

surveys and Internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly. 75(4):709-747.