ICR.0920-12LR-CTG. Media Campaigns. Part B.CLEAN

Community Transformation Grants: Evaluation of Nutrition, Physical Activity, and Obesity-related Television Media Campaigns

New

Supporting Statement

Part B—Collection of Information Employing Statistical Methods

May 10, 2013

Robin Soler, PhD

Contracting Officer Representative (COR)

Division of Community Health

Centers for Disease Control and Prevention (CDC)

4770 Buford Hwy, N.E. MS K-45

Atlanta GA 30341

Telephone: (770) 488-5103

E-mail: [email protected]

TABLE OF CONTENTS

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

B.2 Procedures for the Collection of Information

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

B.4 Tests of Procedures or Methods to be Undertaken

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

References

LIST OF ATTACHMENTS

Attachment Number

1a. Public Health Service Act

1b. Authorizing Legislation: American Recovery and Reinvestment Act of 2009

1c. Patient Protection and Affordable Care Act

1d. Prevention and Public Health Fund

1e. ACA Section 4201, Community Transformation Grants

2a. Federal Register Notice

2b. Summary of Public Comments

3. Community Transformation Grants—Strategic Directions

4. List of Community Transformation Grant Awardees

5a. Screening and Consent Process

5b. Screening and Consent Process (representative screen shots)

6a. Nutrition, Physical Activity, and Obesity Survey (outline of content including each question’s rationale and source)

6b. Nutrition, Physical Activity, and Obesity Survey (representative screen shots)

7. Research Now’s Panel Recruitment Procedures

8. E-mail Notifications and Reminder

9. RTI IRB Approval Notice

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe and Sampling Methods

Respondent Universe

The respondent universe for this Information Collection Request (ICR) is the Research Now (RN) General Population panel, a large online panel of adults in the general U.S. population. A detailed description of RN’s panel recruitment methodology is provided with this submission (Attachment 7). Specifically, RN uses as its sampling frame a broad variety of loyalty programs and related consumer- and business-focused programs. Panel members residing in the following areas across the United States will be sampled to assess the impact of television media campaign advertisements targeting nutrition, physical activity, and obesity (NPAO) on audience reactions, knowledge, attitudes, beliefs, intentions, and behaviors:

1. Intervention Population -Community Transformation Grant (CTG) awardee areas where significant televised obesity prevention media campaigns are being delivered (CTG media-intervention areas).
   
   Based on preliminary discussions with CTG awardees, we have identified three awardees that have implemented or are planning to implement NPAO-targeted media campaigns during a time period that is proximal to our estimated timeframe for data collection. A total of 61 counties or county equivalents are included in this group.

2. Comparison 1 Population - CTG awardee areas where no CTG- or CPPW-sponsored obesity prevention media campaigns are being delivered (CTG media-control areas). A total of 1,076 counties or county equivalents are included in this group.

3. Comparison 2 Population - Areas not covered by either CTG or Communities Putting Prevention to Work (CPPW) (Program-control areas). A total of 1,990 counties or county equivalents are included in this group.

Sample Selection

Although RN makes an effort to oversample those population subgroups that are underrepresented, this is not considered a population representative frame. Even so, CDC considers the frame to be sufficiently large and varied to provide insight into whether there are fairly large differences in outcomes across the geographic areas with different intensities of federal support for obesity-related media campaigns. A stratified simple random sample will be used to select the study sample from within the RN sampling frame. The contractor will first identify the U.S. Federal Information Processing Standard (FIPS) county codes that correspond to CTG media-intervention and CTG and non-CTG control areas. FIPS codes identify counties and county equivalents in the United States and are publicly available. Counties or county equivalents will be assigned to one of four strata based on their geographic location being contained within (1) Intervention Population - CTG awardee areas with NPAO television media-interventions areas (i.e., 3 awardees for a total of 61 counties), (2) Comparison 1 Population - CTG awardee areas with no NPAO-targeted television media campaigns, (3) Comparison 2 Population - non-CTG/ non-CPPW areas, and (4) populations not used in the evaluation - CPPW, non-CTG areas. The stratification is applied to the RN panel such that all members of the panel fall into mutually exclusive and exhaustive strata. Counties in the fourth stratum are excluded from the sampling frame and subsequent analyses.

Next, a simple random sample of panel members will be drawn from each evaluation stratum (i.e., Intervention Population, Comparison 1 Population, Comparison 2 Population) Every stratum will be sampled at different sampling fractions, such that the study sample consists of approximately 2,325 adults from the CTG media-intervention stratum (proportionately distributed across the three awardees planning NPAO-targeted media campaigns), 2,325 adults from the CTG media-control stratum, and 2,325 adults from the non-CTG/CPPW stratum. Emphasis is placed on sampling from CTG media-intervention communities to allow assessment of variation in key outcomes of interest among the three CTG media-intervention awardees. Based on data from previous studies we have conducted using online panels from RN and other online panel providers, we conservatively anticipate that approximately 60%–65% of eligible respondents who consent to participate will complete the survey. We estimate needing to screen approximately 19,500 panel members to yield a total sample size of approximately 6,975.

To account for certain differences between the three populations sampled for this evaluation, we will use propensity scoring, a statistical technique used in the evaluation of other media campaigns^18,19 that offers a means of creating statistically similar intervention and comparison groups and is used in nonrandomized observational studies to estimate the probability that a person will belong to a particular group. Propensity scores will be developed to better equalize the three sampled populations (CTG media intervention, and two comparison populations) so that obesity-related outcomes across groups could be attributed to campaign activities with reduced bias from potential confounders.

Statistical Power

The target sample sizes for each group (i.e., CTG media-intervention and two comparison populations) were chosen to ensure that we have good power to detect moderate differences in key outcomes of interest for comparisons between (1) all CTG media-intervention respondents and CTG media-control respondents (Evaluation Questions 1, 2) and (2) all CTG media-intervention respondents and non-CTG/CPPW media-control respondents (Evaluation Questions 1, 2).

These illustrative power calculations assume 80% power, a Type I error rate of 0.05 (alpha = 0.05), and an unequal weighting effect of 1.5 (a consequence of the unequal probabilities of selection across strata). Sample size calculations were performed to determine the number of interviews needed to detect effect sizes similar to those that have been reported in recent studies of the effect of media campaigns on NPAO-related behaviors including taking action to eat healthy foods, drink healthy beverages, avoid sugar-sweetened beverages, and engage in regular physical activity. For example, previous evaluations of media campaigns have demonstrated relationships of this magnitude between self-reported campaign awareness and changes in NPAO-related behaviors.²⁰

We determined that with a sample size in each group of at least 2,325 respondents, the study may be powered to detect a 5% absolute difference in proportions between any two groups (e.g., CTG media-intervention vs. CTG media-control or non-CTG/CPPW media-control groups), depending upon the extent to which the underlying assumptions are met. For example, if the proportion of adults in the control groups reporting any past-week consumption of high-calorie beverages is 40% (based on 2010 Behavioral Risk Factor Surveillance System data), the study may have as much as 80% power to detect a minimum improvement to 35% among respondents in the CTG media-intervention group.

Several decisions about assumptions that guided our power analysis were intended to err in favor of a larger sample size to enable detection of potentially smaller effect sizes from the campaigns than previously obtained in other studies. This decision was guided, in part, on the heterogeneity of media campaign approaches and dosage across intervention areas in CTG. These assumptions increased our confidence that we could reasonably detect effects comparable to those detected in other NPAO-targeted media-interventions using the sample size we identified. As noted earlier, our sample design is also based on conservative assumptions about survey response. Exhibit B.1.1 provides estimates of the number of available respondents in the RN panel and our estimated sample sizes. Across all three evaluation groups, the panel provides sufficient numbers to ensure meeting our sample goals.

Exhibit B.1.1 Illustrative Sample Size (Interviews Completed) Needed to Ensure 80% Power to Detect 5% Difference in Proportions between Groups**

Group	# of Counties or County Equivalents Represented	Interviews Completed
CTG Media-Intervention Areas	61	2,325
CTG Media-Control Areas	1,076	2,325
Non-CTG/CPPW Media-Control Areas	1,990	2,325
Total	3,127	6,975

** Notwithstanding the limitations of the sampling frame and differences among the geographic areas in factors that affect media exposure and receptiveness to obesity programs.

B.2 Procedures for the Collection of Information

When the study is assigned to the sampled e-mail addresses, individuals will receive e-mail notification that the survey is available for completion. Nonrespondents will receive an e-mail reminder from RN requesting their participation in the survey 7 to 10 days following the initial e‑mail notification. See Attachment 8 for study e-mail notifications and reminders. The surveys will be self-administered and accessible any time of day for a designated period. Participants can complete the survey only once. Eligible participants will include adults (ages 18 and above) in the United States. Attachment 5a provides the complete list of questions and advisements for the screening and consent process, and Attachment 5b provides representative screen shots of the Web-based instrument. Attachment 6a provides the complete list of questions for the main information collection, annotated with notes about each question’s rationale and source. Attachment 6b provides representative screen shots of the Web-based instrument.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

For this study, some nonresponse can be expected. Survey nonresponse may arise from noncontact or refusals, while item nonresponse may arise from respondent fatigue toward the end of the survey. Nonresponse is a potentially serious methodological threat to the interpretation of the study findings, particularly if it occurs differentially across subpopulations (i.e., nonignorable nonresponse). To reduce the potential for both survey and item nonresponse bias, several strategies will be used and are presented below.

Maximizing Response Rates

The following procedures will be used to maximize cooperation and participation in this study:

Participants will be offered $5 in RN currency (equivalent to $5 cash) for completion of the survey. The gift is intended to recognize the effort placed on participants, encourage their cooperation, and convey appreciation for contributing to this important study. This gift amount is typical of all RN surveys of comparable length. E-mail reminders (Attachment 8) will be sent to all sampled participants who do not complete their assigned survey within a given period of time after it is assigned. RN will provide a toll-free telephone number to all sampled individuals and invite them to call with any questions or concerns about any aspect of the study. RN data collection staff will work with RTI project staff to address concerns that may arise.

Dealing with Nonresponse

To investigate the impact of survey nonresponse and determine appropriate solutions, simple descriptive statistics, such as counts and frequencies, will be tabulated for respondents and nonrespondents. Nonrespondent statistics will be tabulated overall and by subtype (refusal vs. not contacted). Response rates will be calculated and comparisons between respondents and nonrespondents on sociodemographic characteristics (e.g., age, sex, and race/ethnicity) and other relevant factors. Statistical weighting techniques to minimize the potential bias resulting from nonresponse will be considered. RN collects information about respondent demographic characteristics that can be used to facilitate these procedures.

In addition to dealing with survey nonresponse, strategies will be implemented to mitigate potential threats arising from item nonresponse. For variables with less than 10% missing data, an imputation strategy may be applied to estimate the missing data based on the distribution of each individual’s baseline characteristics such as age, sex, race/ethnicity, education, and household income. If the survey data are poorly collected, resulting in substantial missing data for some variables (>20%), the characteristics of the missing data will be carefully examined and handled with appropriate strategies.

Sample Weighting

The process for preparing the data for analysis involves completing:

• an assessment of nonresponse bias (i.e., which determines the ability of results to be generalized to the target population);

• a final assessment of data quality including checking for item completeness, accuracy, plausibility (with respect to compatibility with other data collected for the individual, validity checks, and comparison of summary statistics with expected distributions) and instituting corrective action (e.g., imputation for missing data, setting impossible values to missing); and

• calculation of sampling weights.

In all analyses, data will be weighted to account for the unequal probability of selection and response. There are three steps in creating the sampling weights:

1. Calculate the initial weights as in the inverse of the probability of selection with an adjustment for unknown eligibility.

2. Adjust for nonresponse.

3. Poststratify.

The weighting procedure applies a standard post-stratification adjustment based on demographic distributions from the most recent (October 2010) data from the Current Population Survey (CPS). Benchmark distributions for Internet access used in this weight are obtained from the most recent (October 2009) special CPS supplemental survey measuring Internet access. A logistic regression model is fit predicting the probability of response using the following CPS data as predictors:

• Proportion of Hispanics in the block group in which the address is located

• Ratio of households that are owner occupied to rentals in the block group in which the address is located

• Proportion of the population with a bachelor’s degree or higher in the block group in which the address is located

• Proportion of the population in poverty in the block group in which the address is located)

Estimation of differences in key outcomes between media-intervention versus control groups will have to account for the potential confounding effects of group imbalances on important sociodemographic or policy attributes. This will be accomplished using generalized propensity scores methods.^21,22 Briefly, the three study groups (CTG media-intervention, CTG media-control, and non-CTG/CPPW media-control) will be modeled as multinomial outcomes in a generalized logit model, with a suitable set of predictors (covariates) to explain the nonrandom assignment into these groups. The probabilities (propensity scores) of assignment to each group at baseline will be used as covariates in all subsequent models to compare media campaign effectiveness. By adjusting for preexisting imbalances, any differences in NPAO-related outcomes may be delineated with mitigation of potential confounding variables. Covariates used to generate propensity scores will be linked to awardee area characteristics.

Although these weighting procedures will help account for preexisting imbalances in sociodemographic characteristics among intervention groups, differences in the presence and intensity of additional NPAO-targeted interventions among groups may also potentially confound study results. As part of the comprehensive evaluation of the CTG initiative, we will have access to a performance monitoring dataset providing information about the type and intensity of other CTG efforts. By integrating these data, we will be able to statistically account for some program-level differences among CTG awardees. However, we cannot feasibly account for the range of non-CTG NPAO-targeted efforts that may be occurring in tandem with CTG interventions or in areas not covered by the CTG initiative. Further, because groups were not randomly assigned to condition and because of a lack of baseline assessment, we cannot rule out the possibility that additional, unexamined factors may confound study results. We recognize these as limitations inherent in such a quasi-experimental study design.

B.4 Tests of Procedures or Methods to be Undertaken

Prior to launching the survey, we will field an eight-case pretest of the survey instrument. The pretest survey will be identical to the instrument that will be used in this evaluation and approved by the Office of Management and Budget (OMB) with the exception of a few additional questions to assess overall clarity of instrument questions and respondents’ opinions on any aspects of the survey that were not clear. The purpose of the pilot test will be twofold: (1) to assess technical aspects and functionality of the survey instrument, and (2) to identify areas of the survey that were either unclear or difficult to understand. Once this pretest is completed, RN will create a data file for analysis by RTI. This data file will contain diagnostic data on average time of survey completion, survey completion patterns (e.g., are there any concentrations of missing data?), and other aspects related to the proper function of the survey. We will also examine data on pilot test measures that will be used to assess the clarity of item wording and ease of understanding.

In addition to the aforementioned eight-case pretest, RTI will conduct rigorous testing of the online survey instrument prior to its fielding. RTI researchers will have access to an online test version of the instrument that will be used to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and all survey questions are worded correctly and to the specification of the instrument approved by OMB.

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals internal to the Centers for Disease Control and Prevention (CDC) have been consulted on the design and statistical aspects of this information collection and on plans for data analysis:

Name	Organization	Contact Information
Terry O’Toole, Public Health Analyst	Division of Nutrition, Physical Activity, and Obesity (DNPAO), National Center for Chronic Disease Prevention and Health Promotion (NCCDPHP)	Phone: 770-488-5937 E-mail: [email protected]
Rachel Dooley, MPH, ORISE Fellow	DNPAO; NCCDPHP	Phone: 770-488-6023 E-mail: [email protected]
Jan Jernigan, PhD, Health Scientist	DNPAO; NCCDPHP	Phone: 770-488-5224 E-mail: [email protected]
Suzanne Gates, PhD, Health Communication Analyst	Division of Community Health (DCH); NCCDPHP	Phone: 770-488-7580 E-mail: [email protected]
Robin Soler, PhD, Evaluation Team Lead, RSEB	DCH; NCCDPHP	Phone: 770-488-5103 E-mail: [email protected]

The following individuals external to CDC have been consulted on the questionnaire development, statistical aspects of the design, and plans for data analysis:

Name	Organization	Contact Information
Erik Crankshaw, PhD, Research Associate	RTI International	Phone: 919-316-3809 E-mail: [email protected]
Matthew Eggers, MPH, Public Health Analyst	RTI International	Phone: 919-990-8380 E-mail: [email protected]
Kevin Davis, MPH, Senior Research Economist	RTI International	Phone: 919-541-5801 E-mail: [email protected]
Darigg Brown, PhD, Public Health Analyst	RTI International	Phone: 770-407-4937 E-mail: [email protected]
Daniel Zaccaro, PhD, Research Biostatistician	RTI International	Phone: 919-541-6310 E-mail: [email protected]
Jonathan Blitstein, PhD, Research Psychologist	RTI International	Phone: 919-541-7313 E-mail: [email protected]
Burton Levine, PhD, Statistician	RTI International	Phone: 919-541-1252 E-mail: [email protected]
Debra Holden, PhD Research Scientist	RTI International	Phone: 919-541-6491 E-mail: [email protected]
Diane Catellier, PhD Associate Project Director	RTI International	Phone: 919-541-6447 E-mail: [email protected]
Todd Rogers, PhD CTG EE Science Director	RTI International	Phone: 415-848-1374 Email: [email protected]

The following individuals will conduct data collection and analysis:

Name	Organization	Contact Information
Erik Crankshaw, PhD, Research Associate	RTI International	Phone: 919-316-3809 E-mail: [email protected]
Matthew Eggers, MPH, Public Health Analyst	RTI International	Phone: 919-990-8380 E-mail: [email protected]

References

18. Hornik, R., MacLean, D., Cadell, D., et al. (2003). Evaluation of the National Youth Anti-Drug Media Campaign: 2003 report of findings: Executive summary. Washington, DC: Westat. Report prepared for the National Institute on Drug Abuse (contract N01DA-8-5063).

19. Huhman, M., Potter, L. D., Wong, F. L., Banspach, S. W., Duke, J. C., & Heitzler, C. D. (2005). Effects of a mass media campaign to increase physical activity among children: Year-1 results of the VERB campaign. Pediatrics, 116, 277–284.

20. Snyder, L. (2007). Health communication campaigns and their impact on behavior. Journal of Nutrition Education and Behavior, 39, S32–S40.

21. D’Agostino, R.B., Jr. (1998). Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Statistics in Medicine, 17(19), 2265–2281.

22. Preisser, J. S., Young, M. L, Zaccaro, D. J, & Wolfson, M. (2003). An integrated population-averaged approach to the design, analysis and sample size determination of cluster-unit trials. Statistics in Medicine, 22(8), 1235–1254.