Supporting Statement Part B 2018

Part B. Statistical Methods (used for collection of information employing statistical methods)

1. Respondent Universe and Sampling Methods

In order to obtain our sample of U.S. adults for the consumer study, we will use a nonprobability panel provided by Research Now. Research Now uses an “invitation-only” methodology to build panels, an approach that yields high panel quality and representativeness while guarding against duplication, fraudulent respondents, and professional survey takers. The sample of U.S. physicians for the health care provider study will be recruited from a panel maintained by SERMO. SERMO has a proprietary database of more than 1.8 million doctors and allied health professionals who have opted in to participate in research studies, including over 800,000 through their partnership with Doctor Directory. Physicians are added to the database through a variety of means, including direct mail campaigns and doctors who opt in to research opportunities while registering to list their practice on Doctor Directory. The information in the database is rigorously verified on an ongoing basis using external sources, such as the AMA database, state licensing numbers, and quarterly updates from the U.S. Drug Enforcement Administration/National Provider Identifier (DEA/NPI).

The focus of this survey is to assess the impact of disclosures on assessments of pharmaceutical efficacy claims using experimental methods rather than making population-based inferences. Therefore, a nonprobability panel is a reasonable choice. Nevertheless, we will ensure a demographically diverse sample with respect to gender, age, race/ethnicity, and education.

Tables 1 and 2 provide a summary of the soft quotas that will help guide recruitment of participants in this study. Soft quotas will not be used to strictly enforce a quota or limit for any category and will not necessarily reflect population-representative sampling. Instead, they will help to guide recruiting a diverse sample during the course of the study. Soft quotas will be assessed at the study level; throughout the recruitment process, the respondent pool will be evaluated to adjust for any demographic over or under sampling. These requirements will be communicated to recruiters who will then make the adjustments to reach the required target populations.

Table 1: Summary of Screening Soft Quotas for Consumer Study¹

Category	Classification	Soft Quotas
Gender	Male	50%
	Female	50%
Race/Ethnicity	White, Non-Hispanic	72%
	Black, Non-Hispanic	13%
	Asian, Non-Hispanic	5%
	Other/Two or more races, Non-Hispanic	10%
	Hispanic	16%
Age2	18–24	13%
	25–44	34%
	45–64	34%
	65 or Older	20%
Education	No college/some college	60%
	College graduate	40%

U.S. Census. (2016). American Fact Finder. Retrieved from http://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

² Note: Adjusted age to total 100%

Table 2: Summary of Screening Soft Quotas for Physician Study³

Category	Classification	Soft Quotas
Gender3	Male	67%
	Female	33%
Race/Ethnicity4	White, Non-Hispanic	49%
	Black, Non-Hispanic	4%
	Asian, Non-Hispanic	12%
	Hispanic	4%
	Other/Unknown5	31%
Age3	39 or under	21%
	40–49	24%
	50–59	24%
	60 or Older	31%

³ Young, A. et al. (2014). A Census of Actively Licensed Physicians in the United States. Journal of Medical Regulation, 101:2, 8–23. Retrieved from http://www.fsmb.org/Media/Default/PDF/Census/2014census.pdf

⁴ Association of American Medical Colleges. Diversity in the Physician Workforce: Facts & Figures 2014. Retrieved from http://aamcdiversityfactsandfigures.org/section-ii-current-status-of-us-physician-workforce/

⁵ Flexible for quota purposes

2. Procedures for the Collection of Information

Part A of the supporting statement described the rationale for conducting the study. The proposed study seeks to address the following general research question: are healthcare professionals (HCPs) and consumers able to use disclosures to appropriately frame information in efficacy claims in prescription drug promotion?

Our specific research questions are as follows:

• Do HCPs and consumers remember and comprehend the three promotional claims differently depending on the presence or absence of a disclosure and which disclosure it is?

• Do HCPs and consumers perceive the benefits of the products differently depending on the presence or absence of a disclosure and which disclosure it is?

• How well do HCPs and consumers notice, remember, and comprehend the disclosures in the different conditions?

To address these questions, we have designed two studies, one with HCPs and one with consumers. Both studies have three 1x3 designs, with level of disclosure as a between-subjects factor. Double lines in the tables below delineate each 1x3 sub-study. Participants will view one promotional piece for each type of claim in random order, with the level of disclosure also randomized for each claim type.

Study A: HCPs

Type of Claim	Level of Disclosure
	Weak	Strong	Control
Scope of Treatment	Evidence Only	Evidence + Conclusion	None
Ease of Use	Example	Specific Requirements	None
Statistical Significance	Evidence Only	Evidence + Conclusion	None

Study B: Consumers

Type of Claim	Level of Disclosure
	Weak	Strong	Control
Scope of Treatment	Evidence Only	Evidence + Conclusion	None
Ease of Use	Example	Specific Requirements	None
Statistical Significance	Evidence Only	Evidence + Conclusion	None

Claim and Disclosure Manipulations

All promotional pieces for the two studies will be based on actual materials that OPDP reviewers have identified. Drug names and identifying details will be changed, and the drugs will be positioned as treatments for common conditions that participants are likely to be familiar with, such as hypertension, diabetes, and COPD. Disclosures for each type of claim will be similarly formatted and placed (e.g., as footnotes) and will be of similar length and level of specificity. Each promotional piece will consist of two pages: the first will provide context and will remain consistent across disclosure conditions, and the second will include the claim (also held constant across disclosure conditions) and the disclosure being tested. The drug name and branding elements will vary between claim types.

Type of Claim: Scope of Treatment

• Claim example: “Medicines that lower your blood pressure also lower your risk for heart attack and stroke,” suggesting some effect on cardiovascular events not studied by clinical trials

• Disclosure examples:

  • Evidence Only: “Drug X has not been tested as treatment or prevention for cardiovascular events, including heart attack and stroke.”

  • Evidence + Conclusion: “Because Drug X has not been tested as treatment or prevention for cardiovascular events, including heart attack and stroke, it is only approved as a treatment for high blood pressure.”

  • Control: no disclosure

Type of Claim: Ease of Use

• Claim example: “Drug Y is a simple-to-take, once daily tablet that you can take anytime, anywhere.”

• Disclosure examples:

  • Evidence Only: “Take Drug Y with 8 oz. of water and choice of food, containing less than 7% fat and approximately 250 calories.”

  • Evidence + Conclusion: “Efficacy is dependent on proper metabolization; Drug Y must be taken with 8 oz. of water and choice of food, containing less than 7% fat and approximately 250 calories.”

  • Control: no disclosure

Type of Claim: Statistical Significance

• Claim example: “In a clinical trial, people taking Drug Z had 30% less chance of having an acute COPD exacerbation. Patients treated with Drug Z had an average of 0.8 incidents between weeks 8 and 16 of the study, compared to 1.0 incidents for patients treated with Drug Q and 1.4 incidents for patients taking a placebo.”

• Disclosure examples:

  • Evidence Only: “Data showed consistent trends in this direction. Results were not statistically significant.”

  • Evidence + Conclusion: “Data showed consistent trends in this direction. Results were not statistically significant; the observed decrease in acute COPD events might not represent a real effect of treatment with Drug Z.”

  • Control: no disclosure

Specific Hypotheses

1. Main Effects of Disclosure Level

   1. Attention: Participants in the disclosure conditions will be more likely to say they noticed a disclosure than participants in the control condition.

   2. Recall: Participants in the evidence + conclusion disclosure condition will be more likely to remember the information in the disclosure than those in the evidence only condition. Participants in both disclosure conditions will be more likely to recall information in the disclosures than participants in the control condition.

   3. Comprehension: Participants will show the greatest accuracy in comprehension of the claims in the evidence + conclusion disclosure condition, and the least accuracy in the control condition.

   4. Valence: Participants in the control condition will be the most positive about the drug. Participants in the evidence + conclusion disclosure condition will be the least positive about the drug.

   5. Interest: Participants in the control condition will have the greatest intention to seek additional information about the drug. Participants in the evidence + conclusion disclosure condition will have the least intention to seek additional information.

   6. Behavioral intention: Participants in the control condition will be most likely to say they would request the drug if they or a loved one were diagnosed with the disease it is intended to treat (consumers) or that they would prescribe the drug to patients with the disease (HCPs). Participants in the evidence + conclusion disclosure condition will be least likely to request or prescribe the drug.

2. Disclosure Effects by Type of Claim

   1. Scope of Treatment Claim

      1. Efficacy belief: Participants in the control condition are most likely to think that the drug is successful at treating cardiovascular events beyond those that were studied by clinical trials; participants in the evidence + conclusion disclosure condition are least likely to think this.

   2. Ease of Use Claim

      1. Ease of use belief: Participants in the control condition are most likely to believe the drug is easy to take; participants in the evidence + conclusion disclosure condition are least likely to believe the drug is easy to take.

   3. Statistical Significance Claim

      1. Efficacy belief: Participants in the control condition will judge the scientific evidence of the drug’s efficacy as strongest; participants in the evidence + conclusion condition will judge it as weakest.

3. Moderators for Consideration

   1. Education/Training

      1. For consumers, years of education will be related to memory and comprehension measures, such that those with higher levels of education will remember and comprehend more claim and disclosure information.

      2. For HCPs, additional education—such as earning a PhD in addition to an MD, or having training or experience with clinical trials—will be associated with higher attention to and comprehension of claims and disclosures.

   2. Cognitive load: For both HCPs and consumers, awareness, recall, and comprehension of disclosures will decrease as cognitive load increases.

Note: Given the revisions to the disclosure conditions herein, some proposed moderators may not be necessary. Others, such as education and training differences for HCPs, will depend on obtaining sufficient variation within the sample.

Analysis Plan

During descriptive analysis, we will calculate frequency distributions and check the apparent validity of the data (i.e., range checks, frequency of missing responses, or response distribution). For continuous/ordinal variables, statistical output will include means, medians, standard deviations, ranges, and counts. For categorical variables, output will include counts and percentages.

In addition to frequency distributions, we will conduct three other types of analyses during this step. First, we will calculate reliability of composite variables and multi-item scales to determine if the individual items hang together as composite measures. Specifically, we will calculate Cronbach’s alpha for each composite variable. If alpha for a composite measure or scale does not meet our pre-established threshold of 0.75, we will discuss whether to use single-item measures rather than the composite or to consider such composites as indices (because of a theoretical reason to consider an aggregate measure regardless of item correspondence) in hypothesis testing.

Finally, we will conduct a non-response analysis to compare the distribution patterns of responders with known population distributions. These comparisons will be limited to those variables for which we have population information (e.g., sex, age, race/ethnicity).

After descriptive analyses, we will test hypothesized relationships by conducting chi-square tests for categorical outcomes and t-tests and ANOVAs for continuous outcomes to control within subsequent models for such characteristics with notable differences across groups.

We will define a set of planned contrasts to address specific research questions and hypotheses, conducting these planned comparisons based on hypothesized relationships or post-hoc comparisons to identify significant differences between specific experimental groups. To adjust for multiple comparisons, we will apply a post-hoc family-wise error-rate adjustment, such as a Bonferroni correction. If assumptions are violated for the ANOVA or categorical models mentioned above, nonparametric tests will be employed, such as the Kruskal-Wallis for independent samples or Welch’s ANOVA. Both alternatives help detect statistical differences should assumptions such as normality or equal variances not be met.

In the main test, statistical output will include, as appropriate, F or chi-square statistics, test degrees of freedom, p values, mean or proportional differences, and standardized effect sizes.

Power

We conducted a priori power analyses to ensure we obtained a sufficient sample to detect statistically significant differences in the outcome measures of interest across the different experimental conditions. The pretests, assuming the need for power of .80, alpha probability of .05, and a small effect size (f = .10), will require a sample of 500 participants for each of the two pretest experiments (total N = 1,000). For the main studies, given the experimental design and assuming a power of .90, alpha of .05, and small effect size (f = .10), we propose obtaining a sample of 1,500 participants for each study (total N = 3,000).

2. Methods to Maximize Response Rates and Deal with Non-Response

Both the pretest and main studies will be administered via Internet. To help ensure that the participation rate is as high as possible, FDA and the contractor will:

• Design a protocol that minimizes burden (reasonable in length, clearly written, and with appealing graphics).

• Use incentive rates that meet industry standards. In addition to offsetting respondent burden, using market-rate incentives tends to increase response rates, reduce sampling bias, and reduce nonresponse bias.

• Use government sponsorship on the survey invite to increase response rate. An experiment conducted by FDA and RTI¹ found that among endocrinologists, response rates were 6 percentage points higher when FDA was disclosed as the sponsor in the survey invitation than when no sponsor was listed. However, due to concerns raised in the public comments that mentioning FDA could potentially influence subjects’ responses to study questions, we will ensure that all materials reference the U.S. Department of Health and Human Services rather than FDA.

Participants in the pretest and main studies will be convenience samples, rather than probability-based samples of U.S. adults. Rather, the strength of the experimental design used in this study lies in its internal validity, on which meaningful estimates of differences across manipulated conditions can be produced and generalized. This is a counterpoint to observational survey methodologies where estimating population parameters is the primary focus of statistical analysis. The recruitment procedures in this study are not intended to fit the criteria for survey sampling, where each unit in the sampling frame has an equal probability of being selected to participate. In an observational survey study, response rates are often used as a proxy measure for survey quality, with lower response rates indicating poorer quality. Nonresponse bias analysis is also commonly used to determine the potential for nonresponse sampling error in survey estimates. However, concerns about sampling error do not generally apply to experimental designs, where the parameters of interest are under the control of the researcher—rather than being pre-established characteristics of the participants—and each participant has an equal probability of being assigned to any of the experimental conditions.

Generally, there are several approaches to conducting a nonresponse bias analysis, such as comparing response rates by subgroups, comparing respondents and nonrespondents on frame variables, and conducting a nonresponse follow-up study². For the proposed project, we will examine nonresponse for its descriptive value by comparing our full sample with population estimates for age, race, gender.

3. Test of Procedures or Methods to be Undertaken

Nine cognitive interviews were conducted per questionnaire to assess questionnaire flow and wording. We plan to conduct two pretests on a larger scale to ensure the main studies will run smoothly. We propose to test 500 individuals in each pretest.

4. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor, Fors Marsh Group, will collect and analyze the data on behalf of FDA as a task order under Contract HHSF223201510002B. Shane Mannis, Ph.D., 571-444-1109, is the Project Director for this project. Data analysis will be overseen by the Research Team, Office of Prescription Drug Promotion (OPDP), Office of Medical Policy, CDER, FDA, and coordinated by Amie C. O’Donoghue, Ph.D., 301-796-0574, and Kevin R. Betts, Ph.D., 240-402-5090.

1 Aikin, KJ; Betts, K; Boudewyns, V; Stine, A; & Southwell, B. (2016). Physician responsiveness to survey incentives and sponsorship in prescription drug advertising research. Annals of Behavioral Medicine, 50(Suppl.), s251.

2 Office of Management and Budget, Standards and Guidelines for Statistical Surveys, September, 2006. www.whitehouse.gov/sites/default/files/omb/inforeg/statpc. Last accessed April 18, 2013.

9