Supporting Statement B - Study of Clinical Efficacy Information

Study of Clinical Efficacy Information in Professional Labeling and Direct-to-Consumer (DTC) Print Advertisements for Prescription Drugs

0910-Number

SUPPORTING STATEMENT B

Submitted by

Division of Drug Marketing, Advertising, and Communications

Center for Drug Evaluation and Research

Food and Drug Administration

November, 2010

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

For the physician portion of the study 550 (50 for the pretest, 500 for the main study) participants will be recruited from a panel of over 100,000 physician members of the American Medical Association.  Active members of the physician panel are defined as panelists who have started a survey within the past 12 months or joined the panel and activated their account via double opt-in in that time period. The number of active members as defined above is at least 65% of our total panelists. Respondents are prevented from participating in a particular survey more than once.

For the consumers study, 2,100 (100 for the pretest, 2,000 for the main study) participants will be recruited from a panel of 686,000 consumers. Each panel member will complete a prescreening questionnaire, and we will recruit participants who indicated that they have been medically diagnosed with chronic pain. If necessary, we will oversample certain population segments so that the overall sample is in proportion to the U.S. adult population with chronic pain on gender, race/ethnicity, education, and income. At least 20% of the sample will have achieved a high school education or less.  The online sample is drawn from the Internet Panel according to the project needs and randomly, or using sampling techniques according to the quotas set in the questionnaire, and the anticipated response rate per quota group. The contractors will conduct checks of incidence and actual response rates, modifying the sampling plan as needed. 

2. Procedures for the Collection of Information

Design Overview

This research will be conducted in two concurrent, independent parts. The first part will involve 2,000 consumers in an examination of variations of the display page of print DTC ads for a fictitious drug that closely approximates an existing drug for chronic pain and heart attack reduction. In the second part, 500 general practitioners will review and evaluate a fictitious “approved” label for the same conditions. This design will allow us to compare consumers’ perceptions of efficacy with a more objective measure of the true efficacy of the drug as measured by physician perceptions of clinical efficacy from labeling.

Consumer Study. In this part, men and women who have been diagnosed with chronic pain will be recruited and will view one version of a DTC ad for a drug that treats chronic pain (treatment claim) and has also been shown to reduce the risk of heart attack (prevention claim). This medical condition and this type of drug afford us the ability to maintain various realistic manipulations of placebo level and type of claim, as explained below.

Participants will be randomly assigned to see one of 14 DTC print ads and will answer questions about the effectiveness and safety of the fictitious drug advertised in them. Risk information will remain constant in all experimental conditions. These 14 experimental conditions will be created by examining two independent variables (placebo rate [3 levels: small difference, large difference, none], and framing [2 levels: single, mixed]) for two different types of claims (treatment, prevention). The prevention claim study will include two additional cells that reflect a very large difference between the test drug and the placebo; this is explained in greater detail below. Please note that the numbers and particular wording describing efficacy seen in Table 4 are for illustration only. Pretesting will determine actual numbers and wording used (Please see Appendix D for the pretest questionnaire).

The structure of the two factorial designs is illustrated in Table 4.

Table 4. Consumer study design: How many experienced the following events.

		Treatment Claim Study			Prevention Claim Study
		Frame			Frame
		Single	Mixed		Single	Mixed
Placebo	Small Difference	• 30/100 on Milarix reduced pain • 20/100 without Milarix reduced pain	• 30/100 on Milarix reduced pain; 70/100 saw no improvement • 20/100 without Milarix reduced pain; 80/100 saw no improvement		• Milarix reduced the risk of heart attack. 96/100 people avoided a heart attack while on Milarix. 95/100 people avoided a heart attack without Milarix.	• Milarix reduced the risk of heart attack. 96/100 people avoided a heart attack while on Milarix; 4/100 had a heart attack. 95/100 people avoided a heart attack without Milarix; 5/100 had a heart attack.
	Large Difference	• 30/100 on Milarix reduced pain • 3/100 without Milarix reduced pain	• 30/100 on Milarix reduced pain; 70/100 saw no improvement • 3/100 without Milarix reduced pain; 97/100 saw no improvement		• Milarix reduced the risk of heart attack. 96/100 people avoided a heart attack while on Milarix. 91/100 people avoided a heart attack without Milarix.	• • Milarix reduced the risk of heart attack. 96/100 people avoided a heart attack while on Milarix; 4/100 had a heart attack. 91/100 people avoided a heart attack without Milarix; 9/100 had a heart attack.
	None	• 30/100 on Milarix reduced pain	• 30/100 on Milarix reduced pain; 70/100 saw no improvement		• Milarix reduced the risk of heart attack. 96/100 avoided a heart attack on Milarix.	• Milarix reduced the risk of heart attack. 96/100 avoided a heart attack on Milarix.; 4/100 had a heart attack.
	Very Large Difference				• Milarix reduced the risk of heart attack. 96/100 people avoided a heart attack while on Milarix. 85/100 people avoided a heart attack without Milarix.	• Milarix reduced the risk of heart attack. 96/100 people avoided a heart attack while on Milarix; 4/100 had a heart attack. 85/100 people avoided a heart attack without Milarix; 15/100 had a heart attack.

We will investigate variations in the presentation of benefit claims in two different types of claims: treatment and prevention. Treatment claims usually involve symptoms that may be alleviated by taking a given prescription drug. This type of claim is directly testable and, depending on the condition, somewhat observable by patients. If bothersome symptoms do not go away, a patient can return to the healthcare provider with this information and pursue additional options for treatment. In comparison to prevention claims, drugs that treat symptoms typically show objectively observable percentages of people who experience relief.

Prevention claims are important but potentially harder to communicate due to their long-term nature. A drug that prevents a negative future event may not alleviate any symptoms at all. Patients may feel no benefit from the drug and must trust their healthcare provider and the data, as much as they can process it, that the drug is providing a positive benefit for them. For many conditions, the events being prevented are relatively rare, and thus the numbers used to describe them are often very small. For example, a cholesterol drug that reduces the risk of heart attack from 3 out of 100 to 2 out of 100 may not seem objectively large, but from a public health perspective it has enormous consequences for millions of people and the healthcare system in general. We chose to test this type of claim to determine whether consumers are sensitive to the magnitude of the benefit in these clinically meaningful but objectively small outcomes. Although we will examine the current issues in both treatment and prevention claims, we do not intend to test comparisons between the two.

The second variable of interest is communication of placebo information. Three levels will be examined. In addition to testing a control condition with no placebo information, we will examine a small and large difference between drug and placebo rate to better understand if and how consumers use placebo information. We see three possibilities: 1) people use placebo rates correctly, such that the group shown a large difference between drug and placebo will demonstrate higher perceived efficacy than the group shown a small difference between drug and placebo; 2) people use the placebo rates as a peripheral cue to mean “scientific information” and thus do not process the content of the information, so there will be no differences between participants shown small and large differences between drug and placebo on perceived efficacy but both will be higher than the no placebo group; and 3) people do not find the numbers meaningful or cannot process them, so the ratings of participants in the small and large difference groups will not differ from one another and they will not differ from the no placebo group. In an attempt to make our claims as realistic as possible in the prevention design, we will maintain fairly low placebo rates in the large difference and small difference placebo conditions. However, to provide confidence that our research manipulations are operating as we expect, we will also have two additional conditions in the prevention design in which the placebo rate is very large—higher than could reasonably be expected but large enough to be objectively noticeable (e.g., risk of heart attack on Milarix, 4/100; risk of heart attack on placebo, 15/100).

Finally, we will examine the addition of mixed framing to the traditional use of a single positive frame in a DTC ad. Mixed framing provides the number of people who benefited and the number of people who did not benefit, whereas positive, or single, framing provides only the number of people who benefited. Only a few studies have actually measured this mixed approach¹ although risk communication guides recommend the use of mixed framing to create more accurate perceptions.² Although a completely balanced design would also include a negative framing condition (which would provide only the number of people who did not benefit), we feel it is unrealistic to create an ad that would suggest, for example, that “Drug X did not work for 70% of people in clinical trials,” so we have chosen not to include negative framing in our investigation.

In this part of the project, we are most interested in consumers’ perceived efficacy and safety, which we can then descriptively compare with ratings physicians will provide based on the prescribing information, described in the next section. We will also ask participants questions to measure their accuracy with regard to claims, their recall of the information in the ad, and characteristics that may influence their responses, such as demographics, numeracy, and knowledge about their medical condition.

After completing the main part of the study, consumer participants will be asked to complete a task examining the impact of a qualitative frame on perceptions of risk. The summary of product risks on the display page of a DTC prescription drug ad is typically accompanied by a title or call out, designed to draw the reader’s attention to the information. There is no standard language for that call out, though many sponsors choose to use the title “Important Risk Information” or “Important Safety Information.” Research has shown that words used to frame information can influence the interpretation of the information;³ for example, describing the information as “safety” information may lead a reader to one particular interpretation of the riskiness of the product, compared to a reader who sees the information framed as “risk” information. In this task, participants will read the risk section from a DTC prescription drug ad that uses the title “Important Risk Information” or “Important Safety Information” and be asked their perceptions of the product’s risks.

Physician Study. In this part, five hundred general practitioners⁴ will participate in an Internet survey lasting no longer than 20 minutes. They will complete two tasks during this time. In the first task, they will evaluate a prescription drug label written in the content and format labeling rule format⁵ (also known as the prescribing information, written for healthcare practitioners) for the fictitious drug described in the consumer study. All physicians will see the same risk information profile. To provide a match for the variations of information in the DTC ads the consumers will read, physicians will be randomly assigned to see prescribing information that varies in terms of placebo rates in clinical trials and will be randomly assigned to answer questions about either the treatment or prevention indications of a fictitious drug called Milarix in a 2 x 2 manner as follows:⁶

Table 5: Physician Study Design

		Type of Claim
		Treatment Claim	Prevention Claim
Difference between Drug and Placebo	Small Difference
	Large Difference

As part of this task, we will obtain timing and sequence information on which sections of the label physicians examine (Pretest questionnaire available in Appendix E). This will enable us to have a deeper understanding of physicians’ processing of the prescribing information, including which sections they read, how long they spend on each section, and the order in which they read the sections. We are not aware of existing literature on this topic. Additionally, physicians will answer questions about the efficacy and safety of the drug and quantitative questions about the benefit shown in the clinical studies (as described in the label). These questions have been designed such that they can be reasonably compared with the responses of consumers who will answer the same questions after viewing a corresponding DTC ad (see “Consumer Study” section above).

Physicians will also be asked to perform a separate judgment task. In this separate task, physicians will see four versions of a print DTC ad for a fictitious product that treats high cholesterol. The versions will vary in terms of the presence or absence of placebo and the single or mixed frame. Physicians will rank the ads in order of their preference for the display of clinical data and how useful they believe the ads would be for their patients. To reduce burden, the physician sample will be randomly split in this task, such that half of the physicians see the four ad versions with treatment claims and the other half see the four ad versions with prevention claims. Type of claim is described in greater detail in the consumer experiment section. In this task, the main measure of interest is not how effective physicians think the drug is but rather whether placebo information and framing manipulations alter their view of what is better for communicating numerical information to patients.

Thus, this research will provide us with a rich data set in order to address several questions: 1) how physicians process clinical efficacy information and how they use approved product label information; 2) what physicians’ preferences are for alternative DTC ad presentations; and 3) which variations of information in DTC ads bring consumers closer to or farther away from the conclusions of the physicians regarding the same drugs.

Procedure

All parts of this study will be administered over the internet. A total of 2,000 consumer interviews and 500 physician interviews will be completed. In the consumer study, consumer participants will be randomly assigned to view one version of a DTC prescription drug print ad which consists of a display page and the accompanying brief summary page. Following their perusal of this document, they will answer questions about their recall and understanding of the benefit and risk information, their perceptions of the benefits and risks of the drug, and their intent to ask a doctor about the medication. As part of a separate task, they will also read a short description of another prescription drug’s risks framed either as “safety” or “risk” information and answer three questions about it.

In the physician study, physician participants will be randomly assigned to see one version of the prescription drug prescribing information and will answer questions about their recall and perceptions of the document. They will then see four versions of a DTC ad for an unrelated product and will rank these ads in terms of perceived scientific accuracy and perceived ease of patient understanding.

For both parts, demographic and numeracy information will be collected. In addition, consumers will answer questions about their familiarity with their medical condition and physicians will answer questions about their practice and career. The entire procedure is expected to last approximately 20 minutes. This will be a one-time (rather than annual) information collection.

Participants

Data will be collected using an Internet protocol. Approximately 2,000 consumers who have been diagnosed with chronic pain will be recruited for the consumer study. Five hundred general practitioners who see patients at least 50% of the time and have been in practice for more than three years will be recruited for the physician study. Because the task presumes basic reading abilities, all selected participants must speak and read English fluently. Participants must be 18 years or older.

Hypotheses

Consumer Study

We will conduct two separate study arms with consumers; one will evaluate responses to a treatment claim and the other will evaluate responses to a prevention claim. We will not test comparisons between the treatment and prevention designs.

Both arms of the study will investigate the role of placebo rates, framing, and any interaction between the two variables on measures such as perceived efficacy and risk, the likelihood of taking the drug, and specific benefit accuracy questions. It should be noted that we will measure perceived risk because it is an important variable, but we do not expect any of our manipulations to influence perceived risk.

Placebo Hypotheses. One goal of this research is to determine whether consumers attend to placebo information when learning about the effectiveness of a new drug. We have manipulated placebo rates in three ways such that each consumer will see either 1) no placebo information, 2) a small difference between drug and placebo efficacy rate, or 3) a large difference between drug and placebo efficacy rate. If participants process placebo rates, we should see differences among all conditions in perceived efficacy and likelihood of taking the drug. If participants use placebo rates as a heuristic or peripheral cue, we expect that participants who see a small or large difference between the drug and placebo rates will differ from the no placebo condition in perceived efficacy and likelihood of taking the drug, but not from each other. Finally, if participants do not process placebo rates at all, we would expect to find no differences among conditions in perceived efficacy and likelihood of taking the drug.

We expect numeracy and severity of condition to moderate the effect of placebo rate, such that those with higher numeracy and more severe conditions will be more likely to process the placebo information and thus it is more likely that that we will see differences among all placebo conditions in high numeracy participants compared with low numeracy participants and less severe conditions.

It should be noted that our prevention claim design includes an additional level of placebo (very large difference between drug and placebo), representing an extra high prevalence of heart attacks without treatment. We added this condition to ensure that the numbers were distinct enough to find an effect. The other conditions may stretch the bounds of realism, but are within reasonable limits as to how many heart attacks might be expected in the general population. This condition provides a research check to investigate effects. Thus, we expect that participants in this condition will be more likely than the other conditions to perceive the drug as effective and will report a higher likelihood of taking the drug.

Framing Hypotheses. We are investigating whether providing a mixed framing presentation provides additional information for consumers above and beyond that provided by a single frame, as is currently typical. We expect that consumers in the mixed frame condition will reveal better benefit accuracy than those in the single frame condition. We also predict that participants in the mixed frame condition will spend more time looking at the ad, as there is more information to absorb.

For framing, we have competing hypotheses about the role of numeracy. The mixed frame may help low numeracy participants understand the numerical information better; if so, then we would expect that differences between low and high numeracy participants to be greater in the single frame, compared to mixed frame, conditions. However, the mixed frame may present too many numbers for low numeracy participants to process, in which case we would not expect numeracy to moderate the effect of framing.

Interaction of Placebo and Framing Hypotheses. We will investigate interactions between placebo rates and framing for completeness, but at this time these analyses are exploratory.

Safety/Risk Terminology Hypotheses. The second task for consumers is to read a short paragraph about the risks of a different prescription drug for another medical condition and to answer questions about it. Participants will be randomly assigned to see either the title “Important Risk Information” or “Important Safety Information;” all other information in the paragraph will be identical. We predict that perceived risk will be greater when the word risk is used rather than the word safety.

Physician Study

Both parts of the physician study are exploratory. The purpose of the first task, wherein physicians look through the prescribing information and answer questions about it, is to obtain data on which sections of the label physicians look at, how long they spend on each section, and in what order they look at information. We will examine this by itself and also in relation to covariates such as length of time in practice, number of prescriptions written per week, and familiarity with the drug class. We will show half of the physicians a label for which the difference between drug and placebo rates are small and the other half a label for which the difference between drug and placebo rates are large; we will randomly assign physicians to answer questions about the treatment or prevention indications. If physicians attend to placebo information as we expect them to, we predict that physicians who see the label for the drug with the small difference will perceive it as less effective than the drug with the large difference and will report less intention to prescribe it. Nevertheless, if physicians do not attend to placebo rates, we will see no differences in these variables, because the treatment rate itself is held constant.

The second task that physicians will complete is a ranking task of four different DTC ads for an unrelated drug product. They will rank order the versions on the basis of accuracy of scientific information and ease of patient understanding. In addition to an overall ranking, we will examine the rankings in relation to covariates such as attitudes toward DTC, length of time in practice, and number of prescriptions written per week.

Physician and Consumer Comparison

Physicians will read through the prescribing information for the fictitious drug Milarix and answer specific questions about the benefits of the drug. Consumers will read through a DTC ad for Milarix and will answer the same questions about the drug. We will examine the responses to determine whether there is any concurrence between the responses of the physicians and the responses of the consumers. We predict that consumers who see the mixed frame presentation will have responses closer to those of the physicians than consumers who see the single frame presentation. Any comparisons we make will be explicitly defined as exploratory in nature.

Analysis Plan

Consumer Study

Treatment Claim Study Arm. Composite measures will be created for perceived efficacy (questions 1, 2, 7, 9, and 10), and perceived risk (questions 5, 6, and 8) if analyses with Cronbach’s alpha reveal that these measures have acceptable reliability (α > .70).

Five separate 3 x 2 (placebo rate x framing) ANOVAs will be conducted for each dependent variable of interest: perceived efficacy, the likelihood of taking the drug, perceived risk, benefit accuracy, and time spent looking at the first page of the ad. We will conduct these analyses both with and without covariates (e.g., demographic characteristics) included in the model. In addition, we will test whether any main effects are moderated by other measured variables (e.g., numeracy). If the main effects are significant, we will conduct pairwise-comparisons to determine which conditions are significantly different from one another. Because we have multiple comparisons, we will make Bonferroni adjustments as needed to an initial alpha of .05.

Prevention Claim Study Arm. Composite measures will be created for perceived efficacy (questions 1, 2, 7, 9, and 10), and perceived risk (questions 5, 6, and 8) if analyses with Cronbach’s alpha reveal that these measures have acceptable reliability (α > .70).

Five separate 4 x 2 (placebo rate x framing) ANOVAs will be conducted for each dependent variable of interest: perceived efficacy, the likelihood of taking the drug, perceived risk, benefit accuracy, and time spent looking at the first page of the ad. We will conduct these analyses both with and without covariates (e.g., demographic characteristics) included in the model. In addition, we will test whether any main effects are moderated by other measured variables (e.g., numeracy). If the main effects are significant, we will conduct pairwise-comparisons to determine which conditions are significantly different from one another. Because we have multiple comparisons, we will make Bonferroni adjustments as needed to an initial alpha of .05.

Supplementary Investigation of Safety/Risk Terminology. For the examination of safety/risk wording that all consumer participants will see, a t-test will be conducted between participants who see the two different headlines to determine if they demonstrate a significant difference in perceived risk or likelihood of taking the drug.

Physician Study

Descriptive statistics will be obtained on timing variables, sections selected, and order variables overall and by covariates, including demographics, time in practice, familiarity with medical condition, and number of prescriptions written per week. T-tests will be conducted to determine whether physicians who see the high (large) placebo version differ from those who see the low (small) placebo version in perceived efficacy and likelihood to prescribe.

In the second task, rankings will be calculated for the perceived scientific accuracy of four DTC ad versions overall. We will also investigate the rankings by placebo condition and covariates, including the same covariates as above plus attitude toward DTC, to determine if there are any differences although this analysis is exploratory. We will conduct the same analysis for the perceived ease of patient understanding of the four DTC ads.

Physician and Consumer Comparison

We will examine whether presenting information in a single frame or a mixed frame brings consumers closer or farther away from physician ratings. For example, among those who saw a treatment claim with a high placebo rate, were consumers who saw a mixed frame closer to the responses of the physicians than those consumers who saw a single frame? Physicians who see either small or large difference between drug and placebo rates and who see either prevention or treatment claim information will be analyzed in four separate groups, as will consumers. Within each group, responses to specific benefit accuracy questions will be combined into a composite measure if Cronbach’s alpha displays acceptable reliability (α > .70).

Power

The following assumptions were made in deriving the sample size for the consumer study: 1) 0.05 alpha and 0.90 power and 2) an effect size between small and medium. The tables below show the sample size required to detect differences with effect sizes ranging from conventionally “small” (f = 0.1) to almost “medium” (f = 0.25) for the largest comparison we plan to analyze.

Table 6. Power Analysis Calculation for Consumer Treatment Claim Study

A priori power analysis to determine sample size needed in F tests (ANCOVA: fixed effects, main effects, and interactions) to achieve power of 0.90 (Faul et al., 2007).7
		Effect size f*
Input
		0.10	0.12	0.14
	α error probability	0.05	0.05	0.05
	Power ( 1 – β error probability)	0.90	0.90	0.90
	Numerator df	2	2	2
	Number of groups	6	6	6
	Number of covariates	5	5	5
Output
	Noncentrality parameter λ	12.69	12.70	12.72
	Critical F	3.00	3.01	3.01
	Denominator df	1,262	875	642
	Total sample size	1,269	882	649
	Actual power	0.90	0.90	0.90

*An effect size of 0.10 is traditionally considered small, whereas an effect size of 0.25 is considered medium (Cohen, 1988).⁸ Here we have shown three different effect sizes centering around small to medium effects to show that we will be able to detect fairly small effects.

We will have 140 participants per cell, with a total of 840 participants in the 6 cells represented in the table (2 x 3).

Table 7: Power Analysis Calculation for Consumer Prevention Claim Study

A priori power analysis to determine sample size needed in F tests (ANCOVA: fixed effects, main effects, and interactions) to achieve power of 0.90 (Faul et al., 2007).9
		Effect size f*
Input
		0.10	0.12	0.14
	α error probability	0.05	0.05	0.05
	Power ( 1 – β error probability)	.90	.90	.90
	Numerator df	3	3	3
	Number of groups	8	8	8
	Number of covariates	5	5	5
Output
	Noncentrality parameter λ	14.22	14.24	14.25
	Critical F	2.61	2.61	2.62
	Denominator df	1,415	982	720
	Total sample size	1,422	989	727
	Actual power	0.90	0.90	0.90

*An effect size of 0.10 is traditionally considered small, whereas an effect size of 0.25 is considered medium (Cohen, 1988).¹⁰ Here we have shown three different effect sizes centering around small to medium effects to show that we will be able to detect fairly small effects.

We will have 145 participants per cell, with a total of 1,160 participants in the 8 cells represented in the table (2 x 4).

The following assumptions were made in deriving the sample size for the physician study: 1) 0.05 alpha and 0.90 power and 2) an effect size between medium and large. The tables below show the sample size required to detect differences centering around conventionally “medium” (f = 0.25) and heading toward “large” (f = 0.50) effects for the largest comparison we plan to analyze, which is a t-test.

Table 8: Power Analysis Calculation for Physician Study

A priori power analysis to determine sample size needed in independent groups t- test to achieve power of 0.90 (Faul et al., 2007).11
		Effect size f*
Input
		0.25	0.30	0.35
	α error probability	0.05	0.05	0.05
	Power ( 1 – β error probability)	0.90	0.90	0.90
	Allocation ratio	1	1	1
Output
	Noncentrality parameter δ	3.25	3.25	3.25
	Critical t	1.96	1.97	1.97
	Df	674	468	344
	Sample size group 1	338	235	173
	Sample size group 2	338	235	173
	Total sample size	676	470	346
	Actual power	0.90	0.90	0.90

*An effect size of 0.25 is traditionally considered medium (Cohen, 1988).¹² Here we have shown three different effect sizes centering around medium and heading toward large effects to show that we will be able to detect medium to large effects.

We will have 250 participants per cell, with a total of 500 participants in the 2 cells represented in each t-test.

3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response

Response rates can vary greatly depending on many factors including the sample composition, panel type, invitation content, time of day and incentive offering. In addition, outside factors including email filters, recipient ISP downtime and general conditions on the Internet can impact response rates.  We calculate response rate as ratio of the number of surveys started by the number of panelists contacted by invitation. Response rates range from 15% to over 50%, with an expected 20-25% response rate for typical surveys conducted with these internet panels. To help ensure that the participation rate is as high as possible, FDA and the contractor will:

• Design an experimental protocol that minimizes burden (short in length, clearly written, and with appealing graphics);

• Administer the experiment over the Internet, allowing respondents to answer questions at a time and location of their choosing;

• Sending out two email reminders after the initial invitation. 

• Provide respondents with a helpdesk link that they can access at any time for assistance.

Additionally, the Panel leverages the social media concept and has developed ‘panel communities’ in order to maximize member engagement and overcome challenge of declining survey response rates and multi-panel membership.

4. Test Procedures

The contractor will run nine participants through the procedure to assess blatant glitches in questionnaire wording, programming, and execution of the study. We will also conduct pretests with 50 physicians and 100 consumers before running the main studies to ensure that stimuli and questionnaire wording is clear. Finally, we will run the main studies as described elsewhere in this document.

5. Individuals Involved in Statistical Consultation and Information Collection

The contractor, Decision Partners, and its subcontractor, Penn, Schoen, and Berland, will collect the information on behalf of FDA as a task order under Contract HHSF223200510007I. Sara Eggers is the Project Director for this project, telephone (919) 419-8939. Data analysis will be conducted primarily by the Research Team, Division of Drug Marketing, Advertising, and Communications (DDMAC), Office of Medical Policy, CDER, FDA, and coordinated by Amie C. O’Donoghue, Ph.D., 301-796-0574 and Kathryn J. Aikin, Ph.D., 301-796-0569.

APPENDIX A

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Food and Drug Administration

[Docket No. FDA-2010-N-0266]

Agency Information Collection Activities; Proposed Collection; Comment Request; Study of Clinical Efficacy Information in Professional Labeling and Direct-to-Consumer (DTC) Print Advertisements for Prescription Drugs.

AGENCY: Food and Drug Administration, HHS.

ACTION: Notice.

SUMMARY: The Food and Drug Administration (FDA) is announcing an opportunity for public comment on the proposed collection of certain information by the agency. Under the Paperwork Reduction Act of 1995 (the PRA), Federal agencies are required to publish notice in the FEDERAL REGISTER concerning each proposed collection of information and to allow 60 days for public comment in response to the notice. This notice solicits comments on Study of Clinical Efficacy Information in Professional Labeling and Direct-to-Consumer (DTC) Print Advertisements for Prescription Drugs. This study is designed to investigate efficacy and effectiveness information of prescription drugs as conveyed to healthcare providers through approved labeling and to consumers through print advertisements.

DATES: Submit written or electronic comments on the collection of information by [insert date 60 days after date of publication in the FEDERAL REGISTER].

ADDRESSES: Submit electronic comments on the collection of information to http://www.regulations.gov. Submit written comments on the collection of information to the Division of Dockets Management (HFA-305), Food and Drug Administration, 5630 Fishers Lane, Rm. 1061, Rockville, MD 20852. All comments should be identified with the docket number found in brackets in the heading of this document.

FOR FURTHER INFORMATION CONTACT:

Elizabeth Berbakos,

Office of Information Management (HFA-710),

Food and Drug Administration,

5600 Fishers Lane,

Rockville, MD 20857,

301-796-3792.

SUPPLEMENTARY INFORMATION: Under the PRA (44 U.S.C. 3501-3520), Federal agencies must obtain approval from the Office of Management and Budget (OMB) for each collection of information they conduct or sponsor. "Collection of information" is defined in 44 U.S.C. 3502(3) and 5 CFR 1320.3(c) and includes agency requests or requirements that members of the public submit reports, keep records, or provide information to a third party. Section 3506(c)(2)(A) of the PRA (44 U.S.C. 3506(c)(2)(A)) requires Federal agencies to provide a 60-day notice in the FEDERAL REGISTER concerning each proposed collection of information before submitting the collection to OMB for approval. To comply with this requirement, FDA is publishing notice of the proposed collection of information set forth in this document.

With respect to the following collection of information, FDA invites comments on these topics: (1) Whether the proposed collection of information is necessary for the proper performance of FDA's functions, including whether the information will have practical utility; (2) the accuracy of FDA's estimate of the burden of the proposed collection of information, including the validity of the methodology and assumptions used; (3) ways to enhance the quality, utility, and clarity of the information to be collected; and (4) ways to minimize the burden of the collection of information on respondents, including through the use of automated collection techniques, when appropriate, and other forms of information technology.

Study of Clinical Efficacy Information in Professional Labeling and

Direct-to-Consumer (DTC) Print Advertisements for Prescription Drugs

Section 1701(a)(4) of the Public Health Service Act (42 U.S.C. 300u(a)(4)) authorizes the Food and Drug Administration (FDA) to conduct research relating to health information. Section 903(b)(2)(c) of the Federal Food, Drug, and Cosmetic Act (FD&C Act) (21 U.S.C. 393(b)(2)(c)) authorizes FDA to conduct research relating to drugs and other FDA regulated products in carrying out the provisions of the FD&C Act.

FDA regulations require that an advertisement that makes claims about a prescription drug include a “fair balance” of information about the benefits and risks of the advertised product, in terms of both content and presentation (21 CFR 202.1(e)(5)(ii)). In past research FDA has focused primarily on the risk component of the risk-benefit ratio. In the interest of thoroughly exploring the issue of fair balance, however, the presentation of effectiveness, or benefit, information is equally important.

The Federal Food, Drug, and Cosmetic Act (the Act) requires that manufacturers, packers, and distributors (sponsors) who advertise prescription human and animal drugs, including biological products for humans, disclose in advertisements certain information about the advertised product's uses and risks.¹³ By its nature, the presentation of this risk information is likely to evoke active trade-offs by consumers, i.e., comparisons with the perceived risks of not taking treatment, and comparisons with the perceived benefits of taking a treatment.¹⁴ Since FDA has an interest in fostering safe and proper use of prescription drugs, an activity that engages both risks and benefits, an in-depth understanding of consumers’ processing of this information is central to this regulatory task.

Research and guidance to sponsors on how to present benefit and efficacy information in prescription drug advertisements is limited. For example, “benefit claims,” broadly defined, appearing in advertisements are often presented in general language that does not inform patients of the likelihood of efficacy and are often simply variants of an “intended use” statement.¹⁵ In a content analysis of DTC advertising,¹⁶ the researchers classified the “promotional techniques” used in the advertisements. Emotional appeals were observed in 67% of the ads while vague and qualitative benefit terminology was found in 87% of the ads. Only 9% contained data. For risk information, however, half the advertisements used data to describe side-effects, typically with lists of side-effects that generally occurred infrequently.

FDA regulations require that prescription drug advertisements that make (promotional) claims about a product also include risk information in a “balanced” manner (21 CFR 202.1(e)(5)(ii)), both in terms of the content and presentation of the information. This balance applies to both the front (aka “display”) page of an advertisement, as well as the brief summary page. However, beyond the “balance” requirement limited guidance and research exists to direct or encourage sponsors to present benefit claims that are informative, specific, and reflect clinical effectiveness data.

The purpose of this project is to (1) understand how physicians process clinical efficacy information and how they interpret approved product label information;¹⁷ (2) determine physician preferences for alternative presentations of clinical efficacy information in direct-to-consumer (DTC) advertising; and (3) examine how different presentations of clinical efficacy information in DTC advertising affect consumers’ perceptions of efficacy and safety. Specifically, we are interested in how physicians and consumers make risk/benefit assessments and particularly, how consumers make such judgments in response to variations in the efficacy presentations in the “display” (first) page of a DTC print ad. A particular concern is whether certain presentations cause consumers to form skewed perceptions or unfounded risk/benefit tradeoffs. Therefore, we will investigate to what extent consumers, when provided with efficacy information, form perceptions that correspond with clinically-based physicians’ assessments of the benefits, risks, and benefit/risk tradeoffs of the same drugs. These studies will inform FDA’s thinking regarding how manufacturers may provide useful and non-misleading efficacy information in DTC print advertisements.

Design Overview

This study will be conducted in two concurrent, independent parts. The first part will involve 2,500 consumers in an experimental examination of variations of the display page of print DTC ads for two fictitious drugs, closely approximating existing drugs for overactive bladder (OAB) and benign prostatic hyperplasia (BPH). In the second part, 600 general practitioners will review and evaluate a fictitious “approved” label for the same conditions. This design will allow us to compare consumers’ perceptions of efficacy with a more objective measure of the true efficacy of the drug as measured by physician perceptions of clinical efficacy from labeling.

Consumer experiment. In this part of the study, women who have been diagnosed with or are at risk for OAB (self-designated based on relevant symptoms) will be recruited and will view one version of a DTC ad for a drug to treat OAB. Men who have been diagnosed with or are at risk for BPH (self-designated based on relevant symptoms) will be recruited and will view one version of a DTC ad for a drug to treat BPH. Although the two conditions are somewhat specific to gender (men can suffer from OAB but it is much more prevalent in women), they share many of the same symptoms and characteristics. These medical conditions afford us the ability to maintain various realistic manipulations of placebo level and type of claim, as explained below. The graphical elements and construction of the two ads will be comparable yet still realistic.

Consumers will be randomly assigned to see one of twelve DTC print ads within their respective medical condition and will answer questions about the effectiveness and safety of the fictitious drug advertised in them. These twelve experimental conditions will be created by examining three independent variables in the following manner: type of claim (2 levels: treatment, prevention), placebo rate (3 levels: high, low, none), and framing (2 levels: single, mixed). Please note that the numbers describing efficacy seen in the table are for illustration only. Actual numbers used will be determined by pretesting.

		Treatment Claim Study			Prevention Claim Study
		Frame			Frame
		Single	Mixed		Single	Mixed
Placebo	High	• 30/100 on Drug X reduced urinary frequency and urgency • 20/100 without Drug X reduced urinary frequency and urgency	• 30/100 on Drug X reduced urinary frequency and urgency; 70/100 saw no improvement • 20/100 without Drug X reduced urinary frequency and urgency; 80/100 saw no improvement		• Diagnosed with bladder cancer on Drug X: 4/100 • Diagnosed with bladder cancer without Drug X: 5/100	• Diagnosed with bladder cancer on Drug X: 4/100; Not diagnosed with bladder cancer on Drug X: 96/100 • Diagnosed with bladder cancer without Drug X: 5/100; Not diagnosed with bladder cancer without Drug X: 95/100
	Low	• 30/100 on Drug X reduced urinary frequency and urgency • 3/100 without Drug X reduced urinary frequency and urgency	• 30/100 on Drug X reduced urinary frequency and urgency; 70/100 saw no improvement • 3/100 without Drug X reduced urinary frequency and urgency; 97/100 saw no improvement		• Diagnosed with bladder cancer on Drug X: 4/100 • Diagnosed with bladder cancer without Drug X: 9/100	• Diagnosed with bladder cancer on Drug X: 4/100; Not diagnosed with bladder cancer on Drug X: 96/100 • Diagnosed with bladder cancer without Drug X: 9/100; Not diagnosed with bladder cancer without Drug X: 91/100
	None	• 30/100 on Drug X reduced urinary frequency and urgency	• 30/100 on Drug X reduced urinary frequency and urgency; 70/100 saw no improvement		• Diagnosed with bladder cancer on Drug X: 4/100	• Diagnosed with bladder cancer on Drug X: 4/100; Not diagnosed with bladder cancer on Drug X: 96/100
Extra High Efficacy					• Diagnosed with bladder cancer on Drug X: 4/100 • Diagnosed with bladder cancer without Drug X: 15/100	• Diagnosed with bladder cancer on Drug X: 4/100; Not diagnosed with bladder cancer on Drug X: 96/100 • Diagnosed with bladder cancer without Drug X: 15/100; Not diagnosed with bladder cancer without Drug X: 85/100

We will investigate variations of numerical presentation in two different types of claims: treatment and prevention. Treatment claims usually involve symptoms that may be alleviated by taking a given prescription drug. This type of claim is directly observable and somewhat testable by patients. If bothersome symptoms do not go away, a patient can return to the healthcare provider with this information and pursue additional options for treatment. In general, drugs that treat symptoms typically show substantial percentages of people who experience relief.

Prevention claims are important but due to their long-term nature, potentially harder to communicate. A drug that prevents a negative future event may not alleviate any symptoms at all. Patients may feel no benefit from the drug and must trust their healthcare provider and the data, as much as they can process it, that the drug is providing a positive benefit for them. The nature of these claims is such that the event being prevented is relatively rare, and thus the numbers used to describe them are often very small. For example, a cholesterol drug that reduces the risk of heart attack from 3 out of 100 to 2 out of 100 may not seem objectively large, but has enormous consequences for millions of people and the healthcare system in general. We chose to test this type of claim to determine whether consumers are sensitive to the magnitude of the benefit in these clinically meaningful but objectively small and usually asymptomatic outcomes. While we will examine the current issues in both treatment and prevention claims, we do not intend to make comparisons between the two.

The second variable of interest is communication of a placebo rate. Three levels will be examined. In addition to testing a control condition with no placebo information, we will utilize a high and low placebo rate to better understand if and how consumers use placebo information. We see three possibilities: 1) people use placebo numbers correctly, such that the low placebo group demonstrates higher perceived efficacy than the high placebo group; 2) people use the placebo numbers as a peripheral cue to mean “science” so there are no differences between high and low placebo groups on perceived efficacy but both are higher than the no placebo group; and 3) people do not find the numbers meaningful or cannot process them, so the high and low groups do not differ from one another and they do not differ from the no placebo group. In an attempt to make our claims as realistic as possible, we will maintain fairly low rates of prevention in the prevention conditions. For this reason, in addition to the 12 cells above, we will also have an additional control cell in which the effectiveness rates are quite high—higher than could reasonably be expected but high enough to be objectively noticeable (e.g., risk of bladder cancer on Drug X, 4/100; risk of bladder cancer on placebo, 15/100). This additional condition will provide confidence that our research manipulations are operating as we expect.

Finally, we will examine the addition of mixed framing to the traditional use of a single positive frame in a DTC ad. Mixed framing provides the number of people who benefited and the number of people who did not benefit, whereas positive framing provides only the number of people who benefited. Only a few studies have actually measured this mixed approach ¹⁸ although risk communication guides recommend the use of mixed framing to create more accurate perceptions.¹⁹ Although a completely balanced design would also include a negative framing condition (which would provide only the number of people who did not benefit), we feel it is unrealistic to create an ad that would suggest, for example, that “Drug X did not work for 70% of people in clinical trials,” so we have chosen not to include negative framing in our investigation.

In this part of the project, we are most interested in consumers’ perceived efficacy and safety, which we can then compare with ratings physicians will provide based on the prescribing information, described in the next section. We will also ask consumers questions to measure their accuracy with regard to claims, their recall of the information in the ad, and demographic questions that may influence their responses, such as knowledge about their medical condition and their level of numeracy.

Physician Study. Six hundred general practitioners²⁰ will participate in an Internet survey lasting no longer than 20 minutes. They will complete two tasks during this time. In the first task, they will evaluate a prescription drug label (also known as the prescribing information, written for healthcare practitioners) for one of the two fictitious drugs described in the consumer study below. To provide a match for the variations of information in the DTC ads the consumers will observe, physicians will be randomly assigned to see prescribing information that varies in terms of claim type, placebo rates in clinical trials, and the medical condition the drug treats (OAB or BPH).

As part of this task, we will obtain timing and sequence information on which sections of the label physicians examine. This will enable us to have a deeper understanding of physicians’ processing of the prescribing information. We are not aware of existing literature on this topic. Additionally, physicians will answer questions about the efficacy and safety of the drug and quantitative questions about the benefit shown in the clinical studies (as described in the label). These questions have been designed such that they can be reasonably compared with the responses of consumers who will answer the same questions after viewing a corresponding DTC ad.

In the second task, physicians will see four versions of a print DTC ad for a fictitious product for high cholesterol and will rank the ads in order of how representative of the clinical data as the physicians know it the ads are and how useful they believe the ads would be for their patients.²¹ The four versions will be selected to mirror the versions of the OAB/BPH drug that consumers will see in the consumer experiment (i.e., low placebo, frame).

Thus, this research will provide us with a rich data set in order to address several questions: (1) how physicians process clinical efficacy information and how they use approved product label information; (2) how physicians’ interpretations of clinical efficacy information relate to their preferences for alternative DTC ad presentations; and (3) which variations of information in DTC ads bring consumers closer to or farther away from the conclusions of the physicians regarding the same drugs.

FDA estimates the burden of this collection of information as follows:

The total respondent sample for this data collection is 3,400. We estimate the response burden to be 20 minutes in the first part and 15 minutes in the second part, for a burden of 906 hours.

The response burden chart is listed below.

Table 1.--Estimated Annual Reporting Burden1
21 CFR Section	No. of Respondents	Annual Frequency per Response	Total Annual Responses	Hours per Response	Total Hours	Total Capital Costs	Total Operating & Maintenance Costs
Physician survey-pretest	100	1	1	20/60	33
Physician survey-main study	600	1	1	20/60	200
Consumer experiment-pretest	200	1	1	15/60	50
Consumer experiment-main study	2,500	1	1	15/60	625
Total	3,400				908

¹ There are no capital costs or operating and maintenance costs associated with this collection of information.

Please note that on January 15, 2008, the FDA Division of Dockets Management Web site transitioned to the Federal Dockets Management System (FDMS). FDMS is a Government-wide, electronic docket management system. Electronic comments or submissions will be accepted by FDA only through FDMS at http://www.regulations.gov.

Dated: ______________________.

_______________________________

APPENDIX B

Consumer Questionnaire

Sample: Men and women who have been diagnosed with chronic pain

N = 2000

2 Studies: one with treatment claims, one with prevention claims

Treatment study: claim Reduced symptoms of pain

6 conditions (140 per cell: 70 men, 70 women)

Prevention study: claim  Reduces the likelihood of heart attack

8 conditions (145 per cell: 72 men, 73 women)

Context: New drug, Milarix (lexisalicylic acid and milaristatin calcium)

Variables for both studies:

• 3 (Placebo: small difference, large difference, none)

• 2 (Frame: positive, mixed)

• Prevention study also has two conditions to represent extremely high efficacy (e.g., Milarix risk 4/100 – placebo risk 15/100) for research control

*Administer informed consent procedures.*

Consumer Questionnaire:

[PROGRAMMER: We need to record time in milliseconds spent on each screen (including questions) throughout protocol.]

[PROGRAMMER: Randomly assign participants to conditions as described above.]

Introductory language on at least three screens (to obtain baseline reading speeds).

Thank you for taking time from your busy schedule to take part in this research. Your answers will remain confidential – that is, the responses of all participants will be looked at together and your personal responses will not be traced to your name.

[PROGRAMMER: New screen]

This study is about a new product for treating chronic pain. Please look at the following magazine ad as you normally would on your own and answer the questions that follow.

[PROGRAMMER: New screen]

On the next screen you will see a magazine ad for a new prescription drug to treat chronic pain, Milarix (lexisalicylic acid and milaristatin calcium). You will first see the two pages of the ad in their entirety and then you will be able to click on different parts of the ad to read them thoroughly if you are interested. This means you will be able to explore each section in further detail by clicking on the section that interests you.

[PROGRAMMER: Display proper ad version. Record time spent on each page separately as well as time spent on each section, and order of sections chosen.]

(Perceived Benefit)

[PROGRAMMER: Randomize the order of Q1 and Q2]

Q1. Based on the ad you saw, how effective would Milarix be for you?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

Q2. Based on the ad you saw, how well would Milarix work for you?

1 2 3 4 5 6 7

Not at all Moderately Very

well well well

Q3. (open-ended) Please explain why you rated the effectiveness of Milarix as you did.

Possible codes:

Numbers

People in ad

Text

Side effects

Other

Q4. How likely would you be to take this drug if your doctor prescribed it?

Very likely

Somewhat likely

Somewhat unlikely

Very unlikely

Q1a. What about the drug caused you to select that answer? (open-ended)

Possible codes:

Numbers

People in ad

Text

Side effects

Other

(Perceived Safety)

[PROGRAMMER: Randomize the order of Q5 and Q6]

Q5. Based on the ad you saw, how safe do you think Milarix would be for you?

1 2 3 4 5 6 7

Not at all Moderately Very

Safe safe safe

Q6. Based on the ad you saw, how risky do you think Milarix would be for you?

1 2 3 4 5 6 7

Not at all Moderately Very

risky risky risky

(Perceived Comparative Benefit)

[PROGRAMMER: Participants who see treatment version will see Q7T; participants who see prevention version will see Q7P]

Q7T. Compared with other drugs that treat chronic pain, how effective do you think Milarix is?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

Q7P. Compared with other drugs that reduce the risk of heart attacks, how effective do you think Milarix is?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

(Perceived Comparative Safety)

[PROGRAMMER: Participants who see treatment version will see Q8T; participants who see prevention version will see Q8P]

Q8T. Compared with other drugs that treat chronic pain, how safe do you think Milarix is?

1 2 3 4 5 6 7

Not at all Moderately Very

Safe safe safe

Q8P. Compared with other drugs that treat chronic pain, how safe do you think Milarix is?

1 2 3 4 5 6 7

Not at all Moderately Very

Safe safe safe

[PROGRAMMER: New screen]

Now you will see the same ad that you saw earlier. Please look at this ad again, this time focusing specifically on how effective this drug is.

[PROGRAMMER: Display proper ad version. Record time spent on each page separately as well as time spent on each section, and order of sections chosen.]

Please answer the following specific questions based on what you learned from the Milarix ad.

[PROGRAMMER: Participants who see treatment version will see Q9T; participants who see prevention version will see Q10P]

Q9T. After reading the information, what is your sense of how much this product will reduce symptoms of pain on average?

1 2 3 4 5 6 7

Not much Moderately A great

at all deal

Q10P. After reading the information, what is your sense of what reduction in heart attack risk has been seen with this drug?

1 2 3 4 5 6 7

Not much A moderate A great deal

reduction amount of reduction of reduction

[PROGRAMMER: All participants see Q11]

Q11. What about the information caused you to respond as you did in the last question? (open-ended)

Possible codes:

Numbers

People in ad

Text

Side effects

Other

(Specific Benefit Accuracy)

[PROGRAMMER: For Questions 12-13, participants in the treatment study will see only questions 12T-13T. Participants in the prevention study will see only questions 12P-13P. In both cases, randomize the order of Q12-Q13]

Q12T. According to the information you just read, if 100 people take Milarix, how many will experience less pain?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q13T. According to the information you just read, if 100 people take no treatment, how many will experience less pain?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q12P. According to the information you just read, if 100 people take Milarix, how many will have a heart attack?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q13P. According to the information you just read, if 100 people take no treatment, how many will have a heart attack?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

[PROGRAMMER: New screen]

Now we would like you to read some information about another drug. This drug is used to treat high cholesterol. Please read through the following information and answer the questions that follow.

[PROGRAMMER: Show a randomly assigned half of participants Version 1 (Important Safety Information) and the other half Version 2 (Important Risk Information). These versions will be identical except for the headline and read as follows:

Votrea is not for everyone, including people with liver problems and women who are nursing, pregnant, or may become pregnant. You need simple blood test to check for liver problems. If you develop fever, unexplained weakness, or confusion, tell your doctor right away as these might be signs of a rare but potentially life threatening condition called TTP, which has been reported sometimes in less than 2 weeks after starting therapy. Also tell your doctor if you are taking other medications, or if you have any muscle pain or weakness, as this may be a sign of another rare but serious side effect. Common side effects include diarrhea, joint pain, and tiredness.]

Q14. How risky or safe do you think Votrea is?

1 2 3 4 5 6 7

Very Somewhat Somewhat Very

safe safe risky risky

Q15. How serious do you think the risks of Votrea are?

1 2 3 4 5 6 7

Not at all Moderately Very

serious serious serious

Q16. How likely would you be to take Votrea if you needed to lower your cholesterol, given the information you just read?

1 2 3 4 5 6 7

Not at all Moderately Very

likely likely likely

Q17. (Objective numeracy) Now here are some questions that require you to use numbers to solve the problem. Some are easy and others are more difficult. No calculators please- we’d like you to answer on your own. Remember, almost everyone will have trouble with these questions, so don’t be upset if some are difficult—just do your best!

[PROGRAMMER: DO NOT randomize Q19a-f]

1. What number is the correct answer:

8 + 4 + 11 = ?

1. 14

2. 19

3. 21

4. 23

5. 32

6. Don’t know

2. What is the correct answer:

17 – 8 + 4 = ?

a. 11

1. 13

2. 21

3. 23

4. 29

5. Don’t know

3. What is the correct answer:

100 x 10 x 10 = ?

1. 100

2. 1,000

3. 10,000

4. 100,000

5. 1,000,000

6. Don’t know

4. Imagine that you flip a fair coin 1,000 times. What is your best guess about how many times the coin would come up heads in 1,000 flips?

___ times out of 1,000 [PROGRAMMER: set acceptable range from 0 to 1,000]

5. In the BIG BUCKS LOTTERY, the chance of winning a $10 prize is 1%. What is your best guess about how many people would win a $10 prize if 1,000 people each buy a single ticket to BIG BUCKS LOTTERY?

________ people [PROGRAMMER: set acceptable range from 0 to 1,000]

6. In ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1,000. What percent of tickets to ACME PUBLISHING SWEEPSTAKES will win a car?

___ percent

Q18. (Subjective Numeracy 1st part) For each of the following questions, please check the box that best reflects how good you are at doing the following things:

1. How good are you at working with fractions?

2. How good are you at working with percentages?

3. How good are you at calculating a 15% tip?

4. How good are you at figuring out how much a shirt will cost if it is 25% off?

Q19. (Subjective Numeracy 2^nd part) For each of the following questions, please check the box that best reflects your answer:

1. When reading the newspaper, how helpful do you find tables and graphs that are part of a story?

2. When people tell you the chance of something happening, do you prefer that they use words (“it rarely happens”) or numbers (“there is a 1% chance)?

3. When you hear a weather forecast, do you prefer predictions using percentages (“there will be a 20% chance of rain”) or predictions using only words (“there is a small chance of rain today”)?

4. How often do you find numerical information to be useful?

Now please answer the following questions for classification purposes.

Q20. What year were you born? ______________

Q21. How severe is your chronic pain now? Would you describe it as:

Very mild

Mild

Moderate

Serious

Very serious

Q22. In general, how much do you feel you know about your chronic pain condition? Would you say you know:

A lot

A good bit

Some

Only a slight amount

Nothing at all

Q23. How would you rate your familiarity with prescription treatments for chronic pain?

Very familiar

Somewhat familiar

Somewhat unfamiliar

Not familiar at all

Q24. Are you currently taking a prescription medicine for chronic pain?

Yes

No

Don't know or uncertain

Q25. Have you ever seen any advertising for Milarix before today?

Yes

No

Q26. How many hours in a typical week do you use the internet for work purposes, if at all?

_____ hours

Q27. How many hours in a typical week do you use the internet for personal use, if at all?

_____ hours

Q28. Overall, how do you feel about ads on television, in magazines, or on the internet for prescription medicines?

Very positively

Somewhat positively

Has not affected the quality at all

Somewhat negatively

Very negatively

Q29. What kind of device did you take this survey on?

Desktop

Laptop

Notebook

Hand-held device

Other

Q30. Please select the range that includes your total annual household income before taxes.

􀂉         Less than $35,000   

􀂉         Between $35,000 and $70,000

􀂉         Over $70,000

Q31. Please enter your 5-digit zip code.

The purpose of this research is to learn about how people feel about and understand how well prescription drugs work from information provided in ads. In order to get a real-life reaction to this information, we created a brand to use in this study. MILARIX is not a real product and it is not available for sale. Please see your healthcare professional for questions about your health and your medical conditions.

APPENDIX C

Physician Questionnaire

Sample: Primary care physicians

N = 500

• Half will see the treatment claim; half will see the prevention claim (random assignment)

• Half will see a small difference between drug and placebo rate; half will see a large difference between drug and placebo rate (random assignment)

		Type of Claim
		Treatment Claim	Prevention Claim
Difference between Drug and Placebo	Small Difference	n = 125	n = 125
	Large Difference	n = 125	n = 125

Total N = 500

*Administer informed consent procedures*

Physician Questionnaire:

[PROGRAMMER: Record time in milliseconds spent on each screen (including questions) throughout protocol.]

[PROGRAMMER: Randomly assign participants to conditions as described above.]

Introductory language on at least three screens (to obtain baseline reading speeds).

Thank you for taking time from your busy schedule to contribute to this research. Your answers will remain confidential.

This study is about alternative methods of presenting prescription drug information. You will review information on a new (fictitious) product and make prescribing decisions as well as answer questions about the information you saw.

[PROGRAMMER: New screen]

On the next screen you will see the highlights section of the prescribing information for a fictitious new prescription drug, Milarix (lexisalicylic acid and milaristatin calcium). The document contains hyperlinks. This means you will be able to read the important information in the highlights section and explore each section in further detail by clicking on the section that interests you.

[PROGRAMMER: New screen]

Please read through this prescribing information as you would if you were learning about any new prescription-only product for the first time.

[PROGRAMMER: Display highlights section that will have hyperlinks to further information about each section. Record time spent on highlights section, time spent on each section, and order of sections chosen.]

(Perceived Benefit)

[PROGRAMMER: Randomize the order of Q1 and Q2]

Q1. If this were a real drug, how likely would you be to prescribe this drug to your patients?

Very likely

Somewhat likely

Somewhat unlikely

Very unlikely

Q1a. What about the drug caused you to select that answer? (open-ended)

Possible codes:

Efficacy

Safety

Convenience

Too new

Other

Q2. Based on the prescribing information you read, how effective would Milarix (lexisalicylic acid and milaristatin calcium) be for your patients?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

Q3. Based on the prescribing information you read, how well would Milarix (lexisalicylic acid and milaristatin calcium) work for your patients?

1 2 3 4 5 6 7

Not at all Moderately Very

well well well

Q4. (open-ended) Please explain why you rated the effectiveness of Milarix (lexisalicylic acid and milaristatin calcium) as you did.

Possible codes:

Efficacy

Safety

Convenience

Too new

Other

(Perceived Safety)

[PROGRAMMER: Randomize the order of Q5 and Q6]

Q5. Based on the prescribing information you read, how safe would Milarix (lexisalicylic acid and milaristatin calcium) be for your patients?

1 2 3 4 5 6 7

Not at all Moderately Very

Safe safe safe

Q6. Based on the prescribing information you read, how risky would Milarix (lexisalicylic acid and milaristatin calcium) be for your patients?

1 2 3 4 5 6 7

Not at all Moderately Very

risky risky risky

(Perceived Comparative Benefit)

[PROGRAMMER: Randomize the order of Q7 and Q8]

Q7a. Compared with other drugs that treat chronic pain, how effective do you think Milarix (lexisalicylic acid and milaristatin calcium) is?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

Q7b. Compared with other drugs that reduce the risk of heart attack, how effective do you think Milarix (lexisalicylic acid and milaristatin calcium) is?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

(Perceived Comparative Safety)

Q8a. Compared with other drugs that treat chronic pain, how safe do you think Milarix (lexisalicylic acid and milaristatin calcium) is?

1 2 3 4 5 6 7

Not at all Moderately Very

Safe safe safe

Q8b. Compared with other drugs that reduce the risk of heart attack, how safe do you think Milarix (lexisalicylic acid and milaristatin calcium) is?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

[PROGRAMMER: New screen]

Now you will see the same prescribing information that you saw earlier. Please refer to this prescribing information again, this time focusing specifically on how effective this drug is.

[PROGRAMMER: Display highlights section that will have hyperlinks to further information about each section. Record time spent on highlights section, time spent on each section, and order of sections chosen.]

Please answer the following specific questions based on what you learned from the Milarix (lexisalicylic acid and milaristatin calcium) prescribing information.

[PROGRAMMER: Participants who see treatment version will see Q9; participants who see prevention version will see Q10]

Q9. Based on the prescribing information you read, what is your sense of how much this product will reduce pain symptoms on average?

1 2 3 4 5 6 7

Not much Moderately A great

at all deal

Q10. Based on the prescribing information you read, what is your sense of what amount of reduction in heart attack risk can be expected with this drug?

1 2 3 4 5 6 7

Not much A moderate A great deal

reduction amount of reduction of reduction

[PROGRAMMER: All participants see Q11]

Q11. What about the information caused you to respond as you did in the last question? (open-ended)

Possible codes:

Efficacy

Safety

Convenience

Too new

Other

(Specific Benefit Accuracy)

[PROGRAMMER: For Questions 12-13, participants in the treatment study will see only questions 12T-13T. Participants in the prevention study will see only questions 12P-13P. In both cases, randomize the order of Q12-Q13]

Q12T. According to the information you just read, if 100 people take Milarix (lexisalicylic acid and milaristatin calcium), how many will experience less pain?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q13T. According to the information you just read, if 100 people take no treatment, how many will experience less pain?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q12P. According to the information you just read, if 100 people take Milarix (lexisalicylic acid and milaristatin calcium), how many will have a heart attack?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q13P. According to the information you just read, if 100 people take no treatment, how many will have a heart attack?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

[PROGRAMMER: New screen]

Now you will see four different versions of a magazine ad directed at patients. The ad is for another drug, a fictitious drug for high cholesterol, Votrea (trevastatin calcium). These four different versions represent different ways to present the data from the prescribing information to patients.

[PROGRAMMER: New screen]

Q14. There are many ways to present scientific data. Some are better than others. After looking at each of the four versions, please rank them from best to worst in terms of how well the ad represents the scientific information. In other words, your first selection will be the one you believe best represents the data, your second choice will be the one you believe is the second-best, and so forth.

To view each version in more detail, please click on the page and it will enlarge. You will be able to zoom in and out for your ease of viewing.

[PROGRAMMER:

1. Display all four versions of Votrea ad on screen. As participants click on a version, bring that version to a full screen view. Maintain some sort of zoom capacity so that participants can enlarge sections for ease of reading. Please adjust instructions to participants as appropriate, depending on the procedure you put in place.

2. Please arrange a format whereby participants can then select each version in their chosen order.

3. A randomly selected half of the participants will see treatment claim versions of the ad:

Version name 1

Version name 2

Version name 3

Version name 4

A randomly selected half of participants will see prevention claim versions of the ad:

Version name 5

Version name 6

Version name 7

Version name 8

]

Q14a. (open-ended) What about the different versions caused you to rank them this way?

Possible codes:

Format

Layout

Numbers

Text

Other

Q15. Now, looking at the same four versions, please rank these in order of ease of understanding for the typical patient. In other words, your first selection will be the version you think the typical patient will most readily understand, your second selection will be the second-most understandable version, and so forth.

[PROGRAMMER: Execute the same procedure as above. Participants will see the same versions in Q16 and Q17.]

Q15a. (open-ended) What about the different versions caused you to rank them this way?

Possible codes:

Format

Layout

Numbers

Text

Other

[PROGRAMMER: Randomize Qs 16-17. Q16 and 16a should be on the same screen at the same time. Q17 and 17a should be on the same screen at the same time.]

Please indicate your agreement with the following statements.

Q16. I can effectively communicate risk numerically (probability, percent).

Strongly agree

Somewhat agree

Somewhat disagree

Strongly disagree

Q16a. I consider this important to my practice.

Strongly agree

Somewhat agree

Somewhat disagree

Strongly disagree

Q17. I can effectively communicate risk qualitatively (‘high,’ ‘low’).

Strongly agree

Somewhat agree

Somewhat disagree

Strongly disagree

Q17a. I consider this important to my practice.

Strongly agree

Somewhat agree

Somewhat disagree

Strongly disagree

Q18. (Objective numeracy) Now here are some questions that require you to use numbers to solve the problem. Some are easy and others are more difficult. No calculators please- we’d like you to answer on your own.

[PROGRAMMER: DO NOT randomize Q18a-c]

1. Imagine that you flip a fair coin 1,000 times. What is your best guess about how many times the coin would come up heads in 1,000 flips?

___ times out of 1,000 [PROGRAMMER: set acceptable range from 0 to 1,000]

2. In the BIG BUCKS LOTTERY, the chance of winning a $10 prize is 1%. What is your best guess about how many people would win a $10 prize if 1,000 people each buy a single ticket to BIG BUCKS LOTTERY?

________ people [PROGRAMMER: set acceptable range from 0 to 1,000]

3. In ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1,000. What percent of tickets to ACME PUBLISHING SWEEPSTAKES will win a car?

___ percent

Now please answer the following questions for classification purposes.

Q19. What year were you born? ______________

Q20. How many years have you been in practice? _________

Q21. Thinking about prescriptions of all kinds, about how many prescriptions do you write in an average week, including hospital and institutional orders?

_____ per week

Q22. How many prescriptions for chronic pain do you write in an average week?

_____ per week

Q23. How many prescriptions for cardiovascular outcomes do you write in an average week?

_____per week

Q24. About what percentage of your patients are you treating for chronic pain?

_____%

Q25. About what percentage of your patients are you treating for cardiovascular risk factors?

_____%

Q26. How would you rate your familiarity with prescription treatments for chronic pain?

Very familiar

Somewhat familiar

Somewhat unfamiliar

Not familiar at all

Q27. How would you rate your familiarity with prescription treatments for improving cardiovascular outcomes?

Very familiar

Somewhat familiar

Somewhat unfamiliar

Not familiar at all

Q28. How many hours in a typical week do you use the internet for work purposes, if at all?

_____ hours

Q29. How many hours in a typical week do you use the internet for personal use, if at all?

_____ hours

Q30. Are you part of any of the following health-care arrangements? You may say yes to more than one. (check all that apply)

A solo practice

A small group practice or partnership

A multispecialty group practice

A health maintenance organization or HMO

A preferred provider list or network of physicians

None of the above

Q31. Overall, how would you say direct-to-consumer advertising has affected your patients and your practice?

Very positively

Somewhat positively

Has not affected the quality at all

Somewhat negatively

Very negatively

Q32. What caused you to answer as you did in the previous question? (open-ended)

Possible codes:

Helps dialog

Hinders dialog

Takes too much time

Creates false expectations

Other

Thank you for taking the time to participate in this study. This study has been designed by the Food and Drug Administration to explore the ways physicians use approved prescription drug labels in an attempt to improve these documents for future use. Your participation has been valuable.

APPENDIX D

Consumer Pretest Questionnaire

N = 100

*Administer informed consent*

Pretest Questionnaire

Quilarix (lexitacisprin) is a new prescription drug that helps reduce symptoms of chronic pain. It comes in tablet form and is generally taken two times a day. Possible side effects include stomach upset, dizziness, and dry mouth.

Q1. Please read the following statements (each participant sees one version and answers corresponding questions):

Version A

• 30 out of 100 people on the drug Quilarix reduced their pain symptoms.

• 20 out of 100 people on placebo reduced their pain symptoms.

a. What do these statements mean to you?

Possible codes (to be applied for all versions):

• Quilarix works/more people who took Quilarix had effect

• Quilarix does not work

• Clinical trial/a test was used

• Don’t know

b. What does “placebo” mean here?

Possible codes (to be applied for all versions):

• Full understanding

• Reference to clinical trial/test/experiment/research

• People did not take a drug

• People did not do anything to fix symptoms

• It’s another drug

• Don’t know

• Blood sugar/diabetes reference

• Incorrect other than “it’s another drug?”

Version B

• 30 out of 100 people on the drug Quilarix reduced their pain symptoms.

• 20 out of 100 people on sugar pill reduced their pain symptoms.

a. What do these statements mean to you?

b. What does “sugar pill” mean here?

Version C

• 30 out of 100 people on the drug Quilarix reduced their pain symptoms.

• 20 out of 100 people without Quilarix reduced their pain symptoms.

a. What do these statements mean to you?

b. What does “without Quilarix” mean here?

Version D

• 30 out of 100 people on the drug Quilarix reduced their pain symptoms.

• 20 out of 100 people with no treatment reduced their pain symptoms.

a. What do these statements mean to you?

b. What does “with no treatment” mean here?

2. Please provide a guess as to how many people in the US have a heart attack in a given year.

_____ out of 100.

3. Cobyrel (cobyrexen) is a prescription drug that has been shown to reduce the risk of having a heart attack in people who are at risk for heart problems. It is available in three different dosage forms: as a pill, a shot, or a patch applied to the skin. Possible side effects of Cobyrel include nightsweats, headache, and muscle weakness.

Please read the following statements (each participant sees all three versions on the screen at once and they can access the versions as they answer the questions below):

Version A

Cobyrel (cobyrexen) reduced the risk of heart attack.

• 96/100 people avoided a heart attack while on Cobyrel.

• 95/100 people avoided a heart attack without Cobyrel.

Version B

Cobyrel (cobyrexen) reduced the risk of heart attack.

• 96/100 people did not have a heart attack while on Cobyrel.

• 95/100 people did not have a heart attack without Cobyrel.

Version C

Cobyrel (cobyrexen) reduced the risk of heart attack.

• 4/100 people had a heart attack while on Cobyrel.

• 5/100 people had a heart attack without Cobyrel.

1. What do these statements mean to you?

Possible codes:

• All same

• Different versions mean different things

• Drug is effective

• Drug is not effective

• Don’t know

2. Do the statements say the same thing? Why or why not?

Possible codes:

• Yes

• No

-difference in effectiveness

-difference in riskiness

• Don’t know

3. Please rank the statements from most to least on the following attributes:

-Understandable

-Believable

-Clear

-Easy to read

-Persuasive

When researchers want to know if a drug works, they conduct a clinical trial. In clinical trials, some people are given the real drug and others are given a fake drug (a placebo). No one knows who got what. The researchers then look to see if people who got the real drug do better than people who did not get the real drug. Sometimes there is a big difference and people who got the real drug do much better than people who got the fake drug, meaning that the real drug works very well. Sometimes there is a small difference and people who got the real drug only do a little better than people who got the fake drug, meaning that the real drug works a little bit, but does not do much more than not taking any drug at all would.

4. Please read the following sentences and fill in the blanks based on what you think a BIG difference would be between the drug Quilarix and a fake drug.

a.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __5__ out of 100 people without Quilarix reduced their pain symptoms.

b.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __25_ out of 100 people without Quilarix reduced their pain symptoms.

c.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __50_ out of 100 people without Quilarix reduced their pain symptoms.

5. Please read the following sentences and fill in the blanks based on what you think a SMALL difference would be between the drug Quilarix and a fake drug.

a.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __5__ out of 100 people without Quilarix reduced their pain symptoms.

b.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __25_ out of 100 people without Quilarix reduced their pain symptoms.

c.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __50_ out of 100 people without Quilarix reduced their pain symptoms.

6. Please read the following sentences and fill in the blanks based on what you think NO DIFFERENCE would be between the drug Quilarix and a fake drug.

a.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __5__ out of 100 people without Quilarix reduced their pain symptoms.

b.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __25_ out of 100 people without Quilarix reduced their pain symptoms.

c.

• _____ out of 100 people on the drug Quilarix reduced their pain symptoms.

• __50_ out of 100 people without Quilarix reduced their pain symptoms.

7. What does “fake drug” mean to you? (open-ended)

Possible codes to be determined.

8. How likely do you think it would be that 5 out of 100 people would experience less pain if they did not take any drug at all?

1	2	3	4	5
Not at all likely		Somewhat likely		Very likely

9. How likely do you think it would be that 25 out of 100 people would experience less pain if they did nothing to treat it at all?

1	2	3	4	5
Not at all likely		Somewhat likely		Very likely

10. How likely do you think it would be that 50 out of 100 people would experience less pain if they did nothing to treat it at all?

1	2	3	4	5
Not at all likely		Somewhat likely		Very likely

The purpose of this research is to develop materials to learn about how people feel about and understand how well prescription drugs work from information provided in ads. In order to get a real-life reaction to this information, we created brands to use in this study. QUILARIX and COBYREL are not real products and are not available for sale. Please see your healthcare professional for questions about your health and your medical conditions.

This concludes the project today. Thank you for your time.

APPENDIX E

Physician Pretest

Sample: Primary care physicians

N = 50

• Half will see the treatment claim; half will see the prevention claim (random assignment)

• Half will see a small difference between drug and placebo rate; half will see a large difference between drug and placebo rate (random assignment)

		Type of Claim
		Treatment Claim	Prevention Claim
Difference between Drug and Placebo	Small Difference	n = 12	n = 13
	Large Difference	n = 13	n = 12

Total N = 50

*Administer informed consent*

Pretest Questionnaire:

[PROGRAMMER: Record time in milliseconds spent on each screen (including questions) throughout protocol.]

[PROGRAMMER: Randomly assign participants to conditions as described above.]

Introductory language on at least three screens (to obtain baseline reading speeds).

Thank you for taking time from your busy schedule to contribute to this research. Your answers will remain confidential.

This study is about alternative methods of presenting prescription drug information. You will review information on a new (fictitious) product and make prescribing decisions as well as answer questions about the information you saw.

[PROGRAMMER: New screen]

On the next screen you will see the highlights section of the prescribing information for a fictitious new drug, Milarix (lexisalicylic acid and milaristatin calcium). The document contains hyperlinks. This means you will be able to read the important information in the highlights section and explore each section in further detail by clicking on the section that interests you.

[PROGRAMMER: New screen]

Please read through this prescribing information as you would if you were learning about any new prescription-only product for the first time.

[PROGRAMMER: Display highlights section that will have hyperlinks to further information about each section. Record time spent on highlights section, time spent on each section, and order of sections chosen.]

[PROGRAMMER: Randomize the order of Q1 and Q2]

Q1. How thoroughly did you read the PI? (check all that apply)

___I did not read any of it

___I skimmed the highlights section

___I read the highlights section thoroughly

___I clicked on and skimmed a few links

___I clicked on and read only a few links, but I read those links thoroughly

___I clicked on and skimmed many links

___I clicked on and read many links thoroughly

___I clicked on and read every link

Q2. How similar is this to how much information you usually read about a new drug?

___I read more information than I usually read about a new drug

___I read about the same amount of information

___I read less information than I usually read about a new drug

Q3. How easy or difficult was it for you to find the information you were interested in?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither Easy nor Difficult	Somewhat Difficult	Very Difficult

Q4. How believable was the information in this PI?

1	2	3	4	5
Not at all believable		Somewhat believable		Very believable

Q4a. What made you answer the previous question as you did? (open-ended)

Possible Codes:

Drug not realistic

Sections of PI not realistic

Was told it was fictitious

Formatting

Other

[PROGRAMMER: New screen]

Now you will see the same prescribing information that you saw earlier. Please refer to this prescribing information again, this time focusing specifically on how effective this drug is.

[PROGRAMMER: Display highlights section that will have hyperlinks to further information about each section. Record time spent on highlights section, time spent on each section, and order of sections chosen.]

Please answer the following specific questions based on what you learned from the Milarix (lexisalicylic acid and milaristatin calcium) prescribing information.

Q5. How easy or difficult was it to find information about the effectiveness of the drug?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither Easy nor Difficult	Somewhat Difficult	Very Difficult

Q6. How easy or difficult was it to distinguish your task in the second viewing of the PI from your task in the first viewing of the PI?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither Easy nor Difficult	Somewhat Difficult	Very Difficult

[PROGRAMMER: New screen]

Now you will see four different versions of a magazine ad directed at patients. The ad is for another drug, a fictitious drug for high cholesterol, Votrea (trevastatin calcium). These four different versions represent different ways to present the data from the prescribing information to patients.

[PROGRAMMER: New screen]

Q7. There are many ways to present scientific data. Some are better than others. After looking at each of the four versions, please rank them from best to worst in terms of how well the ad represents the scientific information. In other words, your first selection will be the one you believe best represents the data, your second choice will be the one you believe is the second-best, and so forth.

To view each version in more detail, please click on the page and it will enlarge. You will be able to zoom in and out for your ease of viewing.

[PROGRAMMER:

1. Display all four versions of Votrea ad on screen. As participants click on a version, bring that version to a full screen view. Maintain some sort of zoom capacity so that participants can enlarge sections for ease of reading. Please adjust instructions to participants as appropriate, depending on the procedure you put in place.

2. Please arrange a format whereby participants can then select each version in their chosen order.

3. A randomly selected half of the participants will see treatment claim versions of the ad:

Version name 1

Version name 2

Version name 3

Version name 4

A randomly selected half of participants will see prevention claim versions of the ad:

Version name 5

Version name 6

Version name 7

Version name 8

]

Q7a. (open-ended) What about the different versions caused you to rank them this way?

Possible Codes:

Format

Visuals

Numbers

Colors

Other

Q8. How easy or difficult was this task?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither Easy nor Difficult	Somewhat Difficult	Very Difficult

Q9. How clear were the instructions for this task?

1	2	3	4	5
Very Clear	Somewhat Clear	Neither Clear nor Unclear	Somewhat Unclear	Very Unclear

Q10. Now, looking at the same four versions, please rank these in order of ease of understanding for the typical patient. In other words, your first selection will be the version you think the typical patient will most readily understand, your second selection will be the second-most understandable version, and so forth.

[PROGRAMMER: Execute the same procedure as above. Participants will see the same versions in Q16 and Q17.]

Q10a. (open-ended) What about the different versions caused you to rank them this way?

Possible Codes:

Format

Visuals

Numbers

Colors

Other

Q11. How easy or difficult was this task?

1	2	3	4	5
Very Easy	Somewhat Easy	Neither Easy nor Difficult	Somewhat Difficult	Very Difficult

Q12. How clear were the instructions for this task?

1	2	3	4	5
Very Clear	Somewhat Clear	Neither Clear nor Unclear	Somewhat Unclear	Very Unclear

Q13. Now here are some questions that require you to use numbers to solve the problem. Some are easy and others are more difficult. No calculators please- we’d like you to answer on your own.

[PROGRAMMER: DO NOT randomize Q20a-c]

7. Imagine that you flip a fair coin 1,000 times. What is your best guess about how many times the coin would come up heads in 1,000 flips?

___ times out of 1,000 [PROGRAMMER: set acceptable range from 0 to 1,000]

8. In the BIG BUCKS LOTTERY, the chance of winning a $10 prize is 1%. What is your best guess about how many people would win a $10 prize if 1,000 people each buy a single ticket to BIG BUCKS LOTTERY?

________ people [PROGRAMMER: set acceptable range from 0 to 1,000]

9. In ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1,000. What percent of tickets to ACME PUBLISHING SWEEPSTAKES will win a car?

___ percent

Q14. How did you feel about being asked the previous three questions? (open-ended)

Possible codes:

Positive

Negative

Other

Now please answer the following questions for classification purposes.

Q21. What year were you born? ______________

Q22. How many years have you been in practice? _________

Q23. About what percentage of your patients are you treating for chronic pain?

_____%

Q24. About what percentage of your patients are you treating for cardiovascular risk factors?

_____%

Q25. How would you rate your familiarity with prescription treatments for chronic pain?

Very familiar

Somewhat familiar

Somewhat unfamiliar

Not familiar at all

Q26. How would you rate your familiarity with prescription treatments for improving cardiovascular outcomes?

Very familiar

Somewhat familiar

Somewhat unfamiliar

Not familiar at all

Q27. How many hours in a typical week do you use the internet for work purposes, if at all?

_____ hours

Q28. How many hours in a typical week do you use the internet for personal use, if at all?

_____ hours

Q29. Overall, how would you say direct-to-consumer advertising has affected your patients and your practice?

Very positively

Somewhat positively

Has not affected the quality at all

Somewhat negatively

Very negatively

Q30. What caused you to answer as you did in the previous question? (open-ended)

Possible codes:

Helps dialog

Hinders dialog

Takes too much time

Creates false expectations

Other

Q31. Do you have any thoughts or comments on this study? (open-ended)

Thank you for taking the time to participate in this study. This study has been designed by the Food and Drug Administration to develop materials to explore the ways physicians use approved prescription drug labels in an attempt to improve these documents for future use. Your participation has been valuable.

This concludes the study. Thank you for your time.

1 For a literature review, see Moxey, A., O’Connell, D., McGettigan, P., & Henry, D. (2003). Describing treatment effects to patients: How they are expressed makes a difference. Journal of General Internal Medicine, 18, 948-959.

2 Fagerlin, A., Ubel, P.A., Smith, D.M., & Zikmund-Fisher, B.J. (2007). Making numbers matter: Present and future research in risk communication. American Journal of Health Behavior, 31, S47-S56; Schwartz, L.M., Woloshin, S., & Welch, H.G. (1999). Risk communication in clinical practice: Putting cancer in context. Monograph of the National Cancer Institute, 25, 124-133.

3 See Armstrong, K., Schwartz, J.S., Fitzgerald, G., Putt, M., & Ubel, P.A. (2002) Effect of framing as gain versus loss on understanding and hypothetical treatment choices: survival and mortality curves. Medical Decision Making, 22, 76-83; Dunegan, K.J. (1993) Framing, cognitive modes, and imagery theory: Toward an understanding of a glass half full. Journal of Applied Psychology, 78, 491-503; Rothman, A.J., & Salovey, P. (1997) Shaping perceptions to motivate healthy behavior: The role of message framing. Psychological Bulletin, 121, 3-19; Smith, S.M., & Petty, R.E. (1996) Message framing and persuasion: A message processing analysis. Personality and Social Psychology Bulletin, 22, 257-268; Tversky, A., & Kahneman, D. (1981) The framing of decisions and the psychology of choice. Science, 211, 453-458.

4 Including internists, general practitioners, and family practitioners.

5 See 21 CFR 201.56, 201.57, 201.58, 201.80.

6 Physicians in the treatment conditions will not be compared with physicians in the prevention conditions.

7 Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A, (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.

8 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2^nd Ed). Hillsdale, NJ: Lawrence Erlbaum & Associates, Inc.

9 Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A, (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.

10 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2^nd Ed). Hillsdale, NJ: Lawrence Erlbaum & Associates, Inc.

11 Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A, (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175-191.

12 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2^nd Ed). Hillsdale, NJ: Lawrence Erlbaum & Associates, Inc.

13 For prescription drugs and biologics, the Act requires advertisements to contain "information in brief summary relating to side effects, contraindications, and effectiveness" (21 U.S.C. 352(n)).

14 See Schwartz, L., Woloshin, S., Black, W., & Welch, H.G. (1997). The role of numeracy in understanding the benefit of screening mammography. Annals of Internal Medicine, 127(11), 966-72.

15 Woloshin, S., & Schwartz, L. (2001). Direct to consumer advertisements for prescription drugs: What are Americans being told. Lancet, 358, 1141-46.

16 Woloshin, S., & Schwartz, L. (2001). Direct to consumer advertisements for prescription drugs: What are Americans being told. Lancet, 358, 1141-46.

17 As part of this effort, a qualitative mental models procedure was completed that helped us determine how physicians think about the efficacy of potential pharmaceutical options (OMB Control No. 0910-0649).

18 For a literature review, see Moxey, A., O’Connell, D., McGettigan, P., & Henry, D. (2003). Describing treatment effects to patients: How they are expressed makes a difference. Journal of General Internal Medicine, 18, 948-959.

19 Fagerlin, A., Ubel, P.A., Smith, D.M., & Zikmund-Fisher, B.J. (2007). Making numbers matter: Present and future research in risk communication. American Journal of Health Behavior, 31, S47-S56; Schwartz, L.M., Woloshin, S., & Welch, H.G. (1999). Risk communication in clinical practice: Putting cancer in context. Monograph of the National Cancer Institute, 25, 124-133.

20 Including internists, general practitioners, and family practitioners.

21 To reduce burden, the physician sample will be split in this task, so that half of the physicians see the four ad versions with treatment claims and the other half see the four ad versions with prevention claims. Type of claim is described in greater detail in the consumer experiment section.

1