Part B

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

The universe for this experimental study is members of the Knowledge Networks Internet panel.  Knowledge Network’s Internet panel consists of 48,000 adult panel members who are systematically recruited by random-digit dialing (RDD) or by using address-based sampling.  Households without existing Internet service are also eligible, and Knowledge Networks provides these members with laptops or Web TVs to enable their participation. The sample is nationally representative and statistically accurate.  Typically, panel members receive 3-4 invitations per month to participate in research projects. 

The 4,500 participants for this study will be drawn from the pool of 48,000 panel members. All panel members complete prescreening questionnaires on a variety of topics, and we will recruit participants who indicated that they have been medically diagnosed with high cholesterol.  If necessary, we will oversample certain population segments so that the overall sample is in proportion to the U.S. adult population on gender, race/ethnicity, education, and income.  At least 20% of the sample will have achieved a high school education or less. 

The Agency does not intend to generate nationally representative results or precise estimates of population parameters from the experimental study.  The study will use a convenience sample rather than a probability sample.  Despite the attempt to match between the study’s sample and the respondent universe in four demographic characteristics, matching is used solely to produce a sample with a reasonable degree of diversity in key demographic characteristics. 

Rather, the strength of the experimental studies lies in its internal validity, on which meaningful estimates of differences across experimental conditions can be produced and generalized.  As discussed in the following sections, the agency has taken commonly accepted measures to enhance internal validity of the study.  Examples of these measures include random assignment of respondents and conditions, counterbalancing condition assignments within the sample, and use of comparison conditions and relevant covariates.

2. Procedures for the Collection of Information

Overview of Design

This study will be conducted in two concurrent parts: one examining quantitative information in DTC print advertisements and the other examining such information in DTC television advertisements. Three factors will be examined: drug efficacy, statistical format, and visual format.

We will investigate two levels of drug efficacy (low versus high), defined by a quantifiable, objective metric that can be conveyed in graphical representations of the drug versus the comparator reference drug (in this case, placebo). Specifically, high efficacy will be defined by a large, noticeable difference compared with no treatment, whereas low efficacy will be defined by a minimal difference between the drug and no treatment. We will examine two levels of efficacy to determine whether participants can accurately distinguish between these levels within various formats.

We will investigate five statistical formats, defined as the type of statistical information conveyed: frequency, percent, frequency plus percent, relative frequency, and frequency plus relative frequency. Based on existing literature, we will use the frequency statistical format in all of our visual formats for consistency.

Visual format is defined as various methods through which efficacy can be visually represented. We have chosen to investigate four different formats: pie chart, bar chart, table, and pictograph.

Additionally, we will have a control condition with no specific efficacy information provided. Please see Appendix 3 for the operationalization of each of these conditions. The factors will be combined in a partially crossed factorial design as follows:

		Statistical Format
		Frequency	Percent	Frequency + Percent	Relative Frequency	Frequency + Relative Frequency
Efficacy	Low
	High

AND

		Visual Format
		None	Pie Chart	Bar Chart	Table	Pictograph
Efficacy	Low
	High

+ 1

No Statistical Format/No Efficacy

Procedure

This study will be administered over the internet. A total of 2,250 interviews involving print ads will be completed. Participants in this part of the study will be randomly assigned to view one version of the magazine promotion page and the brief summary page of a prescription drug ad. Following their perusal of this document, they will answer questions about their recall and understanding of the benefit and risk information, their perceptions of the benefits and risks of the drug, and their intent to ask a doctor about the medication.

A total of 2,250 interviews involving television ads will be completed. Participants in this part of the study will be randomly assigned to view one version of a television ad twice and answer the same questions described above.

For both parts, demographic and health care utilization information will be collected. The entire procedure is expected to last approximately 20 minutes. This will be a one-time (rather than annual) information collection.

Participants

Data will be collected using an Internet protocol. Participants will all have reported that a healthcare professional has diagnosed them with high cholesterol and will represent a range of education levels. Because the task presumes basic reading abilities, all selected participants must speak English as their primary language. Participants must be 18 years or older.

Hypotheses

Preface

The proposed research has two main objectives. First, we plan to test several statistical formats to determine whether the presentation of efficacy information in different formats affects perceptions of efficacy. The risk communication literature suggests that presenting numerical risk information as an absolute frequency (e.g., N out of 100) may be the most easily understood format (Fagerlin et al., 2007¹). Percent, and a combination of absolute frequency and percent, represent increasingly complex statistical formats; however, they may not differ from the baseline of absolute frequency for average consumers. In contrast, the risk communication literature suggests that presenting numerical risk information as a relative frequency (e.g., 10 times higher) is a markedly more complex statistical format that biases perceptions (Fagerlin et al., 2007). Thus, presenting efficacy information as a relative frequency, compared to absolute frequency, may affect perceptions of efficacy. Presenting the combination of absolute frequency and relative frequency may mitigate this effect.

Second, we plan to test several visual formats to determine whether the presentation of a visual format, in conjunction with the presentation of absolute frequency information, affects perceptions of efficacy. The risk communication literature suggests that the addition of visual formats such as bar charts, tables, and pictographs increase people’s understanding of numerical information (Ancker et al., 2006²; Lipkus & Hollands, 1999³). However, not all visual formats are always helpful; for instance, pie charts may only help when people are comparing proportions (Lipkus, 2007⁴). Thus, presenting efficacy information with a bar chart, table, and pictograph—but not necessarily with a pie chart—may affect people’s understanding of efficacy information, in comparison to when there is no visual format.

Measuring numeracy will allow us to assess the magnitude of these effects across participants. Similarly, the separate TV and print portions of the study will allow us to assess the magnitude of these effects across these modalities.

Specific Hypotheses

Efficacy effects in print and TV ads

(1) Behavioral intentions, attitude toward drug, and perceived efficacy will be higher in high efficacy conditions than in low efficacy conditions.

(2) We will explore whether there are differences between the no efficacy condition (control) and the low and high efficacy condition on behavioral intentions, attitude toward drug, and perceived efficacy.

(3) Benefit accuracy will be higher in the low and high efficacy conditions than in the no efficacy condition. There will be no difference between the low and high efficacy conditions.

(4) The effects tested in 1 & 3 will be modified by numeracy, such that high numeracy participants will be more likely to show these effects than will low numeracy participants.

(5) Risk recall will not differ by efficacy level (no, low, high).

(6) Perceived risk will be lower in the high efficacy condition compared with the low efficacy condition because, according to the Affect Heuristic (Slovic & Peters, 2006⁵), people perceive things that are more beneficial as less risky.

Statistical format effects in print and TV ads

(1) We will test competing hypotheses for behavioral intentions, attitude toward drug, and perceived efficacy.

(1a) Overestimation hypothesis: The first hypothesis rests on the assumption that in the absence of any quantitative information people overestimate the effectiveness of drugs. Accordingly, we would predict that behavioral intentions, attitude toward drug, and perceived efficacy will be higher for participants in the no statistical format condition, compared to all other statistical format conditions. Support for this interpretation will be found if estimates of the benefits are higher in the no statistical format condition than in all other statistical format conditions.

(1b). Peripheral cue hypothesis: The competing hypothesis rests on the assumption that any statistical information will be used as a peripheral cue; that is, participants will not process the quantitative information provided in the various statistical formats but will rather view it as “scientific proof” of the drug’s efficacy. Accordingly, we would predict that behavioral intentions, attitude toward drug, and perceived efficacy will be lower for participants in the no statistical format condition, compared to all other statistical format conditions. Support for this interpretation will be found if, in addition to perceived efficacy effects, estimates on attitude toward the ad “peripheral cue” measures—ratings of how believable, persuasive, informative, etc, the ad is--are lower in the no statistical format condition than in all other statistical format conditions.

(2) Based on the risk communication literature, we predict that the absolute frequency, percent, and absolute frequency and percent conditions may not differ on behavioral intentions, attitude toward drug, or perceived efficacy. However, we predict that behavioral intentions, attitude toward drug, and perceived efficacy will be higher in the relative frequency condition than in the absolute frequency, percent, absolute frequency + percent, and absolute frequency + relative frequency conditions.

(3) The effects tested in hypotheses 1-2 will be modified by numeracy. For instance, we expect that the difference between the relative frequency and the absolute frequency + relative frequency conditions will be greater for high numeracy participants than for low numeracy participants (because high numeracy participants will be more likely to use the additional information provided by the absolute frequency).

(4) Benefit accuracy will be lowest in the no statistical format condition and highest in the absolute frequency condition (Slovic, Monahan, & MacGregor, 2000⁶). Tests of other relations between statistical formats will be exploratory. For instance, we might see information overload with some formats (e.g., absolute frequency & relative frequency) which impedes benefit accuracy.

(5) The effects tested in (4) will be modified by numeracy, such that low numeracy participants will show greater differences in benefit accuracy across statistical formats than will high numeracy participants (Peters, Vastfjall, et al., 2006⁷).

(6) We expect that risk recall will not differ by statistical format, but we will conduct exploratory analyses to determine whether information overload impedes risk recall.

(7) We expect that perceived risk will be lowest in the relative frequency condition if perceived benefit is indeed highest in this condition (see Slovic & Peters, 2006, footnote 10).

Visual format effects in print and TV ads

(1) We will test competing hypotheses for benefit accuracy, behavioral intentions, attitude toward drug, and perceived efficacy.

(1a) Visual information facilitation hypothesis: The first hypothesis rests on the assumption that participants will, to the extent possible, process and use the information in the visual formats. The risk communication literature suggests that visual representations of risk can increase understanding, and that people have a more difficult time processing this kind of information in pie charts, as compared to other visual formats. Therefore, our first hypothesis is that benefit accuracy will be higher in the bar chart, table, and pictograph conditions—but not necessarily the pie chart condition--than in the no visual format condition. Tests of other relations between visual formats will be exploratory.

(1b) Information overload hypothesis: Alternatively, there may be no differences across visual formats on behavioral intentions, attitude toward drug, perceived efficacy, or benefit accuracy if the visual serves as a distraction or is too much information to process.

(1c) Peripheral cue hypothesis: Behavioral intentions, attitude toward drug, and perceived efficacy—but not benefit accuracy—may be higher in all visual conditions than in the no visual condition if the visual information serves as a peripheral cue.

(2) The effects tested in hypothesis 1 will be modified by numeracy. For instance, we expect that high numeracy participants will be more likely to process the information in the visual formats, and thus more likely to show the pattern of effects outlined in 1a, compared to low numeracy participants.

(3) We expect that perceived risk and risk recall will not differ by visual format but we will conduct exploratory analyses to determine whether information overload impedes risk recall.

Analysis Plan

We will conduct the following analyses separately for the print and television versions of the ad.

Efficacy effects in print and TV ads: We will conduct ANOVAs to test whether the no statistical format/no efficacy condition differs from the low and high efficacy condition on the dependent measures (i.e., benefit accuracy, behavioral intentions, attitude toward drug, perceived efficacy, perceived risk, and risk recall, peripheral cue measures). We will conduct these analyses both with and without covariates (e.g., demographic and health characteristics) included in the model. In addition, we will test whether any main effects are moderated by other measured variables (e.g., numeracy, demographic and health characteristics). If the main effect of efficacy is significant, we will conduct pairwise-comparisons to determine which conditions are significantly different from one another. We will also conduct planned comparisons in line with our hypotheses (see above). In addition, the main effect of efficacy (low vs. high) and any interaction it has with statistical format or visual format will be tested in the ANOVAs presented in the following two sections.

Statistical format effects in print and TV ads: We will conduct ANOVAs to test whether the no statistical format/no efficacy condition differs from the other statistical format conditions on the dependent measures. In addition, we will examine the main effect of statistical format in ANOVAs predicting our dependent measures from statistical format, efficacy level, and their interaction. We will conduct these analyses both with and without covariates included in the model. In addition, we will test whether any main effects are moderated by other measured variables. If the main effect of statistical format is significant, we will conduct pairwise-comparisons to determine which conditions are significantly different from one another. We will also conduct planned comparisons in line with our hypotheses (see above).

Visual format effects in print and TV ads: To test our hypotheses regarding visual format, we will examine the main effect of visual format in ANOVAs predicting our dependent measures from visual format, efficacy level, and their interaction. We will conduct these analyses both with and without covariates included in the model. In addition, we will test whether any main effects are moderated by other measured variables. If the main effect of visual format is significant, we will conduct pairwise-comparisons to determine which conditions are significantly different from one another. We will also conduct planned comparisons in line with our hypotheses (see above).

Power

The following assumptions were made in deriving the sample size: (1) 0.05 alpha and 0.90 power and (2) an effect size between small and medium. The table below shows the sample size required to detect differences with effect sizes ranging from conventionally “small” (f = .1) to almost “medium” (f = .25) for the largest comparison we plan to analyze (2 x 5). Specifically, visual and statistical format will never be crossed and we plan to conduct the print and broadcast parts of the study as entirely separate analyses.

Table: A priori power analysis to determine sample size needed in F tests (ANCOVA: fixed effects, main effects, and interactions) to achieve power of .90 (Faul et al., 2007).8
		Effect size f*
Input
		.10	.13	.20
	α error probability	.05	.05	.05
	Power ( 1 – β error probability)	.90	.90	.90
	Numerator df	9	9	9
	Number of groups	10	10	10
	Number of covariates	4	4	4
Output
	Noncentrality parameter λ	19.92	19.98	20.20
	Critical F	1.88	1.89	1.90
	Denominator df	1,981	1,171	494
	Total sample size	1,992	1,182	505
	Actual power	.90	.90	.90

*An effect size of .10 is traditionally considered small, whereas an effect size of .25 is considered medium (Cohen, 1988).⁹ Here we have shown three different effect sizes centering around small to medium effects to show that we will be able to detect fairly small effects.

We will have 118 participants per cell, with a total of 1,180 participants in the 10 cells represented in the table (2 x 5). The table shows that our sample size of 1,180 per portion of the study will be sufficient to detect effects as small as .13.

3. Methods to Maximize Response Rates and to Deal with Issues of Non-Response

This experimental study will use an existing Internet panel to draw a sample.  The panel comprises individuals who share their opinions via the Internet regularly.  The participation rate for similar studies is 65-70% percent without additional efforts to convert non-respondents.  To help ensure that the participation rate is as high as possible, FDA will:

• Design an experimental protocol that minimizes burden (short in length, clearly written, and with appealing graphics);

• Administer the experiment over the Internet, allowing respondents to answer questions at a time and location of their choosing;

• Email a reminder to the respondents who do not complete the protocol four days after the original invitation to participate is sent;

• Provide a toll-free hotline for respondents who may have questions or technical difficulty as they complete the experiment. 

4. Test Procedures

The contractor will run nine participants through the procedure to assess questionnaire wording, basic glitches in the programming and execution of the study. This pretest is designed to ensure that questionnaire wording is clear and that procedures for viewing stimuli and proceeding through the experiment are as planned.

5. Individuals Involved in Statistical Consultation and Information Collection

The contractor, RTI International, will collect the information on behalf of FDA as a task order under the Quick-Turn-Around Research Services contract. Doug Rupert, MPH, is the Project Director for this project, telephone (919) 541-6495. Data analysis will be conducted primarily by the Research Team, Division of Drug Marketing, Advertising, and Communications (DDMAC), Office of Medical Policy, CDER, FDA, and coordinated by Amie C. O’Donoghue, Ph.D., 301-796-0574 and Kathryn J. Aikin, Ph.D., 301-796-0569.

APPENDIX 2

Questionnaire, Quantitative Study (for both print and broadcast)

[PROGRAMMER: We need to record time in milliseconds spent on each screen (including questions) throughout protocol.]

Interview Protocol.

(Present Informed Consent Form. Participants will be blind to FDA’s sponsorship.)

Thank you for agreeing to participate in this study today.

Make sure you are comfortable and can read the screen from where you sit. This study is about advertising for a new product. We will show you an ad for a new product and ask you some questions about it. Your answers are anonymous, which means that no one will ever connect your name with your answers. Your help is valuable and we thank you.

[PROGRAMMER: New screen]

Next you will see an ad for a new product.

[Instructions for print version] Even though it is on a computer screen, please read the ad as you would in a magazine if you saw an ad for a product that you might be interested in for yourself. You can take as much time as you want to look over the ad. The ad has two pages. You can flip back and forth between pages using the BACK and FORWARD buttons if that is how you would normally read this ad. Once you are finished reading, please click “next” to move on to the next part of the study. [PROGRAMMER: Record time in milliseconds spent on each page of each print ad.]

[Instructions for TV version] Even though it is on a computer screen, please watch the ad as you would on television if you saw an ad for a product that you might be interested in for yourself. After viewing the ad, the program will instruct you to move on to the next part of the study. [PROGRAMMER: Show ad twice]

Now please answer the following questions based on the ad you saw.

Q1. Do you remember seeing an ad for Votrea?

Yes

No (terminate)

Not sure (terminate)

OMB Control No.

Q2. What type of product is Votrea? [PROGRAMMER: randomize responses]

Over the counter drug

Prescription drug

Herbal supplement

Lens cleaner

Q3. What condition does Votrea treat?

High blood pressure

High cholesterol

Migraine headaches

Seasonal allergies

Q4. What are the benefits of Votrea?

(open ended)

(Gist Comprehension)

Q5. Based on the information in the ad, does Votrea work better than not taking any treatment?

Yes

No

Not sure

(Perceived Benefit)

Q6. Based on the information in the ad, how effective would Votrea be for you?

1 2 3 4 5 6 7

Not at all Moderately Very

effective effective effective

Q7. Based on the information in the ad, how well would Votrea work for you?

1 2 3 4 5 6 7

Not at all Moderately Very

well well well

(Specific Benefit Accuracy)

OMB Control No.

Please answer the following specific questions based on what you learned from the Votrea ad.

Q8. If 100 people take Votrea, how many will lower their bad cholesterol to normal levels?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q9. What percentage (%) of people who take Votrea will lower their bad cholesterol to normal levels? For example,

If no one will lower their bad cholesterol to normal levels, enter 0.

If everyone will lower their bad cholesterol to normal levels, enter 100.

If some but not all will lower their bad cholesterol to normal levels, enter a number between 0 and 100 that reflects the percentage.

______ percent (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q10. How many more times effective is Votrea than no treatment in lowering bad cholesterol? Enter a number to show how effective Votrea is compared to no treatment. For example,

If taking Votrea is no more effective than no treatment, enter 0.

If taking Votrea is two times more effective than no treatment, enter 2.

If taking Votrea is three times more effective than no treatment, enter 3.

______ times more effective (fill in the blank. PROGRAMMER: set acceptable range from 0 to unlimited upper bound)

Q11. If 100 people take no treatment, how many will lower their bad cholesterol to normal levels?

______ people (fill in the blank. PROGRAMMER: set acceptable range from 0 to 100)

Q12. This ad had a picture or visual showing how well Votrea works.

Yes

No

Not sure

[PROGRAMMER: randomize Q9a-c]

OMB Control No.

Q13. (Behavioral Intention) Please rate how likely or not likely you are to do each of the following behaviors using the scale on this page.

	Not at all likely	Somewhat likely	Very likely	Extremely likely
a. Talk to your doctor about Votrea
b. Look for more information about Votrea
c. Ask your doctor to prescribe Votrea
d. Take Votrea if prescribed

Q14. (Recall of benefits) Please check which of the following were mentioned in the ad as benefits of taking Votrea.

[PROGRAMMER: randomize Q10a-h]

	Yes	No
a. Votrea works with diet and exercise.	x
b. Votrea can lower bad cholesterol to normal levels.	x
c. Votrea works for people with several common risk factors for heart disease.	x
d. Votrea takes the place of diet and exercise.		x
e. Votrea can reduce the risk of diabetes.		x
f. Votrea is the #1 prescribed medication for high cholesterol.		x
g. High cholesterol is a risk factor for heart disease.	x
h. Votrea can raise good cholesterol to normal levels.		x

Q15. What do you remember about the risks of Votrea?

(open ended)

Q16. Based on the information in the ad, how safe would Votrea be for you?

OMB Control No.

1 2 3 4 5 6 7

Not at all Moderately Very

Safe safe safe

Q17. Based on the information in the ad, how risky would Votrea be for you?

1 2 3 4 5 6 7

Not at all Moderately Very

risky risky risky

Q18. (Recall of risks) Please check which of the following were mentioned in the ad as risks of taking Votrea.

[PROGRAMMER: randomize Q14a-i]

	Yes	No
a. TTP is a risk when you take Votrea	x
b. You need blood tests when taking Votrea.	x
c. People with liver problems should not take Votrea.	x
d. Votrea may cause muscle pain or weakness.	x
e. A common side effect of Votrea is blurry vision.		x
f. A common side effect of Votrea is tiredness.	x
g. A common side effect of Votrea is dizziness.		x
h. People with kidney problems should not take Votrea.		x
i. A common side effect of Votrea is joint pain.		x

[PROGRAMMER: randomize order of Q19-Q20]

Q19. Compared to other treatments you could take for high cholesterol, how well do you think Votrea works?

Much better

Somewhat better

About the same

Somewhat worse

OMB Control No.

Much worse

Q20. Compared to other treatments you could take for high cholesterol, how safe or risky do you think Votrea is?

Much safer

Somewhat safer

About the same

Somewhat riskier

Much riskier

(Affect toward drug)

Q21. How good or bad do you feel about this product?

Very bad

Somewhat bad

Neither bad nor good

Somewhat good

Very good

[PROGRAMMER: Randomize order of Q22a-e]

Q22. (Attitude toward the ad) Please rate your agreement or disagreement with each of the following statements.

	Strongly agree	Somewhat agree	Neither agree nor disagree	Somewhat disagree	Strongly disagree
The ad clearly communicated the benefits of Votrea.
The ad clearly communicated the risks of Votrea.
The ad evenly balanced the risks and benefits of Votrea.
The ad communicated the benefits of Votrea in a way that was easy to understand.
T OMB Control No. he ad communicated the risks of Votrea in a way that was easy to understand.

[PROGRAMMER: Randomize order of Q23a-d]

Q23. (Attitude toward drug) Please rate your agreement or disagreement with each of the following statements.

	Strongly agree	Somewhat agree	Neither agree nor disagree	Somewhat disagree	Strongly disagree
The risks and negative effects outweigh the benefits and positive effects of Votrea.
The benefits and positive effects of Votrea outweigh the risks and negative effects
I could deal with the side effects if I reduced my bad cholesterol with Votrea.
Reducing my bad cholesterol to normal levels would not be enough to balance the risks and negative effects from Votrea.

[PROGRAMMER: Randomize order of Q24a-h]

Q24. (Peripheral cue) This ad had information about how well Votrea works. To what extent do you agree or disagree that the information was:

	Strongly agree	Somewhat agree	Neither agree nor disagree	Somewhat disagree	Strongly disagree
Noticeable
Persuasive
Believable
Distracting
Confusing
Scary
Attractive
Informative

[

OMB Control No.

PROGRAMMER: Randomize order of Q25a-h]

Q25. (Peripheral cue) This ad had information about the risks and side effects of Votrea. To what extent do you agree or disagree that the information was:

	Strongly agree	Somewhat agree	Neither agree nor disagree	Somewhat disagree	Strongly disagree
Noticeable
Persuasive
Believable
Distracting
Confusing
Scary
Attractive
Informative

Q26. (Skepticism) How likely is it that the benefit claim presented in this ad is true?

1 2 3 4 5 6 7 8 9

Not at all Extremely

Likely Likely

Q27. (Skepticism) How skeptical are you about the truth of the benefit claim in this ad?

1 2 3 4 5 6 7 8 9

Not at all Extremely

Skeptical Skeptical

Q28. (Subjective health literacy) How often do you need to have someone help you when you read instructions, pamphlets, or other written material from your doctor or pharmacy?

Never

Rarely

Sometimes

Often

Always

Q29. (Objective numeracy) Now here are some questions that require you to use numbers to solve the problem. Some are easy and others are more difficult. No calculators please- we’d like you to answer on your own. Remember, almost everyone will have trouble with these questions, so don’t be upset if some are difficult—just do your best!

[PROGRAMMER: DO NOT randomize Q29a-f]

OMB Control No.

1. What number is the correct answer:

8 + 4 + 11 = ?

1. 14

2. 19

3. 21

4. 23

5. 32

6. Don’t know

2. What is the correct answer:

17 – 8 + 4 = ?

a. 11

1. 13

2. 21

3. 23

4. 29

5. Don’t know

3. What is the correct answer:

100 x 10 x 10 = ?

1. 100

2. 1,000

3. 10,000

4. 100,000

5. 1,000,000

6. Don’t know

4. Imagine that you flip a fair coin 1,000 times. What is your best guess about how many times the coin would come up heads in 1,000 flips?

___ times out of 1,000 [PROGRAMMER: set acceptable range from 0 to 1,000]

5. In the BIG BUCKS LOTTERY, the chance of winning a $10 prize is 1%. What is your best guess about how many people would win a $10 prize if 1,000 people each buy a single ticket to BIG BUCKS LOTTERY?

________ people [PROGRAMMER: set acceptable range from 0 to 1,000]

OMB Control No.

6. In ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1,000. What percent of tickets to ACME PUBLISHING SWEEPSTAKES will win a car?

___ percent

Q30. (Subjective Numeracy 1st part) For each of the following questions, please check the box that best reflects how good you are at doing the following things:

1. How good are you at working with fractions?

2. How good are you at working with percentages?

3. How good are you at calculating a 15% tip?

4. How good are you at figuring out how much a shirt will cost if it is 25% off?

Q31. (Subjective Numeracy 2^nd part) For each of the following questions, please check the box that best reflects your answer:

1. When reading the newspaper, how helpful do you find tables and graphs that are part of a story?

2. When people tell you the chance of something happening, do you prefer that they use words (“it rarely happens”) or numbers (“there is a 1% chance)?

OMB Control No.

3. When you hear a weather forecast, do you prefer predictions using percentages (“there will be a 20% chance of rain”) or predictions using only words (“there is a small chance of rain today”)?

4. How often do you find numerical information to be useful?

Q32. Are you currently taking a prescription medicine for high cholesterol?

Yes

No

Don't know or uncertain

Q33. How severe is your high cholesterol now? Would you describe it as:

Very mild

Mild

Moderate

Serious

Very serious

Q34. In general, how much do you feel you know about high cholesterol? Would you say you know:

A lot

A good bit

Some

Only a slight amount

Nothing at all

OMB Control No.

Q35. In general, how much do you feel you know about treatments for high cholesterol? Would you say you know:

A lot

A good bit

Some

Only a slight amount

Nothing at all

Q36. Roughly, what is your total cholesterol level? Your best guess is OK.

_____

Q37. Have you ever seen any advertising for Votrea before today?

Yes

No

Don’t Remember

Q38. Did you get any help on the questions that had numbers in them from anything or anyone (e.g., calculator, other person, website)? It’s okay if you did—we just want an honest answer.

Yes (write source_____________)

No

Q39. Are you:

• Hispanic or Latino

• Not Hispanic or Latino

Q40. Which of these best represents your ethnic group? You may choose one or more. Would you say that you are:

• American Indian or Alaska Native

• Asian

• Black or African-American

• Native Hawaiian or Other Pacific Islander

• White

• Other

• Don’t know

• Prefer not to answer

Q41. Gender

1 Male 2 Female

OMB Control No.

Q42. How many years of education have you had?

Completed grade school

Completed middle school

Completed high school

Some college

College degree

Some postgraduate work

Postgraduate degree (M.A., Ph.D., M.D., J.D., etc.)

[Include if not available in KN panel database.]

Q43. How often do you exercise?

Q44. Occupation

Q45. What year were you born? _______

[End time: ___________________ ]

You have been very helpful. Thank you very much for your participation!

OMB Control No.

APPENDIX 3

Sample Statistical Formats

NO VISUAL, NO EFFICACY

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

NO VISUAL, HIGH EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Percent:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65% of people lowered their bad cholesterol to normal levels versus 2% of people with no treatment.

Combination of Absolute Frequency and Percent:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65 out of 100 people (65%) lowered their bad cholesterol to normal levels versus 2 out of 100 people (2%) with no treatment.

Relative Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, people were 33 times more likely to lower their bad cholesterol to normal levels, compared to no treatment.

Combination of Relative Frequency and Percent:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65% of people lowered their bad cholesterol to normal levels versus 2% of people with no treatment—that’s 33 times more effective.

BAR GRAPH, HIGH EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

PIE CHART, HIGH EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Votrea No treatment

TABLE, HIGH EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Clinical Trial Results
		Votrea	No Treatment
Outcome: Lowered cholesterol to normal levels	Yes	65	2
	No	35	98
	Total	100 patients	100 patients

PICTOGRAPH, HIGH EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 65 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Votrea

* * * * * * * * * *

* * * * * * * * * *

* * * * * * * * * *

* * * * * * * * * *

* * * * * * * * * *

* * * * * * * * * *

* * * * * ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

No treatment

* * ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

NO VISUAL, LOW EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Percent:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10% of people lowered their bad cholesterol to normal levels versus 2% of people with no treatment.

Combination of Absolute Frequency and Percent:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10 out of 100 people (10%) lowered their bad cholesterol to normal levels versus 2 out of 100 people (2%) with no treatment.

Relative Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, people were 5 times more likely to lower their bad cholesterol to normal levels, compared to no treatment.

Combination of Relative Frequency and Percent:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10% of people lowered their bad cholesterol to normal levels versus 2% of people with no treatment—that’s 5 times more effective.

BAR GRAPH, LOW EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

PIE CHART, LOW EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Votrea No treatment

TABLE, LOW EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Clinical Trial Results
		Votrea	No Treatment
Outcome: Lowered cholesterol to normal levels	Yes	10	2
	No	90	98
	Total	100 patients	100 patients

PICTOGRAPH, LOW EFFICACY

Absolute Frequency:

Votrea reduces bad cholesterol for people with several common risk factors for heart disease. (super: starting from an average bad cholesterol of 160 mg/dL).

With Votrea, 10 out of 100 people lowered their bad cholesterol to normal levels versus 2 out of 100 people with no treatment.

Votrea

* * * * * * * * * *

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

No treatment

* * ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° ° °

1 Fagerlin, A., Ubel, P.A., Smith, D.M., & Zikmund-Fisher, B.J. (2007). Making numbers matter: Present and future research in risk communication. American Journal of Health Behavior, 31, Suppl 1: S47-56.

2 Acker, J.S., Senathirajah, Y., Kukafka, R., & Starren, J.B. (2006). Design features of graphs in health risk communication: A systematic review. Journal of the American Medical Information Association, 13, 608-618.

3 Lipkus, I., & Hollands, J.G. (1999). The visual communication of risk. Journal of the National Cancer Institute Monographs, 25, 149-163.

4 Lipkus, I. (2007). Numerica, verbal, and visual formats of conveying health risks: Suggested best practices and future recommendations. Medical Decision Making, 27, 697-713.

5 Slovic, P., & Peters, E. (2006). Risk perception and affect. Current Directions in Psychological Science, 15, 322-325.

6 Slovic, P., Monahan, J., & MacGregor, DG. (2000). Violence risk assessment and risk communication: the effects of using actual cases, providing instruction, and employing probability versus frequency formats. Law and Human Behavior, 24, 271-96.

7 Peters, E., Vastfjall, D., Slovic, P., Mertz, CK, Massocco, K., & Dickert, S. (2006). Numeracy and decision making. Psychological Science, 17, 407-13.

8 Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A, (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavor Research Methods, 39, 175-191.

9 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2^nd Ed). Hillsdale, NJ: Lawrence Erlbaum & Associates, Inc.

33