Supporting Statement A-1

Experimental Study of Format Variations in the Brief Summary of Direct-to-Consumer (DTC) Print Advertisements

0910-Number

SUPPORTING STATEMENT

Submitted by

Office of Prescription Drug Promotion

Center for Drug Evaluation and Research

Food and Drug Administration

February, 2012

A. JUSTIFICATION

1. Circumstances Making the Collection of Information Necessary

Section 3507 of the Patient Protection and Affordable Care Act of 2010 requires The Secretary of Health and Human Services, acting through the Commissioner of Food and Drugs, to determine whether providing quantitative summaries of benefits and risks in a standardized format in print promotional materials for prescription drugs would “improve health care decision making by clinicians and patients and consumers” (Appendix A). This study directly addresses this concern.¹

Section 502(n) of the Federal Food, Drug, and Cosmetic Act specifies that ads for prescription drugs and biological products must provide a true statement of information “in brief summary” about the advertised product’s “side effects, contraindications, and effectiveness.” The prescription drug advertising regulations (21 CFR § 202.1(e)(3)(iii); Appendix B) specify that the information about risks must include each specific side effect and contraindication from the advertised drug’s FDA-approved labeling, including the Warnings, Precautions, Adverse Reactions, and other relevant sections. Some of the current approaches to fulfilling the brief summary requirement, while adequate from a regulatory perspective, result in ads that may be difficult to read and understand when used in consumer-directed promotion.

In recent years, FDA has become concerned about the adequacy of the brief summary in DTC print advertisements for prescription drugs. Because the regulations do not specify how to address each risk, sponsors can use discretion in fulfilling the brief summary requirement under § 202.1(e)(3)(iii). Frequently, sponsors print in small type, verbatim, the risk-related sections of the approved product labeling (also called the package insert, professional labeling, prescribing information, and direction circular). This labeling is written for health professionals, using medical terminology. While adequate to fulfill the brief summary requirement for print advertisements, this method may not be the most ideal. Research has shown that while many consumers will make the effort to read the brief summary in prescription drug print advertisements if they are especially interested in the drug, as a general rule consumers typically read little or none of the brief summary information.² Health practitioners themselves have indicated they often have difficulty finding information they actively seek in package inserts (see 65 FR 80733 at 81082, December 22, 2000, for a discussion of studies supporting the use of a highlights section in physician labeling). There may be other ways to fulfill this requirement that improve consumers’ ability to find and comprehend the information in this important document.

Evidence suggests that both information content and the format in which that content is presented will impact comprehension. For instance, research with the format of over-the-counter (OTC) drug labels,³ the nutrition facts label,⁴ and other information formats⁵ demonstrates that information presented with section headings, graphics (such as bullets), and other design elements is more easily read than information presented in paragraph format.

Research conducted by FDA and others has examined the content and format of the brief summary specifically. For instance, FDA conducted a series of relevant studies (OMB control numbers 0910-0591 and 0910-0611). Schwartz, Woloshin, and Welch have compared one format for adding quantitative and qualitative benefit and risk information to the brief summary.⁶ Specifically, Schwartz et al. designed a prescription drug facts box similar in format to the Nutrition Facts panel and OTC Drug Facts panel. The box contains a number of elements, including qualitative and quantitative (both absolute frequency and absolute difference) information about benefits and risks. This study showed that consumers who were provided efficacy information in a prescription drug facts box were more likely to correctly choose the product with the higher efficacy than consumers who saw the brief summary using medical language from the Prescribing Information PI. However, it is unclear which elements of the drug facts box are necessary to improve consumer understanding. For instance, it is not known whether simply adding efficacy rate information to a consumer-friendly brief summary would be sufficient to enable consumers to understand a product’s efficacy, or whether qualitative summations are necessary as well.

The current study will add to previous research by systematically examining these different elements to determine whether and how to add qualitative and quantitative benefit and risk information to the brief summary⁷. The results of this study will inform FDA of the usefulness and parameters of various format and content options for the brief summary and will help address Section 3507 of the Patient Protection and Affordable Care Act of 2010.

2. Purpose and Use of the Information Collection

This project will involve a two-part web-based experiment with consumers, one examining consumers’ understanding of efficacy information, and the other examining consumers’ understanding of risk information. The purpose of this two-part project is to gather data for the Federal Government to address 1) whether quantitative or qualitative information provides differentially useful information for consumers, 2) whether a box format itself is enough to improve consumer understanding, and 3) whether these factors vary depending on the efficacy or riskiness of the drug. Part of FDA’s public health mission is to ensure the safe use of prescription drugs; therefore it is important to communicate the risks and benefits of prescription drugs to consumers as clearly and usefully as possible. External researchers have proposed a format for the brief summary in print advertisements for prescription drugs.⁸ We propose to tease apart their effects to determine which of the tested format variables improve communication of information in the brief summary.

Data will be collected by an independent contractor and shared with FDA electronically. No personally identifiable information will be sent to FDA. All information that can identify individual respondents will be maintained by the independent contractor in a form that is separate from the data provided to FDA. The data shared with FDA will be used to answer the research questions.⁹

3. Use of Improved Information Technology and Burden Reduction

Automated information technology will be used in the collection of information for this study. The contracted research firm will collect data through Internet administration. One hundred percent (100%) of participants will self-administer the Internet survey via a computer, which will record responses and provide appropriate probes when needed. In addition to its use in data collection, automated technology will be used in data reduction and analysis. Burden will be reduced by recording data on a one-time basis for each respondent, and by keeping surveys to less than 20 minutes.

4. Efforts to Identify Duplication and Use of Similar Information

As discussed in section A.1, Schwartz, Woloshin, and Welch (2005) examined a drug facts format for use as a brief summary in prescription drug advertising. This study held promise as it introduced a possible improvement to a document that FDA would like to examine in terms of consumer understanding. The study did not tease apart particular aspects of the new format, however, and as such, what particular factors improved consumer understanding were not clear. FDA conducted a study comparing a similar brief summary format to three other formats (OMB control number 0910-0611; see explanation of the study, “Experimental Evaluation of Variations in Content and Format of the Brief Summary in Direct-to-Consumer Print Advertisements for Prescription Drugs” below), showing that this type of format was well-received by consumers. However, this study focused only on the presentation—the box itself—and not on the content (for instance, quantitative information on prescription drug risks and benefits). To our knowledge, no subsequent studies have followed up on the Schwartz et al. study to further examine the results of the changes introduced.

The FDA has an on-going research program related to consumer understanding of Direct-to-Consumer (DTC) advertising of prescription drugs. The three studies most relevant to the study under review are outlined below. We do not have any other studies on the addition of quantitative information to DTC ads in development. However, depending on the results of the study under review, we may conduct another study in relation to this research program in the next few years.

1. Evaluation of Consumer-Friendly Formats for Brief Summary in Direct-to-Consumer (DTC) Print Advertisements for Prescription Drugs: Study 1 (FDA-2005-N-0016) and Experimental Evaluation of Variations in Content and Format of the Brief Summary in Direct-to-Consumer Print Advertisements for Prescription Drugs (FDA-2006-N-0111)

To improve its understanding of how consumers use the brief summary and explore ways in which it might be improved, the Food and Drug Administration (FDA) conducted studies to address the following three questions:

• Does the risk information presented on the first page of the ad (the display page) influence the way that people read through and understand the information in the brief summary, and what topics do people think are important in the brief summary?

• Do additional details and context about side effects negatively affect the reading and understanding of other risk information in the brief summary?

• How do alternative formats compare with the commonly used format for the brief summary? 

We found that the addition of a serious risk to the display page and the addition of frequency and duration information about side effects in the brief summary did not negatively affect the understanding of the risk information as a whole, including the most serious warnings and precautions. The format of the information had several effects. For instance, participants who viewed the Drug Facts format were better able to recall risks than those who saw the Traditional format.

2. Presentation of Quantitative Effectiveness and Risk Information to Consumers in Direct-to-Consumer (DTC) Broadcast and Print Advertisements for Prescription Drugs (FDA-2009-N-0263)

In this study, we examined the communication of complicated efficacy information.  Specifically, we examined whether quantitative efficacy information can be successfully added to television and print advertisements to maximize the audience's understanding of the risk and benefit information in the piece.  We investigated the level of product efficacy (high or low), the statistical format of that information (frequency, percent, frequency plus percent, relative frequency or frequency plus relative frequency), and ways in which that information can be expressed visually (pie chart, bar chart, table, or pictograph). This study focused exclusively on the inclusion of quantitative efficacy information, not risk, and did not alter the brief summary in any way. Data collection for this study is complete and we are reviewing preliminary results.

3. Study of Clinical Efficacy Information in Professional Labeling and Direct-to-Consumer Print Advertisements for Prescription Drugs (FDA-2010-N-0266)

This study will investigate the communication of effectiveness information on the main advertising (display) page of print advertisements.  We will examine whether adding placebo information and whether changing the framing of the information helps consumers understand the information.  Separately, we will examine how physicians use the prescribing information documents, and specifically how they assess efficacy information in this document.  Data collection for this study is underway.

5. Impact on Small Businesses or Other Small Entities

No small businesses would be involved in this data collection.

6. Consequences of Collecting the Information Less Frequently

The proposed data collection is one-time only. There are no plans for successive data collections.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This collection of information fully complies with 5 CFR 1320.5. There are no special circumstances.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A 60 day Federal Register Notice was published in the Federal Register on August 31, 2010, vol. 75, No. 168; pp. 53312-53314 (see Appendix C). FDA received four public comments. In the following section, we outline the observations and suggestions raised in the comments and provide our responses.

Comment 1. Several of this comment’s suggestions related to participant demographics. We agree that the study design should include the variables of age, education, ethnicity, and race; these are included in the questionnaire. We will ask whether participants can read, understand, and speak English. This comment suggested measuring health literacy; instead, we will measure subjective health literacy and the related concept of numeracy, which is relevant for this research as we are studying the comprehension of quantitative information. To clarify, we will not limit our sample to those who are currently being treated with a prescription drug for the condition being assessed; however, the questionnaire includes questions about prescription drug use.

A question in this comment asked what our primary research questions are. As stated in the 60-day notice, the current study will add to previous research by systematically examining the different elements in the drug facts box tested in previous research¹⁰ to determine whether and how to add qualitative and quantitative benefit and risk information to the brief summary. Specifically, we will test whether the inclusion of a qualitative label and/or the inclusion of quantitative information affects consumers’ understanding of the information and their perceptions of the product.

One question related to the test direct-to-consumer (DTC) advertisements to be used in the study. We have contracted with an organization that produces realistic ads and stimuli to ensure that we will show respondents realistic materials.

This comment requested that FDA provide clarity on the timing and strategy for the conduct of this study with respect to other planned studies. To clarify, this study will begin after two related studies¹¹ have been conducted. The results from these studies inform the execution of this study (see the response to Comment 3 for more details). The study will not be superseded by related research results, as none of the other research examines the drug facts box format for the brief summary.

Finally, the comment recommends that FDA publish findings from this study and previous studies on the Office of Prescription Drug Promotion (OPDP) webpage. We agree and have taken steps to publish reports from our previous research on the OPDP webpage.¹² For instance, we reported the results of the first study mentioned in A.4 (Evaluation of Consumer-Friendly Formats for Brief Summary in Direct-to-Consumer (DTC) Print Advertisements for Prescription Drugs: Study 1 (FDA-2005-N-0016) and Experimental Evaluation of Variations in Content and Format of the Brief Summary in Direct-to-Consumer Print Advertisements for Prescription Drugs, (FDA-2006-N-0111) on the OPDP webpage. When the current project is concluded, we will post the findings on the OPDP webpage as well.

Comment 2. This comment states that there was not enough detail in the 60-day federal register notice. The questionnaire, which has information about how questions will be asked and how behavioral intention will be assessed, was available upon request during the first comment period and will continue to be available during the second comment period. Information about how risk information will be portrayed, what statistical analyses will be performed, subject recruitment, and pretest content is addressed below.

The comment states that the previous notice did not describe the criteria for determining the amount and type of risk and benefit information to provide in the box format. We agree that a major challenge of the Drug Facts Box format is deciding the amount and content of risk information to include; however, this type of study cannot address this issue. To replicate and extend past research, we will use the Drug Facts Box from a previous study¹ with slight modifications to the risk information (e.g., the addition of a serious risk, different rates of side effects in the placebo and active drug groups).

A related comment noted that product labeling is multi-faceted and recommended that conclusions should be flexible to address these wide variations in product attributes. We agree that product labeling is multifaceted and will tailor our conclusions to acknowledge that we tested one simple version of the Drug Facts Box.

Another question noted that qualitative terms depend on many factors. We agree; however, this study does not address the feasibility of creating qualitative terms but rather tests whether qualitative terms affect consumer comprehension. As requested, we will note this in our conclusions.

Another suggestion was to consider a label format that includes multiple endpoints. As a first step, we plan to study a simple version of the Drug Facts Box, with one indication. If consumers cannot understand the information in a Drug Facts Box with one indication, they are not likely to understand the information in the Drug Facts Box with multiple indications. In addition, testing an ad with one endpoint is realistic as drug ads often promote only one indication even if a drug has multiple indications.

This comment recommends that we consider implementing a cross-over study design to address inter-patient variability. Conducting a cross-over design would significantly increase study length, and repeated exposure to the same stimuli with minor changes may affect participants’ responses. We have conducted power analyses and believe we can find interpretable results without conducting a cross-over design.

This comment suggested considering caregivers and consumers who do not have the medical condition treated by the drug. To ensure that our participants are motivated to consider the information presented in the study and to conserve resources, we will limit our sample to people who have the medical condition of interest.

The final question in this comment asked how the tools were qualified or validated for their intended use. Cognitive testing will be used to test questionnaire items prior to their use, and similar items have been used in our previous studies. The items have face validity and several are drawn from well-tested items used in the psychology literature.¹³ Finally, we will pre-test the study manipulations.

Comment 3. Much of this comment focused on previous research. First, this comment requests that we disclose the results of previous research. As stated in the response to Comment 1, we agree and have taken steps to publish findings from our previous research on the OPDP webpage.¹⁴ As stated in response to Comment 1, as research projects develop we will take results of previous research in account. As an example, the following paragraph describes how the findings from the studies outlined in Section A.4 were used to guide the design of the proposed research.

The first study mentioned in A.4 (“Experimental Evaluation of Variations in Content and Format of the Brief Summary in Direct-to-Consumer Print Advertisements for Prescription Drugs”) showed that a “drug facts box” format was well-received by consumers. However, this study focused only on the presentation—the box itself—and not on the content (for instance, quantitative information on prescription drug risks and benefits). The proposed study builds on that study by examining the addition of quantitative benefit and risk information into the drug facts box format. In addition, preliminary results from the second study mentioned in A.4 (“Presentation of Quantitative Effectiveness and Risk Information to Consumers in Direct-to-Consumer (DTC) Broadcast and Print Advertisements for Prescription Drugs”) suggest that providing quantitative benefit information in DTC advertising does not overwhelm participants and gives us confidence to proceed with a study testing the inclusion of quantitative information in the brief summary. Finally, pretest results from the third study mentioned in A.4 (“Study of Clinical Efficacy Information in Professional Labeling and Direct-to-Consumer Print Advertisements for Prescription Drugs”) showed that the term “placebo” is understood as well as or better than alternatives (e.g., “sugar pill”) so we will use the term “placebo” in this study.

This comment included three statements about the details of the proposed study. First, the comment questioned why we chose to test percents and frequencies and not relative differences in this study. We focus on percents and frequencies because we are replicating and extending previous research on a Drug Facts Box¹⁵ which included percents and frequencies but not relative differences. The study found that the Drug Facts Box outperformed a traditional brief summary. The Drug Facts Box tested had several elements that differed from the traditional brief summary, including percents, frequencies (i.e., XX/100), and qualitative labels. From these results it is not possible to tell which elements of the Drug Facts Box were responsible for the effects found. This study aims to test systematically the elements of the Drug Facts Box to determine which, if any, improves consumer comprehension.

Second, this comment pointed out that the differences in the stimuli should be stated as percentage points, not as percentages. We will make this change to our stimuli.

Third, the comment asks whether the risk and benefit information will be presented in the same mathematical expression and whether they will be presented independently. To clarify, when participants see benefit information in a certain information type (or mathematical expression, for example, percents), they will also see risk information in that same information type (for example, percents). However, the efficacy level (from smallest to largest effect) will be manipulated in one design and the risk level (from smallest to largest effect) will be manipulated in a separate design.

Comment 4. The first recommendation is to redesign the study such that participants would view the study materials and then answer questions about the materials only after consulting with a physician. This is not feasible or ethical. We cannot ask participants to incur the financial and personal (time) cost of visiting a doctor to discuss a treatment for the purposes of research. We cannot ethically ask them to go to their doctor to discuss a fictitious drug (nor would the doctor be able to discuss a fictitious drug with them) and we cannot ethically recommend a real product for them to discuss with their doctor. Aside from the feasibility and ethical issues, this is an unnecessary step to answer our research questions about participants’ comprehension of a widely disseminated written form of information. Moreover, the assumption behind this recommendation, that physician consultations are the “context in which prescription drug advertisements are actually used,” is questionable. DTC advertising does not exist solely in the confines of a doctor’s office; rather, DTC advertising targets consumers outside of a doctor’s office, with the goal of prompting consumers to ask their physicians about the product. Therefore, clear communication of risks and benefits is needed for consumers before a consultation with a physician.

This comment lists a number of practical issues surrounding how to create Drug Facts Boxes and notes that this study will provide limited practical information on how to format the brief summary for drugs with multiple indications, multiple studies, or multiple outcomes. We agree that there are several practical issues surrounding the utility of the Drug Facts Box; however, these issues are outside the scope of the proposed study. This study does not address how information would be chosen for inclusion in Drug Facts Boxes but rather whether and how consumers can understand the information presented. As stated in the response to Comment 2, our first step will be to study a simple version of the Drug Facts Box, with one indication.

Another recommendation from the comment is to include conditions that test relative difference. We agree that relative difference is an interesting way to present quantitative information and are currently studying this presentation in another study.¹⁶ However, as noted in the response to Comment 3, in this study we are systematically testing the elements of the Drug Facts Box presented in past research¹⁷ to determine which, if any, improves consumer comprehension.

Another comment suggested that, along with testing the qualitative label, “fewer people taking Drug X had symptom Y,” we should also test the qualitative label, “more people taking Drug X received effective relief from symptom Y.” Unfortunately we do not have the resources to test multiple qualitative labels in this study; however, we will test the qualitative label suggested by the commenter in place of our original language.

The final comment recommends eliminating the “largest effect” cells. We agree that these cells may be unrealistic and plan to use pretests to determine the number of levels and the content of the levels (e.g., the differences used) to be included in the main study.

External Reviewers

In addition to public comment, DDMAC sent materials and received comments from four individuals for external peer review. These individuals are:

• Angela Fagerlin, Ph.D., University of Michigan

• Sarah Fein, Ph.D., Center for Food Safety and Applied Nutrition, FDA

• Lisa Schwartz, M.D., Dartmouth Medical School

• Steven Woloshin, M.D., Dartmouth Medical School

9. Explanation of Any Payment or Gift to Respondents

Internet panel participants receive points for completing a survey. One thousand points (approximately monetary equivalence of $1) will be awarded. Members are allowed to use their points to exchange for vouchers and gifts from a partner network.

10. Assurance of Confidentiality Provided to Respondents

No personally identifiable information will be sent to FDA. All information that can identify individual respondents will be maintained by the independent contractor in a form that is separate from the data provided to FDA. The information will be kept in a secured fashion that will not permit unauthorized access. Confidentiality of the information submitted is protected from disclosure under the Freedom of Information Act (FOIA) under sections 552(a) and (b) (5 U.S.C. 552(a) and (b)), and by part 20 of the agency’s regulations (21 CFR part 20.63).¹⁸ These methods will all be approved by FDA’s Institutional Review Board (Research Involving Human Subjects Committee, RIHSC) prior to collecting any information.

All respondents will be provided with an assurance of confidentiality to the extent allowable by law. The Internet Panel includes a Panel Privacy Policy that is easily accessible from any page on the site. A link to the Privacy Policy will be included on all survey invitations. The Panel complies with established industry guidelines and states  that members’ personally identifiable information will never be rented, sold, or revealed to third parties except in cases where required by law. These standards and codes of conduct comply with those set forth by American Marketing Association, the Council of American Survey Research Organizations, and others. In addition, a consent form will be displayed before participants begin the survey (Appendix G). The consent form states that participation is voluntary.¹⁹

All electronic data will be maintained in a manner consistent with the Department of Health and Human Services’ ADP Systems Security Policy as described in the DHHS ADP Systems Manual, Part 6, chapters 6-30 and 6-35.²⁰ All data will also be maintained in consistency with the FDA Privacy Act System of Records #09-10-0009 (Special Studies and Surveys on FDA Regulated Products).²¹

11. Justification for Sensitive Questions

This data collection will not include sensitive questions. The complete list of questions is available in Appendix D.

12. Estimates of Annualized Burden Hours and Costs

The total annual estimated burden imposed by this collection of information is 6,221 hours for this one-time collection (Table 1). These estimates are based on a 16-minute FDA study using the Synovate panel entitled “Nutrition Facts Experimental Study I,” which had the goal of a final sample with similar demographic proportions to the 2010 Census with regard to age, race/ethnicity, gender, and education.  

Table 1.--Estimated Annual Reporting Burden1
Activity	No. of Respondents	No. of Responses per Respondent	Total Annual Responses	Average Burden per Response (in Hours)2	Total Hours
Sample outgo (pretests & main study)	197,480	--	--	--	--
Number to complete the screener (35%)	69,118	1	69,118	2/60	2,304
Number eligible for survey (20%)	13,824	--	--	--	--
Number to complete the survey (85%)	11,750	1		20/60	3,917
Total					6,221

¹ There are no capital costs or operating and maintenance costs associated with this collection of information.

² Burden estimates of less than 1 hour are expressed as a fraction of an hour in the format "[number of minutes per response]/60".

Table 3. --Estimated Annualized Burden Costs
Type of Respondent	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
General public	6,221	$18.681	$116,208
Total			$116,208

¹Based on the 2010 median weekly income of $747 for both sexes, as reported by the Department of Labor, ftp://ftp.bls.gov/pub/special.requests/lf/aat39.txt

13. Estimates of Other Total Annual Costs to Respondents and Record Keepers

There are no costs to respondents. There are no record keepers.

14. Annualized Cost to the Federal Government

The total estimated cost to the Federal Government for the collection of data is $889,528 ($296,509 per year for three years). This includes the costs paid to the contractors to create stimuli, program the study, draw the sample, collect the data, and create a database of the results ($769,528). The task order was awarded as a result of competition. Specific cost information other than the award amount is proprietary to the contractor and is not public information. The cost also includes FDA staff time to design and manage the study, to analyze the resultant data, and to draft a report ($120,000; 15 hours per week for 3 years).

15. Explanation for Programs Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule

Conventional statistical techniques for experimental data, such as descriptive statistics, analysis of variance, and regression models, will be used to analyze the data. See section B below for detailed information on the design, hypotheses, and analysis plan. The Agency anticipates disseminating the results of the study after the final analyses of the data are completed, reviewed, and cleared. The exact timing and nature of any such dissemination has not been determined, but may include presentations at trade and academic conferences, publications, articles, and Internet posting.

Table 4. Project Timetable

Task	Estimated Completion Date
External Peer Review	March, 2011
RIHSC Review	April, 2011
30-day FR notice publication	April, 2011
OMB Review of PRA package	June, 2011
Data Collection	July, 2011
Receipt of Data and Methods Report from Contractor	December, 2011
Data Analysis	January, 2012
Draft Report	March, 2012
Internal Review of Draft Report	April, 2012
Revisions	May, 2012
Final Report	June, 2012

17. Reason(s) Display of OMB Expiration Date is Inappropriate

No exemption is requested.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to the certification

1 Note: This information collection is not related to the American Recovery and Reinvestment Act of 2009 (ARRA).

2 Aikin, K.J., Swasy, J.L. and Braman, A.C. (2004). Patient and Physician Attitudes and Behaviors Associated with DTC Promotion of Prescription Drugs: Summary of FDA Survey Research Results, Final Report. Available at http://www.fda.gov/downloads/Drugs/ScienceResearch/ResearchAreas/DrugMarketingAdvertisingandCommunicationsResearch/UCM152860.pdf. Last accessed August 12, 2010.

3 Aikin, K.J. (1998). Consumer Comprehension and Preference for Variations in the Proposed Over-The-Counter Drug Labeling Format, Final Report; Vigilante, W.J. & Wogalter, M.S. (1997). The preferred order of over-the-counter (OTC) pharmaceutical label components. Drug Information Journal, 31, 973-988.

4 Levy, A.S., Fein, S.B. & Schucker, R.E. (1992). More effective nutrition label formats are not necessarily more preferred. Journal of the American Dietetic Association, 92(10), 1230-1234.

5 Lorch, R. & Lorch, E. (1995). Effects of organizational signals on text-processing strategies. Journal of Educational Psychology, 87(4), 537-544; Lorch, R. & Lorch, E. (1996). Effects of organizational signals on free recall of expository text. Journal of Educational Psychology, 88(1), 38-48; Lorch, R., Lorch, E. & Inman, W. (1993). Effects of signaling topic structure on text recall. Journal of Educational Psychology, 85(2), 281-290.

6 Schwartz, L.M., Woloshin, S., & Welch, H.G. (2009). Using a drug facts box to communicate drug benefits and harms: Two randomized trials. Annals of Internal Medicine, 150(8). Available online at http://www.annals.org/cgi/content/full/0000605-200904210-00106v1. Last accessed August 12, 2010.

7 For a full discussion of the information to be collected, see Section B.2 and Appendix G. This satisfies section D.b.1 of the OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002.

8 Schwartz, L.M., Woloshin, S., & Welch, H.G. (2009). Using a drug facts box to communicate drug benefits and harms: Two randomized trials. Annals of Internal Medicine, 150(8). Available online at http://www.annals.org/cgi/content/full/0000605-200904210-00106v1. Last accessed August 12, 2010.

9 This paragraph satisfies sections D.b.2 and D.b.3 of the OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002.

10 Schwartz, L.M., Woloshin, S., & Welch, H.G. (2009). Using a drug facts box to communicate drug benefits and harms: Two randomized trials. Annals of Internal Medicine, 150(8), 516-527. Available online at http://www.annals.org/content/150/8/516.full. Last accessed December 30, 2010.

11 FDA-2009-N-0263 (January 5, 2010). Presentation of Quantitative Effectiveness and Risk Information to Consumers in Direct-to-Consumer (DTC) Broadcast and Print Advertisements for Prescription Drugs; FDA-2010-N-0266. (June 16, 2010). Study of Clinical Efficacy Information in Professional Labeling and Direct-to-Consumer Print Advertisements for Prescription Drugs.

12 http://www.fda.gov/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDER/ucm090276.htm

13 For example, behavioral intentions: Webb, T.L., & Sheeran, P. (2006). Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychological Bulletin, 132, 249-268.

14 http://www.fda.gov/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/CDER/ucm090276.htm

15 Schwartz, L.M., Woloshin, S., & Welch, H.G. (2009). Using a drug facts box to communicate drug benefits and harms: Two randomized trials. Annals of Internal Medicine, 150(8), 516-527. Available online at http://www.annals.org/content/150/8/516.full. Last accessed December 30, 2010.

16 FDA-2009-N-0263 (January 5, 2010). Presentation of Quantitative Effectiveness and Risk Information to Consumers in Direct-to-Consumer (DTC) Broadcast and Print Advertisements for Prescription Drugs; FDA-2010-N-0266. (June 16, 2010).

17 Schwartz, L.M., Woloshin, S., & Welch, H.G. (2009). Using a drug facts box to communicate drug benefits and harms: Two randomized trials. Annals of Internal Medicine, 150(8), 516-527. Available online at http://www.annals.org/content/150/8/516.full. Last accessed December 30, 2010.

18 This section states: “(a) The names or other information which would identify patients or research subjects in any medical or similar report, test, study, or other research project shall be deleted before the record is made available for public disclosure. (b) The names and other information which would identify patients or research subjects should be deleted from any record before it is submitted to the Food and Drug Administration. If the Food and Drug Administration subsequently needs the names of such individuals, a separate request will be made.”

19 This satisfies section D.b.4.1 and D.b.4.2 of the OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002.

20 This satisfies section D.b.4.3 of the OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002.

21 This satisfies section D.b.4.4 of the OMB Guidance for Implementing the Privacy Provisions of the E-Government Act of 2002.

20