Quant Claims SSB FINAL MAY 2024

United States Food and Drug Administration

A Survey on Quantitative Claims in Direct-to-Consumer Prescription Drug Advertising

OMB Control No. 0910-NEW

SUPPORTING STATEMENT

Part B. Statistical Methods

1. Respondent Universe and Sampling Methods

Adults aged 18 or over will be eligible for participation. To obtain a nationally representative sample, we will use address-based sampling (ABS). Westat, the contractor, will utilize the vendor Marketing Systems Group (MSG) to provide a frame for the ABS sample. MSG obtains addresses from the U.S. Postal Service Computerized Delivery Sequence File and appends information that can be used in stratification, such as demographic characteristics at the Census block level.

We will use a stratified random sample of 2,993 addresses on the ABS frame, estimating 1,100 completed surveys (assuming a 40 percent response rate and 5 percent bad addresses or returned mail). The stratification variable, educational attainment, was chosen to ensure a nationally representative sample of the general adult population. The precision obtained from a stratified random sample is not the same as a simple random sample of 1,100 completed surveys because of both within-household selection and nonresponse adjustment. Within-household selection of one adult means that a household weighting factor has to be assigned, which is the number of adults in the household (i.e., number of adults that the sampled adult is representing in the household).

We defined 10 Census Tract strata, of roughly equal size (sample size 300), based on a mean educational attainment score for the Census Tract. These Census Tract scores were defined based on the distribution of adults 25+ at 16 educational attainment levels from no schooling at all (score=0) through High School diploma only (score=1) through bachelor’s degree only (score=3) to advanced degree (score=5). Census tracts with the lowest mean educational levels were in the first stratum, Census tracts with the highest mean educational levels were in the tenth stratum, and so forth. This stratification guarantees that Census tracts with lower or higher mean educational attainment levels will not be over- or underrepresented in the national sample, and a balance on educational attainment will be achieved.

2. Procedures for the Collection of Information

Survey Overview

The 20-minute survey contains questions about respondents’ perceptions and understanding of several quantitative claims drawn from direct-to-consumer (DTC) ads in the marketplace (Appendix A). We will also measure other potentially important variables such as demographics and numeracy. The survey questions were informed by consumer feedback elicited in one-on-one interviews (OMB control number 0910-0847, “Quantitative Claims in Direct-to-Consumer Prescription Drug Advertising: Consumer Interviews”). In 2022, we conducted 24 qualitative semi-structured interviews with adult consumers from the general US adult population. We conducted 12 interviews with individuals with lower educational attainment and 12 interviews with individuals with higher educational attainment. The goal of the interviews was to examine how consumers interpret different quantitative claims that are used in DTC promotional materials (see Appendix A for the claims). Our analysis revealed several areas where consumers may misinterpret quantitative information about prescription drugs. For example, participants often misinterpreted claims with qualifiers describing a minimum or maximum threshold.

Procedure

The data collection protocol will include mailing invitations to potential respondents to complete a web-based survey followed by a paper survey mailing in the last contact, with a total of four contacts (Appendices B through E). While the primary mode of response is expected to come from the web survey, we will use a paper survey in the final mailing to attain a higher response rate and ensure responses from hard-to-reach populations with lower education, older consumers, and people with low socioeconomic levels who may not have access to the web. This four-contact, mixed-mode approach along with incentives is a cost-effective approach to achieve a high response rate while also ensuring diversity of respondents.

We will first send a prenotification letter on FDA letterhead. The letter will describe the study purpose and the importance of the responses, introduce Westat as the contractor, and inform recipients of the upcoming opportunity to participate. The letter will include a link to an FDA website describing the survey (Appendix F). One week after the prenotification letter is mailed, we will send a survey invitation letter that includes a unique survey login, instructions on who within the household should complete the survey (i.e., the adult household member with the next birthday), a $5 prepaid incentive, and a promised additional incentive of $10 upon completion of the survey for 75 percent of the sample (see Part A for details of the incentive experiment).

We will conduct a soft launch of the invitation letter to approximately 10 percent of the sample to test and evaluate procedures for the web survey. Any issues discovered will be corrected and tested as quickly as possible. We will subsequently mail the remaining survey invitations. The third mailing will be a sealed postcard reminder with survey login information to all nonrespondents. The fourth mailing will include a paper copy of the survey with a return address envelope. A final mailing with the $10 postpaid incentive will be sent to individuals in the promised incentive groups who completed the survey and who did not select the egift card option (Appendix G).

Analysis Plan

We will analyze the survey data to provide descriptive statistics (e.g., frequencies, percentages, composite scores, means) as well as breakouts by subgroups of adult consumers of interest (e.g., age, gender, race/ethnicity, education). These descriptive statistic results will be weighted to represent the adult consumer population. We will use NVivo to facilitate coding and analysis of the open-ended responses. Two trained coders will code a subset of the open-ended responses, and Cohen’s kappa will be used to calculate inter-rater reliability.

Nonresponse bias in a survey can be substantial when three conditions hold: (1) the response rate is relatively low, (2) there are considerable differences in response propensity across important subgroups of the population, and (3) the difference between the characteristics of respondents and nonrespondents is relatively large. We will conduct nonresponse bias analysis according to OMB’s “Standards and Guidelines for Statistical Surveys.” We will compare respondent characteristics with known population distributions, including comparisons of demographic data available from our sampling frame and variables of interest from the most current American Community Survey (ACS). We will calibrate the sampling weights to the ACS totals to help improve the precision of the estimates (see more details below). After adjusting for population differences (e.g., web respondents are likely younger), we will conduct analyses to determine if there are differences between web and paper responses.

Weighting

Sampling weights must be developed to account for the within-household adult selection. In addition, the weights for the survey must be adjusted for nonresponse and coverage errors, although the coverage error is expected to be small given the design. Data from the sampling frame can be used in the formation of nonresponse adjustment cells. This technique can reduce the risk of nonresponse bias if the adjustment cells are selected carefully. We will conduct a final calibration of the weights or post-stratification adjustment. Westat uses raking to make these final adjustments, and the control totals can be based on ACS estimates. This technique can simultaneously reduce nonresponse bias, reduce coverage error, and increase the precision of the estimates (i.e., reduce the variance). Given the nature of response to advertising claims, it is likely that gender, age, education, and race/ethnicity may all be potentially important adjusters related to response propensity and response to advertising claims.

Power

We assume an overall 1.5 design effect (the inflation of variance from weighting effects against a simple random sample). With this sample size and design effect, 1,100 completed surveys will provide a ±3.6-percentage 95 percent confidence interval around a sample percentage of 50 percent. If we compare a subgroup that is one-third the population against its complement, we will have 80 percent power to detect an 11-percentage point difference in these subgroup survey item percentages (with better power for larger subgroups and less power for smaller ones). This assumes a two-sided 95-percent test of the null hypothesis of no difference, at a null percentage of 50 percent, with better power for other percentages.

The standard error is √((p*(1-p)*DEFF)/n), where p is the population prevalence, n is the responding sample size, and DEFF is a design effect. The initial sample size is the responding sample size divided by response rate (adjusted for bad addresses). Power calculations assume sample sizes of 1100/3 and 2*1100/3 for the two compared subgroups and independence between the subgroups.

3. Methods to Maximize Response Rates and Deal with Non-response

To help ensure that the participation rate is as high as possible, FDA will:

• Send four different contacts to participants

• Send a small cash incentive with the initial request to respond

• Administer the survey in two modes (web and mail), allowing respondents to answer questions in the mode, time, and location of their choosing

• Design the survey to minimize burden (e.g., short in length, clearly written, and with appealing graphics)

4. Test of Procedures or Methods to be Undertaken

Data collection will begin with a soft launch to test and evaluate procedures. The soft launch will utilize the same procedures and methods as the full data collection but utilize only a small sample. In addition, findings from the qualitative interviews (OMB control number 0910-0847, “Quantitative Claims in Direct-to-Consumer Prescription Drug Advertising: Consumer Interviews”) were used to refine the questions and response options implemented in the survey.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The contractor, Westat, will collect and analyze the data on behalf of FDA as a task order under contract 75F40120A00018. Naomi Yount, Ph.D., 301-610-8842, is Westat’s project director for this project. Data collection and analysis will be overseen by the OPDP Research Team, Office of Medical Policy, Center for Drug Evaluation and Research, FDA, and coordinated by Helen W. Sullivan, Ph.D., MPH, 301-796-4188 and Kevin Betts, Ph.D., 240-485-6252.

3