Supporting Statement Part B

United States Food and Drug Administration

Center for Tobacco Products

Survey of Risk Factors of Lithium-Ion Batteries used in ENDS

OMB Control No. 0910-0810

GEN IC SUPPORTING STATEMENT PART B

Part B. Statistical Methods

1. Respondent Universe and Sampling Methods

The respondent universe for this study is current users of electronic nicotine delivery systems (ENDS) users aged 21 or older in the U.S. The final sample for the online survey will include 6,100 adults who are current ENDS users recruited from the online survey vendor Dynata. We define current ENDS use as using on at least 1 of the past 30 days. We will limit the sample to adults ages 21 and older to ensure that participants are of legal age to purchase ENDS products. The sample drawn here is designed primarily to provide information on risk factors for incidents of overheating, fire, or explosion (O/F/E) of the lithium-ion batteries in ENDS, perceptions of O/F/E incidents, and the prevalence of O/F/E incidents among ENDS users.

The panel provider, Dynata, has a proven and demonstrated ability to orchestrate and support the sampling plan specifications of this study. They have had experience recruiting similar populations for other surveys on tobacco use.

Sampling Methods

The sample provided will be of a sample of ENDS users aged 21 or older. Participants will be recruited via a survey panel composed of individuals in the United States who have been prescreened for their willingness to participate in online surveys.

We wish to collect data from a sufficient number of current ENDS users to obtain at least 12 cases of serious incidents of O/F/E (see Sample Size section for additional information). Given the absence of nationally representative data, the true prevalence of O/F/E experiences among current ENDS users is unknown and can only be estimated based on existing data about injury reports, we can adjust this goal as needed or implement quotas designed to increase the likelihood of recruiting participants who have experienced O/F/E. We will make a determination partway through data collection as to whether and how to implement quotas based on revised screening criteria. While we anticipate that implementing more restrictive quotas would increase response from the harder-to-reach individuals with O/F/E experience, this may result in a lower overall sample size as it will likely require screening out individuals who have not experienced O/F/E but would otherwise qualify for participation.

The convenience sample provided by the Dynata online survey panel is considered suitable for the purposes of this study. The sample will have diversity with respect to age, gender, education, and race/ethnicity. Because the study sample is drawn from a non-probability-based panel, the study sample will not be nationally representative of the U.S. ENDS user population. However, the data will be weighted to align the sample with demographic benchmarks from the National Health Interview Survey (NHIS). After we collect data from approximately 3,050 eligible respondents, we will examine whether we have sufficient representation of O/F/E incidents (i.e., approximately 6 serious O/F/E cases). If we do not meet these anticipated thresholds, we will use a modeling procedure to predict the probability that an eligible respondent in the sample has experienced O/F/E based on variables collected during screening. We will use the predicted probabilities of O/F/E to create strata, and then we will create quotas based on the predicted probability of O/F/E. Those quotas will be used for the recruitment of the remaining 3,050 respondents. Data will be weighted to account for these sample quotas and will be further calibrated to population benchmarks from the NHIS as described above.

Sample Size

Researchers have estimated 1,022 emergency room visits from ENDS burn injuries in 2016 (Consumer Product Safety Commission, 2022). The sales rate of ENDS was estimated to be 1500 units per 100,000 per month in 2016, equating to roughly 570,000 total ENDS sales in 2016 in the US (Wang et al., 2018). Based on this estimation, at least 1 in 570 ENDS products (0.2%) is expected to cause a field failure (i.e., overheating, fire, or explosion incidents). At least 12 serious incidents are needed in order to conduct a qualitative analysis of the incidents (Hennink & Kaiser, 2022). Given the estimated failure rate of ENDS and the fact that we anticipate that current ENDS users will have purchased at least 1 ENDS product in the past year, the goal sample of 6,100 ENDS users is needed for the survey in order to include at least 12 ENDS users who have experienced serious O/F/E.

To obtain a final sample of approximately 6,100 ENDS users 21 years old or older, we will need to screen approximately 61,000 potential participants. In prior national surveys conducted by RTI with the same Dynata panel, approximately 10% of adult panel members were current ENDS users; Dynata anticipates the same 10% incidence in this study. In addition, we request an additional 1,000 to account for potential overages, resulting in a total of 62,000 individuals screened.

We note that is not possible to calculate an exact participation rate or response rate because there are no study-specific invitations sent to panelists, nor is there a sample frame (because this is a non-probability panel-based sample). Thus, there is no denominator from which to calculate a participation rate or response rate.

2. Procedures for the Collection of Information

Dynata regularly emails panel members to notify them that new surveys are available in their panelist portal and providing a unique link to access their portal. Once this study has launched, it will be available to adults as one of the potential opportunities listed in their portal. There is no specific invitation for the study.

When potential participants select the study from their panel portal, they will be directed to the screener. The screener asks for their willingness to answer a few questions to determine eligibility. Potential respondents who agree complete the screener. If eligible, they are routed to the online consent form. If they consent, they complete the online survey. Participation in the study is voluntary, and no effort will be made to convert refusals.

Because the study sample is drawn from a non-probability-based panel, the study sample will not be nationally representative of the U.S. ENDS user population. However, the data will be weighted to align the sample with demographic benchmarks from the National Health Interview Survey (NHIS), a population-representative survey that includes demographic and tobacco/ENDS use variables. To weight the sample, we will use a calibration model which forces the sum of the weights to equal population totals estimated from the NHIS, which can reduce bias due to selection error, coverage error, and nonresponse error. We can use NHIS to estimate the population totals of ENDS users for the various distributions used in the calibration model. We will conduct analyses to determine the best combination of calibration distributions, but we anticipate the final model will include variables for gender, age category, race/ethnicity, educational attainment, and current cigarette smoking status. We will use SUDAAN Proc WTADJUST to implement the calibration, duplicating the calibration methodology often referred to as proportional iterative fitting.

Unusual Problems Requiring Specialized Sampling Procedures

We seek to include at least 12 participants who have experienced serious O/F/E incidents. Given that the true prevalence of O/F/E experiences among current ENDS users is unknown, we can adjust this goal as needed. If we are having difficulty recruiting ENDS users who have experienced O/F/E, we have the option to institute quotas partway through data collection that would enhance the likelihood that the ENDS users we recruit have experienced O/F/E. Specifically, after the first 3,000 responses have been collected, we will pause data collection and examine the number of respondents reporting having experienced O/F/E. If, after reviewing the first 3,000 cases, we have not recruited a sufficient number of individuals with O/F/E experience and anticipate we will fall short of our O/F/E sample targets, we may implement quotas based on revised screening criteria to maximize response from those who have experienced O/F/E. To inform the screening criteria adjustment, we will conduct regression analyses to identify factors that are most strongly associated with having experienced O/F/E, such as ENDS device type, history of ENDS use, or sociodemographic characteristics.

3. Methods to Maximize Response Rates and Deal with Non-response

Several features of this study have been designed to maximize participant response rate and questionnaire completion.

• Reminders: Dynata regularly sends panel members notifications that new surveys are available in their portals. These can prompt people to log in to their profile, thus reminding them of the availability of this survey.

• Mobile Phone Responsiveness: All of the materials will be optimized for performance on a mobile phone, in addition to other electronic devices such as tablets and laptops. Ensuring that the surveys are optimized for mobile phone performance will reduce non-response and drop-off due to technical issues related to compatibility of the instruments with the mobile phone format.

• Token of Appreciation: In this study, we will use a token of appreciation in the form of nonmonetary points that are accrued and redeemable with the panel company for cash or other rewards. Dynata panelists will receive nonmonetary points valued at approximately $3.00-$4.00 for completion of the survey. Numerous empirical studies have also shown that a token of appreciation can significantly increase response rates in cross-sectional studies and reduce attrition in longitudinal studies. (e.g., Abreu & Winters, 1999; Castiglioni, Pforr, & Krieger, 2008; Shettle & Mooney, 1999; Singer, 2002).

4. Test of Procedures or Methods to be Undertaken

Prior to this study, the evaluation contractor (RTI International) will conduct cognitive interviews with nine ENDS users to assess comprehensibility and ease of use of the survey instrument. The results of the cognitive interviews will help refine the final survey to minimize burden and improve quality.

In addition, RTI will conduct rigorous internal testing of the online survey instrument prior to its fielding. RTI will review the online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB.

RTI will also conduct a “soft launch” of the online survey. After collecting responses from 100 participants, we will pause data collection to examine whether eligibility criteria, skip patterns, and quotas are operating as expected. If there are no issues detected, we will include these 100 participants in the final sample. After any issues are resolved, we will re-open the survey.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals inside the agency have been consulted on questionnaire development and data collection implementation.

Azieb Kidanu

Office of Science

Center for Tobacco Products

Food and Drug Administration

11785 Beltsville Drive

Beltsville, MD 20705

Phone: (301) 796-1248

E-mail: [email protected]

Chelsea McKinney

Office of Science

Center for Tobacco Products

Food and Drug Administration

11785 Beltsville Drive

Beltsville, MD 20705

E-mail: [email protected]

Alexander Persoskie

Office of Science

Center for Tobacco Products

Food and Drug Administration

11785 Beltsville Drive

Beltsville, MD 20705

Phone: (240) 402-6864

E-mail: [email protected]

The following individuals outside of the agency have been consulted on survey development and data collection implementation.

Erik Crankshaw

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-316-3809

E-mail: [email protected]

Matthew Eggers

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: 919-990-8380

E-mail: [email protected]

Laura Baum

RTI International

2987 Clairmont Road NE

Atlanta, GA 30329-4434

Phone: 770-234-5017

E-mail: [email protected]

References

Abreu, D. A., & Winters, F. (1999). Using monetary incentives to reduce attrition in the survey of income and program participation. Proceedings of the Survey Research Methods Section (pp. 533-538).

Castiglioni, L., Pforr, K., & Krieger, U. (2008, December). The effect of incentives on response rates and panel attrition: Results of a controlled experiment. Survey Research Methods (Vol. 2, No. 3, pp. 151-158).

Consumer Product Safety Commission. National Electronic Injury Surveillance System (NEISS) 2002-2021 on NEISS Online Database, released April, 2022. Generated at https://www.cpsc.gov/cgibin/NEISSQuery/home.aspx

Hennink M, Kaiser BN. Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Soc Sci Med. 2022;292:114523. doi:10.1016/j.socscimed.2021.114523

Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. Survey Nonresponse, 51, 163-177.

Shettle, C., & Mooney, G. (1999). Monetary incentives in US government surveys. Journal of Official Statistics, 15(2), 231.

Wang, Teresa et al., National and State-specific unit sales and prices for electronic cigarettes, United States, 2012-2016, Prev Chronic Dis 2018;15:170555.

6