Supporting Statement Part B Trainee Scholar Survey 2020

United States Food and Drug Administration

Center for Tobacco Products, Food and Drug Administration Funded Trainee/Scholar Survey

OMB Control No. 0910-NEW

B. Statistical Methods (used for collection of information employing statistical methods)

1. Respondent Universe and Sampling Methods

The respondent universe for the evaluation comprises a census of all current and former U.S. Food and Drug Administration (FDA) Center for Tobacco Products (CTP) tobacco regulatory science trainees and scholars for whom current contact information is available. This includes Tobacco Centers of Regulatory Science (TCORS) trainees and trainees on other CTP-funded projects. The Center for Evaluation and Coordination of Training and Research (CECTR) Training Working Group established definitions for a CTP-funded trainee (TCORS and non-TCORS) and former trainee (TCORS and non-TCORS). Table 1 provides a description for trainees included in the Funded Trainee/Scholar Survey.

Table 1. Description of Trainees Included in the Funded Trainee/Scholar Survey

Role	Description
TCORS trainee	Graduate or post-doc trainee funded under the TCORS Training Core Graduate or post-doc trainee who has participated on a TCORS grant Graduate or post-doc trainee funded by TCORS funding but not out of the TCORS Training Core
Non-TCORS CTP-funded trainee	Graduate or post-doc trainee funded under the CTP grant Graduate or post-doc trainee who has participated on the CTP funded grant
Former TCORS trainee	A previous graduate or post-doc trainee who was funded under the TCORS Training Core, and/or participated on a TCORS grant, and/or received TCORS funding but not out of the TCORS Training Core.
Former Non-TCORS CTP-funded trainee	A previous graduate or post-doc trainee who worked on a CTP grant, and/or was funded under a CTP grant, and/or participated on a CTP funded grant.

• Principal Investigator or Training Director trainee contact information request and notification email. We estimate that all trainees will have a unique principal investigator (PI) or training director; therefore, we anticipate emailing 350 PIs and training directors requests for trainee contact information and survey notifications. It is likely that PIs or training directors will have multiple trainees; however, assuming the possibility of one trainee per PI or training director will help to avoid underestimating the burden.

• Lead letter and email invitation. We expect to have a total of 350 trainees across all training programs for whom current contact information is available. One hundred percent of these trainees will receive a lead letter and email invitation to participate in the survey.

• Informed consent and trainee survey. We expect that 85% of trainees will access the survey link, read the informed consent, and complete the survey. We based this assumption on participation rates of the 2017 trainee survey. Participation rates for the survey ranged from 54% to 85% across CTP and TCORS training programs (70.6% overall). Because the 2019 survey will be open 2-weeks longer than the 2017 survey, we assume an 85% participation rate across all training programs. Also, estimating toward the high end of the previous range will help to avoid underestimating the burden. Thus, we expect to obtain 298 responses to the consent and survey.

• Follow-up email. After the survey is launched, weekly reminder emails will be made to all trainees who have not yet started the survey. We estimate that 25% of invited trainees (88 trainees) will receive up to 2 reminder emails and another 25% (88 trainees) will receive up to 4 reminder emails.

2. Procedures for the Collection of Information

Annually, on behalf of FDA, the Center for Coordination of Analytics, Science, Enhancement, and Logistics (CASEL) (formerly CECTR) emails Tobacco Centers of Regulatory Science (TCORS) administrators/coordinators and CTP-funded principal investigators (PIs) (Appendix D and E) requesting trainee names and email addresses. CASEL will compile this information and share the full list with the contractor conducting the data collection, RTI. CASEL will share the list of contact information by sending a password-protected file to RTI via email, and RTI will save the list in a restricted-access electronic file.

RTI will email potential trainee respondents a lead letter (Appendix F) describing the study and letting them know about a forthcoming email invitation from RTI to participate in the survey (Appendix G). Respondents will use the email link to enter the survey portal and read the informed consent (Appendix C). Respondents will be asked to read the consent and click “yes” to provide consent to participate, and then will access the trainee survey (Appendix H). Around the same time trainees receive the email invitation, PIs and training directors will receive an email asking them to encourage trainees to participate in the survey (although these communications will not indicate whether trainees have participated or not) (Appendix I). After the survey is launched, weekly reminder emails will be made to all trainees who have not yet started the survey (Appendix J). RTI will maintain a file linking contact information and survey IDs so that reminders are only sent to those individuals who have not already completed the survey. The online survey will be available for up to 6 weeks from the time of the initial invitation.

The online survey will be programmed using Qualtrics software, and respondents will access the survey via hyperlink provided in the invitation email, for example https://rti2.az1.qualtrics.com/jfe/form/SV_0ULaMzRKZjrhbr7. Survey links will be unique to each respondent and will take them to the last place they stopped in the survey each time they click on the link. Respondent email addresses are stored in Qualtrics in a "contact list" associated with the survey, which allows Qualtrics to send out invitations and track responses. Data is transmitted from encrypted Qualtrics storage using SSL, accessible only after authentication to the Qualtrics server either via direct login or an API authentication. RTI will store the survey dataset on secure servers that are only accessible by project staff trained in human subjects. At the completion of data collection, the databases will be deleted from the RTI’s Qualtrics account and remain only on RTI’s secure shared drive.

Response data will not be recorded in a manner that is linkable to respondent identifiers. RTI will assign a unique identifier code to each survey respondent. Survey response data will be stored and analyzed by identifier code. Only RTI staff conducting the data collection will have a crosswalk that links study IDs to trainee names. After project completion (September 2023) the crosswalk will be destroyed. No survey participant names or other identifying information will be included in any reports or data sets.

All respondents to the trainee surveys are eligible for inclusion in the analysis. The data will provide details about trainee characteristics and the program components. Specifically, analysis will review the trainee demographics, discipline (pre/post), research focus, and alignment with CTP priority research domains and special interest. The review of program components will offer descriptive analysis of trainee participation, reported usefulness, coverage of topic areas, and mentoring. The analysis will also examine the combination of training and program components relative to the TRS knowledge and skills that specifically focus on scholarly activities and preparation for research, policy, communication, and professional development activities. Collectively, the analysis will offer detail into the variations within and across trainee cohorts, identifying potential trends over time and differences year to year. Using Chi-square analysis and ANOVA, we will analyze the frequency of these characteristics to determine if the differences and changes are significant.

These analyses will provide FDA with a better understanding of how training through CTP-funded TRS projects and activities is developing and preparing new researchers for TRS. The contractor will present results to FDA using traditional bar charts and line graphs with narrative text to explain and interpret the findings.

3. Methods to Maximize Response Rates and Deal with Non-response

Participation rates for the 2017 trainee survey ranged from 54% to 85% across CTP and TCORS training programs (70.6% overall). We are estimating an 85% response rate across training programs for the 2019 survey. To maximize response rates and ensure the project reaches at least an 80% response, we intend to use the following strategies:

• Emailed lead letter from RTI. All potential respondents will receive a lead letter describing the study, informing them of the forthcoming email invitation, and encouraging them to participate.

• Email reminders for trainees. After the survey is launched, weekly reminder emails will be made to all trainees who have not yet started the survey. In the 2017 trainee survey, only periodic email reminders were sent to trainees who had not yet started the survey. Increasing the frequency of reminders will likely increase survey response.

• Notifications to PIs and training directors. PIs and TCORS training directors will receive an email asking them to encourage trainees to participate in the survey.

• Short survey. The low burden of a short, 10-minute survey is intended to increase response rate. The web-survey will also include a progress bar at the top of each screen so respondents can see their progress. Additionally, respondent progress is saved and each time a respondent clicks their unique survey link it will take them to the last place they stopped in the survey (to complete the survey if they did not do so previously).

• Longer field time. The 2017 trainee survey site was open 4 weeks. The 2019 online survey will be available for up to 6 weeks from the time of the initial invitation. Two additional weeks for survey completion will likely increase response rate.

Methods to Deal With Nonresponse

Nonresponse falls into two primary categories. Survey nonresponse occurs when a sampled person does not complete any of the measurement instrument and is essentially not part of the study. Reasons for survey nonresponse include refusal or lack of comprehension/ability to complete the instrument. Item nonresponse, typically referred to as missing data, occurs when individual items on a questionnaire are not completed. Analytical methods for addressing each type are briefly discussed below.

Sampling weights are the most common method to manage survey nonresponse, adjusting for the differences between a sample and the population from which it was drawn. Sampling weights are assigned to each sample member and consist of an initial sampling weight and a factor to adjust for nonresponse. Nonresponse adjustments are applied to the initial sampling weight to compensate for the potential biasing effects of differential nonresponse. A variety of weighting strategies exist, but a model-based weight adjustment is commonly used. This method attempts to reduce bias due to discrepancies between the sampled group and the larger population using a set of covariates to generate weights. Propensity scores are one model-based approach that may be used for survey or unit nonresponse.

Missing data will likely be the result of respondents choosing not to respond to certain items. The impact of missing data will be minimized through the use of maximum likelihood-based estimators that are available in many statistical software packages such as SAS, R, and Mplus. Although likelihood-based methods generally produce results equivalent to those from multiple imputation (another appropriate method for addressing missing data), they have the advantage of not requiring generation of multiple datasets and can directly estimate the models of interest while accounting for missing information. These methods yield unbiased estimates and accurate standard errors without sacrificing cases, and thus maximize statistical power, when missing data are truly random or predicted by other variables in a given model but independent of the potential values of the outcome itself (i.e., missing at random [MAR]).

4. Test of Procedures or Methods to be Undertaken

The 2017 trainee survey was pilot tested with a small sample of TCORS trainees prior to survey administration. The current version of the survey included in this information collection has not been cognitively tested or pretested; however, the survey has only been slightly modified from the 2017 version by FDA CTP in collaboration with RTI.

Furthermore, a similar version of the survey was used in 2015, 2016, 2017, and 2018 in evaluations of the Tobacco Regulatory Science Program.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The agency official responsible for receiving and approving contract deliverables is Tarsha McCrae, Contracting Officer Representative, Center for Tobacco Products, Food and Drug Administration. She has overall responsibility for overseeing the design and administration of the project and reporting of the survey information.

RTI International will conduct the survey data collection under contract with FDA.

Individuals responsible for data collection and analysis are listed in Table 2.

Table 2. Individuals Responsible for Design, Data Collection, and Analysis

Name	Title	Telephone Number
Karen Crotty	Project Director, RTI	919-541-6536
Carol Schmitt	Subject Matter Expert, RTI	970-498-1819
Lauren Grattan	Data Collection, Analysis, and Reporting Lead, RTI	919-541-8893
Ghada Homsi	Statistical Analyst, RTI	919-316-3329
Jessica Pikowski	Survey Programmer, RTI	919-541-6657
Mary Oxendine	Data Collection, Analysis, and Reporting team member, RTI	919-316-3123

4