U.S. Food and Drug Administration
Food and Drug Administration’s Research and Evaluation Survey for the Public Education Campaign on Tobacco among LGBT (RESPECT)
OMB Control No. 0910-0808
SUPPORTING STATEMENT PART A
A. Justification
This information collection supports a Food and Drug Administration (FDA, us or we) survey. The 2009 Family Smoking Prevention and Tobacco Control Act (Tobacco Control Act) (Pub. L. 111–31) amended the Federal Food, Drug, and Cosmetic Act (the FD&C Act) to grant FDA authority to regulate the manufacture, marketing, and distribution of tobacco products to protect public health and to reduce tobacco use by minors. Section 1003(d)(2)(D) of the FD&C Act (21 U.S.C. 393(d)(2)(D)) supports the development and implementation of FDA public education campaigns related to tobacco use. In May 2016, FDA began implementing a public education campaign, This Free Life, to help prevent and reduce tobacco use among LGBT young adults and thereby reduce the public health burden of tobacco. The This Free Life campaign continues to be implemented in 12 U. S. cities and features events, television, radio and print advertisements, digital communications, including videos and social media, and other forms of media. For the purpose of this OMB package, these campaign elements will be referred to as “advertisements” or “ads.”
In support of the provisions of the Tobacco Control Act that require FDA to protect the public health and to reduce tobacco use, FDA requests OMB approval to collect information needed to evaluate FDA’s This Free Life campaign on tobacco use among LGBT. This evaluation is titled the Research and Evaluation Survey for the Public Education Campaign on Tobacco (RESPECT). Comprehensive evaluation of FDA’s public education campaigns is needed to ensure campaign messages are effectively received, understood, and accepted by those for whom they are intended. Evaluation is an essential organizational practice in public health and a systematic way to account for and improve public health actions.
To evaluate the effectiveness of FDA’s This Free Life campaign at reducing tobacco use among LGBT young adults aged 18 to 24, FDA contracted with RTI International (RTI) to conduct the evaluation (RESPECT) using web-based surveys with the target population in the 12 campaign cities and 12 comparison cities. The surveys include measures of tobacco-related knowledge, attitudes, beliefs, intentions, and use as well as measures of audience awareness of and exposure to campaign events and advertisements (Attachment 1). The voluntary surveys also collect information on demographic variables, including sexual orientation, age, sex, race/ethnicity, education, and primary language. Baseline data collection for RESPECT was conducted between February and May 2016, prior to the launch of the This Free Life campaign. Five subsequent waves of data collection were conducted with new (cross-sectional) and returning (longitudinal) respondents. This design facilitated analysis of relationships between individuals’ exposure to campaign activities and baseline to follow-up changes in outcomes of interest between campaign and comparison cities. Information collection for baseline and the first five follow-ups was reviewed and approved by OMB.
FDA will continue to implement the This Free Life campaign in 12 U.S. cities through April 2019. To complete the evaluation of This Free Life, FDA is requesting an extension of the previously approved information collection in order to conduct an additional wave of data collection with the target population. The proposed seventh wave of data collection (i.e., sixth follow-up after baseline) will coincide with the official end of the campaign and serve as an assessment of the campaign at completion. Continued evaluation is necessary to determine the campaign’s impact on outcomes of interest.
As in previous waves, new and returning survey respondents will be invited to complete the online questionnaire (see Attachment 2). New (or cross-sectional) respondents will be recruited at LGBT social venues and via social media (i.e., Facebook). A brief screener used to identify LGBT young adults recruited in person and via social media for the baseline survey is presented in Attachment 3. A similar eligibility screener (Attachment 13) is used at each of the follow-up waves to determine if participants still meet the eligibility criteria to complete the full survey.
The information obtained from the proposed data collection activities is collected from individuals and will be used to inform FDA, policy makers in the United States, prevention practitioners, and researchers about the extent of LGBT young adults’ exposure to the campaign’s activities and the extent to which exposure to these activities is associated with changes in targeted outcomes. While not exhaustive, the list below illustrates a range of purposes and uses for the proposed information collection:
Provide critical data on the reach of the campaign among LGBT young adults in targeted cities, particularly with estimates of the proportion of the population that was exposed to the campaign.
Understand the influence of the campaign on targeted beliefs and attitudes among those evaluated in this study.
Inform FDA, policy makers, and other stakeholders on the impact of the campaign among evaluated cities.
Inform the public about the impact of the campaign in the evaluated cities.
Inform future programs that may be designed for similar purposes.
To achieve these goals, data collection will consist of a baseline survey and six follow-up surveys with young adults as the target audience. The follow-up surveys will be conducted among those young adults who participated previously and with new cross-sectional participants being recruited to make up for attrition. By re-contacting study participants from previous waves, we can reduce the costs of data collection and allow for the possibility of examining changes in study outcomes within individuals over time. However, we anticipate that it will be difficult to retain a sufficiently large proportion of the baseline sample to rely exclusively on a longitudinal design. Young adults are more mobile than older adults (Benetsky et al., 2015) and may move out of the selected study markets or simply be difficult to re-contact. As a result, our goal is to recruit additional LGBT young adults at each wave to ensure the same overall sample size remains constant.
Eligible respondents will be young adults who are 18 to 24 years old and who self-identify as LGBT. The sample will include young adults who self-identify as LGBT, as well as young adults who self-identify as being queer, trisexual, omnisexual, transsexual, gender variant or pansexual (definition from National Health Interview Survey (NHIS) (http://www.cdc.gov/nchs/nhis.htm). FDA’s This Free Life campaign will target up to 15 cities. The RESPECT outcome evaluation data collection will occur in the 12 campaign-targeted cities and 12 similar (“comparison”) cities. The embedded longitudinal cohort will also reduce cost, as well as respondent burden. By re-contacting participants from previous waves of data collection, we will reduce the amount of screening of the population required to reach our target sample compared to collecting an entirely new sample at each wave. The reduced screening reduces overall burden on the population and thus reduces costs.
The outcome study will rely primarily on participants intercepted and invited to complete the screening in LGBT venues to identify eligible young adults, followed by web-based data collection for eligible participants. We will supplement this approach by recruiting young adults through social media. We will advertise in social media and invite young adults aged 18 to 24-years-old to complete the screening survey online. We will then ask eligible young adults to continue on to complete the same web survey completed by participants recruited and screened in person.
The campaign’s target audience consists of young adults who participate in the LGBT community. To determine LGBT status, we will use a series of questions used by the National Health Interview Survey (NHIS) (http://www.cdc.gov/nchs/nhis.htm) to identify individuals as being LGBT. Survey participants will be categorized as LGBT if they self-identify as one or more of the following: lesbian, gay, bisexual, transgender, transsexual, gender variant, queer or pansexual (also referred to as trisexual or omnisexual). Eligible young adults intercepted in person will receive a link by email or text message to complete the web survey. Eligible young adults recruited via social media will continue immediately to the survey from the screener.
This survey will be self-administered online (via the participant’s personal computer or mobile device). The baseline survey had a sample size of 4,057,, with about 60% of the sample (N=2,422) from 12 campaign-targeted cities and about 40% (N=1,635) from comparison cities. The total sample for the follow-up surveys will be approximately 18,665, with roughly the same number of surveys in campaign and comparison cities. We will estimate the proportion of baseline participants expected to complete successive follow-up surveys and supplement that longitudinal sample with new cross-sectional participants to meet our target total sample size. This design permits an analysis of trends in outcomes between young adults in targeted and comparison cities. Compared to a purely cross-sectional design with independent samples, the inclusion of participants from previous waves requires accounting for the over-time correlation in responses from the embedded longitudinal sample. To account for the non-independence of these observations over time, we create unique identifiers for participants and use these to cluster the multiple observations per participant (Wooldridge, 2010; Wears, 2002).
Of the total 22,722 baseline and follow-up surveys, approximately 5,680 (25%) will be completed by young adults recruited through social media.
Information collected in this campaign evaluation will not be generalized to broader or national LGBT populations.
Use of an embedded longitudinal cohort will markedly reduce burden relative to a design consisting solely of cross-sectional surveys. In addition, this outcome study will rely on a partially in-person computer-based screener, social media screener and web surveys for baseline and follow-up data collection. The proposed approach of screening eligible young adults via intercept screeners in LGBT venues and via social media provides a number of methodological advantages, including efficiency in identifying this hard-to-reach population, increased accuracy in measurement of key variables of interest, and reduced burden on study participants. Computerized administration permits the instrument designer to incorporate into the questionnaire routings that might be overly complex or not possible using a paper-based survey. The tablet and web surveys, which will be used to collect intercept screener data and baseline and follow-up surveys, can be programmed to implement complex skip patterns and fill specific wordings based on the respondent’s previous answers. Interviewer and respondent errors caused by faulty implementation of skip instructions are virtually eliminated. Second, computerized and web-based administration increases the consistency of the data. The computerized version of the screener and web-based versions of screener and surveys can be programmed to identify inconsistent responses and attempt to resolve them through respondent prompts. This approach reduces the need for most manual and machine editing, thus saving time and money. In addition, it is likely that respondent-resolved inconsistencies will result in data that are more accurate than when inconsistencies are resolved using editing rules. FDA estimates that 100% of the respondents will use electronic means (either via computerized screeners using a tablet, or web-based screeners and surveys) to fulfill the agency’s request.
Respondents who are screened in person at an LGBT venue will be screened with a self-administered questionnaire programmed on a tablet. All screener data collected in-person will be transmitted via secure encrypted data transmission to RTI’s offices, after which survey item response data will be automatically wiped from all field data collection devices. Respondents will be distinguished in the data only by a unique identifier linking individual screenings and interviews. Identifiers (email address and first name) will be stored, transmitted, and maintained in a data file separate from responses to questions. The computer-assisted self-interview technology for the screener survey permits greater expediency with respect to data processing and analysis (e.g., a number of back-end processing steps, including coding and data entry, will be minimized). Data are transmitted electronically within 48 hours. These efficiencies save time due to the speed of data transmission, as well as receipt in a format suitable for analysis. Finally, this technology permits respondents to complete the interview in privacy. Providing the respondent with a methodology that improves privacy makes reporting of potentially embarrassing or stigmatizing behaviors (e.g., tobacco use, gender identity) less threatening and enhances response validity and response rates.
The in-person computerized sample will be supplemented by a sample of respondents who are recruited through social media. These respondents will be recruited through the social media platform, Facebook, and led to an online screener for the study (see Attachment 3). Respondents will be invited to complete the screener using a web survey programmed and hosted on RTI’s servers. This web survey will have the advantage of immediately notifying respondents if they are eligible for the full study. In addition, use of social media as a recruitment tool will cast a wider net to identify additional, eligible study respondents who are members of this hard-to-reach population.
In an effort to prevent individual participants from completing surveys multiple times to receive additional incentives, we have implemented the following 5-step procedure to identify duplicates and poor-quality surveys for removal:
To prevent any duplicate email addresses that are exact email matches from moving past the screening instrument (an individual email can only enter the baseline survey one time).
After a participant completes the survey, identify exact/almost exact name matches (80%+ name matches for names longer than 5 characters) between email addresses provided during screening (e.g., [email protected], [email protected], [email protected], [email protected]). These email addresses are flagged and reviewed by the project analyst to determine whether the names are similar enough to warrant coding these as suspicious and removed. Surveys that are not deemed as suspicious then go on to Step 3. Duplicate email addresses are then provided to the survey provider as people who should not automatically receive additional incentives beyond what incentives they have already received. If a participant labeled as a duplicate contacts us for their incentive, we will inform them that only one incentive is allowed per respondent.
Check questions with Likert-type scales- if simple straight-lining or other pattern is found then these surveys will be removed. The remaining surveys will go on to Step 4.
Identify low engagement behavior by reviewing the speed of answers, respondents who exceed four or five standard deviations from the mean completion time will be labeled as speeders and their surveys will be removed from the sample.
Identify patterns within the submitted data, including completes from the same IP address in a small window of time and completes from IP addresses known to be sources of malicious software or services.
In addition, to prevent fraudulent cases, we will be implementing the following procedures for the Follow-Up 2 survey:
For social media participants, add a referrer field into the screener. If the referrer field does not contain facebook.com the individual is screened out.
For social media participants add CAPTCHA at the screener. If CAPTCHA is failed at the screener they are screened out.
For social media participants at the screener, if the country code of the IP address is not U.S.-based, they are screened out immediately.
Have returning respondents who were recruited at prior waves (intercept and social media in all cohorts) answer questions in the survey that are asked in the screener. If those answers don’t match they are screened out.
Checks for response straight lining, speeding, and passing attention checks will be conducted. Respondents who do not pass these measures will not be invited to the next wave of the study. We have revised the consent form to inform respondents that they may not be invited back to the study if we determine that their data quality is not sufficient.
Eligible respondents will be routed to the full web survey and given a unique ID to use to enter the survey. Respondents will be able to quit the survey at any time and resume where they left off upon reentry. Respondents will also be emailed a link to resume the survey if they do not complete the survey in one sitting, contact information to ask questions, receive reminders to complete the survey, and receive a virtual gift card upon completion.
Administration of the survey using web methods will help to contain costs, allowing for a sample that is geographically diverse without driving up interviewer costs for travel during data collection.
To date, there has been no in-depth evaluation of FDA’s This Free Life campaign in a real-world setting, and there are no existing data sources that contain measures on awareness of and exposure to the campaign. This proposed information collection therefore does not duplicate previous efforts. In designing the proposed data collection activities, we have taken several steps to ensure that this effort does not duplicate ongoing efforts and that no existing data sets would address the proposed study questions. We have carefully reviewed existing data sets to determine whether any of them are sufficiently similar or could be modified to address FDA’s need for information on the effectiveness of the campaign with respect to reducing LGBT young adult tobacco-related outcomes. We investigated the possibility of using existing data to examine our research questions, such as data collected as part of ongoing national surveillance systems, evaluations of current or past surveys including LGBT young adults, including the National Adult Tobacco Survey (NATS) and the National Health Interview Survey (NHIS). Due to the timing of the campaign, the limited geographic reach of the campaign, and specificity of the target population, none of these existing data sources will be able to provide the necessary data collection needs of the campaign, none will include the necessary in-depth survey questions on awareness of individual ads and other campaign materials, and none contain all of the necessary outcome variables specific to the campaign’s messages.
Respondents in this study aremembers of the general public and specific subpopulations, not business entities. No impact on small businesses or other small entities is anticipated.
Respondents to this collection of information are invited to answer up to seven surveys. While there are no legal obstacles to reduce burden, any lack of information needed to evaluate FDA’s This Free Life campaign may impede the federal government’s efforts to improve public health. Without the information collection requested for this evaluation study, it would be difficult to determine the value or impact of the campaign on the lives of the people they are intended to serve—LGBT young adults. Failure to collect these data could reduce effective use of FDA’s program resources to benefit young adults in the United States. Careful consideration has been given to how frequently the campaign’s intended audience should be surveyed for evaluation purposes. We believe that the proposed outcome study design will provide sufficient data to evaluate the campaign effectively.
There are no special circumstances for this collection of information that require the data collection to be conducted in a manner inconsistent with 5 CRF 1320.5(d)(2). The message testing activities fully comply with the guidelines in 5 CFR 1320.5.
In accordance with 5 CFR 1320.8(d), FDA published a 60-day notice for public comment in the Federal Register on August 2, 2018 (83 FR 37817). The FDA received a total of nine comments from the public, of which five were PRA related.
Comment: Two commenters indicated support for the FDA’s efforts to evaluate media campaigns targeting smoking within the LGBT community.
Response: The FDA appreciates the public’s support of its efforts to meet its mission to promote and protect public health.
Comment: One commenter questioned the need for further data collection on this topic.
Response: The FDA disagrees. This collection of information is necessary for the FDA to meet its mission to promote and protect public, and in its implementation of the Tobacco Control Act.
Comment: One commenter questioned whether the evaluation is collecting sufficient data on the campaign’s impact on the target population’s thinking about smoking.
Response: The campaign is intended to influence the target population’s attitude towards smoking. To evaluate the effectiveness of the campaign, the FDA is asking questions about their tobacco use-related knowledge, attitudes, beliefs, and intentions before and after seeing the campaign’s ads to test whether those have changed over time as a result of exposure to the campaign.
Comment: One commenter questioned the utility of collecting data on smoking among LGBT young adults without first gathering information on smoking rates in this population, and also suggested specific modes for participant recruitment.
Response: Multiple peer-reviewed studies have found that LGBT populations of all age groups are significantly more likely to smoke cigarettes and use other tobacco products compared with non-LGBT populations. The FDA appreciates the detailed review of the evaluation’s recruitment approach. Consistent with the commenter’s recommendation, this information collection recruits participants both online via social media platforms and in person at LGBT social venues. This information collection does not recruit on the street or advertise via television.
Comment: Several comments raised questions about the appropriateness of the target population and implementation approach of the public education campaigns being conducted by the FDA.
Response: The FDA notes that these comments address the content, focus, or implementation of an existing public education campaign, and are therefore outside the scope of this information collection, which is being conducted to evaluate the campaign.
The following individuals inside the agency have been consulted on the design of the campaign evaluation plan, audience questionnaire development, or intra-agency coordination of information collection efforts:
Gem Benoza
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-0088
E-mail: [email protected]
David Portnoy
Office of Science
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 301-796-9298
E-mail: [email protected]
Leah Hoffman
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-7134
E-mail: [email protected]
Janine Delahanty
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-9705
E-mail: [email protected]
Matthew Walker
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-3824
E-mail: [email protected]
Ollie Ganz
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-5389
E-mail: [email protected]
The following individuals outside of the agency have been consulted on questionnaire development. Additionally, input has been solicited and received from FDA on the design of this study, including participation by FDA in meetings with OMB:
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6852
E-mail: [email protected]
Erik Crankshaw
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3809
E-mail: [email protected]
Jennifer Duke
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-2269
E-mail: [email protected]
Jamie Guillory
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3725
E-mail: [email protected]
Kristine Fahrney-Wiant
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5531
E-mail: [email protected]
Jane Allen
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-597-5115
E-mail: [email protected]
Youn Lee
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5536
E-mail: [email protected]
Amy Henes
RTI International
701 13th Street, NW
Washington, DC, 20005
Phone: 202-974-7821
E-mail: [email protected]
Patricia LeBaron
RTI International
230 W Monroe Avenue, Suite 2100
Chicago, IL 60606
Phone: 312-777-5204
E-mail: [email protected]
Pamela Rao
Akira Technologies, Inc.
1747 Pennsylvania Ave NW Suite 600
Washington, DC 20002
Phone: (202) 517-7187
Email: [email protected]
Xiaoquan Zhao
Department of Communication
George Mason University
Robinson Hall A, Room 307B
4400 University Drive, 3D6
Fairfax, VA 22030
Phone: 703-993-4008
E-mail: [email protected]
Joseph Lee
East Carolina University
Belk Building
Greenville, NC
Phone: 919-966-2807
E-mail: [email protected]
Jeff Jordan
Rescue Social Change Group
2437 Morena Blvd.
San Diego, CA 92110
Phone: 619-231-7555 x 150
Email: [email protected]
Mayo Djakaria
Rescue Social Change Group
660 Pennsylvania Avenue SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 120
Email: [email protected]
Dana Wagner
Rescue Social Change Group
660 Pennsylvania Ave SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 331
Email: [email protected]
Brandon Tate
Rescue Social Change Group
6463 Sunset Boulevard, Suite 504
Hollywood, CA 90028
Phone: 619-231-7555 x 152
Email: [email protected]
Participants who complete the in-person screener plus all seven waves of the study during the early response period could receive a combination of cash or virtual gift cards valued at up to $185. Due to difficulty in recruiting intercept respondents, intercept respondents will be paid $10 in cash for completing the screener. Those who are eligible will receive an email invitation to complete the full survey. If they choose to complete the full survey they will receive an online gift card of $20.00, with a $5 bonus (total of $25.00) if they complete the full survey within two days of invitation. Incentives are particularly important in intercept surveys. By definition, intercept respondents are busy doing something else at the time they are intercepted. In the case of this study, when intercepted, respondents will be entering a venue, leaving the venue to go elsewhere, or are likely to be otherwise engaged in interactions with others inside the venue. Unlike a mail or web survey that can be done at the respondent’s leisure, or a telephone or in-person interview that can be scheduled at the respondent’s convenience, in an intercept study the invitation to participate is a relatively immediate one, and respondents are likely to require motivation to stop what they are doing. While there is little published experimental research that examines the effectiveness of incentives vs. no incentives with intercept surveys, there are numerous examples of public health research that has used cash incentives when intercepting respondents at bars or other “party” venues. Incentives within this literature tend to range from $5-$10 for completing a brief survey when entering the venue (e.g., Bourdeau et al. 2015, Guillory et al. 2015, Miller et al. 2003, and Voas et al. 2013) and were typically $20 when both survey data and biological measures were collected upon leaving the venue data (e.g., Bourdeau et al. 2015, Miller et al. 2003, and Voas et al. 2013). A meta-analysis of incentive use during intercept studies in the transportation field suggests that incentives that are paid at the time of completion have a larger impact on response rates than promised incentives (Schaller, 2005).
Respondents who are recruited through Facebook will receive a link to a virtual gift card via email, such as from Visa or Amazon, with a value of $20 upon completion of the survey. Respondents recruited via social media will not receive separate compensation for completing screener instruments. The incentive procedures and amounts for new cross-sectional sample participants at follow-up waves will be identical to the baseline survey. For the longitudinal sample, participants will receive $20 for completing the follow-up survey and an additional $5 if they complete the survey within 48 hours of the invitation to participate.
A more detailed justification for the use of incentives is provided in Attachment 4. The use of modest incentives is expected to enhance survey response rates without biasing responses. A smaller incentive would not appear sufficiently attractive to participants. We also believe that the incentives will result in higher data validity as participants will become more engaged in the survey process. This will also enhance overall response to the baseline and follow-up surveys and reduce attrition at follow-up within the embedded longitudinal cohort. The use of incentives will help ensure that baseline data collection is completed in a timely manner and potentially reduce the amount of additional intercept recruitment time in LGBT venues. Use of incentives within the embedded longitudinal cohort will reduce attrition, which in turn will reduce respondent burden and the cost of follow-up surveys. The specific amount of the proposed incentive is similar to the incentives used in several federally-funded projects, including the National Health and Nutrition Examination Survey (NHANES) (incentives range from $20 to $125 depending on the survey and physical exam components in which respondents agree to participate), and the National Survey on Drug Use and Health (NSDUH) ($30 for 60-minute interview). RTI has also used similar incentives for previous FDA campaign evaluations, including the Evaluation of the Public Education Campaign on Teen Tobacco (ExPECTT), which employs a $20 promised incentive for 30 and 45 minute in-person or web surveys, and the Evaluation of the Fresh Empire Campaign on Tobacco (EFECT), which employs a $25 promised incentive for 30 and 45 minute in-person or web surveys. Additional research studies have used similar incentives to effectively recruit members of the LGBT community, noting that members of the LGBT community are a particularly difficult-to-reach population and that incentives for participating in research are an important component of recruiting LGBT research participants (Meyer & Wilson, 2009). Jones et al. (2008) provided 18 to 30 year old black men who have sex with men $20 gift card incentives for participating in cross-sectional surveys. Silvestre et al. (2006) paid minority men who have sex with men $25 for an HIV epidemiological study. Remafedi and Carol (2005) offered LGBT youth $20 incentives and smoking cessation referrals for participating in interviews for designing tobacco prevention campaigns.
In developing this study, CTP consulted the FDA Privacy Officer to identify potential risks to the privacy of participants and other individuals whose information may be handled by or on behalf of FDA in the performance of this study. Prior to consulting the Privacy Officer, CTP had intentionally designed the study to minimize privacy risks in keeping with the Fair Information Practice Principles (FIPPs) and applying controls selected from the National Institute of Standards and Technology (NIST), Special Publication 800-53, Security and Privacy Controls for Federal Information Systems and Organizations. CTP has also identified privacy compliance requirements and coordinated with FDA’s Privacy Officer to ensure responsible offices in CTP satisfy all requirements. The FDA Privacy Office is currently reviewing the Privacy Impact Assessment.
PII Collection
As part of this study, RTI International, the contractor acting on behalf of FDA, is collecting and maintaining personally identifiable information (PII) about participants who complete the screeners and online baseline and follow-up questionnaires. Potential respondents who consent to complete the new participant screener (intercept and social media) are asked to provide their zip code, month and year of birth date, age, and gender identity to determine eligibility. Eligible respondents are asked to provide their first name, email address, and cell phone number so a survey link can be sent to them. Returning respondents who consent to completing the longitudinal screener are asked to provide their zip code, month and year of birth, age, and gender identity to confirm eligibility. IP address is also collected for all participants completing the online screener. Respondents will be randomly assigned a unique identification number during the screening process to allow response de-identification during analysis and response tracking,
Eligible respondents will be provided a unique survey link in a personal email or text message. Eligible respondents will be routed to the Web survey and enter the survey using their email address. PII or potential PII about respondents collected in both the baseline and follow-up questionnaires includes month and year of birth, race/ethnicity, employment status, education attainment, and annual household income.
Privacy Act Applicability
The information collection is not subject to the Privacy Act of 1974. Hence, no Privacy Act Statement is required to be displayed on the form, website, mobile application or other point at which individuals submit their information.
Data Minimization
The PII collected or used for this study is limited to the minimum necessary to achieve the authorized purpose and produce a valid study. The purpose of the study is to evaluate the This Free Life campaign to help prevent and reduce tobacco use among LGBT young adults and thereby reduce the public health burden of tobacco. The PII is necessary to determine respondent eligibility, invite respondents to participate in future waves of the study, and distribute incentives.
Likewise, any potentially sensitive information gathered from respondents in association with their PII is limited to that which is essential for the study, such as tobacco use and experiences as a member of the LGBT community. Items such as media use are collected because they are established risk factors for tobacco use in LGBT young adults.
FDA has minimized the risk of unnecessary access, disclosure, use or proliferation of PII about respondents. FDA and other parties involved in the study maintain study records containing PII only as long as required (for 3 years after final payment of the contract in accordance with FAR Subpart 4.7). RTI will use a case identification number to identify participants. Access to PII is restricted by role to personnel who must access this information. Sensitive records are kept in a secure location until destruction occurs. RTI has in place standard operating procedures based on RTI Policy to ensure the security and confidentiality of recorded information during all phases of the destruction process, including pickup and transport of records from RTI’s locations to the destruction site. Non-identifiable or de-identified data (i.e., responses to the study, but without any PII) will be sent by the contractor to FDA. No PII will be sent to or be accessible by FDA at any time. Field data collectors and field supervisors sign a detailed data collection agreement at the time they are hired onto the project. This data collection agreement, amongst other things, states that they agree to treat as confidential all information obtained during the interviews or obtained during the course of completing their project-related activities.
Participants who complete the online survey provide their email address, so they can receive a virtual gift card incentive. RTI study staff upload the incentive request file to the incentive provider’s (Creative Group Inc.) encrypted FTP site. The file contains the participants’ email addresses, first names, and case IDs. RTI does not share this information with CTP. Creative Group Inc. does not have access to any other PII or non-PII from the study.
RTI shares the case IDs, first names, and email addresses of new and longitudinal participants with the survey vendor, Qualtrics, so respondents can receive the survey link. The information is uploaded to Qualtrics using SSL encrypted transmissions from RTI servers and is stored on AES-256 encrypted servers at Qualtrics. RTI does not share this information with CTP. RTI will not share PII gathered via this collection with any other individuals or entities.
Notice and Transparency
All subjects are provided notice regarding the collection and use of the information they submit. The purpose of the study and the intended use of the information collected is described on the first screen of the screeners and the baseline and follow-up surveys as part of the informed consent process. Respondents are also informed that information collected in the screeners determines their eligibility for the study and that they must provide consent to complete the screeners. Survey respondents must first read and accept an informed consent form before they can complete the baseline or follow-up surveys.
Individual Participation and Control
Participation in the evaluation of the This Free Life campaign is entirely voluntary. Participants may choose not to join the study and are free to withdraw at any time from the study, including during the course of responding to a questionnaire, without incurring any negative repercussions. For all consent forms, affirmative consent is obtained by clicking an “accept” button below the electronic consent text.
Third-Party Accountability
RTI is held accountable for complying with privacy and security procedures (including reporting data breaches) by its contract with FDA, which requires that RTI complies with 45 CFR part 46 and with the Contractor’s current Federal-wide Assurance (FWA) on file with the Office for Human Research Protections (OHRP), Department of Health and Human Services. The Contractor agrees to provide certification at least annually that the Institutional Review Board has reviewed and approved the procedures, which involve human subjects in accordance with 45 CFR part 46 and the Assurance of Compliance. RTI also has an established protocol in place for privacy breaches that includes the Project Director notifying RTI’s IRB and CTP, who, in turn, notifies RIHSC. In addition, RTI has an Incident Response and Breach Notification Plan in place that activates first responders when an incident occurs, and, as required by law, a breach notification policy with respect to protected health information. RTI subcontractors are accountable via contract terms for all data that it handles, uses, shares and maintains as part of this survey.
Data Security
RTI International’s data security procedures for the Federal Information Processing Standards (FIPS) Low network, which is the RTI network on which the data from the evaluation will be stored, have been reviewed by a FedRAMP certified Third Party Organization and deemed acceptable. This organization issued an Authorization to Operate (ATO) for the FIPS Low network.
RTI’s Institutional Review Boards (IRB) will review and approve the consent forms (Attachments 5, 6a, 6b,14) for the outcome evaluation survey. The IRB’s primary concern is protecting respondents’ rights, one of which is fully maintaining the privacy of respondent information to the fullest extent of the law.
Concern for privacy and protection of respondents’ rights will play a central part in the implementation of the outcome evaluation study and will receive the utmost emphasis. Eligible respondents are asked to provide their first name, email address, and cell phone number so a survey link can be sent to them. The data file for the screener questions are stored separately from the identifying information and the two are linked through a back-end data processing step. This layout is an additional layer of data security, as the transmission of the data and the storage of the data do not allow for linking respondent answers to PII.
Before completing any screener or survey, new and returning respondents are asked to provide informed consent. Specifically, the survey informed consent forms for new and returning respondents state that respondents may stop participation at any time and their answers to the survey questions will be kept private to the fullest extent allowable by law. In the survey informed consent forms, respondents are asked to complete the survey in a place where no one can look over their shoulders and view their answers to ensure their answers are kept confidential and private. Furthermore, self-administration maximizes privacy by giving control directly to the respondent. This allows the respondent to read the questions directly from the computer screen and then key his or her own responses into the computer via the keyboard.
Individuals are recruited in person at LGBT venues and asked to complete the screener, a self-administered questionnaire programmed on a tablet. Screener data collected from field staff is uploaded to Qualtrics via secure encrypted data transmission at the end of every shift. Data are transmitted electronically within 48 hours. Once the data are securely transmitted from the field to Qualtrics, cases and all associated information are automatically removed from the tablet. These efficiencies save time due to the speed of data transmission, as well as receipt in a format suitable for analysis. Finally, this technology permits individuals to complete the screener themselves in privacy, making the reporting of potentially embarrassing or stigmatizing behaviors (e.g., tobacco use, gender identity) less threatening and enhances response validity and response rates.
Screener and contact data collected via social media are collected and housed directly on Qualtrics servers. Both in-person and social media recruited respondents are distinguished in the data only by a unique identifier linking individual screenings and interviews. Identifiers (email address and first name) will be stored, transmitted, and maintained in a data file separate from responses to questions. Data is downloaded from Qualtrics daily for processing and analysis using secured SSL encrypted transmissions. Only authorized RTI staff will have access to this information on a need-to-know basis.
Security for respondents of the Web-based screeners and survey will be assured in a number of ways: (1) each respondent will remain anonymous and will be known only by a unique alphanumeric variable; respondents will be asked to provide their email address to receive survey links and incentives; (2) participants will log onto the secure server hosted by Qualtrics using a unique survey link; (3) respondents will be provided with information about the privacy of their data before they encounter the first survey item; (4) respondents will be required to freely provide their consent to participate before they encounter the first survey item; and (5) respondents will have the option to decline to respond to any item in the survey for any reason. All those who handle or analyze data will be required to adhere to the standard data security policies of RTI.
To ensure data security, all RTI project staff are required to adhere to strict standards. RTI maintains restricted access to all data preparation areas (i.e., receipt and coding). All data files on multi-user systems are under the control of a database manager, with access limited to project staff on a “need-to-know” basis only. No respondent identifiers will be contained in reports to FDA, and results will only be presented in aggregate form.
Implementation of data security systems and processes occur as part of the survey data collection. Data security provisions involve the following:
All data collection activities are conducted in full compliance with FDA regulations to maintain the privacy of data obtained from respondents and to protect the rights and welfare of human research subjects as contained in their regulations. Respondents receive information about privacy protections as part of the informed consent process.
All data collectors are trained on privacy procedures and are prepared to describe them in full detail, if necessary, or to answer any related questions raised by respondents. Training includes procedures for safeguarding sample member information in the field, including securing tablets in the field and while traveling and protecting the identity of sample members.
All field interviewers sign a privacy agreement that emphasizes the importance of respondent privacy and describes their obligations.
All field staff tablets are equipped with encryption software so that only the user or RTI administrators can access any data on the hard drive even if the hard drive is removed and linked to another computer.
Tablets require valid passwords to access any applications.
All data transferred to RTI servers from field staff tablet is encrypted and transferred via a secure (SSL) broadband connection. Similarly, all data entered via the Web-based survey system is encrypted, as the responses will be on a website with an SSL certificate applied. Data are passed through a firewall at RTI and then collected and stored on a protected network share on the RTI Network. Only authorized RTI project staff members have access to the data on the secure network share.
Respondents access the survey with a survey link and complete the survey on a secure server.
All respondents are assured that the information they provide is maintained in a secure manner and will be used only for the purpose of this research. Respondents are assured that their answers will not be shared. Respondents are told that the information obtained from all surveys will be combined into a summary report so that details of individual questionnaires cannot be linked to a specific participant.
Respondents participate on a voluntary basis. The voluntary nature of the information collection is described in the introductory section of the consent process (Attachments 5, 6a, 6b, 14).
The majority of questions asked will not be sensitive in nature. There will be no requests for a respondent’s Social Security Number (SSN). However, it will be necessary to ask some questions that may be considered to be sensitive in nature in order to assess specific health behaviors, such as cigarette smoking and marijuana and alcohol use. While this may be a sensitive question, we feel that it is important to ask respondents about marijuana use because it is a common risk factor that may influence receptivity to the campaign. We have also included questions asking participants about their alcohol use. While these also may be sensitive questions, non-daily smoking while drinking alcohol is a common behavior among young adults and it is important to understand how co-use of alcohol and cigarettes is influenced by the campaign. These questions are essential to the objectives of this information collection. Questions about messages concerning lifestyle (e.g., smoking, current smoking behavior, attempts to quit smoking) and some demographic information, such as race, ethnicity, gender, sexual identity and income, could be considered sensitive, but not highly sensitive. Questions about gender and sexual identity are necessary to determine whether participants identify as LGBT so that we can screen them as eligible participants for the baseline and follow-up surveys. To address any concerns about inadvertent disclosure of sensitive information, respondents will be fully informed of the applicable privacy safeguards. The informed consent (see Attachments 5 and 6a, b) will apprise respondents that these topics will be covered during the survey. This study includes a number of procedures and methodological characteristics that will minimize potential negative reactions to these types of questions, including the following:
Respondents will be informed that they need not answer any question that makes them feel uncomfortable or that they simply do not wish to answer.
Web surveys are entirely self-administered and maximize respondent privacy without the need to verbalize responses.
Participants will be provided with a specific toll-free phone number (linking directly to the RTI IRB Office) to call in case they have a question or concern about the sensitive issue.
Finally, as with all information collected, these data will be presented with all identifiers removed.
As in previous waves, new and returning survey respondents will be invited to complete the online questionnaire. New (or cross-sectional) respondents will be recruited at LGBT social venues and via social media (i.e., Facebook). In-person recruitment will take place in a variety of LGBT venues. The owners or managers of potential recruitment sites will be asked a series of questions to determine the appropriateness of its clientele for participation in the study. For the sixth follow-up, an estimated 30 new venues (10 annualized) will be assessed at 5 minutes per assessment, for an additional 2.5 hours (0.83 annualized). A total of 1,980 venues (660 annualized) will be assessed during the evaluation study, for a total of 165 hours (55 annualized).
Our goal is to recruit 75 percent of the sample via intercept interviews and 25 percent via social media. To obtain the target number of completed sixth follow-up questionnaires, an additional 5,952 adults (1,984 annualized) recruited in person and 1,368 adults (456 annualized) recruited via social media will complete screening questionnaires. For the entire evaluation study, a total of 33,717 adults (11,239 annualized) recruited in person will complete screening questionnaires along with 10,617 adults (3,539 annualized) recruited via social media. The estimated burden to complete the screening questionnaire is 5 minutes (0.083 hour), for a total of 2,799 hours (933 annualized) for in-person recruits and 881 hours (294 annualized) for social media recruits.
Based on analysis of response rates from prior waves of data collection, we expect 65 percent of intercept respondents will be deemed eligible and 50 percent of those will complete the sixth follow-up questionnaire. We expect 30 percent of those recruited via social media will be deemed eligible and complete the sixth follow-up questionnaire. Lastly, we expect 50 percent of returning (or longitudinal) respondents to complete the sixth follow-up questionnaire. We estimate that approximately 1,050 new respondents (350 annualized) and 3,339 returning ( 1,113 annualized) respondents will complete the sixth follow-up questionnaires, for a total of 4,389 responses (1,463 annualized).
OMB previously approved 4,206(1,402 annualized) respondents recruited via social media and 12,795 (4,265 annualized) respondents recruited in person to complete the first five follow-up questionnaires. Adding the sixth follow-ups brings the total estimated number of follow-up questionnaires completed by social media recruits to 5,256 (1,752 annualized) and by in-person recruits to 16,134 (5,378 annualized). At 40 minutes per completed questionnaire, the total burden is 3,507 hours (1,169 annualized) for social media respondents and 10,761 hours (3,587 annualized) for in-person respondents.
OMB also previously approved 393 hours (approximately 132 annualized) for social media respondents and 1,182 hours (394 annualized) for in-person respondents to complete baseline questionnaires. OMB also approved the pilot test of procedures in bars (6 hours [2 annualized]). As these study components are complete, the corresponding burden will not change. Lastly, the original study design included a media tracking component, which included a burden of 414 hours (138 annualized) for completing a 5-minute screening questionnaire and 999 hours (333 annualized) for completing the media tracking questionnaire. However, this component was dropped from the study; hence, the related burden has been deducted from the total study burden.
To accommodate the additional waves of data collection, FDA requests approval to increase the number of burden hours under the existing control number. The previous number of approved responses was 65,706 (21,902 annualized), and the previous burden was 17,574 hours (5,858 annualized). The sixth follow-up adds 11,739 responses (3,913 annualized), which include responses to new venues assessments, screening questionnaires, and the follow-up questionnaires, for a total of 3,540 additional burden hours (1,180 annualized). Removing the media tracking component deducts 6,507 responses (2,169 annualized) and 1,413 burden hours (471 annualized).
Exhibit 1 presents the annualized sample sizes and burden hours for the entire evaluation study. With the addition of the sixth follow-up, the study totals are increasing by 5,232 responses (1,744 annualized) and 2,124 hours (708 annualized) for a new total of 70,938 responses (23,646 annualized) and 19,692 burden hours (approximately 6,566 annualized).
Exhibit 1. Estimated Annual Burden Hoursa
Respondent Type and Activity |
No. of Respondents |
No. of Responses per Respondent |
Total Annual Responses |
Average Burden per Response |
Total Hours |
Venue Owners and Managers |
660 |
1 |
660 |
0.083 (5 minutes) |
55 |
General Population: Pilot test of Procedures in Bars |
27 |
1 |
27 |
0.083 (5 minutes) |
2 |
General population – outcome screener (in person) |
11,239 |
1 |
11,239 |
0.083 (5 minutes) |
933 |
General population – outcome screener (social media) |
3,539 |
1 |
3,539 |
0.083 (5 minutes) |
294 |
LGBT young adults outcome baseline (social media) |
263 |
1 |
263 |
0.500 (30 minutes) |
132 |
LGBT young adults outcome baseline (in person) |
788 |
1 |
788 |
0.500 (30 minutes) |
394 |
LGBT young adults outcome follow-up questionnaire (social media) |
1,752 |
1 |
1,752 |
0.667 (40 minutes) |
1,169 |
LGBT young adults outcome follow-up questionnaire (in person) |
5,378 |
1 |
5,378 |
0.667 (40 minutes) |
3,587 |
Totals |
|
|
|
|
6,566 |
a There are no capital costs or operating and maintenance costs associated with this collection of information.
Respondents participate on a purely voluntary basis and, therefore, are subject to no direct costs other than time to participate. There are also no start-up or maintenance costs. RTI has conducted many smoking-related surveys of similar length among young adults. We examined diagnostic data from each wave of data collection and estimate it will take respondents approximately 40 minutes to complete the follow-up survey. According to the U.S. Department of Labor (DOL) Bureau of Labor Statistics the average hourly wage in 2018 was $26.92. Thus, the estimated total cost to participants will be $175,222.28 (3 years of data collection). The estimated annual value of respondents’ time for participating in the information collection is summarized in Exhibit 2.
Exhibit 2. Estimated Annual Costa
Type of Respondent |
Activity |
Annual Burden Hours |
Hourly Wage Rate |
Total Cost |
Adults 18 and older in the United States
|
Venue Recruitment |
55 |
$26.92 |
$1,480.60 |
Pilot screening |
2 |
$26.92 |
$ 53.84 |
|
Screeners |
1,227 |
$26.92 |
$33,030.84 |
|
Baseline survey |
526 |
$26.92 |
$14,159.92 |
|
Follow-up surveys |
4,756 |
$26.92 |
$128,031.52 |
|
Total |
|
6,566 |
|
$176,756.72 |
aNumbers reflect cost for 1 year of data collection (total data collection period is 3 years).
There are no capital, start-up, operating, or maintenance costs associated with this information collection.
This information collection is funded through a contract with RTI. The total estimated costs attributable to this data collection are $8,039,606 (Exhibit 3). There are additional contract-funded activities occurring before and after this data collection that include project planning and data analysis. Other activities outside this data collection include coordination with FDA and its media contractor, evaluation plan development, instrument development, reporting, RTI IRB, and progress reporting and project management.
Exhibit 3. Itemized Cost to the Federal Government
Government Personnel |
Time Commitment |
Average Annual Salary |
Total |
GS-13 |
25% |
$96,970 |
$24,243 |
GS-14 |
15% |
$114,590 |
$17,189 |
GS-15 |
5% |
$134,789 |
$6,739 |
Total Salary Costs |
$48,171 |
||
Contract Cost |
$7,991,435 |
||
Total |
$8,039,606 |
||
The Food and Drug Administration is submitting this extension request to add a Follow-up 6 to our data collection schedule and refine the data collection documents for Follow-up 6. Follow-Up 6 is scheduled for May 2019 through August 2019. During this follow-up data collection effort, we will continue to re-contact participants and recruit new respondents to offset attrition. The media tracking ICs were removed because they were not needed for the information collection. This led to a decrease in 2,169 responses and 471 hours. The addition of Follow-up 6 increases respondents by 3,913 and burden hours by 1,180.
Data from this information collection will be used to estimate awareness of and exposure to the campaign among LGBT young adults. These estimates will take the form of self-reported ad recognition and recall that assess basic exposure as well as frequency of ad exposure. These estimates will also be calculated separately for each specific campaign advertisement.
Data from this information collection will also be used to examine statistical associations between exposure to the campaign and baseline to follow-up changes in specific outcomes of interest for campaign and comparison groups. We will conduct two primary types of analyses. The first will focus on aggregate changes in outcomes from the baseline to follow-up periods between the campaign and comparison cities. The second analytic approach will focus on individual changes in outcomes as a function of campaign exposure, which will vary within and across campaign and comparison cities. The embedded longitudinal cohort may also permit some longitudinal analysis. The primary outcomes of interest among young adults will be awareness of the campaign as well as tobacco-related beliefs, attitudes, intentions and behaviors. We hypothesize that there should be larger changes in outcomes among individuals with more frequent campaign exposure (i.e., dose-response effects).
In addition to relying on self-reported exposure, we will also utilize measures of market-level campaign intensity, which will be constructed with available data on campaign activities, including traditional and digital advertising and local campaign events. These data will be merged to the survey to provide an additional measure of campaign exposure among study participants. This will allow us to analyze the relationship between the market-level delivery of the campaigns and actual levels of awareness in each sample that is collected. This will also facilitate further analyses of the relationship between exogenous market-level measures of campaign dose and changes in the aforementioned outcome variables of interest.
The reporting and dissemination mechanism will consist of three primary components: (1) summary statistics (in the form of PowerPoint presentations and other briefings) on individual awareness of and reactions to the campaign, (2) a comprehensive evaluation report summarizing findings from this information collection, and (3) at least two peer-reviewed journal articles that document the relationships between campaign exposure and changes in the aforementioned outcomes of interest. The key events and reports to be prepared are listed in Exhibit 4.
Exhibit 4. Project Schedule
Project Activity |
Date |
Baseline Data Collection |
January 2016 – May 2016 |
Wave 2 (Follow up 1) Data Collection |
September 2016 – November 2016 |
Wave 3 (Follow up 2) Data Collection |
April 2017 – June 2017 |
Wave 4 (Follow up 3) Data Collection |
September 2017 – November 2017 |
Wave 5 (Follow up 4) Data Collection |
May 2018 – July 2018 |
Wave 6 (Follow up 5) Data Collection |
October 2018 –January 2019 |
Wave 7 (Follow up 6) Data Collection |
May 2019 – August 2019 |
Preparation of analytic data file |
Approximately 4 weeks after completion of data collection |
Data analysis |
Approximately 5 – 12 weeks after completion of each analytic data file |
Report writing and dissemination |
Approximately 12 – 16 weeks after completion of each analytic data file |
Not applicable. All data collection instruments will display the expiration date for OMB approval of the information collection.
Not applicable. There are no exceptions to the certification statement.
References
Benetsky M.J., Burd C.A., Rapino M.A. (2015). Young Adult Migration: 2007-2009 to 2010-2012. United States Census Bureau, American Community Survey Reports, ACS-31. Washington, D.C.: United States Census Bureau.
Bourdeau, B., Miller, B., Johnson, M., Voas, R. (2015). Method of transportation and drinking among club patrons. Transportation Research Part F, 32, 11-22.
Guillory, J., Johns, M., Farley, S., Ling, P. (2015). Loose Cigarette Purchasing and Nondaily Smoking Among Young Adult Bar Patrons In New York City. American Journal of Public Health, 105(8), e140-e147.
Jones, K.T., Gray, P., Whiteside, Y.O., Wang, T., Bost, D., Dunbar, E., Foust, E., Johnson, W.D. (2008). Evaluation of an HIV prevention intervention adapted for black men who have sex with men. American Journal of Public Health, 98(6), 1043-1050.
Meyer, I.H., Wilson, P.A. (2009). Sampling lesbian, gay, and bisexual populations. Journal of Counseling Psychology, 56(1), 23-31.
Miller, B., Byrnes, H., Branner, A., Johnson, M., Voas, R. (2003). Group Influences on Individuals’ Drinking and Other Drug Use at Clubs. Journal of Study of Alcohol and Drugs, 74(2),280-287.
Remafedi, G., Carol, H. (2005). Preventing tobacco use among Lesbian, Gay, Bisexual, and Transgender Youths. Nicotine & Tobacco Research, 7(2), 249-256.
Schaller, B. (2005). On-Board and Intercept Transit Survey Techniques. Transit Cooperative Research Program (TCRP) No. 63. Washington, DC: Transportation Research Board.
Silvestre, A.J., Hylton, J.B., Johnson, L.M., Houston, C., Witt, M., Jacobson, L., Ostrow, D. (2006). Recruiting minority men who have sex with men for HIV research: Results from a 4-city campaign. American Journal of Public Health, 96(6), 1020-1027.
Voas, R., Johnson, M., Miller, B. (2013). Alcohol and drug use among young adults driving to a drinking location. Drug and Alcohol Dependence, 132, 69-73.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Matthew Farrelly |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |