Supporting Statement A

Development of a Mobile Messaging Intervention for Men who have Sex with Men:

Formative Study

GenIC Information Collection Request under OMB #0920-0840

Section A: Supporting Statement

May 11, 2016

CONTACT

Gordon Mansergh, PhD
Project Officer

Centers for Disease Control and Prevention
Division of HIV/AIDS Prevention
1600 Clifton Road, NE, Mailstop E-37
Atlanta, GA 30333
Phone: 404-639-6135
Fax: 404-639-1950
E-mail: [email protected]

TABLE OF CONTENTS

1. Circumstances Making the Collection of Information Necessary 4

2. Purpose and Use of the Information Collection 5

3. Use of Improved Information Technology and Burden Reduction 6

4. Efforts to Identify Duplication and Use of Similar Information 6

5. Impact on Small Businesses or Other Small Entities 6

6. Consequences of Collecting the Information Less Frequently 6

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 6

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 6

9. Explanation of Any Payment or Gift to Respondents 7

10. Protection of the Privacy and Confidentiality of Information Provided by Respondents 8

11. Institutional Review Board (IRB) and Justification for Sensitive Questions 8

12. Estimates of Annualized Burden Hours and Costs 8

12A. Estimated Annualized Burden Hours 8

12B. Estimated Annualized Burden Costs 10

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers 10

14. Annualized Cost to the Federal Government 10

15. Explanation for Program Changes or Adjustments 11

16. Plans for Tabulation and Publication and Project Time Schedule 11

17. Reason(s) Display of OMB Expiration Date is Inappropriate 12

18. Exceptions to Certification for Paperwork Reduction Act Submissions 12

References: 13

EXHIBITS

Exhibit 12.1: Participant Recruitment and Screening Flow 9

Exhibit 12.2: Estimated Annualized Burden Hours 9

Exhibit 12.3: Estimated Annualized Burden Costs 10

Exhibit 14.1: Annualized Cost to the Government 11

Exhibit 16.1: Project Time Schedule 11

LIST OF ATTACHMENTS

Attachment 1 Recruitment Materials

Attachment 2 Data Collection Instruments

2a. Participant Screening Form

2b. Contact Information Form

2c. Focus Group Guide

2d. In-Depth Interview Guide

2e. Online Screening Form

2f. Online Contact Information Form

Attachment 3 Consent Forms

3a. Focus Group Consent Form

3b. In-Depth Interview Consent Form

Attachment 4 Messages for Testing

Attachment 5 Human Subjects Approvals

5a. Emory IRB Letter of Approval

5b. Public Health Solutions IAA

5c. University of Michigan IAA

• Goals of the study: The purpose of this formative research study is to develop and assess HIV prevention messages tailored for men who have sex with men (MSM) in the United States, including informational and motivational messages about recent advances in biomedical HIV prevention.

• Intended use: Data collected through this study will be used to develop HIV prevention messages for use in an HIV prevention intervention for MSM delivered via a Smartphone application.

• Methods to be used to collect data: Data will be collected from 135 MSM in focus group discussions (9 groups to include 90 men in total) and (45) in-depth interviews.

• The subpopulation to be studied: 135 MSM, including 45 HIV-positive MSM. MSM will be living in Atlanta, Detroit, and New York City.

• How data will be analyzed: Qualitative coding of 9 focus groups and 45 in-depth interview transcripts using computer assisted qualitative data analysis software.

Supporting Statement

A. Justification

1. Circumstances Making the Collection of Information Necessary

The Centers for Disease Control and Prevention’s Division of HIV/AIDS Prevention, (DHAP) requests OMB approval for a formative research study entitled, “Development of a Mobile Messaging Intervention for Men who have Sex with Men: Formative Study” under the Formative Research and Tool Development Generic Clearance (OMB #0920-0840, expires 01/31/2019).

This study will assess preferences of MSM for HIV prevention messages addressing new HIV prevention strategies for delivery via new technology, such as text messages, apps, infographics, and brief videos. Pre-crafted messages based on the scientific merit of current HIV prevention strategies will be tailored during focus groups and in-depth interviews to meet the needs of MSM in terms of mode of delivery, format of content (video, text, infographic), message content (scientific, persuasive), and style of message delivery (formal or informal). We will use the information collected through this formative study to develop mobile messaging intervention for MSM.

Sexual risk behavior among men who have sex with men (MSM) in the U.S. increased from 2005-2011, a group already disproportionally affected by HIV:¹ increases are especially alarming among young MSM² and MSM of color.³ The 20-city National HIV Behavioral Surveillance system reported that in 2011, only 67% of men overall and 62% of young MSM had tested in the past 12 months.^4,5 At the same time, messaging on the range and complexity of HIV prevention options is becoming increasingly challenging as the field of HIV prevention moves from behavioral to biomedical risk reduction. It is estimated that 25% of MSM are at high risk for HIV infection and could be appropriately prescribed the breakthrough biomedical strategy of Pre-Exposure Prophylaxis (PrEP)⁶ but only 2% of MSM in an online survey were currently taking PrEP.⁷ Reasons for poor uptake of PrEP include confusion and misunderstanding in the community about the efficacy of taking a pill to prevent HIV and concern about side effects.⁸ As new information about biomedical prevention options continues to emerge from the scientific community, this information must be translated clearly and concisely to populations at highest risk for HIV infection. New and complex scientific information included in this study are: pre-exposure prophylaxis (PrEP); antiretroviral (ARV) adherence; condom efficacy; and frequency of HIV/STD testing. This complex HIV prevention landscape requires novel multi-component messaging for 1) HIV-positive MSM, 2) high-risk HIV-negative MSM, and 3) lower-risk HIV-negative MSM. Importantly, these three groups are distinct from each other with respect to the prevention messages that will be most appropriate for them. For example, ARV adherence is important for HIV-positive MSM, and PrEP is an important HIV prevention method for high-risk HIV-negative MSM but not for lower-risk MSM (who might benefit from condom and testing frequency messaging). Using new technologies to deliver messages related to these emerging prevention methods will enable easy tailoring of messages so that they will be relevant to MSM from each risk group. Further, research has demonstrated that utilizing current technology, such as Smartphones and tablets, is an increasingly promising approach for reaching MSM and for scaling up interventions addressing HIV risk reduction and medication adherence.⁹ While other studies are also testing messages around new HIV prevention options, the messages developed for those studies are intended for broader or disparate audiences (e.g., health professionals) and are intended to be delivered via disparate mechanisms (e.g., fact sheets). To our knowledge, no study to date has tested brief messages for MSM that will be delivered via smartphone or tablet applications.

2. Purpose and Use of the Information Collection

The purpose of the information collection is to develop and assess HIV status/risk level-specific prevention messages for MSM about new options in HIV prevention, preferred modes of receiving mobile prevention messages, including format, style, and frequency, through qualitative focus group discussions and individual in-depth interviews.

One-hundred and thirty-five (135) MSM, including 45 HIV-positive MSM, 45 lower-risk seronegative MSM, and 45 higher-risk seronegative MSM will be recruited into either focus groups or in-depth interviews. MSM recruited into the study will be diverse (at least 50% MSM of color) and young (at least 50% age 18-29) and living in Atlanta, Detroit, or New York City. Recruitment will be conducted in both online and offline venues in which MSM congregate, including web sites (such as Facebook), bars, community events, and street locations (Attachment 1). In addition, MSM can drop-in or be referred to the study via collaborating clinics and community centers. All potential participants will complete a brief two-phase screening process for eligibility (Attachment 2a-b), which includes an initial screening and collection of contact information, followed by reverification of eligibility prior to consent and data collection. If eligible via reverification, participants will complete the consent process for focus groups or in-depth interviews (Attachments 3a-b).

Two qualitative methodologies will be used to collect information for this study. Focus group discussions will focus primarily on message format and message delivery, and secondarily on message content (Attachment 2c). These discussions will incorporate a pile-sorting activity to generate discussions around men’s general preferences as they relate to message mode of delivery, frequency of delivery, privacy issues, and specific HIV-related content. In-depth interviews will be used to assess the extent to which messages need to be customized and tailored to address variations in prevention needs (Attachment 2d). The key goal of these interviews will be to assess participants’ reactions to specific pre-developed messages (Attachment 4). All data collection instruments have been approved by Emory University IRB (Attachment 5).

The qualitative data collected through this study will be used to develop HIV prevention messages for use in an HIV prevention intervention for MSM, delivered via a Smartphone application. Information collected from this activity will help us to develop core HIV prevention messages and to customize secondary messages based on variations in HIV risk and HIV status. Due to the qualitative nature of the data collected, results will not be generalizable beyond the specific populations and geographic contexts in which they were obtained. CDC and/or its partners may also analyze these data and publish results in peer-reviewed journals.

3. Use of Improved Information Technology and Burden Reduction

The grantee will conduct in-person focus groups and individual interviews with key participants recruited in Atlanta, Detroit, and New York. Telephone interviews or visual remote interviews (such as web or Skype interviews) are not a good vehicle for developing the necessary rapport between interviewer and respondent for a successful qualitative interview on a sensitive topic. Body language and facial cues are critical to understand where additional probing may be needed or should stop, and telephone or web interviews limit the interviewer’s ability to read both. Thus, the grantee will conduct the individual, in-depth interviews (IDIs) in person. After asking for and receiving permission from the respondent, the contracting team will audio-record the interviews and transcribe recordings after the interview. This limits the burden on the respondent (no additional burden after completing the interview) and allows the interviewer to focus on building and maintaining rapport with the respondent.

4. Efforts to Identify Duplication and Use of Similar Information

The focus groups and in-depth interviews will collect information about HIV prevention messaging for complex and new HIV prevention options for MSM in the United States. We are aware that another study is also planning to test messages related to new HIV prevention options (OMB 0920-0572); however, it will be testing messages that differ in format, content, intended audience and intended delivery mechanism. Therefore, the Agency believes this information is not captured elsewhere. The Agency believes no other survey data collection effort has been conducted or has been planned to collect similar information for this population. CDC conducted a review of similar studies prior to the issuance of the Cooperative Agreement, and determined that this study is collecting unique information from the populations. Therefore, our evaluation requires the collection of this new primary data. There would be no reason for another Federal Agency to evaluate this.

5. Impact on Small Businesses or Other Small Entities

This collection request does not involve burden to small businesses or other small entities.

6. Consequences of Collecting the Information Less Frequently

The present study will provide the primary qualitative data needed to tailor HIV status and risk-level-specific HIV prevention messages for different groups of MSM, will test message content, and will identify preferred modes of message delivery and style. If this study were not conducted, it would not be possible to have a contextual understanding of the preferences and needs for mobile-ready risk-level-specific HIV prevention messaging about new HIV prevention strategies, including PrEP, nPEP, and ARV adherence for MSM. The length of data collection is 3-4 months and data will only be collected once.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This data collection effort does not involve any special circumstances.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A 60 day FRN notice to solicit public comments was published for the Generic umbrella collection (0920-0840) in the Federal Register on 06/25/2015, Volume 80, Number 122, Page Number 36540-36541. No public comments were received.

In addition, Emory University, Public Health Solutions, and the University of Michigan were consulted for the development of this study. There were no unresolved issues associated with the consultation process. Aside from the official 60 day public comment period for the Generic data collection, there were no other public contacts or opportunities for public comment on this study.

Patrick Sullivan, Principal Investigator Emory University 1518 Clifton Rd NE, Rm 438 Atlanta, GA 30322 404-727-2038 [email protected]	Aaron Siegler Emory University 1518 Clifton Rd NE, Rm 406 Atlanta, GA 30322 404-712-9733 [email protected]
Eli Rosenberg Emory University 1518 Clifton Rd NE, GCR 472 Atlanta, GA 30322 404-712-8897 [email protected]	Sabina Hirshfield Public Health Solutions 40 Worth Street, 5th Floor New York, NY 10013 646-619-6676 [email protected]
Mary Ann Chiasson Public Health Solutions 40 Worth Street, 5th Floor New York, NY 10013 646-619-6411 [email protected]	Martin Downing Public Health Solutions 40 Worth Street, 5th Floor New York, NY 10013 646-619-6528 [email protected]
Jose Bauermeister University of Michigan 1415 Washington Heights, SPH I, Room 3822 Ann Arbor, MI 48109 734-615-8414 [email protected]	Rob Stephenson University of Michigan 400 North Ingalls Building Ann Arbor, MI 48109 734-615-0149 [email protected]

9. Explanation of Any Payment or Gift to Respondents

Focus group participants and in-depth interview participants will each receive a $40 token of appreciation. In his memorandum for the president’s management council dated January 20, 2006, the Administrator of the Office of Information and Regulatory Affairs of the Office of Management and Budget wrote, “Incentives are most appropriately used in Federal statistical surveys with hard-to-find populations or respondents whose failure to participate would jeopardize the quality of the survey data (e.g., in panel surveys experiencing high attrition), or in studies that impose exceptional burden on respondents, such as those asking highly sensitive questions…” Although there has been some debate on the necessity of offering tokens of appreciation, numerous studies have shown that tokens of appreciation can significantly increase response rates and the use of modest tokens of appreciation is expected to enhance survey response rates without biasing responses. Offering tokens of appreciation is necessary to recruit minorities and historically underrepresented groups in to research.

In a recent study of recruitment and retention of Black men who have sex with men (BMSM) by a Community Based Organization (CBO), recruiters found it difficult to obtain information from the BMSM because many were reluctant to provide their names and contact information because of concerns about being seen giving these personal details to an HIV prevention program.¹⁰ Some of those who were screened provided incorrect contact information, making it difficult or impossible to locate them later. In this study, offering a token of appreciation improved participation among BMSM.¹⁰ A meta-analysis of 95 studies published between January 1999 and April 2005 describing methods of increasing minority persons’ enrollment and retention in research studies found that remuneration enhanced retention among this group.¹¹

Remuneration has been used in other HIV-related CDC data collection efforts, such as for National HIV Behavioral Surveillance (OMB 0920-0770, exp. 5/31/2014), the Transgender HIV Behavioral Survey (OMB 0920-0794, exp. 12/31/2010), and the Testing Brief Messages for Black and Latino MSM Study (OMB 0920-14SY under 0920-0840, exp. 1/31/2019), all of which included hard-to-reach populations and had a similar length of time for completing the client interview as in this proposed research. In all of these other projects, tokens of appreciation were used to help increase participation rates. 

10. Protection of the Privacy and Confidentiality of Information Provided by Respondents

The NCHHSTP PRA Coordinator has reviewed this project and determined the Privacy Act does not apply since personally identifiable information (PII) will not be transmitted to the CDC.

The grantee, Emory University, will be responsible for collecting all data for this study. We will inform respondents that their responses will be kept private to the extent permitted by the law. All respondents interviewed will be informed that the information collected will not be attributable directly to the respondent and will only be discussed among members of the research team. Terms of the CDC Cooperative Agreement authorizing data collection require the grantee to maintain the privacy of all information collected. Accordingly, individuals’ data will be kept private and protected to the extent permitted by law.

As the nature of this study is to better understand how HIV prevention messages are received among men who have sex with men (MSM), we are sensitive to the need to protect personal health information (PHI). To ensure respondents’ PHI is protected, we will take several measures to separate personally identifiable information (PII) from study-related data. Study forms, site reports, and audio tapes will be kept in locked file cabinets at the project sites for the duration of the study. All digitally-administered survey data, including screeners and informed consent forms, will be administered through a secure survey portal, hosted by SurveyGizmo, with whom Emory has established a business associate agreement to ensure HIPAA-compliance. Data stored with Survey Gizmo will be stored on an independent server, never co-mingled with data from other projects or clients. Data (screener data, digitally-provided consent forms, focus group discussion and in-depth interview transcripts) will be stored only with an ID and not with a direct personal identifier such as name or phone number. All focus group discussion and in-depth interview transcripts will be digitally-audio recorded and transcribed verbatim, and all transcriptions will be de-identified. No personal identifiers will be directly associated with any data other than the contact information files. Contact information files will be kept locked in project files or stored within a secure database, and a limited number of project staff will have access to the files. Contact information will be destroyed six months after completion of the study unless participants indicate that project staff may keep it on file for future studies or programs, or they request a copy of the primary results of this study. However, six months after the study is completed, study ID numbers for all participants will be de-linked from contact information in the participant database. Audio tapes will be destroyed 2 years after the end of the study (i.e., two years after the last follow-up assessment is completed). At the end of the study, all data will be sent to CDC via secure file transfer protocols. Only de-identified data will be sent to CDC and at no time will CDC have access to any PII.

11. Institutional Review Board (IRB) and Justification for Sensitive Questions

IRB Approval

This study has been reviewed and approved by the Emory IRB (Attachment 5).

Sensitive Questions

This is an initiative to learn about preferences and needs for risk level-specific HIV prevention messaging for MSM. As such, our study entails collection of sensitive HIV-related information. All study staff will be trained to provide respondents with referrals for prevention and care, such as mental health organizations, as needed. Sensitive questions will be asked to identify the risk level of participants. All other information is related to the content or format of the messages under review by each participant or focus group. We will inform all participants that they may skip any question or stop participation at any time for any reason.

12. Estimates of Annualized Burden Hours and Costs

1. Estimated Annualized Burden Hours

All potential participants will complete a brief, two-phase screening process for eligibility (Attachment 2a-b), which includes initial screening and reverification of eligibility prior to consent and data collection, as outlined in Exhibit 12.1 Men will be recruited either online through web advertisements or in-person through venue-based sampling and outreach, print advertisements, recruitment from local MSM-serving HIV organizations, or word of mouth (Attachment 1). In the first phase of screening, men will consent to screen and complete a brief screening online screening questionnaire (Attachment 2a). Eligible men will be asked to provide contact information (name, phone number and email address) through a separate online questionnaire (Attachment 2b). In the second phase of screening, men will be asked to verify their eligibility before completing the in-person interview or focus group discussion. Those who remain eligible will complete the corresponding informed consent (Attachments 3a-b) and will then participate in either a focus group discussion or in-depth interview.

Exhibit 12.1: Participant Recruitment and Screening Flow

It is expected that 50% of men screened will meet study eligibility and provide contact information, that 75% will schedule and show up for an in-person appointment, and that 95% of these men will remain eligible after reverification. We anticipate initial screening will take 4 minutes (Attachment 2a), providing contact information will take 1 minute (Attachment 2b), and reverification will take 4 minutes (Attachments 2a) to complete. Individual interviews and focus groups will each take about 90 minutes (1.5 hours) total to complete (Attachments 2c-d). The total number of burden hours is 243. Exhibits 12.2 and 12.3 provide further details about how the estimates of burden hours and costs were calculated.

Exhibit 12.2: Estimated Annualized Burden Hours

Type of Respondent	Form Name	No. of Respondents	No. of Responses Per Respondent	Average Burden Per Response (in Hours)	Total Burden Hours
General Public- Adults	Participant Screening (Eligibility) (Att. 2a )	380	1	4/60	26
General Public-Adults	Contact Information Form (Att. 2b)	190	1	1/60	4
General Public- Adults	Participant Screening (Verification) (Att. 2a)	143	1	4/60	10
General Public- Adults	Focus Group (Att. 2c)	90	1	1.5	135
General Public- Adults	In-Depth Interview (Att. 2d)	45	1	1.5	68
Total					243

2. Estimated Annualized Burden Costs

The annualized costs to the respondents are described in Exhibit 12.3. The United States Bureau of Labor Statistics’ employment and wages estimates from May, 2014 (http://www.bls.gov/oes/current/oes_nat.htm) were used to estimate the hourly wage rate for the general public for the purpose of this GenIC request. The total estimated cost of the burden to respondents is approximately $5,518.53. This cost represents the total burden hours of general respondents multiplied by the average hourly wage rate ($22.71).

Exhibit 12.3: Estimated Annualized Burden Costs

Type of Respondent	Form Name	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
General Public- Adults	Participant Screening (Eligibility) (att 2a)	26	$22.71	$590.46
General Public- Adults	Contact Information Form (att 2b)	4	$22.71	$90.84
General Public- Adults	Participant Screening (Verification) (att 2a)	10	$22.71	$227.10
General Public- Adults	Focus Groups (att 2c)	135	$22.71	$3,065.85
General Public- Adults	In-depth Interviews (att 2d)	68	$22.71	$1,544.28
Total $5,518.53

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no other costs to respondents for participating in this survey.

14. Annualized Cost to the Federal Government

The estimated annualized cost to carry out the data collection activities is $676,488. This estimate includes the cost of recruitment, screening, conducting the interviews, analysis and reporting, as well as the total cost of the tokens of appreciation ($40 per completed interview, for a total of $5,400). The CDC staff are primarily responsible for providing technical assistance in the design and implementation of the research; assisting in the development of the research protocol and data collection instruments for CDC Project Determination and local IRB reviews; working with investigators to facilitate appropriate research activities; and analyzing data and presenting findings at meetings and in publications. Data will be collected by members of contractor project staff.

Exhibit 14.1: Annualized Cost to the Government

Expense Type	Expense Explanation	Annual Costs (dollars)
Direct Costs to the Federal Government	CDC, Project Officer (GS-14 0.20 FTE)	$23,362
	CDC Scientist(GS-13, 0.20 FTE)	$19,770
	CDC Scientist(GS-13, 0.10 FTE)	$9,885
	CDC Project Coordinator (GS-12, 0.30 FTE)	$23,471
	Subtotal, Direct Costs	$76,488
Cooperative Agreement Costs	Cooperative Agreement #PS15-002 Costs	$ 600,000
	TOTAL COST TO THE GOVERNMENT	$ 676,488

15. Explanation for Program Changes or Adjustments

This is a new GenIC information collection request (ICR).

16. Plans for Tabulation and Publication and Project Time Schedule

Tabulation will include descriptive characteristics of study respondents collected in the first part of the interview (e.g., city, age, race/ethnicity, HIV-risk group). Data collection will occur between August and November of 2016, analyses will be carried out in November 2016, and the final data set and report will be submitted in December 2016. The project timeline is detailed in exhibit 16.1.

Exhibit 16.1: Project Time Schedule

Activity	Time Schedule
Develop data collection tools, sampling and data plans, study protocol	December 2015 – February 2016
OMB Submission	April 2016
Recruitment	After OMB approval
Data Collection	1-4 months after OMB approval
Data analysis finalized and report drafted	4 months after OMB approval
Final data set and final report submitted to CDC	5 months after OMB approval

Results from this data collection will primarily be used to develop HIV prevention messages for use in an HIV prevention intervention for MSM delivered via a Smartphone application. In addition, we anticipate that multiple manuscripts will be published in peer reviewed journals, presented at national conferences, and provided on conference websites. Links to these publications will be available through the CDC website. In compliance with the CDC policy on data management, we will develop a final, de-identified (names, other PII, and locations will be removed) qualitative database for this study along with the corresponding data documentation, which will be made publicly available within 30 months of the end of data collection.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

We do not seek approval to eliminate the expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exemptions to the certification.

References:

1 Centers for Disease Control and Prevention. HIV Testing and Risk Behaviors Among Gay, Bisexual, and Other Men Who Have Sex with Men — United States. MMWR 2013a; 62(47); 6 of 44, 958-962. Available at: http://www.cdc.gov/mmwr/preview/mmwr.html/mm6247a4.htm. Accessed 9/29/2014.

2 Prejean J, Song R, Hernandez A, Ziebell R, Green T, Walker F, Lin LS, An Q, Mermin J, Lansky A, Hall HI. Estimated HIV incidence in the United States, 2006-2009. PLoS One. 2011;6(8):e17502. Epub 2011/08/10. doi:10.1371/journal.pone.0017502 PONE-D-10-02530 [pii]. PubMed PMID: 21826193; PMCID: 3149556.

3 Centers for Disease Control and Prevention. HIV among Gay and Bisexual Men [online surveillance fact sheet]. May 2012. Available at http://www.cdc.gov/hiv/topics/msm/pdf/msm.pdf, last accessed September 20, 2012.

4 Centers for Disease Control and Prevention. HIV Testing and Risk Behaviors Among Gay, Bisexual, and Other Men Who Have Sex with Men – United States. MMWR Morb Mortal Wkly Rep. 2013;62(47):958-62. PubMed PMID: 24280915.

5 Oster AM, Johnson CH, Le BC, Balaji AB, Finlayson TJ, Lansky A, Mermin J, Valleroy L, Mackellar D, Behel S, Paz-Bailey G. Trends in HIV Prevalence and HIV Testing Among Young MSM: Five United States Cities, 1994-2011. AIDS Behav. 2013. doi: 10.1007/s10461-013-0566-1. PubMed PMID: 23955658.

6 Dawn K. Smith, Michelle Van Handel, Richard J. Wolitski, Jo Ellen Stryker, H. Irene Hall, Joseph Prejean, Linda J. Koenig, Linda A. Valleroy. Vital Signs: Estimated Percentages and Numbers of Adults with Indications for Preexposure Prophylaxis to Prevent HIV Acquisition — United States, 2015. MMWR 27, 2015 / Vol. 64 / No. 46. Pp1291-1295. Accessed February 9, 2016 at http://www.cdc.gov/mmwr/pdf/wk/mm6446.pdf.

7 Kenneth H. Mayer, Catherine E. Oldenburg, David S. Novak, Steven A. Elsesser, Douglas S. Krakower, Matthew J. Mimiaga. Early Adopters: Correlates of HIV Chemoprophylaxis Use in Recent Online Samples of US Men Who Have Sex with Men. AIDS Behav DOI 10.1007/s10461-015-1237-1

8 Katya Corado, Sonia Jain, Jill Blumenthal, Deborah Collins, Shelly Sun, Michael Dube, Michael Menchine, Sheldon Morris, Kathleen Jacobson Prevention Research and Implementation Science Abstract 1921. NHPC, Dec 2015, Atlanta.

9 Muessig KE, Nekkanti M, Bauermeister J, Bull S, Hightow-Weidman LB. A systematic review of recent smartphone, Internet, and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep. 2015 Mar; 12(1): 173-90.

10 Painter, T. M., Ngalame, P. M., Lucas, B., Lauby, J. L., & Herbst, J. H. (2010). Strategies used by community-based organizations to evaluate their locally developed HIV prevention interventions: Lessons learned from the CDC's innovative interventions project. AIDS Education and Prevention. 22(5), 387-401.

11 Yancey, A. K., Ortega, A. N., & Kumanyika, S. K. Effective recruitment and retention of minority research participants. Annual Review of Public Health. 27: 1-28.

14