OMB Approval Memo

FDA DOCUMENTATION FOR THE GENERIC CLEARANCE

OF FOCUS GROUPS (0910-0497)

Focus groups do not yield meaningful quantitative findings. They can provide public input, but they do not yield data about public opinion that can be generalized. As such, they cannot be used to drive the development of policies, programs, and services. Policy makers and educators can use focus groups findings to test and refine their ideas, but should then conduct further research before making important decisions such as adopting new policies and allocating or redirecting significant resources to support these policies.

TITLE OF INFORMATION COLLECTION:

End-User Testing Associated with the “Pregnancy and Lactation Labeling Rule” to Improve Health Communications and Prescribing Decisions in Pregnant Women

DESCRIPTION OF THIS SPECIFIC COLLECTION

1. Statement of need:

The Food and Drug Administration (FDA), Center for Drug and Evaluation Research (CDER) and Center for Biologics Evaluation and Research (CBER) is seeking OMB approval under the generic clearance 0910-0497 to conduct a focus group study, “End-User Testing Associated with the “Pregnancy and Lactation Labeling Rule” to Improve Health Communications and Prescribing Decisions in Pregnant Women.” The objective of this study is to collect qualitative information from health care professionals (HCPs) to obtain their feedback about effective risk messaging in the Pregnancy subsection of the Pregnancy and Labeling Lactation Rule (PLLR) labeling, as well as examining their reactions to an infographic designed to supplement information offered in the Pregnancy subsection.

To help inform HCPs about the impacts prescription drugs may have on pregnancy, the FDA requires specific labeling on prescription drugs. The current regulation, the PLLR, became effective June 30, 2015. PLLR aims to assist HCPs in counseling patients to make informed medical decisions when using a prescription product during pregnancy. However, risk communication when there are limited data coupled with uncertainty presents a significant challenge. A recent study found a majority of participants thought PLLR labeling was “unhelpful” and less than half said it was not “clear.”¹ Other recent research has found that HCPs hold divergent understandings of the purpose, content, and evidence used in vaccine labeling for pregnant women.²

There is minimal published literature on best practices in risk communication in prescription product labeling. Similarly, no published literature exists on best practices regarding graphical representations to convey limitations or risk uncertainty in labeling. FDA proposes to identify best approaches for risk messaging in PLLR subsections of labeling. Findings from the HCP focus groups will provide input into the major themes and gaps in HCP perceptions, interpretations, and understandings of risk messaging in labeling.

The Wave I focus groups described in this memo will be followed by a second wave of focus groups to be conducted in fiscal year 2021. The second wave will explore best approaches for risk messaging in PLLR subsections of labeling with nurse midwives, nurse practitioners, pharmacists, and physician assistants. We will seek a separate OMB review for Wave II.

2. Intended use of information:

Information from the focus groups will be used to assess HCPs’ understanding of a modified PLLR labeling approach (plain language text, quick-take summaries, and infographics) intended to facilitate use of labeling information, and to identify gaps in communication of the risk message in PLLR labeling. Specific study endpoints include:

Primary endpoints:

• Identify and detail major themes that arise as prescribers interpret the risk message based on the PLLR labeling test materials

• Identify and describe gaps in HCP perceptions of the risk message presented in the PLLR labeling test materials

• Evaluate the messaging and comprehension of nonclinical data as it relates to clinical decision-making

Secondary endpoints:

• Qualitatively generate a ranked list from most to least understood statistical estimates from clinical studies that impact decision-making.

• Qualitatively generate a list of best practices of risk messaging in the existing Pregnancy subsections of prescription product labeling.

Knowledge gained from the focus groups can impact medical practice by improving health communications and informing prescribing decisions between HCPs and pregnant women, thereby optimizing obstetrical and perinatal outcomes. This project could also serve as a model for HCP feedback on effective risk messaging in labeling and contribute to other subsections of labeling beyond PLLR.

FDA recognizes that the data collected will only be representative of the participants and will not be generalizable to the population segments characterized by the groups. The data will not be used for the purposes of making policy or regulatory decisions.

3. Description of respondents:

Focus groups will be conducted with 3 segments of HCPs who care for pregnant women: 1. Obstetrician/gynecologists, 2. Family medicine physicians, 3. other specialty physicians (psychiatrists, rheumatologists, neurologists, and gastroenterologists).

All groups will include individuals ages 18 and over and will include participants of diverse ages and races/ethnicities. Groups will also include a diverse mix of years in practice, number of pregnant patients seen each month, medical setting, geographic area, and amount of time per day spent in patient care (See Appendix I).

4. Date(s) to be conducted and location(s):

Focus groups will be conducted in June of 2020, or within approximately six weeks of OMB approval. Given the ongoing COVID-19 pandemic, the groups will be conducted after receipt of OMB approval and when it is safe to do so.

Focus groups will be conducted remotely in order to capture the most diverse range of experience in practice and geographic area.

Participants in the remote focus groups will come from any geographic location in the United States. Remote groups will be conducted using WebEx. Participants will be directed to log on to the focus group at an appointed time.

5. How the information is being collected:

Recruitment Information

All recruitment will be conducted by the contractor Westat, through professional associations corresponding to the 3 HCP segments, including:

• Segment 1: Society for Maternal Fetal Medicine, American College of Obstetricians and Gynecologists

• Segment 2: American Academy of Family Physicians

• Segment 3: American Psychiatric Association, American Academy of Neurology, American College of Gastroenterology, American Gastroenterological Association, American College of Rheumatology

Recruitment for the online focus groups will entail outreach to relevant professional association memberships via email. The associations will review data from their member directories to identify eligible participants and forward contact information for these potential participants to Westat. Working with these member lists provided by the association partners, Westat will send an email invitation to members who meet the study criteria of working with pregnant patients. That email will direct the potential participants to complete an online screener. Participants will be selected to represent a mix of time spent in patient care, years in practice, number of pregnant patients seen each month, and practice location.

Content for the online screener can be found in Appendix I, and for the email in Appendix II.

In the event that online recruitment is not sufficient, recruitment for groups will also be conducted by a professional focus group facility. The recruitment strategy for these facilities will consist of outreach to their proprietary databases. Facility staff will provide all necessary information and instructions to ensure participants log on at the agreed upon date and time. They will conduct recruitment and ensure that the needed number of participants attend their scheduled time slot. The facilities will send confirmation and reminder correspondences to recruited participants to help ensure attendance.

HCPs will be notified of their eligibility upon completion of the screener. Eligible participants will be re-contacted via telephone or email to notify them of their selection and to schedule the focus group. All participants will receive an email confirmation with details on the specific time of the focus group. A confirmation and reminder correspondence will be sent to recruited participants to help ensure attendance.

Focus Group Discussions

A Westat senior social science researcher will serve as a moderator for all focus groups. The moderator will review the items of consent at the beginning of each group and obtain verbal consent from each participant. Individuals who do not give verbal consent will not be able to participate in the focus group.

The moderator will then conduct the group using the attached moderator’s guide (Appendix IV) to ensure that all relevant topic areas are addressed. The moderator’s guide will be versioned to rotate the order in which participants see the labeling examples. The labeling examples and infographic mockup are included as Appendix V. The labeling examples will be displayed using screen sharing through WebEx. Participants will be able to view the examples on their screens and have control, if needed, to scroll through the documents at their own pace. WebEx is easily installed on personal computers through a temporary app. Detailed instructions with a test link will be sent before the focus group to ensure that participants can access the group.

Prior to beginning the discussion, the moderator will ensure that the FDA project staff may observe all the sessions remotely using streaming video technology. Westat will provide both audio and visual recordings of each group, as well as provide a near-verbatim transcript of each discussion, to ensure that participants’ views and opinions are accurately captured. These transcripts will form the basis of the data analysis.

Westat and all contracted vendors (e.g., focus group facilities, transcription vendor) will comply with safeguards for ensuring participant information is kept secure to the extent permitted by law. The last names of the participants will not appear on any focus group materials. Participants will be instructed not to enter their full name when logging onto the WebEx group. Verbatim quotes included in the final report will not be attributed to any individual.

All focus group audio recordings and transcripts delivered to FDA by Westat will be de-identified. The key to the coded data allowing for re-identification will never be released to the FDA, and the FDA will have no need for reidentification of the HCPs participating in the focus groups.

6. Number of focus groups:

Remote focus groups will consist of 5 to 6 participants. Table 1 demonstrates the total number of participants that will participate by segment. Since HCPs have a higher than usual drop-off rate, we will recruit 7 participants per remote group.³ Remaining participants above these participation levels will be dismissed. Remote groups will be 75 minutes. Up to 15 remote focus groups will be conducted.

Table 1. Number of focus groups and participants by segment

Segment	Group	Maximum number of participants per group	Total
Segment 1: OB/GYN	Group 1	6	30
	Group 2	6
	Group 3	6
	Group 4	6
	Group 5	6
Segment 2: Family Medicine	Group 1	6	30
	Group 2	6
	Group 3	6
	Group 4	6
	Group 5	6
Segment 3: Other specialty physicians	Group 1	6	30
	Group 2	6
	Group 3	6
	Group 4	6
	Group 5	6

7. Amount and justification for any proposed incentive:

To prepare for these focus groups, we consulted with facilities that host focus groups to determine incentive rates for each segment of HCPs. The incentives will ensure that we are able to attract a reasonable cross section of participants who meet our screening requirements to participate in the focus groups.

Recognizing the significant time and other burdens involved with participation in research and to convey the importance of the research to participants, honorariums are intended to help defray these “costs” in order to encourage individuals to participate.⁴ Numerous empirical studies have established that an honorarium can significantly increase participation rates. ^{5,6,7,8,9,10,11,12,13,14} This is particularly true for HCPs who are more difficult to recruit as study participants than members of the general population.^15,16

There are many reasons why HCPs are difficult to engage in research, most of which are related to their professional demands and time constraints.¹⁷ For example, many HCPs work irregular hours and must respond to clinical emergencies, making them less available to participate in research that must be scheduled in advance. Relatedly, focus groups must be scheduled to accommodate the needs of a diverse group of eight to 10 participants. Although researchers try to accommodate HCPs’ demanding schedules (e.g., offer multiple timeslots, conduct groups early or late in the day, etc.), it is challenging to find times that do not interfere with their patient care and other required activities.¹⁸ The amount of time required for data collection is another factor that limits HCPs’ participation in research. High patient-volume, back-to-back scheduling, and the need to respond to patient emergencies leaves little time in the day to participate in nonessential activities.¹⁹ For example, a 2015 study found that burden was the primary reason for nonresponse in a study involving a web-based survey, with 60% of physicians saying they had insufficient time to complete the survey requests.²⁰ Building on these earlier findings, a recent survey conducted by The Physicians Foundation²¹ found that physicians saw an average of 20 patients per day, and about 80% said they were overworked/overextended or at capacity. These time constraints are particularly salient for qualitative data collections like focus groups because they tend to be more time consuming than surveys and may require travel to an offsite location (typically 60-90 minutes).

Studies with HCPs have shown that participation rates vary by specialty, gender, and other factors.²² For example, a recent study found that general surgeons (29.6%), pediatricians (29.2%), and psychiatrists (27.1%) were less likely to participate in a web-based survey than neurologists or neurosurgeons (46.6%) and internists (42.9%). Furthermore, Juster and Suzman²³ (1995) found that high incentives reduced nonresponse bias for people with high incomes. These findings are particularly relevant for our study because we intend to recruit physicians from a variety of specialties, including neurologists, gastroenterologists, and psychiatrists as well as others that are even less common, such as rheumatologists.

For this study, we will provide all focus groups participants with a $300 honorarium which OMB approved for the previous FDA research project titled Testing Communications on Biological Products, approved in 2014 under generic clearance 0910-0687. Only if HCPs decline to participate based on their stated reason that the $300 incentive is too low, we will reserve the option to offer them an additional $100 in an attempt to convert these refusals to willingness to participate ($400 maximum). This option will enable FDA to address costs that have risen over the past 5 years and better ensure recruitment of the very specialized and difficult-to-recruit populations. A similar tiered strategy was approved by OMB in 2017 for the Centers for Disease Control and Prevention’s Formative Research to Develop HIV Social Marketing Campaigns for Healthcare Providers (OMB No. 0920-1182). Participants will receive the honorarium by check after the completion of the focus group.

Several studies have explored strategies to improve recruitment of HCPs, and some have examined participation rates by incentive amount and/or type .^{24,25,26,27,28} This research shows that monetary incentives resulted in higher survey response rates compared to nonmonetary incentives and that the value (or perceived value) of the incentive matters, with higher incentives yielding greater participation than lower incentives. Although the incentives in this literature were lower than those proposed here, they involved significantly shorter survey participation and are often from many years ago. However, provision of high-value honorariums is supported by leading qualitative researchers who suggest that focus groups with physicians may require amounts up to or exceeding $500.²⁹ In addition, past experience on other projects our contractor has conducted, and their recent consultation with two national research firms (Plaza Research and Fieldwork, Inc.), show that the amounts offered are consistent with what specialized HCPs such as those we are targeting for this project require to take time out of their already time-constrained clinical practices to participate in these types of research projects. For example, in line with the $300 incentives for the 90-minute focus group participation in the FDA biological products study mentioned above, the contractor also paid specialists incentives of $250 for participating in interviews that were substantially shorter (60 minutes). Similarly, specialists received $250 incentives for participating in a one-hour focus group as part of FDA’s Generic Drug Substitution in Special Populations study (OMB No. 0910-0677; 2017) and in 60-minute telephone interviews for Studies to Enhance FDA Communications Addressing Opioids and Other Potentially Addictive Pain Medications (OMB No. 0910-0695; 2016), an incentive rate that would have amounted to $375 for 90 minutes.

The decision to provide the proposed honorarium amounts is based on the principles set forth in OMB’s guidance on factors that may justify provision of an incentive (Office of Management and Budget, 2006):

• Data quality: One strategy we will implement to improve the quality and robustness of the data is to recruit HCPs with wide-ranging behaviors, specialties, and sociodemographic characteristics. Offering less than the proposed amounts, however, is likely to limit our ability to recruit the diversity of the desired subgroups and provide the breadth of the expertise, experience, and prescribing practices necessary for this project. Prior studies have shown variations in participation rates by medical specialty, age, and race/ethnicity. Furthermore, the difficulties engaging HCPs for research studies have been well-documented. Numerous studies have shown that honorariums can reduce sample bias which occurs when research participants do not represent the diversity of the intended audience (Griffin et al., 2011; Lesser et al., 2001; Singer & Kulka, 2002).^30,31,32 A biased sample will compromise the usefulness and validity of the findings.

• HCP participant burden. Lack of time and competing demands are significant barriers to research participation among HCPs. Although we will aim to limit burden to the extent possible, the study design requires participation in a 75-minute prescheduled focus group session. Participants may need to find coverage for their patients to attend the session and/or work longer days to complete paperwork or follow up with patients. Burden concerns are a common deterrent to participation, particularly among high-volume HCPs or high-demand and/or uncommon specialists, which would lead to sample bias and affect data quality (see above). The honorarium will help offset or reduce concerns about time and other burdens, such as disruptions to patient flow, inconvenience, financial loss, and the need to limit or reschedule patient appointments.

• Past experience: As described previously, the study team has conducted qualitative studies with HCPs, and our experience confirms that they are a very challenging population to recruit. The requested amount is consistent with the amounts provided for the prior FDA studies that involved qualitative research methods with similar populations (see above).

• Improved coverage of specialized respondents or rare groups: The HCP participants in this study are considered specialized because they have specific knowledge and experience related to the topic area. Although the number of professionally active primary care and specialty physicians in the United States are similar, the latter are distributed across multiple specialty types, increasing specialists’ rarity. In 2018, for example, psychiatrists made up just 11% of all specialists and Ob/Gyns made up 11% of all primary care doctors.³³ Other specialties, such as rheumatology face a short-fall in an adequate number of providers.³⁴ These data demonstrate that the pool of potentially-eligible HCPs is limited, and our eligibility criteria impose further eligibility restrictions. Rarity coupled with barriers related to time and other burdens make specialists particularly hard-to-recruit.

8. Questions of a Sensitive Nature:

There will be no questions of a sensitive nature asked of participants.

9. Description of Statistical Methods ( i.e., Sample Size and Method of Selection):

This is a qualitative study using a convenience sample. It does not entail the use of any statistical methods. The qualitative and analytical methods are described below.

Transcribed recordings and notes from all focus groups will be analyzed using standard qualitative methodologies. As notes and transcripts are read, sections of text that indicate or suggest key themes will be marked and provisionally labeled. The labels will form the basis for a coding scheme, for which each code is defined and illustrated. The process of code development will be iterative and will take place over time. The set of codes accumulated during data review will be repeatedly revisited and revised to produce the final coding scheme. Actual data coding will then consist of reviewing the set of notes and transcripts and assigning codes to the sections of text to which they correspond. Data will be coded using NVivo, a software program for analyzing qualitative data. The findings will be summarized in manuscripts and presentations and shared with stakeholders.

BURDEN HOUR COMPUTATION (Number of respondents (X) estimated response or participation time in minutes (/60) = annual burden hours):

Type/Category of Respondent	Total No. of Respondents	Participation Time (minutes)	Burden (hours)
Screener	360	5	30
Segment 1 – 5 groups of up to 6 participants	30	75	37.5
Segment 2 - 5 groups of up to 6 participants	30	75	37.5
Segment 3 - 5 groups of up to 6 participants	30	75	37.5
Total			142.5

REQUESTED APPROVAL DATE: May 1, 2020

Tamara Johnson, MD, MS

FDA Project Lead

Center for Drug Evaluation and Research (CDER)

Division of Pediatric and Maternal Health (DPMH)

[email protected]

301-796-1522

Ila S. Mizrachi

Paperwork Reduction Act (PRA) Desk Officer

PRA Staff

Division of Information Governance

Office of Enterprise Management Service

[email protected]

301-796-7726

FDA CENTER: Center for Drug Evaluation and Research

Center for Biologics Evaluation and Research

Attachments:

Appendix I – Participant Screeners

Appendix II – Recruitment Email

Appendix III – Informed Consent

Appendix IV – Moderator’s Guide

Appendix V – Labeling Examples with Infographic Mockup

1 Namazy, J. 2018. Physician’s Perspective of the New PLLR: Survey Results. https://www.fda.gov/media/111610/download

2 Manca, T., Graham, J., Dube, E., Kervin, M., Castillo, E., Crowcroft, N., Fell, D., Hadskis, M., Mannerfeldt, J., Greyson, D., MacDonald, N, Top, K. (2019.) Developing product label information to support evidence-informed use of vaccines in pregnancy. Vaccine, 37: 7138-7146.

3 Asch, S., Connor, S.E., Hamilton, E.G., and Fox, S.A. (2000). Problems in recruiting community-based physicians for health services research. Journal of General Internal Medicine, 15(8), 591-599.

4 Klabunde, C. N., Willis, G. B., McLeod, C. C., Dillman, D. A., Johnson, T. P., Greene, S. M., & Brown, M. L. (2012). Improving the quality of surveys of physicians and medical groups: A research agenda. Evaluation and the Health Professions, 35(4), 477–506.

5 Abreu, D. A., & Winters, F. (1999). Using monetary incentives to reduce attrition in the survey of income and program participation. Proceedings of the Survey Research Methods Section of the American Statistical Association. 533-538.

6 Aikin, K., Betts, K., Boudewyns, V., Stine, A., & Southwell, B. G. (2016). Physician responsiveness to survey incentives and sponsorship in prescription drug advertising research. Annals of Behavioral Medicine, 50(Suppl.), s251.

7 Dykema, J., Stevenson, J., Day, B., Sellers, S., & Bonham, V. (2011). Effects of incentives and prenotification on response rates and costs in a national web survey of physicians. Evaluation and the Health Professions, 34(4): 434–447.

8 Greenbaum, T. L. (2000). Moderating focus groups: A practical guide for group facilitation. Thousand Oaks, CA: Sage Publications, Inc.

9 Martins, Y., Lederman, R.I., Lowenstein, C.L., Joffe, S., Neville, A., Hastings, B.T., & Abel, G.A. (2012). Increasing response rates from physicians in oncology research: a structured literature review and data from a recent physician survey. British Journal of Cancer, 106(6), 1021-1026.

10 Medway, R. L., & Tourangeau, R. (2015). Response quality in telephone surveys: Do prepaid cash incentives make a difference?  Public Opinion Quarterly, 79(2), 524-543.

11 Mercer, A., Caporaso, A., Cantor, D., & Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys.  Public Opinion Quarterly, 79(1), 105-129.

12 Shettle, C., & Mooney, G. (1999). Monetary incentives in U.S. government surveys. Journal of Official Statistics, 15(2), 231–250.

13 Thorpe, C., Ryan, B., McLean, S.L., Burt, A., Stewart, M., Brown, J.B. (2009). How to obtain excellent response rates when surveying physicians. Family Practice, 26(1), 65-68.

14 VanGeest, J. B., Johnson, T. P., & Welch, V. L. (2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Evaluation of Health Professionals, 30(4), 303–321.

15 Asch S., Connor S.E., Hamilton E.G., & Fox S.A. (2000). Problems in recruiting community-based physicians for health services research. Journal of General Internal Medicine, 15(8):591-599.

16 Cummings, S.M., Savitz, L.A., & Konrad, T.R. (2001). Reported response rates to mailed physician questionnaires. Health Services Research, 35(6), 1347-1355.

17 Asch S., Connor S.E., Hamilton E.G., & Fox S.A. (2000). Problems in recruiting community-based physicians for health services research. Journal of General Internal Medicine, 15(8):591-599.

18 Bakken S, Lantigua RA, Busacca LV, & Bigger JT. (2019). Barriers, enablers, and incentives for research participation: a report from the Ambulatory Care Research Network (ACRN). J Journal of the American Board of Family Medicine, 22(4):436–45.

19 Capko, J. To improve practice work flow, tackle patient scheduling. Retrieved from http://www.physicianspractice.com/scheduling/improve-practice-work-flow-tackle-patient-scheduling. Accessed January 17, 2019.

20 Cunningham, C.T., Quan, H., Hemmelgarn, B., Noseworthy, T., Beck, C.A., Dixon, E. (2015). Exploring physician specialist response rates to web-based surveys. BMC Medical Research Methodology, 15(32). doi: 10.1186/s13104-014-0969-8.

21 Physcian’s Foundation. (2018). America’s Physicians: Overworked and Burning Out. https://thehospitalleader.org/americas-physicians-overworked-and-burning-out/Accessed March 24, 2020.

22 Cunningham, C.T., Quan, H., Hemmelgarn, B., Noseworthy, T., Beck, C.A., Dixon, E. (2015). Exploring physician specialist response rates to web-based surveys. BMC Medical Research Methodology, 15(32). doi: 10.1186/s13104-014-0969-8.

23 Juster, F. T. & Suzman, R. (1995). An overview of the health and retirement study. Journal of Human Resources, 30, S7-S56.

24 Asch, D.A., Christakis, N.A., & Ubel, P.A. (1998). Conducting physician mail surveys on a limited budget: a randomized trial comparing bill versus $5 bill incentives. Medical Care, 36(1):95–99.

25 Deehan, A., Templeton, L., Taylor, C., Drummond, C., & Strang, J. (1997). The effect of cash and other financial inducements on the response rate of general practitioners in a national postal study. British Journal of General Practice, 47(415):87–90. 

26 Pit, S.W., Vo, T., & Pyakurel, S. (2014). The effectiveness of recruitment strategies on general practitioner's survey response rates - a systematic review. BMC Medical Research Methodology, 6(14). doi: 10.1186/1471-2288-14-76.

27 Thomson, C.E., Paterson-Brown, S., Russell, D., McCaldin, D., & Russell, I.T. (2004). Short report: encouraging GPs to complete postal questionnaires - one big prize or many small prizes? A randomized controlled trial. Family Practice, 21(6):697–698.

28 Young, R.A., Fulda, K.G., Suzuki, S., Hahn, K.A., Espinoza, A.M., Marshall, J.D.. (2011). The influence of research compensation options on Practice-based Research Network (PBRN) physician participation: A North Texas (NorTex) PBRN study. Journal of the American Board of Family Medicine, 24(5), 562-568.

29 Krueger, R.A., & Casey, M.A. (2015). Focus Groups: A practical guide for applied research. Thousand Oaks, CA: Sage Publications, Inc.

30 Griffin, J., Simon, A. B., Hulbert, E., Stevenson, J., Grill, J., Noorbaloochi, S., & Partin, M. (2011). A comparison of small monetary incentives to convert survey non-respondents: a randomized control trial. BMC Medical Research Methodology, 11(81). doi: 10.1186/1471-2288-11-81.

31 Lesser, V.M., Dillman, D.A., Carlson, J., Lorenz, F., Mason, R., & Willits, F. (2001) Quantifying the Influence of Incentives on Mail Survey Response Rates and Nonresponse Bias. Paper presented at the annual meeting of the American Statistical Association, Atlanta, GA.

32 Singer, E. & Kulka, R. A. (2002). Paying Respondents for Survey Participation. In M. Ver Ploeg, R. A. Moffitt, & C. F. Citro (Eds.), Studies of Welfare Populations: Data Collection and Research Issues, Washington, D.C.: National Academy Press.

33 Kaiser Family Foundation. (2018a). Professionally Active Specialist Physicians by Field. Retrieved from https://www.kff.org/other/state-indicator/physicians-by-specialty-area/?dataView=1&currentTimeframe=0&selectedRows=%7B%22wrapups%22:%7B%22united-states%22:%7B%7D%7D%7D&sortModel=%7B%22colId%22:%22All%20Other%20Specialties%22,%22sort%22:%22asc%22%7D

34 Sullivan, M. (2016). ACR 2015 Workforce Study: Fewer Rheumatologists, More Patients, and the Struggle to Bridge the Gap. Rheumatology News. https://www.mdedge.com/rheumatology/article/118407/practice-management/acr-2015-workforce-study-fewer-rheumatologists-more. Accessed March 24, 2020.

8