Supporting Statement B_NIOSH Health Worker MH Campaign_2.16.23

Evaluation of the NIOSH Health Worker Mental Health Campaign

Request for Office of Management and Budget Review and Approval

for Federally Sponsored Data Collection

Supporting Statement B

February 16, 2023

Contact Information:

Amia Downes

CDC/NIOSH/OD

(404) 498-2544

[email protected]

Table of Contents

1. Respondent Universe and Sampling Selection

2. Procedures for Collecting Information

3. Methods to Maximize Response Rates and Deal with No Response

4. Tests of Procedures or Methods to Be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

1. Respondent Universe and Sampling Selection

The universe of participants are hospital leaders (specifically, hospital executives and senior management) and healthcare workers (specifically, patient-facing staff). Activities under this information clearance will involve representative, probability samples, and potentially self-selected participants. Respondents will be selected to cover a range of hospital health worker occupations (i.e., physicians, nurses, nurse practitioners, physician assistants, patient care technicians, pharmacists, phlebotomists, and therapists).

To select the sample of hospital healthcare workers and hospital leaders to complete the surveys, JPA/EDC will work with partner organizations (e.g., American Nurses Association, American College of Emergency Physicians, American Hospital Association) that will be disseminating campaign messaging and tools. JPA/EDC will select those partner organizations (up to 11) whose membership is likely to include the main audience—hospital leaders and hospital clinical staff. For example, the American Nurses Association is likely to be a vital partner for reaching nurses, who will comprise the majority of the sample in the healthcare worker survey.

Ultimately, the goal is to collect data from a representative group of 3,000 health workers and 500 high-level healthcare executives that hail from relevant partner network organizations.

Table 1. Goal participation by occupation group

Hospital Healthcare Worker Occupation Groups	Universe (N = 2,903,900)	Wave 1 (n = 1,500)	Wave 2 (n = 1,5000)	Total (n = 3,000)
Nurses (RNs)	1,752,210	900	900	1800
Nurses (LPNs)	78,090	35	35	70
Nurse Practitioners	52,790	30	30	60
Patient Care Technicians3	555,460	285	285	570
Pharmacists	81,500	45	45	90
Phlebotomists	47,470	23	22	45
Physicians	84,910	45	45	90
Physician Assistants	30,020	15	15	30
Social Workers	550	2	3	5
Therapists4	220,900	120	120	240
Hospital Leader Occupation Groups	Universe (N = 208,970)	Wave 1 (n = 250)	Wave 2 (n = 250)	Total (n = 500)
Top Executives	12,890	15	15	30
Other Managers	196,080	235	235	470

JPA/EDC expects the response rate to be approximately 25%, typical of online surveys like this one.¹ They will not use the results to make statements representative of the study universe, to produce statistical descriptions (careful, repeatable measurements), or to generalize the data beyond the scope of the sample. Table 1 provides information on the number of entities in the universe covered by the collection and in the corresponding sample.

To ensure that JPA/EDC achieves their goals for respondents by occupational group, they will work with partner organizations to draw a sample for each group that is significantly larger (four times higher given the anticipated response rate of 25%) than what is needed to draw conclusions about program contributions and effects. For example, a partner organization will draw a sample of at least 360 pharmacists, invite one-third of this group to complete the survey in Wave 1 and forward reminders twice each week. Then the organization will repeat this for Wave 2 until the goal of 90 respondents has been met.

Table 2. Sampling by occupation group

Hospital Healthcare worker Occupation Groups	Universe (N = 2,903,900)	Wave 1 (n = 6,000)	Wave 2 (n = 6,000)	Total (n = 12,000)
Nurses (RNs)	1,752,210	3600	3600	7200
Nurses (LPNs)	78,090	140	140	280
Nurse Practitioners	52,790	120	120	240
Patient Care Technicians3	555,460	1140	1140	2280
Pharmacists	81,500	180	180	360
Phlebotomists	47,470	90	90	180
Physicians	84,910	180	180	360
Physician Assistants	30,020	60	60	120
Social Workers	550	10	10	20
Therapists4	220,900	480	480	960
Hospital Leader Occupation Groups	Universe (N = 208,970)	Wave 1 (n = 1000)	Wave 2 (n = 1000)	Total (n = 2,000)
Top Executives	12,890	60	60	120
Other Managers	196,080	940	940	1880

However, if partner organizations are not able to create a sampling frame and draw a probability sample of eligible participants, JPA/EDC will ask them to divide their mailing lists/listservs into three segments and invite one segment at a time. JPA/EDC will need to quickly triage questionnaire responses in REDCap to determine the percent that are not eligible (screened out via questionnaire); and whether or not they are meeting sampling goals by occupation. Partner organizations will be encouraged to forward reminders twice each week for 4 weeks or until goal participation is met.

2. Procedures for Collecting Information

JPA/EDC will not have access to mailing lists/listservs or personal identifiable information. Instead, they will ask partner organizations to draw the sample and send out the invitation with the link to the survey embedded. JPA/EDC will provide technical assistance on sampling, a draft recruitment/invitation email, and the REDCap-generated hyperlink that can be embedded into invitation email. Participants who click on the link will be taken directly to the consenting page and survey. JPA/EDC will also provide a draft reminder email for the partner organizations to send twice a week for four weeks considering existing communication strategies of particular hospitals (e.g., intranet, weekly newsletters, daily huddles). The reminders are brief and include a link to opt-out.

To confirm that respondents work in hospital settings and have a leadership or patient-facing role, the first two questions in the survey are screening questions about occupational group and work setting. Those who indicate that they do not work in a hospital setting or in one of the occupational groups of interest will be screened out as not eligible. These individuals will not be counted in the denominator for calculating the response rate. If eligible, the respondent will then be able to read through the survey introduction and agree or decline to participate in the survey. If voluntary participation is agreed to, the respondent can proceed with the survey, choosing to refuse to answer a question or stop the survey at any time. This is a one-time survey that will take approximately 10 minutes to complete.

Because JPA/EDC is collecting information via online survey, responses appear in REDCap-generated csv files that can be uploaded into SPSS for further cleaning and analysis. With guidance from an EDC statistician, an EDC research associate will examine data for missing cases and values as well as any anomalies that result from use of online survey platform or data transfer to SPSS. A data dictionary will be created in SPSS variable view with key information on variable name, label, type, and response categories so that others accessing the data for analytical purposes will be able to easily navigate. As appropriate, new variables will be created to facilitate analysis.

3. Methods to Maximize Response Rates and Deal with No Response

In order to maximize response rates:

• Online survey instruments are designed to be completed in less than 10 minutes.

• Online survey instruments include short questions and closed-responses only to expedite their full completion.

• Partner member organizations that include professional organizations, labor unions, and healthcare systems, and are viewed by their constituents as legitimate leaders in the field, will help disseminate the retrospective online surveys to healthcare workers and hospital leaders.

• Data collection periods for the online surveys will be extended over four weeks.

• Eight, brief e-mail messages, sent twice weekly, will remind those who have not yet responded that their responses are critical to improving job conditions for healthcare workers in the United States and include a hyperlink to the online survey.

• Participants will be assured that their responses are anonymous.

Because JPA/EDC anticipates that the response rate will be less than 80%, they will aim to reduce non-response bias by guiding partner organizations to use probability (random) sampling procedures to select participants at random from their mailing lists. As feasible, they will encourage partner organizations to use stratified random sampling according to occupation group so that the sample is representative of the proportion of those occupations in hospital settings. For example, they would expect to survey many more hospital-based nurses than physicians.

To determine whether non-response bias is present, EDC will consider one of two approaches. The first approach assumes that EDC will not have background data on non-respondents. EDC will compare responses of those who responded to the initial request to participate and met eligibility criteria (working in hospital and identify as occupational group of interest) in the online survey with those who responded after several reminders and also met eligibility criteria. This is a common approach to analyzing non-response bias and is based on the premise that those who require more reminders are similar to non-respondents.² Specifically, EDC will examine whether independent variables (e.g., knowledge and beliefs about healthcare worker mental health and well-being) and demographic characteristics (e.g., race/ethnicity, occupation) as well as other variables thought to influence study participation (e.g., years in position) differ by number of reminders required to encourage participation. This procedure and the one that follows can be applied to account for non-response bias if partner network organizations are not able to draw a random sample, stratified by occupation (i.e., they use their entire mailing list/listserv to reach participants).

The second approach assumes that EDC will have some background data on those in the original sample frame, such as demographic variables (e.g., race, gender, ethnicity, age) and other characteristics (e.g., years in the profession, occupational status). These demographic variables serve as imperfect proxies for factors associated with outcomes of interest.³ EDC can compare the summary statistics of the analytic sample (those who complete the survey with valid data) against that of the original sample frame. EDC will summarize the difference or similarity between the analytic sample and the original sample frame descriptively for each of the background variables (e.g., percent Hispanic for the analytic sample vs. percent Hispanic for the original sample frame). EDC will then assess the degree of non-representativeness or representativeness of the analytic sample utilizing statistical criteria (such as a test of significance, or based on the difference in an effect size metric such as Hedges’ g). A variation of this approach is to compare survey respondents and non-respondents (among those to whom the survey was sent) in regard to each of those background characteristics. EDC would then calculate Hedges’ g, to translate the difference between the respondents and non-respondents in the effect size metric. EDC will consider the difference in the effect size metric up to 0.05 as ignorable; between 0.05 to 0.25 as adjustable through weighting; and beyond 0.25 as not adjustable (and therefore acknowledged in the limitation section of the study report).

If the difference between the analytic sample and the original sample frame (or, alternatively, the respondents and non-respondents) are judged ignorable (in all background variables), EDC will not need to make any statistical adjustments to arrive at the “unbiased” results. However, if the difference in any of the background variables are judged not ignorable, EDC will make statistical adjustments such as the use of probability weights. For example, EDC will fit a propensity model with a binary (responded, not responded) as the outcome, and the host of background variables as the predictors. Once a good propensity model is found, EDC will calculate the propensity weights for each respondent.

4. Tests of Procedures or Methods to Be Undertaken

JPA/EDC asked three health workers and healthcare executives to review the data collection instruments to ensure that EDC are using language that resonates with hospital clinical staff and that questions are clear, easy to understand, and appropriate. NIOSH asked three peer reviewers to provide technical review. See Supporting Statement A Section 8.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals designed the data collection methods:

• Kim Dash, PhD, MPH, Education Development Center, Inc., 617.618.2425, [email protected];

• Makoto Hanita, PhD, Education Development Center, Inc., 617.618.2347, [email protected]

• Kristen Quinlan, PhD, Education Development Center, Inc., 617.618.2188 [email protected].

The following individuals will collect the survey data:

• Kim Dash, PhD, MPH, Education Development Center, Inc., 617.618.2425, [email protected];

• Kristen Quinlan, PhD, Education Development Center, Inc., 617.618.2188 [email protected];

• Alissa Bey, PhD, Education Development Center, Inc., 617.618.2381 [email protected]

• Katie Ferro, JPA Health, [email protected], 980.800.2401, ext. 10.

The following individual will provide statistical expertise for the quantitative analysis plan:

• Makoto Hanita, PhD, Education Development Center, Inc. 617.618.2347, [email protected]

1 Wu, M., Zhao, K., & Fils-Aime, F. (2022). Response rates of online surveys in published research: A meta-analysis.

Computers in Human Behavior Reports, 7,100206. https://doi.org/10.1016/j.chbr.2022.100206

2 Duszynski, T. J., Fadel, W., Dixon, B. E., Yiannoutsos, C., Halverson, P. K., & Menachemi, N. (2022). Successive wave analysis to assess nonresponse bias in a statewide random sample testing study for SARS-CoV-2. Journal of Public Health Management and Practice, 28(4), E685-E691. doi: 10.1097/PHH.0000000000001508.

3 While not the only outcome of interest, studies have found associations between socio-demographic factors and burnout and include, for example: Teshome, B. G., Desai, M. M., Gross, C. P., Hill, K. A., Li, F., Samuals, E. A., et al. (2022). Marginalized identities, mistreatment, discrimination, and burnout among US medical students: Cross sectional survey and retrospective cohort study, BMJ, 376, e065984; and Gualano, M. R., Sinigaglia, T., Lo Moro, G., Rousset, S., Cremona, A., Bert, F., & Siliquini, R. (2021). The Burden of Burnout among Healthcare Professionals of Intensive Care Units and Emergency Departments during the COVID-19 Pandemic: A Systematic Review. International Journal of Environmental Research and Public Health, 18(15), 8172.