Ssb

Keeping Cool: Evaluating NIOSH’s Heat Stress Training Module to Inform Resource Development and Research

CDC/ATSDR Formative Research and Tool Development

0920-1154

Supporting Statement B

May 20, 2024

Contact Information:

Kristin Yeoman, MD, MPH
Medical Epidemiologist
CDC/NIOSH
315 E. Montgomery Ave.

Spokane, WA 99207

509-354-8067

[email protected]

Table of Contents

Section

B. Supporting Statement B

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with No Response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects/Individuals Collecting and/or Analyzing Data

Supporting Statement B

1. Respondent Universe and Sampling Methods

This study aims to evaluate mine operators’ use of an online NIOSH heat stress training module to increase mine workers awareness of heat stress and to prevent heat-related illness at work. It is critical to assess safety managers’ and trainers’ use of the module in the field to assess its utility, acceptability, and completeness and to develop additional resources.

Using convenience sampling, NIOSH personnel will recruit two to four U.S. mining or aggregate employers to participate in the study. Using existing professional relationships, we will reach out to occupational safety and health (OSH) managers at mining and aggregate companies to explore their interest in using and providing feedback on the use of the heat stress training module. To participate in the study, OSH managers must agree to determine how and in what ways the training module will be implemented within the context of their organization. NIOSH personnel will only observe and request feedback on employers’ preparation for and provision of the training.

The study is designed to collect feedback from two groups of employees within each company: (1) OSH managers and personnel overseeing and/or providing the heat stress training (i.e., may include other safety personnel, supervisors, foremen, or other trainers); and (2) workers employed at the company and attending the heat stress training sessions provided by the trainers. Participants must speak English and age ≥18 years. There are no other exclusionary criteria. In the first group, we will include up to 20 OSH managers and trainers, across companies, in individual and small group interviews. In the second group, we will include up to 200 workers. All 200 workers will be requested to voluntarily complete a brief survey immediately following the training, and up to 84 workers will additionally be requested to volunteer to participate in small focus groups (4-12 workers each). We expect the majority of focus group participants will have completed the brief survey (i.e., the 84 workers are included in the 200 workers who complete the survey).

2. Procedures for the Collection of Information

NIOSH personnel will conduct a mixed-methods, process evaluation of employers’ use of the NIOSH training module, Keeping Cool: Training to Reduce Heat Stress Incidents, designed to provide workplace education about heat stress to mine workers. Employers’ OSH managers will determine how and in what ways the training module is implemented within the context of their organization. For example, they will determine who serves as trainers, the sites involved, the number of training sessions conducted, and which workers are included in those sessions. This is important to approximate real-world use of the training resources by employers.

As part of the evaluation protocol, NIOSH personnel will use multiple data collection methods including interviews, observation, surveys, focus groups, and training logs. We also will engage multiple informants for feedback including OSH managers and trainers as well as mine workers. Table 1 shows the numbers of participants we anticipate including in the study by participant group and data collection method.

Table 1: Number of Participants by Group

Participant	Data Collection Instrument	Number of Participants per Site	Total Number of Participants in Study
OSH managers and trainers	Interview Scripts (Attachment B)	Up to 6	Up to 20
	Training Log (Attachment E)
Workers	Survey (Attachment C)	Up to 60	Up to 200
	Focus Group Script (Attachment D)	Up to 36 of 60	Up to 84 of 200
Total	-	Up to 42	Up to 220

OSH Managers and Trainers

The size of the operation and how OSH managers decide to implement training within the context of their organization will influence how many managers and trainers are included in the study within each organization. In smaller operations, OSH managers may organize and conduct the trainings, and thus only one manager/trainer will be interviewed. In contrast, OSH managers may oversee the training but designate the training facilitation to one or more other personnel (e.g., other safety staff, supervisors, or foremen). In this case, we will include the OSH manager and potentially multiple trainers.

We will conduct up to five semi-structured interviews with OSH managers and trainers at three different time points:

• Initial interviews: Once with OSH manager at the start of the study (before any training),

• Post-preparation, pre-training session: Once after the trainer prepares for delivery of the training module but prior to facilitating any training, and

• Post-training interviews: Immediately following the first, second, and last (during study period) training session(s), for a maximum of three post-training interviews.

The interview scripts for all three types are included in Attachment B1, B2, and B3. Interviews will be conducted in-person or virtually using Zoom for Government (ZoomGov), a FedRAMP Authorized Software-as-a-Service (SaaS). Interviews will be audio-recorded and then transcribed to facilitate data analysis. Audio-recordings will be deleted after interviews are transcribed and checked for quality.

The following describes the three types of interviews in more detail:

• Initial interviews: The initial interviews will be conducted with OSH managers who oversee the training (i.e., decision makers who determine that heat stress training is necessary and how it will be conducted). Their perspectives are important to understanding the goals and objectives of training and how training fits within the operation’s overall approach to heat illness prevention. The managers may facilitate training or designate other personnel (or external contractors) to deliver the training. These interviews are estimated to last up to 30 minutes.

• Post-preparation interviews: The heat stress training module provides the resources needed to conduct training for trainers who are not experts in heat stress. Therefore, it is important to assess whether trainers feel adequately prepared and are satisfied with the quality and completeness of the materials. Understanding the logistics of preparing for training also has important implications for sustainability. These interviews are expected to last 20 minutes.

• Post training interviews: Evaluators will conduct interviews with OSH managers and trainers after the initial training sessions (estimated up to 40 minutes). If trainers provide additional sessions to workers, evaluators will conduct subsequent brief, follow-up interviews after the second and last trainings during the study period (estimated to be a maximum of 30 additional minutes each). If an employer uses multiple trainers, and investigators are onsite, investigators may conduct some post training interviews in small groups (i.e., 2-3 individuals).

The maximum total time for interviews for any OSH manager or trainer will not exceed 150 minutes across the study period (i.e., maximum of five interviews that total ≤150 minutes). The total time per participant however is likely to be less, particularly when OSH managers designate someone else to do the training.

We also will ask participating OSH managers/trainers to complete a brief training log (Attachment E) capturing information about each training conducted as part of the study. The log will capture the following information about each training: date, number of participants and their work roles, and a brief explanation of why they were selected for training. This log is not intended to capture worker-specific data but rather general information on the types of workers being trained. This information will provide important context to our evaluation of the effectiveness of this training module.

Mine Workers

We will collect feedback from workers who participate in training using surveys and focus groups.

• Surveys (Attachment C): We will ask all workers who participate in the training to complete a brief survey (5 minutes, on paper) immediately following the training. The survey includes eight quantitative, Likert-style items and three open-ended questions. We also ask workers if they have participated in other heat stress training within the past two years.

• Focus Groups (Attachment D): NIOSH staff will also facilitate up to 12 small focus-groups with workers (4-10 participants, average 7 participants, maximum 84 workers total) onsite across employers and operation sites. These will take place within a week or so of the training and will last approximately 30 minutes. Groups may include workers who attended different training sessions. NIOSH personnel will work with OSH managers to schedule these to facilitate participation but to minimize disruption of production schedules. Given varying working conditions and worker roles, NIOSH will likely conduct more focus groups within larger organizations compared to smaller organizations. Focus groups will be audio-recorded and then transcribed to facilitate data analysis. Audio-recordings will be deleted after groups are transcribed and checked for quality.

Observation of Sessions

NIOSH personnel will passively observe as many trainings in-person as possible and take detailed session notes. Observation allows investigators to assess the delivery of the material, the interactions between facilitator and workers, the extent to which the training is delivered as designed, worker engagement, and trainer successes and challenges. This provides a way to triangulate trainer and worker reports with those of an external observer.

Consent

Informed consent will be obtained verbally from OSH managers and trainers participating in the study (Attachment G). Because of the nature of the study, a waiver of documentation of informed consent for the interviews of OSH managers/trainers, has been requested and granted. According to 46.117c(2), this waiver is appropriate because the research presents no more than minimal risk of harm to subjects and involves no procedures for which written consent is normally required outside of the research context. Interviewed OSH managers/trainers will be provided a copy of the informed consent (Attachment G), and NIOSH staff will review it with them prior to interviews but will not require a signature.

A waiver of informed consent for workers participating in focus groups or surveys has been requested, which is appropriate under 46.116(d) given that all four criteria are met:

1) research involves no more than minimal risk to subjects;

2) waiver will not adversely affect the rights and welfare of subjects (i.e., NIOSH is not collecting identifiable or sensitive data in the survey or focus groups, and survey participants will be informed that they do not have to participate and can decline to answer any questions. The NIOSH focus group facilitator will provide a concise, verbal explanation of the study, emphasizing that participation is voluntary, may be withdrawn at any time, shared feedback is confidential, and recording is only for transcriptions and will be deleted afterwards);

3) research could not practicably be carried out without the waiver (i.e., NIOSH investigators will not be present at all trainings and therefore cannot consent workers to complete the survey after the trainings);

4) subjects will be provided with additional information (i.e., training participants will be given principal investigator’s contact information for questions and can access a brief one-page graphical summary of worker feedback after all training sessions are completed and data analysis has been conducted).

3. Methods to Maximize Response Rates and Deal with No Response

Concerns about the effects of heat on mine workers’ health and well-being are growing, and mining constituents have prioritized identification of workplace strategies to mitigate the adverse effects of heat. We anticipate considerable interest by employers in participating. Worker participation can be facilitated when invited to provide feedback and offer solutions on topics that are relevant and important to them.

To maximize response rates, NIOSH researchers will:

• Clearly communicate to participants the purpose of the study, expectations, and future use of findings.

• Emphasize the importance and relevance of participation.

• Offer convenient times and locations for respondents to participate.

• Offer the option of conducting virtual interviews.

• Emphasize the importance of participation if the participant refuses the initial approach.

• Ensure all participants understand that data will be kept confidential and will not identify participants.

• Use guides that are designed to be as easy and non-burdensome as possible. This includes ordering the topics in a logical sequence and discussing only those issues that are needed for analysis purposes.

• Ensure the interviews do not exceed the allotted amount of time to minimize the overall burden.

4. Tests of Procedures or Methods to be Undertaken

For this study, NIOSH Mining Program researchers, including those with medical, behavioral health, and evaluation expertise, internally designed and reviewed the evaluation protocols, data collection instruments, and data analysis plan to ensure readability and relevance to the Mining industry sector. Data collection guides were informed by previous heat stress research and literature.

Research Instruments

OSH Manager and Trainer Interview Guide (Attachments B1, B2, B3)

NIOSH will conduct up to five semi-structured interviews with OSH managers and trainers at three different time points:

• Initial interviews: Once with OSH manager at the start of the study (before any training).

• Post-preparation, pre-training interviews: Once after the trainer prepares for delivery of the training module but prior to facilitating any training.

• Post-training interviews: Immediately following the first, second, and last (during study period) training session(s), for a maximum of three post-training interviews.

Interviews will be conducted in-person or virtually using Zoom for Government (ZoomGov), a FedRAMP Authorized Software-as-a-Service (SaaS). Interviews will be audio-recorded and then transcribed to facilitate data analysis. Audio-recordings will be deleted after interviews are transcribed and checked for quality.

The following describes the three types of interviews in more detail:

• Initial interviews (30 minutes) (Attachment B1): The initial interviews will be conducted with OSH managers who oversee the training (i.e., decision makers who determine that heat stress training is necessary and how it will be conducted). In the interview,NIOSH evaluators will explore OSH managers’ goals and objectives for the training, needs for training, other heat prevention strategies, and which workers they feel need training most.

• Post-preparation interviews (20 minutes) (Attachment B2): The heat stress training module is intended to provide all necessary information needed to facilitate the session. It is important to assess whether trainers feel adequately prepared and are satisfied with the quality and completeness of the materials. Evaluators will also ask about previous training related to heat illness. Understanding the logistics of preparing for training also has important implications for sustainability. In the interview, evaluators will also explore how trainers think workers will react to the training.

• Post training interviews (30-40 minutes) (Attachment B3): Evaluators will conduct interviews with OSH managers and trainers after the initial training sessions. Trainers will be asked what went well and if there were challenges. NIOSH will explore the extent to which workers were engaged and if they offered feedback. NIOSH will also ask for suggestions to improve the training and explore their needs for an implementation guide and other resources to better integrate heat education into their overall worker well-being strategies. If they provide additional sessions to workers, evaluators also will conduct brief, follow-up interviews after the second and last trainings during the study period. If an employer uses multiple trainers, and NIOSH staff are onsite, some post training interviews in small groups (i.e., 2-3 individuals) may be conducted.

Trainer Log (Attachment E)

NIOSH will ask OSH managers/trainers to complete a brief training log capturing information about each training session conducted as part of the study. Trainers will provide training date, number of participants, their work roles, why they were selected for training, and additional notes about the context of training. This log will not capture worker-specific data. This information will provide important context to aid in interpreting the findings.

Worker Survey (Attachment C)

NIOSH will ask all workers who participate in the training to complete a brief survey (5 minutes, on paper) at the conclusion of the training. The survey includes eight quantitative, Likert-style items and three open-ended questions. For example, the items explore workers’ perceptions about the session value, usefulness of interactive discussion, their own heat-related risks, and ability to identify a peer in distress. Workers will also be asked if they have participated in other heat stress training within the past two years.

Worker Focus Group Guide (Attachment D)

NIOSH staff will facilitate up to 12 small focus-groups onsite with workers (4-10 participants, average 7 participants, maximum of 84 total workers) across employers and operation sites. These will take place within a week or so of the training and last approximately 30 minutes. Groups may include workers who attended different training sessions. Workers will be asked about their overall impressions of training, what they learned and how they might apply it in the workplace, and if and how they feel more empowered to recognize and help a peer in distress. NIOSH staff will also explore challenges workers may face in the workplace that make it hard to take heat-related precautions. NIOSH will work with OSH managers to schedule these to facilitate worker participation and to minimize disruption of worker and production schedules. Given varying working conditions and worker roles, NIOSH will likely conduct more focus groups within larger organizations than in smaller organizations. Focus groups will be audio-recorded and then transcribed to facilitate data analysis. Audio-recordings will be deleted after groups are transcribed and checked for quality.

Data Analysis

This study relies heavily on qualitative data and minimally uses quantitative data from an 8-item survey of workers. NIOSH investigators will qualitatively analyze transcripts from the interviews and focus groups individually using Excel and/or a qualitative analysis software. NIOSH will also analyze the open-ended items from the worker survey using an abbreviated analysis approach as open-ended responses often lack the richness and depth obtained through interviews and focus groups.

NIOSH will start with Hamilton’s Rapid Assessment Process (Hamiliton, 2020; Hamilton & Finley, 2019) based on Beebe’s (2001) earlier work. Rapid Assessment Process (RAP) is a team-based, iterative, deductive approach to develop a preliminary understanding of a situation from the “insider’s” perspective. It is a data reduction process that helps to sharpen, sort, focus, and organize data, providing quicker access to preliminary analyses. Data analysis will begin concurrent with data collection. With any method (traditional or rapid), this approach is advised as it often enriches subsequent data collection, because it allows the evaluator to cycle back and forth between existing data and generate enriched data through broadening inquiry if appropriate and asking better clarifying questions (Miles, Huberman, & Saldana, 2020). RAP can illuminate the importance of context in implementation and typically takes less time to critique, reflect, and synthesize the findings. RAP analysis often stands on its own but can also inform subsequent analyses using more traditional, inductive, and intensive coding approaches. Rapid analysis steps include:

• Create a neutral domain name aligned with the question.

• Create a summary template based on the interview script for the team.

• Test the summary template, compare, and revise and establish consistency across team members.

• Summarize transcripts.

• Transfer summaries to a respondent x domain matrix (e.g., using Excel).

Using matrices streamlines the process of identifying similarities, differences, and trends across participant groups (Averill, 2002).

If it is determined that continuing to traditional analysis will be helpful, NIOSH researchers will develop the coding system using an iterative approach with deductive and inductive processes (Miles, Huberman, & Saldana, 2020). The team will independently code a sample of transcripts and then come together to consensually refine codes, operational definitions, and coding rules. A second round with additional transcripts will help to test and refine the coding system. Once the coding system is established, we will conduct multiple coding cycles applying a variety of established codes (e.g., descriptive, In Vivo, process, concept, values).

After these steps of the analysis, NIOSH researchers will examine the data for within and cross-case differences.

Finally, findings will be integrated across data collection techniques to identify cross-method themes, contradictions, and lessons learned.

For quantitative data, NIOSH will conduct descriptive analyses to summarize the Likert items. If appropriate group differences will be assessed within employers as well as multi-level analysis if appropriate to identify similarities and differences across trainers and employers.

Finally, integrative techniques will be used to triangulate the findings and better understand employer and worker perspectives. The findings also will inform NIOSH product development and research to improve employer heat stress prevention.

5. Individuals Consulted on Statistical Aspects/Individuals Collecting and/or Analyzing Data

This information collection relies minimally on quantitative analyses. The project team listed in Table 2 has considerable experience conducting quantitative and qualitative evaluation. If additional assistance or guidance is required regarding data collection, analysis, or management, other internal resources are available in the division.

Study Personnel

Personnel	Title/Organization	Email	Roles
Kristin Yeoman MD, MPH	Medical Epidemiologist, NIOSH/SMRD	[email protected]	Evaluation design, recruitment, data collection, data analysis, reporting
Carol Nixon, PhD, MA	Research Health & Evaluation Scientist	[email protected]	PI, oversees study, medical officer, assists with recruitment, data collection, data analysis, reporting

References

Averill, J. B. (2002). Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qualitative Health Research, 2(6), 855-66.

Beebe, J. (2001). Rapid assessment process: An introduction. Rowman Altamira.

Hamilton A. B. (2020). Rapid qualitative analysis: Updates/developments. U.S. Department of Veterans Affairs. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/video_archive.cfm?SessionID=3846 

Hamilton, A. B., & Finley, E. P. (2019). Qualitative methods in implementation research: An introduction. Psychiatry Research, 280, 112516.

Miles, M. B., Huberman, A. M., & Saldana, J. (2020). Qualitative data analysis: A methods sourcebook, 4^th ed. Los Angeles: Sage.

12