Focus Groups

Appendix R:

Protocol for Task and Cognitive Task Analysis and Information Needs Assessments

Research Summary

Background

This protocol is a subtask for the NIOSH/OMSHR project Assessing and Evaluating Human Systems Integration Needs in Mining. This research will be conducted by members of the Cognitive Engineering (CE) Team in the Human Factors Branch (HFB) of the Office of Mine Safety and Health Research (OMSHR).

With the second highest fatal injury rate in 2010, mining remains one of the most dangerous occupations in the United States (Bureau of Labor Statistics, 2012a). Despite continued efforts in research and regulation, tragedies like Upper Big Branch (2010) and Sago (2006) still highlight a lack of consideration for human systems integration (HSI). HSI is “…a disciplined, unified, and interactive approach to integrate human considerations into system design to improve total system performance and reduce costs of ownership” (Cochrane & Hagan, 2001).

The lack of HSI consideration is becoming a greater problem as the adoptions of the MINER Act of 2006 as well as health and safety initiatives (End Black Lung Campaign) are trending towards mandate of the increased use of safety devices by mine workers (wireless communication systems, personal dust monitor, and proximity detection). These devices offer attractive health and safety benefits – improved tracking and communication, real time monitoring of respirable dust levels, and the prevention of accidental crushing by large mobile machinery. However, while the benefits of such wearable devices are easy to understand within their own context, they inevitably increase the burden on the mine worker who must carry, interact with, and ultimately make decisions with each one of the devices. The increased amount of information being presented by multiple devices divides the miner’s attention and may overload him or her. Overload and distraction may lead to a reduction in his or her understanding of the environment and ability to make informed decisions. It is our goal to understand what the miner has to do to perform his or her job, what information the miner has to have (or would like to have) in order to perform his or her job, and how the miner would like to receive critical information. All of this is necessary for the development of more intuitively usable interfaces.

In order to further understand what the miner has to do to perform his or her job, a task and cognitive task analysis will be conducted. The goal of the task and cognitive task analysis is to learn specific information about the tasks underground coal miners must perform during their job as well as when during a task critical decisions must be made and what factors affect the miners ability to do his or her job. In addition, two questionnaires will be used to assess what information mine workers think is necessary in order to perform their jobs, what information they would like to have that they currently are not getting, and their preferences for how this information should be presented to them. In the following protocol, the data collection procedure will be discussed and the three data collection instruments will be described.

RESEARCH TEAM

Tasks described in the protocol below will be performed by members of the team including (1) Brianna Eiter, PhD, (2) Jennica Bellanca, MS, (3) Justin Patts, BS, (4) Lisa Steiner, MS, and (5) Brendan Demich.

RESEARCH TASKS

Participants invited to take part in this study will all be miners currently working at an underground coal mine. These mine workers will either work as continuous miner operators or as a fire bosses (see Research Participants for a full description). Participants will be invited to take part in only one (1) of the following series of tasks: (1) Task and Cognitive Task Analysis, Subject Matter Expert Questionnaire and General Preference Questionnaire; (2) Subject Matter Expert Questionnaire and General Preference Questionnaire; (3) General Preference Questionnaire. The following is a brief description of the procedure that will be used for each of the aforementioned series of tasks.

• Task and Cognitive Task Analysis, Subject Matter Expert Questionnaire, and General Preference Questionnaire.

Data collection for this series of tasks will take place either at the mine where the mine worker is employed or at the Mining Technology and Training Center (MTTC, Ruff Creek, Pennsylvania). Continuous miner operators and fire bosses who are experienced (and considered experts) and who are less experienced (have fewer years mining experience, see Research Participants for a full description) will be asked to complete these three tasks. Prior to the start of data collection, participants will be asked to read and sign an informed consent (see Informed Consent and Appendix A). The Task and Cognitive Task Analysis will then be administered by a NIOSH researcher. This task is comprised of three (3) parts: the task diagram, the knowledge audit and the scenario interview (see Instruments for a full description and Appendix B). This task should take approximately 2 hours to complete. Once the Task and Cognitive Task Analysis is complete, participants will be given the Subject Matter Expert Questionnaire (see Appendix C). The NIOSH researcher will go through the instructions with the participant and then work with the participant to complete the questionnaire. We have chosen to administer this questionnaire in this way because there are a variety of questions included on this measure. In order to minimize confusion, we feel it is necessary to sit with the participant while he/she is working on the questionnaire. This questionnaire should take approximately 45 minutes to complete. Finally, participants will be given the General Preference Questionnaire (see Appendix D). The NIOSH researcher will go through the instructions for how to complete the questionnaire with the participant. The questions included on the General Preference Questionnaire are more straight forward; once the participant indicate that he/she understand the instruction then he/she will be left alone to complete this questionnaire. This should take approximately 45 minutes to complete.

• Subject Matter Expert Questionnaire and General Preference Questionnaire.

Data collection for this series of tasks will take place at the mine where the mine worker is employed or at the Mining Technology and Training Center (MTTC, Ruff Creek, Pennsylvania). Continuous miner operators and fire bosses who are experienced (and considered experts) and who are less experienced (have fewer years mining experience, see Research Participants for a full description) will be asked to complete these two tasks. Prior to the start of data collection, participants will be asked to read and sign an informed consent (see Informed Consent and Appendix E). Participants will first be given the Subject Matter Expert Questionnaire (see Appendix C). The NIOSH researcher will go through the instructions with the participant and then work with the participant to complete the questionnaire. We have chosen to administer this questionnaire in this way because there are a variety of questions included on this measure. In order to minimize confusion, we feel it is necessary to sit with the participant while he/she is working on the questionnaire. This questionnaire should take approximately 45 minutes to complete. Finally, participants will be given the General Preference Questionnaire (see Appendix D). The NIOSH researcher will go through the instructions for how to complete the questionnaire with the participant. The questions included on the General Preference Questionnaire are more straight forward; once the participant indicate that he/she understand the instruction then he/she will be left alone to complete this questionnaire. This should take approximately 45 minutes to complete.

• General Preference Questionnaire.

Data collection for this task will take place at the mine where the mine worker is employed or at the Mining Technology and Training Center (MTTC, Ruff Creek, Pennsylvania). We are not targeting specific job positions for this data collection; therefore, those mine workers invited to complete only the General Preference Questionnaire can work in any position in an underground coal mine. Prior to the start of data collection, participants will be asked to read and sign an informed consent (see Informed Consent and Appendix F). Participants will then be given the General Preference Questionnaire (see Appendix D). The NIOSH researcher will go through the instructions for how to complete the questionnaire with the participant. The questions included on the General Preference Questionnaire are more straight forward; once the participant indicate that he/she understand the instruction then he/she will be left alone to complete this questionnaire. This should take approximately 45 minutes to complete.

INSTRUMENTS

Task and Cognitive Task Analysis

For the purposes of this study, a modified version of the Applied Cognitive Task Analysis (ACTA, Militello & Hutton, 1998) method will be used. When compared to a more traditional CTA approach (for instance, Clark et al., 2008), approximately the same information is obtained from participants; however, the CTA is accomplished in less time with the participants and with less training for the interviewer.

Task and Cognitive Task Analyses will be broken into three parts: the task diagram, the knowledge audit and the scenario interview (see Appendix B for both the continuous miner operator and fire boss cognitive task analyses). Information obtained from a current NIOSH/OMSHR study [HSRB 13-OMSHR-03XP: “Information Needs of Underground Coal Miners”] will be used to guide the decision about which tasks to focus on during the task diagram (these tasks are therefore not currently identified on the Task and Cognitive Task Analyses, see Appendix B). Using the task diagram, NIOSH researchers will then ask the SME to identify which of the tasks and subtasks that require cognitive skill to perform (e.g. to make a judgment or decision, to problem solve or to multi-task). The goal of this is to encourage the miner to pinpoint the aspects of the job that are cognitively demanding and require judgments or critical thinking to occur. A knowledge audit will then be conducted to identify ways in which expertise is used in a domain and the audit is to capture the most important aspects of expertise while streamlining data collection. The tasks and sub-tasks discussed during the knowledge audit will be consistent with those used during the task diagram. During the knowledge audit, a series of probe questions are asked, probe questions are identical across both participants and job title – CM operators and fire bosses will be asked the same questions. Questions/probes will be used to begin a discussion about specific tasks. Additional unplanned questions will also be asked during the interview to solicit information about critical cues participants’ uses while performing a task as well as the strategies used and decisions made. The final step in the CTA analysis includes the use of a scenario interview. This task is based on the presentation of a challenging scenario to the participants. Two job specific scenarios will be given to each job title, participants will be asked to determine major events that occur within the scenario and to discuss the judgments and decisions he or she would make related to each of the identified major events.

Research Questionnaires

Two research questionnaires were created by NIOSH researchers for these tasks. These questionnaires provide a quantitative and qualitative means to collect information about tasks and subtasks performed by miner’s working in specific job categories. The questionnaires also allow for the assessment of the importance of providing different types of information to miners working in an underground coal mine. Questions on the questionnaires will be a combination of close and open-ended questions (see below for a fuller description of each questionnaire). Close-ended questions will require respondents to describe their opinions using choices that are provided by using a Likert-type scale. Open-ended questions will give the participant the opportunity to provide his or her opinion regarding a certain topic. This tool is effective in collecting data from a larger and broader sample of individuals than can be contacted through focus groups and observations alone. Three research questionnaires will be used. Below is a brief description of each questionnaire.

Subject Matter Expert Questionnaire

The goal of this questionnaire is to determine how subject matter experts (e.g., experienced continuous miner operators) and non-experts (e.g., less experienced miners) working as either continuous miner operators or fire bosses prefer to have information about their work environment, the location of themselves, others and equipment communicated to them while they are working. There are several types of questions included on the questionnaire: demographic questions, general preference questions, information on a daily basis questions, and information during an emergency questions. Demographic questions (e.g., total number of years working as a miner, number of years working in current position, etc.) are open ended questions and are included so that NIOSH researchers can determine the work characteristics of the sample as well as which mine workers are subject matter experts. General preference questions are based on a Likert scale (e.g., on a scale from 1 to 7, rate how helpful you find each of these options), and information on a daily basis and during and emergency questions are a combination of ranking options (e.g., put these options in order from most important to least important) and open ended questions (See Attachment F). There are a total of 41 questions on the questionnaire and it should take approximately 45 minutes to complete.

General Preference Questionnaire

The goal of this questionnaire is to determine how and when miners working in an underground coal mine prefer to have information about their work environment (e.g., gas, dust, and airflow levels), the location of themselves, others and equipment communicated to them while they are working. There are several types of questions included on the questionnaire: demographic questions, current information availability questions, and in an ideal world, information availability questions, and vitals preference questions. Demographic questions (e.g., total number of years working as a miner, number of years working in current position, etc.) are open ended questions and are included so that NIOSH researchers can determine the work characteristics of the sample. Current information availability and in an ideal world, information availability questions are based on a Likert scale (e.g., on a scale from 1 to 7, rate how critical this information is to you) and also multiple choice (e.g., choose only one option or choose all that apply). Vitals preference questions are a combination of yes/no questions and open ended questions. There are a total of 63 questions on the questionnaire and it should take approximately 45 minutes to complete (See Attachment E).

IRB ISSUES

RESEARCH PARTICIPANTS

All participants invited to take part in the tasks included in this protocol will be underground coal miners. Because there are three separate tasks included in this protocol, the research participants will be broken into three separate groups (or samples).

Task: Task and Cognitive Task Analysis

Individual interviews will be conducted with underground coal miners who work as either continuous miner (CM) operators or fire bosses. The goal is to interview CM operators and fire bosses with several years of experience, thus making them subject matter experts (SMEs) at their respective jobs, and CM operators and fire bosses with less experience, thus making them less experienced (or novices) at their respective jobs. SMEs have a different perspective, decision making process and way of performing their job compared to novices, or CM operators and fire bosses with less experience. By interviewing both SMEs and novices, we will define both the similarities and differences in thought and decision making processes that occur while these miners perform specific job tasks.

Research subjects will be currently employed coal miners working in the contiguous United States with the following criteria:

SUBJECT MATTER EXPERT MINER

1. Working more than 20 hours a week at an underground coal mine

2. Have at least 3 years of experience in their current position

3. Have at least 3 years of experience at their current mine company

4. Have at least 8 years of total mining experience

5. Have trained/apprenticed at least one other miner at current position

6. Have approval from the mine/safety director to engage in the study

OR

NOVICE (less experienced) MINER

1. Working more than 20 hours a week at an underground coal mine

2. Have recently been trained/apprenticed for current position – within a year

3. Have at least 3 years of experience at their current mine company

4. Have at least 5 years of total mining experience

5. Have approval from the mine/safety director to engage in the study

Task: Subject Matter Expert and General Preference Questionnaires

The fire bosses and continuous miner operators who take part in the Task and Cognitive Task Analysis will also be invited to complete the Subject Matter Expert and General Preference Questionnaires. Additional participants who meet the aforementioned criteria will also be invited to complete both questionnaires. Specifically, 30 new participants will be asked to take part in the task, 15 of those participants will be continuous miner operators (half subject matter experts and half novice) and 15 will be fire bosses (half subject matter experts and half novice). The same criteria listed above will be applied to determine which mine worker qualifies as a subject matter expert and which qualifies as a novice.

Task: General Preference Questionnaire

The goal of this task is to understand what information underground coal workers require on a daily basis, during an emergency, and also to begin to understand the mine workers’ perceptions of the use of vital signs (e.g., heart rate). In order to fully understand mine workers’ preferences and perceptions, it is necessary to not only collect data from fire bosses and continuous miner operators, but also other positions in the mine as well (e.g., shuttle car operators, scoop operators, roof bolters, etc.).

SUBJECT RECRUITMENT

NIOSH researchers will recruit subjects through safety directors and mine management working at mine sites as well as through the staff working at MTTC to ensure willingness to participate. Mine management will be approached through a phone call and potentially a follow-up email based on existing professional relationships. Staff at MTTC will also be contacted via the telephone and follow-up emails will be sent if necessary. Lisa Steiner, along with the other members of this team, has long-standing relationships with various mines and mining organizations (i.e. UMWA) across the country from which we can recruit. A script to be used in these initial contacts can be found in appendix G (there are 3 scripts included, one for each of the research tasks). If mines and the staff at MTTC are interested in participating, volunteers will be recruited that meet the criteria described above.

Individual subject recruitment will be initiated by mine management or the safety director working at a mine or through the staff at MTTC. NIOSH researchers will meet with the prospective subject(s) either at the mine site or at MTTC. NIOSH researchers will introduce the study (using the script in appendix H – there are three scripts included, one for each of the research tasks) and obtain their informed consent (appendix A, E, and F depending on the task the participants are completing) if they wish to proceed. A talent waiver will also be obtained to permit the capture of the subject’s voice and image (appendix I). If the talent waiver is refused, the study will still continue without the use of the voice recorder, and no pictures of the subject will be taken. No data will be collected underground. That means that data will be collected in an office, either in a mine safety representative’s office, the training office at the mine site, or at the MTTC facility.

Task and Cognitive Task Analysis: A maximum of 20 participants will be recruited for the Task and Cognitive Task Analysis – 10 continuous miner operators and 10 fire bosses. Half of the continuous miner operators (5) will be subject matter experts and half will be less experienced miners. Half of the fire bosses (5) will be subject matter experts and half will be less experienced miners.

Subject Matter Expert Questionnaire: A maximum of 50 participants will be recruited to take part in the Subject Matter Expert Questionnaire, 20 of those participants will have also taken part in the Task and Cognitive Task Analysis. The 30 additional participants will be continuous miner operators (15, half are subject matter experts and half will be less experienced miners) and fire bosses (15, half are subject matter experts and half will be less experienced miners).

General Preference Questionnaire: A maximum of 25 participants will be recruited to take part in only the General Preference Questionnaire. Miners working in positions other than continuous miner operator and fire boss will be recruited.

INFORMED CONSENT

NIOSH researchers will review the informed consent (see Appendix A, E, and F) with the participant (s) taking part in the study. The informed consent will be specific for the task the miner is asked to complete. The participant (s) will be given the option to participate. Upon agreement, the participants will then be given the option to take a copy of the consent form home with them for future reference. The consent form that will be used can be found in appendix C. The consent form is rated at a 12.1 Flesch-Kincaid Reading scale, which is acceptable since miners have a high school level of education. Following the obtainment of informed consent, the miners will also be asked to sign a talent release waiver to enable voice recording during the task and cognitive task analysis and subject matter expert questionnaire (appendix I). If the talent waiver is refused, the study will still continue without the use of the voice recorder.

BENEFITS AND RISKS

There is no direct benefit to the miners for participating in this study, other than the satisfaction of assisting with research that will influence the development of future interfaces in mining.

The activities of this study pose no more than minimal risk to human subjects.

INCENTIVES

A $25 gift certificate to a local retailer will be offered as an incentive for participation in this voluntary program. This amount is an appropriate compensation and does not provide undue incentive. An estimated 1 hour of burden time outside of normal working hours will be required to participate in this study, where approximately 45 minutes of the testing will actually occur outside of normal work hours, and we will ask questions during their typical breaks, which we approximate at 15 – 20 minutes of time. The median hourly wage for a continuous miner operators in coal is $24.98 per hour and the median of all construction and extraction occupations in coal is $24.41 per hour (Bureau of Labor Statistics, 2012b). Since our subject pool is expected to be in the upper range of these salaries due to their increased experience and status (training ability and value) and are being paid for their normal hours of work, we believe we are compensating them fairly for their participation. Subjects choosing to withdraw will be compensated according to their percentage of participation.

DATA MANAGEMENT

The photographs and notes collected from this study will be stored on an encrypted hard drive, on a password-protected computer, locked in the principle investigator’s (PI’s) office. Only the PI and key personnel will have access to the data. No personally identifiable information will be able to link notes with the participants.

ANALYSIS PLAN

The quantitative and qualitative data collected in this study will be used to build a more complete understanding of the tasks and subtasks underground coal miners perform, the decision making processes that occur while performing these tasks, the type(s) of information the mine worker requires to safely perform his job, and the means with which that information is delivered.

Task and Cognitive Task Analyses and Subject Matter Expert Questionnaires are interview based data collections. If the miner consents, these interviews will be voice recorded. Therefore, to begin data analysis, the voice recordings collected at the mine site and MTTC will be transcribed in-house and integrated with any handwritten notes. Then, the combined data will be separated by job title and task and experience level (subject matter expert and less experience miner). For the Task and Cognitive Task Analysis, critical decision points, tasks that require increased cognitive demand and other cognitive factors such as attention and cognitive workload will be identified. A task diagram will be created from this data. Additionally, comparisons will be made between responses elicited from subject matter experts and less experienced mine workers. Coded data will then be analyzed using summary statistics (mean, standard deviation, correlation) to quantify the importance and frequency of device use as well as the most frequent sources of problems. The unreduced qualitative data will also be examined for trends, root causes of problems, and any other implications of informational needs that surface.

The data collected using the General Preference Questionnaire will be first be recoded onto a master spreadsheet. This coded data will be analyzed using summary statistics (mean, standard deviation, correlation) to determine what information mine workers find most critical on a day to day basis as well as during an emergency. Because we are asking mine workers to provide information about the job they perform, we will be able to further separate this data by position to see if specific job titles require different types of information at different times.

All of the data from the Task and Cognitive Task Analyses, Subject Matter Expert Questionnaire, and General Preference Questionnaire, reduced and raw, will also be used to support the design of secondary experiments in the HSI project, such as the redesign of mining interfaces, the development of an integrated development interface and laboratory studies to test these interfaces.

SECURITY

There will be no personally identifiable links between the observational notes and the subject. The voice recording, release waivers, and consent forms that do contain personal information will be stored in a locked office and on a secure server, where only the PI and key personnel will have access.

EMERGENCY PROCEDURES

NIOSH researchers will receive “site-specific” safety training, which will identify the particular and unusual hazards (if any) that are unique to the mining facility. When data collection occurs at a mine site, mine management will review their safety and emergency evacuation procedures with NIOSH researchers before the tests are conducted. No specific procedures need be introduced for the miners who will volunteer as subjects. Mine employees will follow their normal emergency procedures. When data collection occurs at the Mining Technology and Training Center (MTTC, Ruff Creek, Pennsylvania), NIOSH researchers and participating mine workers will follow all safety procedures currently in place at the facility.

References

Bureau of Labor Statistics. (2012a). Census of Fatal Occupational Injuries in 2010. Washinton, DC.

Bureau of Labor Statistics. (2012b, March 27). May 2011 National Industry-Specific Occupational Employment and Wage Estimates Retrieved January 30, 2013, 2012, from http://www.bls.gov/oes/current/oes_nat.htm

Cochrane, C. B., & Hagan, G. J. (2001). Glossary: Of Defense Acquisition Acronyms & Terms. Fort Belvoir, Virginia: Defense Acquisition University Press.