Note to Reviewer - Developing self-administered questions to collect occupational injury and illness classification data

September 14, 2020

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu and Struther Van Horn Office of Survey Methods Research
SUBJECT:	Submission of Materials for “Developing self-administered questions to collect occupational injury and illness classification (OIICS) data”

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 255 hours.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction

As part of its efforts to address concerns about underreporting in the Survey of Occupational Injuries and Illnesses (SOII), BLS is investigating the development of a household survey of workplace injuries and illnesses (Household Survey of Occupational Injuries and Illnesses, HSOII). The primary objective of the HSOII is to provide complementary measures on work-related injuries and illnesses.

An essential part of the HSOII is data on the injury or illness itself (Nature, Part) and the cause of the injury or illness (Source, Secondary source, Event). In the SOII, establishment respondents provide this information on behalf of employees. Typically, a safety officer or other trained employee records details of the injury or illness in a safety log mandated by the Occupational Safety and Health Administration (OSHA), and reports the information to BLS as an open-ended text narrative about the injury or illness. In some cases, BLS or state partners re-contact the respondent for additional information. After data are collected, a coder assigns Occupational Injury and Illness Classification System (OIICS) codes¹. There are currently hundreds of OIICS codes organized by dozens of rules of code selection; accurate and reliable coding requires substantial training and review by expert coders.

In the HSOII, household respondents will be asked to report their own personal workplace injuries and illnesses. Entering open-ended text narratives about the injury or illness is likely to be burdensome for respondents to report and difficult for BLS to code accurately. Data quality of open-ends may also be poor because respondents will not have had any training with OIICS and may not include in an open-ended narrative the information needed to assign the best-fitting OIICS code. To address these concerns, BLS is interested in developing a self-administered method for collecting OIICS classifications directly from HSOII respondents whereby respondents can select from a list of predefined workplace injury and illness categories. Selecting from a list of predefined injury and illness categories would ease reporting and reduce burden but runs the risk of introducing measurement bias if the list does not fully capture injury and illness events and how respondents verbalize their injuries and illnesses.

Given the complexity of the OIICS as a multi-tiered classification structure with hundreds of codes, the program office acknowledges that the current system would be too burdensome for self-coding by untrained respondents. To address this issue, a simplified OIICS structure must be developed that uses plain language and targets a coarser level of detail.

The purpose of the research described in this clearance is to cognitively test the revised OIICS classification system to ensure that it holds together conceptually and is easy for respondents to understand and use. As part of the testing, we will also develop the questions that will guide respondents to apply the rules of selection so that they are focused on the relevant information for OIICS coding. After completing these preliminary studies, we plan to conduct additional pre-testing of the full process of respondents self-selecting the appropriate OIICS category. Within the context of the broader development of the HSOII survey², the scope of this study is limited only to the self-coding of the OIICS elements; question wording and survey design for the survey beyond the self-coded OIICS elements will be developed in other studies.

2. Methodology

Participants for the proposed testing will be recruited through Amazon’s Mechanical Turk (mTurk)³ and TryMyUI⁴ for think-alouds, card-sorting tasks, and navigation tasks. The various tests are listed below:

Study	Purpose
Study 1: Question wording	Understand how people think about workplace injuries and illnesses and how to target the information needed for OIICS.
Study 2: Card Sort of OIICS categories	Determine how well exemplars fall into OIICS categories.
Study 3: Navigation of OIICS categories	Determine how to order OIICS categories (developed in Study 2) and display any supplemental information.

We expect that some modifications may be made to the instruments or protocols during the course of the study based on initial feedback. For example, some existing categories may be renamed, whereas others might be consolidated or reordered. However, the goals of the testing, overall design, and scope of the content will remain the same.

Study 1: Question wording

This study focuses on understanding how people think about workplace injuries and illnesses and how to target the information of interest. This study will be used to develop question wording for a self-administered method to collect OIICS information (nature, part, event, source and secondary source).

Participants will be first asked to provide a brief description of their own workplace injury or illness. This will be used to identify the type of information and level of detail participants find salient when describing their own injury or illness.

Participants will then be given a randomized set of 10 written vignettes that briefly describe the work situations and workplace injuries or illnesses of different fictional characters (Appendix 1). The vignettes were designed to use the most common natures, parts, events, and sources based on 2018 SOII data. A subject matter expert from the program office reviewed the vignettes to ensure they are realistic and relevant and coded the vignettes to provide a reference for assessing participant accuracy. Participants will then be asked a series of questions about each vignette, using ‘think-aloud’ techniques, to see if they can correctly identify the OIICS elements based on the current question wording (Appendix 2). If participant responses do not match the subject matter expert codes, that is an indication that the question wording is not focusing participants on the target information needed. We will use that feedback to revise the question wording to redirect participants.

Study 2: Card Sort of OIICS categories

This study focuses on the response options, or OIICS categories themselves, to determine how well exemplars fall into OIICS categories. Card sorting will help determine how well the revised OIICS captures injury and illness data. Card sorting will also help identify areas of conceptual confusion.

Participants will sort a list of injury or illness information into predefined categories (based on OIICS categories) using Optimal Workshop software⁵. They will also have the option of creating and naming new categories if they believe the current set of categories is inadequate. Participants will each do separate card sorts for each OIICS element (Part, Nature, Event, and Source). The protocol for the card sorting study, including the injury and illness examples to be sorted into categories and debriefing questions, is included in Appendix 3. A revised set of injury and illness categories, based on the planned redesigned OIICS^6,7, is included in Appendix 4. As testing progresses, the list of predefined categories may change based on feedback from participants.

Card sorting is useful for discovering how people expect to see information categorized and also how they differ in their understanding of different concepts. Results from this study will help us to understand which OIICS codes are consistently misplaced or placed in newly created categories and which OIICS codes are consistently placed together in the correct category; feedback from this study will be used to restructure and rename categories in order to be usable for respondents.

Study 3: Navigation of revised OIICS codes

This study will focus on how to order OIICS categories (developed in Study 2) and display any supplemental information to ensure that a participant can find the OIICS category they are looking for. Correctly categorizing injury and illness information will be a critical step in the completion of a self-administered survey. The navigation, or tree testing, method will tell us how easily people can categorize injury and illness information using the revised OIICS classification system.

Participants will navigate the list of OIICS categories resulting from Study 2 to select a code that they believe best matches a given injury or illness nature, part of body, source, or event. We plan to administer this study using Optimal Workshop (the same platform as used in Study 2 for the card sorting task) as a traditional tree test. However, if we find high error rates (i.e., many participants classifying things in the wrong category) then we plan to switch to the SurveyMonkey platform in order to add concurrent probes, which are not possible to program on the Optimal Workshop platform.. These probes will target the specific OIICS elements that appear to have problems. The protocol for this study, including debriefing questions and probes, is included in Appendix 5.

This study will provide insight into whether the categories make sense to people, whether content is grouped logically, whether the presence of an “other” or “unknown” category has an effect on data quality, and whether people can find the category they want easily and quickly. Results from the study will be used to refine how the OIICS classification system is displayed in the self-administered HSOII.

3. Participants

Study 1

For the TryMyUI testing, participants will be recruited as a convenience sample from TryMyUI of adults (18 years and older) residing in the U.S; this study is focused on internal validity rather than representativeness of any population. We will recruit 20 participants for the question wording task. Participants will be screened for having had a workplace injury or illness. If we are unable to recruit the target number of these participants from the TryMyUI population, we will screen for participants who have recently done work for pay. TryMyUI presents the screening criterion as a question that the participant has to read and answer aloud before proceeding to the description of the study. The screener survey questions are included in the protocol (Appendix 2).

Studies 2 and 3

Participants will be recruited as a convenience sample from Amazon Mechanical Turk of adults (18 years and older) residing in the U.S.; this study is focused on internal validity rather than representativeness of any population. We will recruit up to 110 participants for the card sorting task (Study 2) in order to have sufficient sample to identify patterns in categorization decisions and allow for repeat testing after making revisions based on early feedback. We will recruit up to 625 participants for the navigation task (Study 3) in order to have sufficient coverage of the wide range of categories and types of injuries and illnesses (any single participant completes only a subset of 10 vignettes) and to allow for repeat testing after making revisions based on early feedback.

To ensure that higher quality mTurk participants are recruited, a brief screener survey will be used to exclude participants that do not meet data quality standards; that is, participants who give nonsense answers to an open-ended item such as participant’s job title (e.g., responses such as “good” or “thank you”) will not be able to participate in the main survey. The same screener survey will be used to recruit participants for both Studies 2 and 3, though a participant will only be allowed to participate in one study. The screener survey questions are included in Appendix 6.

While screeners typically aim to identify a certain demographic or characteristic, this screener is focused on data quality. The effectiveness of the screener method, however, is unknown. We propose embedding an additional study to learn more about the effectiveness of using data quality screeners for collecting higher quality data in a web survey using a non-probability panel. We will collect navigation task data (Study 3) from an additional 100 participants beyond what is needed for the substantive HSOII study. These 100 participants will be recruited from the subset of participants who fail the data quality screener. Researchers will blindly evaluate the data quality of all participants and compare the findings of those who pass and those who fail the data quality screener.

4. Burden Hours

These studies will require a total of 255 burden hours.

Study 1

The survey will take no longer than 30 minutes, totaling 10 burden hours across 20 participants. This time includes screening, which consists of the participant explaining aloud how they satisfy the screening criteria and is only recorded for participants who complete the task. The surveys will be administered completely online at the time and location of the participant’s choosing.

Studies 2 and 3

We anticipate that the initial screener survey will take no longer than 2 minutes, totaling 35.3 burden hours across 1058 participants. We expect to need to screen 1058 participants to get 110 participants for Study 2 and 625 participants for Study 3. The additional 100 participants for the study of screener effectiveness will be recruited from the subset of participants who fail the screener, and do not affect the calculations for total number of participants to be screened. Study 2 will take no longer than 15 minutes per participant for a total of 27.5 burden hours. Study 3 will also take 15 minutes and total 181.3 burden hours. The surveys will be administered completely online at the time and location of the participant’s choosing.

Respondents	# of Participants Screened	Minutes per participant for Screening	Total Screening Burden	Maximum number of Participants	Minutes per participant for data collection	Total Collection Burden	Total Burden (Screening + Collection)
Study 1 (TryMyUI)				20	30	10.0 hours	10.0 hours
Screener - mTurk (mTurk)	1058	2	35.3 hours				35.3 hours
Study 2 (mTurk)				110	15	27.5 hours	27.5 hours
Study 3 (mTurk)				725 (625 + 100)	15	181.3 hours	181.3 hours
Total							254.1 hours

5. Payment to Participants

Study 1

Participants will receive $10 through TryMyUI, as mandated by the TryMyUI platform. A total of $200.00 (20 participants) will be paid directly to TryMyUI for participant fees.

Studies 2 and 3

We will run an initial screener survey on Amazon Mechanical Turk for which participants will be paid $0.10 (1058 participants, $105.80). For the longer main data collection, participants will be paid $2.00 (835 participants, $1670.00 total). These payments are typical for the platform. A total of $1775.80 will be paid directly to Amazon Mechanical Turk for participant fees.

6. Data Confidentiality

Study 1

Participants will give their consent through the TryMyUI service. In addition, we will include the following text in the introduction:

Please read the following silently to yourself:

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 30 minutes to complete. This study is being administered by TryMyUI.com and SurveyMonkey and resides on a server outside of the BLS Domain. BLS cannot guarantee the protection of responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Screener Survey for Studies 2 and 3

Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the survey, they will be given a link to the survey, which is hosted by SurveyMonkey. The data collected as part of this survey will be stored on SurveyMonkey servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently-approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. We estimate that it will take on average 2 minutes to complete this survey. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Studies 2 and 3

Participants who meet the eligibility criteria will be able to complete the main task in either Study 2 or Study 3. Participants will be given a link to the survey. We plan to use the Optimal Workshop software for both Studies 2 and 3. However, if necessary, we may need to change platforms for Study 3, in which case the pledge will be changed to SurveyMonkey. The data collected as part of this study will be stored on either Optimal Workshop or SurveyMonkey servers. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently-approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. We estimate that it will take on average 15 minutes to complete this survey. This survey is being administered by [Optimal Workshop/SurveyMonkey] and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

7. Attachments

Appendix 1: Vignettes

Appendix 2: Study 1 protocol

Appendix 3: Study 2 protocol

Appendix 4: OIICS

Appendix 5: Study 3 protocol

Appendix 6: Screener survey for mTurk

Appendix 1: Vignettes (Study 1, Study 3)

Number	Vignette Text
1	Miriam works in a veterinary clinic in a rural area as a veterinarian. Miriam was walking to the computer to enter notes after a surgery on an animal. One of the veterinarian assistants spilled a sample from an animal on the floor, which Miriam slipped in. Miriam fell onto the floor, landing on her wrist, resulting in a sprained wrist.
2	Tina is an operator of a cutting machine in a large printing plant. One day, as Tina was carrying materials to her machine, Tina caught the sleeve of her shirt on a shelving unit in the plant, causing her arm to bump against the shelving unit, hard. Tina’s arm has bruising and a cut where it hit the shelving unit.
3	William is a janitor for Modern Family Realtors, a local business with a large office building. He was doing maintenance on the 3rd floor. While on a ladder changing a lightbulb, William fell to the ground. William hit his head on the ground and suffered a concussion.
4	Marcella is a graphic design artist for a large printing company in a metropolitan area. The printing company sent Marcella out in a company car to meet with potential clients. While stopped at an intersection, another vehicle hit the company car. Marcella’s neck was injured when she suffered whiplash during the impact of the accident.
5	Edwin is a residential interior/exterior painter for a family business, Johnson & Sons Painters. Due to the repetitive nature and strain of holding a paint brush and painting, his wrist starts to hurt. Edwin visits the doctor and he is diagnosed with tendonitis.
6	Jackie is a cashier at a local organic garden, who helps water and sell seasonal plants and vegetables. While watering the plants, Jackie slips and runs into a wheelbarrow used to transport plants. Due to running into the wheelbarrow, Jackie suffers bruising to her stomach
7	Jeff is a lawyer for an innovative software corporation. He was assisting the company’s intellectual property lawyer to prepare and file software patents. A storm caused the power in the building to surge, which led to his work computer electrocuting him.
8	Randall is a chemist-toxicologist specialist at an independent testing laboratory. A co-worker next to Randall accidentally mixed together chemicals that led to a chemical explosion, which included a fire. Due to the explosion and fire, Randall ended up receiving burns on his hands as well as inhaling smoke.
9	Regina is a security officer at a private security agency, who helps guard a local building overnight. While on the job she witnessed a co-worker getting into a physical altercation with someone trying to break into the building. This led to her co-worker losing their life. After witnessing the event, Regina was diagnosed by a mental health professional with post-traumatic stress disorder.
10	Harry is a statistician who works with a group of other statisticians in a federal agency. While taking the stairs to his office, Harry tripped and fell down a few stairs, landing on his backside. As a result of the fall, Harry’s tailbone was bruised, which caused him difficulties sitting.

Number	Vignette Text
11	Stephanie is Executive Director of a city Police Athletic League. Stephanie is responsible for coordinating a variety of youth development programs. While helping demonstrate some soccer moves for a youth soccer league, Stephanie fell into a hole in the ground in the soccer field. Stephanie ended up with a sprained left ankle due to twisting her ankle in the hole.
12	Arnold is a carpenter for a custom cabinetry company. While working on building a cabinet, Arnold is struck by a hammer that was dropped by a co-worker. The hammer hit Arnold on his face, causing his lower jaw to become fractured.
13	Nicholas performs general maintenance and repairs for a school. While repairing the school’s boiler system, Nicholas’s pant leg becomes caught on a bolt that is sticking out of the boiler door. As Nicholas tried to remove his pants from the bolt, his leg brushed against the boiler causing a second degree burn on his thigh.
14	Helen is a therapist who was helping local schools prepare and implement Individualized Education Program (IEP) plans for children in need of speech therapy. While lifting a child into a seat for testing, Helen strained her back. When Helen went to the doctor, she was told that she tore ligaments in her lower back, which was causing the pain in her lower back.
15	Philip is a trained nurse working in a rural hospital. While helping to sedate a patient who was injured, the patient became agitated and struck Philip in the face. The patient’s hand and fingernail ended up catching Philip in the eye, causing a scratch to his cornea.
16	Therese is a senior architect at the local city’s building and zoning agency. Therese needed to deliver paperwork to another city government building. As she was walking across the street to the building, she was struck by a car. As a result of being struck by the car, Therese cracked several of her ribs and suffered lacerations and bruising to her torso.
17	Marvin works full time collecting water samples and measuring air quality for his state environmental agency. While collecting air samples, Marvin ended up inhaling toxic fumes. Afterwards, Marvin developed lung cancer, which doctors attributed directly to his inhaling toxic substances at work.
18	Lawrence is an automotive technician who maintains or repairs the vehicles (police, fire, ambulance, etc.) owned by his town in an automotive repair shop. While getting ready to repair a car, the auto lift failed and pinned Lawrence to the ground. As a result of being struck by the auto lift, Lawrence suffered a fractured pelvis.
19	Fred is an attorney specializing in taxation and investment. He works for a law firm, with multiple other attorneys. While retrieving office supplies from the supply closet, a loose shelf struck Fred in the nose, causing a nose bleed.
20	Harriet drives semi-trucks for a shipping company. She drives a company-owned semi-truck when she is working. Helen was in an accident with another vehicle. The airbag deployment and impact of the accident caused Helen to sprain her right wrist.

Appendix 2: Study 1 protocol

Screener Question: Have you ever had a workplace injury or illness? [if necessary: Last week, did you do any work for pay?]

Screener Instructions: Please state out-loud how you meet the requirements by providing a brief description of the workplace injury or illness. [if necessary: …by providing a brief description of your job title.]

General Statement Information: Part of the mission of the Bureau of Labor Statistics is to collect information on work-related injuries and illnesses. This study is part of our research to improve a survey on workplace injuries and illnesses.

Please read the following silently to yourself:

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 30 minutes to complete. This study is being administered by TryMyUI.com and SurveyMonkey and resides on a server outside of the BLS Domain. BLS cannot guarantee the protection of responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Scenario Text: You will be asked to read brief scenarios of workplace injuries and illnesses of people we’ve made up and given first names to. After reading about each scenario, please answer a few questions about the injury or illness. We want to learn how people like you might answer questions like these on a real survey and so your own personal answer is the one we want, just based on what each scenario tells you. There are no “right” or “wrong” answers to any of the questions. The purpose of this research is to get feedback to improve question wording by making the questions as clear and easy to understand as possible, so please give us your honest feedback as you work through the tasks. Please think aloud as you answer each question. Please proceed to the following link (insert Survey Monkey link).

Remember, please think aloud as you answer each question. Please tell us if you have any difficulty answering a question, and explain why.

<Page break>

First, please think about your own workplace injury or illness.

<Page break>

Q1. What event led to the injury or illness?

<Page break>

Q2. What object or substance directly harmed the employee?

<Page break>

Q3. What would the medical diagnosis if the employee went to see a doctor, or in other words, what is the nature of the injury or illness?

<Page break>

Q4. Were multiple parts of the body injured?

<Page break>

[If Yes to Q4]

Q5a. If yes, which parts of the body were injured?

Q5b. If you had to choose just one part of the body to say was injured, which part would you choose and why? Please describe how you came to that answer.

Q5c.Was one part of the body was more severely injured than others?

[If No to Q4]

Q6. Which part of the body was injured?

<Page break>

Now, please read the read the following scenario silently. In your own words, can you summarize this scenario?

<Each vignette will remain on display as the following questions are asked for each vignette. We will randomize 10 vignettes per respondent>

<Repeat questions 1-6 as above for each vignette>

Post-Test Survey:

Q1: Were there any unfamiliar terms used in the questions we asked or in the scenarios? If so, what were those terms?

Q2: Thinking about your own workplace injury or illness, would you be able to answer these questions about your own workplace injury or illness? Why or why not?

Q3: How long ago did your injury/illness occur?

Q4: Any suggestion for changes to the questions we asked?

Appendix 3: Study 2 protocol

Welcome

This study is part of our research to improve a survey on workplace injuries and illnesses.

In this task you will be asked to sort workplaces injuries or illnesses into categories. If you believe none of the existing categories are appropriate, you can create and name new categories.

The study should take about 15 minutes. Please only start the study when you will be able to complete the whole study without interruption.

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently-approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. We estimate that it will take on average 15 minutes to complete this survey. This survey is being administered by Optimal Workshop and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

<page break>

Instructions

Take a look at the list on the left.

We’d like you to sort the items on the left into the groups on the right.

There is no right or wrong answer. Just do what comes naturally.

Click “Finished” at the top right corner of the screen when you’re done.

<page break>

Card sorting task – Each participant will be asked to complete two of the below card sorting tasks. The Groups are based on the higher-level OIICS category groupings (Appendix 4). The Items are examples of concepts from the most detailed level of OIICS categories, with variations to labels to reduce as much as possible any superficial similarity between items. The order of the items will be randomized when displayed to participants.

Task	Groups	Items
Nature	Physical injury Infectious or parasitic disease Tumor or cancer Other disease or disorder of the body Mental health disorder Other or unknown	Brain injury Injury to an internal organ Hernia Damage to a prosthetic or medical device Effect of exposure to poison, toxin or allergen Bacterial disease Fungal infection Cancer or tumor Disease or disorder affecting brain or spinal cord Cataract (causes blindness) Heart disease Asthma Lung cancer Stomach ulcer Blood clots Contact dermatitis Carpal tunnel syndrome Multiple symptoms but no diagnosis Abnormal results but no diagnosis Did not receive any test results or diagnosis
Source	Vehicle Ground, walkway, or stairs Body of water Person, plants, or animal Object, including machine, equipment, or materials Structure or part of a structure Hazardous or infectious materials, including fumes Other or unknown	Aircraft Industrial vehicles Unattached vehicle parts Byproduct of plant or animal Container, including tank, barrel, and box Furniture and fixtures Portable ladders and stairs Recreation and athletic equipment Audio/Visual equipment Medical and surgical instruments and equipment Measurement and scientific devices Office equipment Adhesives, sealants, and paints Door, window or other structural part of the building Confined space, including elevators Building system, including electrical or security House, office, or retail building Confined space, including tunnels and mines Explosives Gases Flammable and combustible liquids and solids Drugs General consumer chemical products
Event	Transportation Explosion or fire Violence or harm by a person Slip, trip or fall Exposure Impact or contact Overexertion including fainting Other or unknown	Incident involving a water vehicle Incident involving a non-motorized vehicle Intentional injury by self Witnessing a violent act Slip, trip or stumble without a fall Exposure to radiation and noise Exposure to temperature extremes Exposure to air or water pressure change Exposure to harmful substances Exposure to low oxygen levels without harmful substances Exposure to a stressful or traumatic event Trapped in a collapsed space Struck by a moving object Contact with an object, including machines or doors Contact with a person by accident Extended vibration, friction, pressure, or jarring Moving an object, including lifting, pushing and steering Position of the body, including sitting and standing Repeated motion Illness or other condition of the body
Part	Head Neck or throat Torso (mid-section) Shoulders, arms or hands Hips, legs or feet Prosthetic or other medical device Other or unknown	Brain Skull or scalp Face, including eyes, nose, mouth Ears Throat or other part of the inside of the neck Outside of the neck Chest, including ribs, internal organs Back, including spine, spinal cord Abdomen Pelvic region Shoulders or collarbone Arms Wrists Hands Hips Legs Ankles Feet Functioning of the entire body, such as cardiovascular system

<page break>

Feedback

Were any of the terms used in the task unfamiliar or confusing to you? If so, please tell us which ones.

[open text entry]

Did you have any difficulty completing the task? If so, please describe why it was difficult.

[open text entry]

What is your mTurk Worker ID number?

[open text entry]

Thank you! Please copy the code 5681354 and paste it into the mTurk HIT window to validate that you have completed this study.

Appendix 5: Study 3 protocol

Welcome

This study is part of our research to improve a survey on workplace injuries and illnesses. You’ll be helping us to develop a classification system.

You will be asked to read brief descriptions of workplace injuries and illnesses – for example, what happened to the worker or what caused the injury – and then select categories from a list to describe what happened.

The study should take about 15 minutes. Please only start the study when you will be able to complete the whole study without interruption.

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently-approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. We estimate that it will take on average 15 minutes to complete this survey. This survey is being administered by [Optimal Workshop/SurveyMonkey] and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

<page break>

[Optimal Workshop Protocol]

Instructions

Here’s how it works:

• Read the information presented to you about a workplace injury or illness, like “slipped and fell on the floor” or “heavy box”.

• Click through the lists to find the category that fits best. Keep drilling down through the lists selecting the best option until the message “I’d find it here!” appears and then click on it.

• If you go down the wrong path, you can go back by clicking one of the preceding links.

[SurveyMonkey Protocol]

Instructions

Here’s how it works:

• Read the information presented to you about a workplace injury or illness, like “slipped and fell on the floor” or “heavy box”.

• Click through the lists to find the category that fits best. Keep drilling down through the lists selecting the best option.

• If you go down the wrong path, you can go back by clicking the “Go Back” button.

<page break>

[Tree testing task – a selection of up to 16 of the below items will be displayed to a given participant. The number of items displayed per participant will be reduced if concurrent probes are added or the task takes longer than the anticipated 15 minutes.]

Tree	See Appendix 4

Task Set	Item
Nature	Broken arm Fainting Paralysis, or loss of ability to move one side of the body Whiplash, or sudden neck strain Mosquito bite Stroke Burns and smoke inhalation Choked on food Carpal tunnel syndrome Low back pain Nose bleed Anxiety
Source	Crate Wall Wheelbarrow Light bulb Self, falling Other, falling Loading dock Hole in ground Knife Car, as a pedestrian Driveable airport stairs
Event	Hit by a nail from a nail gun Carrying heavy boxes Stress from witnessing a fight Jump from a moving truck Slipped on ice but didn't fall Fall from a chair Fall out of a stationary truck trailer Toe run over by a powered wheelchair Dust flying in the air Clothing caught in machine Transferring a patient to a bed
Part	Eyes Lower back Brainstem Jaw Stomach (from a digestion disorder) Tailbone Stomach (from a punch in the stomach) Groin

<page break>

If necessary, add probes at the start and end of an OIICS element navigation task (e.g., before beginning to navigate through the Nature categories, after selecting the final category for Nature) to understand possible causes for misclassification:

Before navigating through the tree categories:

What information from the vignette will you try to classify for this question?

[open text entry]

After selecting a final category:

Did you find the category that you were looking for?

Yes

No

[if no] Please briefly describe how you made your final category selection.

[open text entry]

<page break>

Feedback

Would you have liked to see examples of what kinds of injuries or illnesses are included in or excluded from a category?

Yes

No

No opinion

Were any of the terms used in the task unfamiliar or confusing to you? If so, please tell us which ones.

[open text entry]

Did you have any difficulty completing the task? If so, please describe why it was difficult.

[open text entry]

What is your mTurk Worker ID number?

[open text entry]

Thank you! Please copy the code 4545528 and paste it into the mTurk HIT window to validate that you have completed this study.

Appendix 6: Screener survey for mTurk

This screener survey will be used to determine eligibility for Studies 2 and 3. The open-ended items will be used to identify fraudulent responses (i.e., participants giving nonsense answers indicating poor data quality) and those participants will be excluded from the main survey.

Welcome! Thanks for your interest in this survey.

This is a short screener survey to identify individuals who are eligible for our main survey. The main survey will:

• Launch on mTurk next week

• Take about 15 minutes

• Pay $2.00

• Be available only to those workers who qualify through this screener survey

This survey should take about 2 minutes.

Please do not use your browser's back button. 

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently-approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. We estimate that it will take on average 2 minutes to complete this survey. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

<page break>

Please do your best to answer the questions accurately. We are looking for a range of participants. Answering “no” to any of the below questions will not necessarily mean you are ineligible for the main survey.

1. What is your mTurk Worker ID?

[Open text entry]

2. What is your age?

[Open text entry]

3. What is your sex?

Male

Female

4. Are you Hispanic, Latino or Spanish?

Yes

No

5. What is your race? Please select all that apply.

White

Black or African American

American Indian or Alaska native

Asian

Native Hawaiian or Pacific Islander

6. Last week, did you do any work for pay?

Yes

No

<page break>

If Q6 = Yes then

7. Thinking of your main job, what kind of work do you do, that is, what is your occupation?
   Do not include any names or personal information.

[Open text entry]

8. What was the most burdensome single HIT you have ever completed? Please briefly describe the task.

Do not include any names or personal information.

[Open text entry]

Thank you for participating! Here is your completion code. Please paste this code into the HIT window to verify your participation.

[random numeric code]

1 Occupational Injury and Illness Classification Manual v2.01 (2012). https://wwwn.cdc.gov/Wisards/oiics/Doc/OIICS%20Manual%202012%20v201.pdf

2 BLS completed a pilot telephone HSOII survey in 2018 and, based on the findings from that study, has made several changes to the scope and design of the HSOII. At this time, the goal of the HSOII is now to collect data as a complement rather than as a replicate of the SOII. Additionally, BLS is pursuing a web mode survey rather than a telephone mode survey. Currently, the questionnaire is undergoing major revisions, of which this study is a part. Additional pre-testing of the HSOII questionnaire is planned, as time and resources allow. There is no target implementation date for the launch of the HSOII.

3 https://www.mturk.com/

4 https://www.trymyui.com/

5 https://www.optimalworkshop.com/

6 The OIICS coding structure is undergoing a comprehensive revision to include new and emerging workplace conditions, capture additional case characteristics, and make other improvements (https://www.federalregister.gov/documents/2017/10/13/2017-22188/comment-request).

7 Given that the fielding date of the HSOII is likely to be after the implementation of the redesigned OIICS, we used the latest draft available of the redesigned OIICS for this study. We anticipate that any changes to the revised OIICS will be minor; however, if there are major changes, the self-coded version of the OIICS may need to be re-tested.

21