Note to Reviewer - HSOII Reference Period

September 3, 2020

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Office of Survey Methods Research
SUBJECT:	Submission of Materials for “Investigating injury and illness reporting data quality across reference periods”

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 252 hours.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction

As part of its efforts to address concerns about underreporting in the Survey of Occupational Injuries and Illnesses (SOII), BLS is investigating the development of a household survey of workplace injuries and illnesses (Household Survey of Occupational Injuries and Illnesses, HSOII). The primary objective of the HSOII is to provide complementary measures on work-related injuries and illnesses.

In the SOII, establishment respondents provide information about the workplace injuries and illnesses of employees. Typically, a safety officer or other trained employee responds to the survey. In many establishments, these injuries and illnesses by law must be reported to the Occupational Safety and Health Administration (OSHA), and records of injuries and illnesses are updated throughout the reporting 12 month reference period.

In the HSOII, household respondents will be asked to report their own personal workplace injuries and illnesses. The respondents may not refer to records or documentation; we anticipate that most respondents will rely solely on recall from memory.

From a data quality perspective, we expect that shorter reference periods (e.g. 3 months) will result in higher quality data. If the injury or illness is recent, respondents will be more likely to recall details. However, the number of individuals in the population who have had an injury or illness within a shorter reference period is low, which drives up survey costs. Furthermore, reference periods may have other effects on injury and illness reporting, such as on the types of injuries or illness reported (e.g., respondents may report their most severe injury or illness). The survey program is interested in investigating data quality over a range of reference periods to inform a decision as to what reference period to use in the survey.

This study aims to understand whether the data quality for the injury and illness concepts measured by the HSOII varies by reference period. Within the context of the broader development of the HSOII survey¹, the scope of this study is limited only to the impact of reference period on recall from memory of injury and illness information; question wording and survey design will be developed in other studies. Participants from a non-probability panel with workplace injuries within a range of reference periods will answer HSOII questions. Analyses will explore whether participants had difficulty recalling the requested information and whether the distributions of HSOII measures vary by reference period. Although the data collected using a non-probability panel may not be generalizable to the target survey population, we expect that the comparisons between groups within this study will yield insight into which reference period provides the best balance of data quality and sample size. The findings from this study will inform our understanding of how reference period may affect data quality in the HSOII.

2. Methodology

Ideally, a study investigating the impact of reference period for workplace injuries and illnesses on data quality would randomly assign participants to answer surveys with different reference periods. However, given that we anticipate in-scope workplace injuries and illnesses will be very rare (up to 3.1% incidence rate in the 2018 HSOII pilot telephone survey²; 3.1% incidence rate in the 2018 SOII³), this ideal design is infeasible because it would result in significant loss of potential data. For example, if a participant had an injury 18 months ago but was randomly assigned to a survey with a shorter reference period, then he or she would be screened out of the survey.

Instead, an initial screening survey will be used to identify participants with workplace injuries and illnesses within the longest reference period of interest: the last 24 months (Appendix 1). Those participants will then be eligible to complete the main survey (Appendix 2).

Within the main survey, participants will be asked an injury or illness screener question with a randomly assigned reference period: 3 months, 12 months, or 24 months. The 3 month reference period represents a short reference period expected to yield high quality data. The 12 month reference period represents the current reference period used in the establishment SOII. The 24 month reference period represents a long reference period expected to lower survey costs. If the participant does have an injury or illness within the reference period to report, then he or she will enter a brief description for the injury or illness that they wanted to report. Then, these participants will be asked to briefly list all other workplace injuries or illnesses in that same reference period. From these two pieces of information, we will know all of the injuries and illnesses that were eligible to be reported and which one the participant chose to report on.

Participants without injuries or illnesses in the shorter reference periods who therefore respond negatively to the screener will then be asked whether they had an injury or illness in the next longest reference period. Participants initially asked about the last 3 months, will next be asked about the last 12 months; participants initially asked about the last 12 months, will next be asked about the last 24 months. Participants who respond positively will be asked to enter a brief description for that injury or illness they intend to report and then list all other workplace injuries or illnesses in that same reference period. This protocol continues until the participants reach the 24 month reference period, where participants who do not have an injury or illness within the last 24 months exit the survey. The figure above illustrates this design. This design reduces the loss of data that would have occurred under simple random assignment to reference period by still allowing data collection even after a negative answer to the randomly assigned screener. These data will be used to investigate the quality of data reported from the different reference periods.

After the screener questions are completed and eligibility has been determined by the researchers, the main survey will become available to eligible participants only. Participants must then choose to participate in the main survey. Those who participate in the main survey will be asked an abridged version of the HSOII questionnaire targeting the first injury or illness that was reported during screening. The questionnaire items will be used to investigate whether data quality is affected by reference period. For example, investigating whether participants reporting on injuries and illnesses from 24 months ago are unable to recall details of the nature of the injury or illness. Additional measures of interest include the approximate date of the reported injury or illness, why the participant selected this injury or illness to report on, and the participant’s confidence in the accuracy of responses. In addition, Questions 39 to 43 at the end of the questionnaire seek to capture information about any response difficulties. These probes are not asked concurrently through the survey because we are not targeting specific items for probing, and adding probes to every item would increase respondent burden significantly.

Prior to the administration of the main survey to all eligible participants, a group of 10 eligible participants who pass the screener will be asked to complete the survey as part of what we are calling the ‘pilot’ group. This pilot group of 10 participants will be asked to complete the full survey; however their data will not be included in the final analysis. Instead, their answers to Questions 39 to 43 will inform whether the question wording, response options, and instructions were clear and help us determine whether to make adjustments to the questions, response options, and instructions prior to launching the main survey for all eligible participants.

We expect that some modifications may be made to the main survey between the ‘pilot’ group and the administration to the remainder of eligible participants.

After modifications are made, if necessary, the main survey will be open to all remaining eligible participants. This is the set of participants from which the analysis will be done. Analyses will focus on comparisons between groups. Because participants sampled from a non-probability sample are not representative of the target survey population, we will not make inferences to incidence rates in the population. Instead, we will focus on internal validity and look for differences between reference period groups.

3. Participants

Participants will be recruited as a convenience sample from Amazon Mechanical Turk of adults (18 years and older) residing in the U.S.; this study is focused on internal validity rather than representativeness of any population.

This research design requires an initial sample of 4,000 participants in order to identify a sufficient sample of participants with workplace injuries or illnesses. This number is based on the estimated 3.1% overall rate of injuries and illnesses among full-time workers in a 12 month reference period⁴. An initial sample size of 4,000 is anticipated to yield a sample of 240 participants with an injury or illness in the last 24 months. The actual number of participants in the final survey is unknown and dependent on the response rate to the survey invitation and the number of participants with a workplace injury or illness, but we anticipate it will not exceed 300.

An additional 10 participants will be recruited for initial pilot tests of the questionnaire, also on Amazon Mechanical Turk.

4. Burden Hours

We anticipate that the initial screener survey will take no longer than 3 minutes, totaling 200 burden hours across 4,000 participants. Based on our own pre-testing of the instrument, we anticipate that the survey will take no longer than 10 minutes to complete. The main survey will require a total of 52 burden hours, across 10 pilot participants and 300 final data collection participants. In total, the study will require 4,310 participant-completes or responses (some participants will take both the initial screener survey and the main survey) and 252 burden hours. The surveys will be administered completely online at the time and location of the participant’s choosing.

# of Participants Screened	Minutes per participant for Screening	Total Screening Burden	Maximum number of Participants	Minutes per participant for data collection	Total Collection Burden	Total Burden (Screening + Collection)
4,000	3	200	300 + 10 (pilot)	10	52	252

5. Payment to Participants

We will recruit 4,000 participants from the Amazon Mechanical Turk database for the initial screener survey, for which participants will be paid $0.10 ($400.00 total). This is a typical amount for a screener survey of this length.

For the main survey collection, both pilot and final data collection participants will be paid $2.50 (estimated 310 participants, $775.00 total). This is slightly higher than a typical rate offered on the Amazon Mechanical Turk platform for similar tasks (e.g., $1.50). We propose the use of this higher incentive amount to help achieve the response rates we need to complete our analysis because past experience and this rare population suggest that might otherwise be difficult. In a 2016 study on the same recruitment platform, a follow-up survey (advertised as being of similar length to the current proposed survey but instead paying $0.75) yielded only a 65% response rate. In addition, with only a 3.1% prevalence rate (as previously discussed) in the general population, this population is rare.

A total of $1,175.00 will be paid directly to Amazon Mechanical Turk for participant fees.

6. Data Confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk. Once participants are recruited into the study, they will be given a link to the survey, which is hosted by SurveyMonkey. The data collected as part of this study will be stored on SurveyMonkey servers.

Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality. The duration of the survey will be displayed as either 3 minutes for the screener survey or 10 minutes for the pilot and main data collection surveys.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: March 31, 2021). Without this currently-approved number, we could not conduct this survey. We estimate that it will take on average [3/10] minutes to complete this survey. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

7. Attachments

Appendix 1: Initial screener survey questions

Appendix 2: Main survey questions – to be completed by both the ‘pilot’ group and the remaining eligible participants

1 BLS completed a pilot telephone HSOII survey in 2018 and, based on the findings from that study, has made several changes to the scope and design of the HSOII. At this time, the goal of the HSOII is now to collect data as a complement rather than as a replicate of the SOII. Additionally, BLS is pursuing a web mode survey rather than a telephone mode survey. Currently, the questionnaire is undergoing major revisions, of which this study is a part. Additional pre-testing of the HSOII questionnaire is planned, as time and resources allow. There is no target implementation date for the launch of the HSOII.

2 HSOII Pilot Analysis (2018). Presentation by Michie Dresser, OSHS.

3 Incidence rates of nonfatal occupational injuries and illnesses by industry and case types, 2018. https://www.bls.gov/iif/oshwc/osh/os/summ1_00_2018.htm

4 Incidence rates of nonfatal occupational injuries and illnesses by industry and case types, 2018. https://www.bls.gov/iif/oshwc/osh/os/summ1_00_2018.htm

3