Study Limitations

Attachment L-5:

Study Limitations

Selection and Recruitment Bias: To judge selection bias, the prior and future loss histories of three groups will be compared:

• establishments that choose not to participate (or do not respond to solicitation)

• establishments who agree to participate but are not chosen for participation

• establishments who participate

Within establishments, the injury/ illness data will be compared to characterize differences between individuals who participate versus those individuals who choose not to participate or who decide to remove themselves from the study.

Addressing Threats to Internal Validity: The randomized multiple baseline experimental design for this study will address many but not all threats to internal validity inherent in non or quasi experimental before/ after studies that have been commonly used in intervention research. Below is a summary how this study will address common threats.

Selection Threat: A selection threat may occur when the apparent intervention effect could be due to other differences between study groups. A randomized design in this study will be used to ensure that the control and intervention groups are similar, especially for variables (such as establishment size and prior loss history) that can affect the measured outcomes.

History Threat: A “history threat” arises when events which are external to the intervention, but which still may impact outcome, occur after the intervention is put into place. The main advantage of the multiple baseline design proposed for this study is that it reduces potential history threats. With a basic control group design, intervening factors could impact outcomes rather than the intervention. If the introduction of the intervention is staggered and the observed effects on outcomes are also staggered, history effects as an explanation are unlikely. Even though the multiple baseline design approach is used, history effects are still possible. To address history effects further, a detailed log of intervening controls, and other changes to management personnel, work processes, structure or pace, and management labor relations will be collected at both control and intervention establishments during establishment interviews at all measurement periods (pre-intervention, post implementation, post annual, 2^nd post annual). As well, a number of factors external to the establishment will be tracked during the study periods, including changes in state or national legislation (especially those that impact OSH record keeping and the Ohio workers compensation system) and the general business cycle.

Diffusion Threat: A diffusion threat occurs when the intervention provided to one experimental group “diffuses” to the other. This can commonly occur if the intervention study is conducted within the same company (e.g. placing a control in one line but not another). In this study, this threat will be reduced because the interventions will be placed in entirely different establishments. Rivalry or resentment threat: A rivalry threat can occur when participants in the control group react in some way to not receiving the intervention. In this study, this threat will be reduced because the interventions will be placed in entirely different establishments. As well, all participating companies will eventually receive the intervention.

Instrumentation/ Reporting: An instrumentation threat may arise anytime that the methods for measuring outcomes changes during the course of the study. This threat will be reduced in this study by ensuring that the same protocol for outcome measurements (assessed by questionnaires and workers compensation data analysis). For example, for workers compensation data outcomes, it is possible that overall changes in the workers compensation system could effect case definitions (especially for lost time cases). As mentioned in the section on history threats section, a diary of any workers compensation system changes will be kept. Substantial compensation changes require advance notification to stakeholders. It is likely with this advance notification that compensation outcomes can still be calculated the same way even if changes to the overall system occur.

Regression to the Mean: This threat occurs when the intervention involves groups with high injury rates and the observed decrease in rates is the natural tendency of rates to return to an average level rather than an effect of the intervention tested. The randomized multiple baseline design with blocking in this study will address this issue by ensuring that control and intervention groups are similar in pre-intervention injury rates. Regression to the mean will also be addressed by assessing the WC injury rate for two years, pre- and post-intervention on a quarterly basis (four times per year). In this way, the WC outcome is not a single data point but rather a rate trend.

Testing Threat: A testing threat may occur when the measurement process itself affects outcomes. This can arise especially with any measurements that require establishment and employee participation (such as onsite risk factor assessments, questionnaires). In this study, this threat will be reduced because the intervention and control groups will be treated in similar ways. For example, although interventions will not be placed into the groups at the same time, the same outcomes measures, individual questionnaires, and establishment questionnaires will be collected during onsite visits scheduled (pre-intervention, post implementation, post annual, 2^nd post annual).

Placebo (Hawthorne) Effect Threat: Related to the testing threat, the placebo effect occurs when the participants’ attitudes about whether the control will be successful impact the outcome, rather than the impact alone. Another related effect is the Hawthorne effect where the involvement of external researchers impacts outcomes. As described for the testing threat, the placebo or Hawthorne threat will be reduced in this study because the intervention and control groups will be treated in similar ways, except for the timing of the intervention. Participating establishments and individuals will be fully informed (as required by HSRB) as to the rationale and scope of the testing well in advance of the actual data collection. It will be assured that the purpose of the study is not for production or compliance reasons, but rather OSH.

Maturation Threat: A maturation threat may arise when the natural changes (e.g. workforce aging, and work experience changes) of the treatment groups may impact outcomes in addition to or rather than the intervention. In this study, this will be addressed by collecting information on participant age and job experience and including these terms in the outcome modeling.

Dropout Threat: This threat occurs when a sufficient number of individuals drop from treatment groups to alter the group characteristics that may in turn impact outcomes. Common problems are a “survivor effect” where only relatively healthy participants remain in the study or a positive bias when the remaining participants look more favorably upon the intervention than those that drop out. This threat will be addressed in this study by conducting an “early exit” survey to understand the motivations of participant individuals or establishments who decide to remove themselves from the study. As well, establishment level workers compensation data will still be collected for all non-participating establishments, participating establishments, and drop-outs. In the event that an employee drops out of the study or moves to a non-impacted task, a replacement volunteer employee will be recruited. The same baseline questionnaires described above will be administered to all replacement participating employees