EVALUATING THE EFFECTIVENESS OF
OCCUPATIONAL SAFETY AND HEALTH PROGRAM ELEMENTS IN THE
WHOLESALE RETAIL SECTOR
Request for Office of Management and Budget (OMB) Review and Approval
for a Federally Sponsored Data Collection
Supporting Statement B
Steve Wurzelbacher, Ph.D.
Research Industrial Hygienist
Project Officer
National Institute for Occupational Safety and Health
Division of Surveillance, Hazard Evaluations, and Field Studies
1090 Tusculum Avenue, Mail Stop R14
Cincinnati, Ohio 45226
513-841-4322 (tel)
513-841-4486 (fax)
ICRO DO review September 16, 2015
Table of Contents
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B1. Respondent Universe and Sampling Methods 3
B2. Procedures for the Collection of Information 6
B3. Methods to Maximize Response Rates and Deal with Nonresponse 14
B4. Tests of Procedures or Methods to be Undertaken 15
B5. Individuals Consulted on Statistical Aspects and/or Analyzing Data 16
This is a request for an extension of information collection request (ICR) number 0920-0949 (expiration date is 10/31/2015) from the National Institute for Occupational Safety and Health (NIOSH), Centers for Disease Control and Prevention (CDC). The renewal is being requested for an additional 3 years. For the current study, NIOSH and the OBWC are examining the association between survey-assessed OSH program elements (organizational policies, procedures, practices) and workers compensation (WC) outcomes in a stratified sample of OBWC-insured wholesale/ retail trade (WRT) firms. Crucial OSH program elements with particularly high impact on WC losses will be identified in this study and disseminated to the WRT sector. The proposed research involves a firm-level survey of a series of organizational metrics considered to be potential predictors of injury and illness WC claim rates and duration in a stratified sample of OBWC-insured WRT firms in Ohio. The first year survey data collection has been completed, and there are expected to be up to 4,104 participant firms in the second year surveys. A nested study at 60 firms is asking multiple respondents at each firm to participate.
For this study, the target population (people, groups or workplaces which might benefit from the MSD interventions being tested) includes United States WRT establishments [North American Industry Classification System (NAICS) industry codes 42-45]. The sampling frame (segment of the target population) includes OBWC-insured WRT establishments. The study sample (people, work groups or workplaces chosen from the sampling frame) includes OBWC-insured WRT establishments who volunteer to participate in this OBWC-NIOSH collaboration research project.
The overall purpose of this study is to examine the association between survey-assessed OSH program elements (organizational policies, procedures, practices) and WC outcomes in a large stratified sample of WRT firms from 2014-2015. The long-term research goal is to build a scientifically-grounded evidence base for benchmarking leading indicators of firm organizational and management behavior in OSH that is relevant to all WRT firms. This will support the OBWC in guiding prevention efforts in Ohio. To accomplish this, 4 specific aims and 7 research questions are proposed.
Aim1: Identify a reliable and valid set of firm level measures of organizational and management metrics relevant to OSH and usable by OBWC
Research Question 1: What is the internal consistency reliability of the two OSH program evaluation scales (OPPQ-52 and OSHA Form 33) that comprise the survey used in this study?
Research Question 2: Do the two OSH program evaluation scales measure independent concepts?
Aim 2: Examine the relationships between OBWC claim rates and organizational and management metrics
Research Question 3: What is the relationship between past injury and illness claim rates and the OSH program evaluation scales after controlling for a range of covariates?
Research Question 4a: What is the relationship between the current OSH program evaluation scales and injury and illness claim rates in the follow-up years after controlling for a range of covariates and adjusting for past injury and illness rates?
Research Question 4b: Which current OSH program evaluation scales are the key predictors of injury and illness rates in the follow-up years?
Aim 3: Demonstrate a scientifically-grounded procedure for collecting valid firm-level estimates of organizational metrics, aggregating the data and disseminating benchmarking information to all key stakeholders
Research Question 5: Is there a significant difference in the characteristics of firms who choose to participate and those who choose not to participate that would affect the generalizability of the information?
Research Question 6: Does the position of the person in the organization providing the information affect the content of the information provided?
Aim 4: Examine the relationship between organizational metrics and organizational injury prevention and loss control activity supported by OBWC
Research Question 7: What is the relationship between past organizational injury prevention and loss control activity supported by OBWC and the current OSH program evaluation scales after controlling for a range of covariates?
Sampling Strategy
Firm Sampling: It is anticipated that organizational and management behavior may vary depending on the sector of the firm and its size. To obtain more precise estimates of population quantities (Lohr 1999), and to obtain reasonably precise estimates within subgroups, a stratified sampling strategy is being used in this study with strata defined by a combination of industrial sectors (wholesale, retail) and firm size (< 20 versus, 20-99, 100-499, 500+) to seek representative samples of firms within each of these 2x4 =8 strata. Overall, it has been determined that approximately 3,000 firms would be adequate to meet sample size needs (Attachment J). The goal is to have a 5% sample from each stratum. To achieve this, small and medium-sized firms are being over sampled (20% for <20 and 20-99 and 10% for 100-499) and large firms (500+) are being fully enumerated. It is expected that only about 40% of small firms will agree to participate and 70% of medium and large firms (based on Amick 2000). Using these estimates, a total of 3,731 (2,702 firms in retail and 1,024 in wholesale) are expected to participate (Attachment K). Adding a 10% uncertainty factor, this equates to 4,104 expected firms.
Since the sampling scheme uses different sampling probabilities depending on which stratum a firm is in, sampling weights are being used for some of the analyses proposed (specifically Research Questions 3, 4a,b and 7 and in the production of the benchmarking reports). The weights are being constructed to represent the inverse of the probability of being sampled (Lohr 1999), and can be used when making inferences that apply to the population as a whole. Analyses focusing on the measurement properties of the proposed survey instruments are not using sampling weights since the focus is how the survey items relate to one another. It is also recognized that firms will not agree to participate or not participate randomly (e.g., it is expected smaller firms or firms with significant on-going injury and illness problems is less likely to participate). Therefore, some post-sampling stratification weights are being developed as a result of answering Research Question 5.
Key Informants Sampling: Regarding the selection of key informants within firms, protocols established by Amick (2000; 2004) are being followed and during the first contact (email or phone call) researchers are asking to be directed to the person in the organization most knowledgeable about both OSH and disability management policies and practices. If multiple contacts are given within the firm, the informant are being randomly selected from a list of potential respondents until one of the contacts agrees to participate in the survey.
Nested Study Sampling: The research team recognizes the significance of the question of who is the best informant, and Research Question 6 specifically addresses this issue. Preliminary results from the Ontario Organizational Indices work in 800 firms (Amick personal communication), has suggested it does not matter whether the owner, vice president, head of health and safety or member of the Joint Health and Safety Committee responded. Instead, the reliability of the organizational metrics varied by whether the informant had worked on health and safety issues in the organization. For the nested study of who is the best informant, 60 firms representative of sector and size are being sought. Firms are being selected to maximize the generalizability of the findings (e.g., having small and large firms). Firms in each WRT sub-sector (n=2) and by size (n=4) are being be sought. Sampling is random within the 8 cells without replacement. Recognizing there are significant differences by firm size, researchers attempt to identify: a manager not knowledgeable about health and safety (e.g., financial officer), a manager knowledgeable about health and safety, a manager knowledgeable about disability management, Health and Safety Committee members if appropriate, and a supervisor. Sample size calculations (Attachment K) indicate that 60 is a reasonable number for the nested study.
Sample Size Summary Requirements
The sample size estimations are described in detail in Attachment K. In summary, a sample of approximately 3,700 firms will meet all sample size requirements for Research Questions 3 and 5. In addition, as part of the nested study, 60 randomly selected firms is being administered the organizational questionnaire to obtain sufficient sample for Research Question 6.
Data Collection
Draft Survey Pilot-Testing: The combined survey was first pilot-tested using a web portal on a small stratified sample of 9 OBWC-insured WRT firms using the person responsible for OSH at the surveyed firm as the respondent.
Final Survey Administration: Information is being collected over two one year windows. The first survey has been collected and the second year is currently being completed. Firms identified in year 1 are being followed over year 2 and the same respondent or same type of respondent (e.g. same or similar job duties and/or job title) is being contacted. If firms are no longer in business, researchers are resampling firms within strata. Data collection procedures follow the protocol Amick (2000) used in an earlier study that resulted in a 70% firm-level response rate.
Nested Study: Researchers are approaching 60 firms during the second year of final survey administration (e.g. January- December 2015) to participate in the nested study. Up to five different individuals within each firm are being asked to complete the same set of questionnaires.
Non-Responder Follow-up Interview: Researchers are conducting a 5-minute phone interview (Attachment L) of a 10% random sample of non-participating firms.
Dependent Variables
The primary outcomes are being the workers compensation (WC) claims metrics aggregated to the firm-level. Seven years of WC data (5 years of claim data before the first questionnaire and 2 years of data after the second questionnaire) are being collected from each participating establishment. There are four different dependent variables to be used:
1: Total Claim Rate: This rate will be the primary outcome as it is the key metric many stakeholders use to assess OSH performance. For each year under study, this rate will be constructed as the count of allowed claims (both lost-time and no-lost-time) from a firm with accident date during the year of interest divided by an estimate of the number of full-time equivalents (FTEs) at the firm during the year of interest.
2: Lost-time Claim Rate: This outcome is like total claim rate, but includes only allowed lost time claims in the numerator.
3: No-Lost-Time Claim Rate: This outcome is like total claim rate, but includes only allowed no-lost-time claims in the numerator.
4: MSD Lost Time Claim Rate: This rate will be like the lost-time claim rate, but will include only allowed lost time claims for musculoskeletal disorders (MSDs) in the numerator. The MSD definition will align with the following Bureau of Labor Statistics description: “Musculoskeletal Disorders (MSDs) include cases where the nature of the injury or illness is sprains, strains, tears; back pain, hurt back; soreness, pain, hurt, except the back; carpal tunnel syndrome; hernia; or musculoskeletal system and connective tissue diseases and disorders, when the event or exposure leading to the injury or illness is bodily reaction/bending, climbing, crawling, reaching, twisting; overexertion; or repetition. Cases of Raynaud’s phenomenon, tarsal tunnel syndrome, and herniated spinal discs are not included. Although they may be considered MSDs, the survey classifies these injuries and illnesses in categories that also include non-MSD cases.” (BLS, 2010)
Potential Covariates
Covariates are being obtained from the OBWC administrative data: firm size, firm sector, and firm geographic area. The first three will be measured as indicator variables. Covariates will also be obtained from the survey. Respondents are being asked whether the firm has experienced downsizing and the OBWC data are being used to compare the number of employees employed in the most recent year versus three years prior to construct measures of downsizing. OBWC is providing a record of whether firms have participated in selected OBWC programs (e.g. Safety Councils, Drug Free Safety Program, and onsite loss control visits, as described in a protocol previously approved by the NIOSH Human Subject Review Board (approval number HSRB11-DSHEFS-01XP) for “Ohio Bureau of Workers Compensation Intervention Programs Historical Effectiveness”).
Data Analysis
In preliminary analysis the data are being examined for missing values and a merged de-identified data set will be created. Analyses for each research question are described below.
Research Question 1: Cronbach’s alpha and the corrected item total correlation (ITC) will be used to assess scale internal consistency (Cronbach, 1951). An ITC of each item with its theoretical subscale should be at least 0.40 (Ware 1997). Cronbach’s alphas should be greater than 0.7, but some proposed scales measure broad concepts with few items and thus alphas may be in the 0.6 to 0.7 range. These measures may have high test-retest correlations, illustrating the importance of multiple reliability tests. Internal consistency is a measure of the precision, while the test-retest is a measure of repeatability. Thus, a scale can be performing well with less than optimal precision but strong repeatability (Striener, 1995; Bollen and Lennox, 1991). The properties of the of the OSH program evaluation scales (OPPQ-526, OSHA Form 33) will be examined and the final scales that emerge from the confirmatory factor analysis (CFA).
Research Question 2: To assess whether the organizational and management metrics contain independent concepts, CFA will be used. Researchers will start with the hypothesized measurement model with the OSH program evaluation scales. Fit will be assessed using 5 indices: 1) the overall 2 statistic (Bollen 1989), 2) the Root Mean Square Error of Approximation [RMSEA, (Steiger and Lind 1980)], 3) the non-normed fit index [NNFI, a.k.a. TLI (Bentler 1990, Tucker and Lewis 1973)], 4) the comparative fit (CFI) and, 5) normed fit (NFI) indices (Bentler 1990). Model fit will be deemed acceptable if the overall 2 test was non-significant (Bollen 1989); the RMSEA fell between 0.06 and 0.08 (Browne and Cudeck 1992); and the other three indices had estimated values between 0.90-0.95. A well fit model includes: a non-significant overall 2 test (Bollen 1989); a RMSEA between 0.01 and 0.06 (Hu and Bentler 1999); a NNFI, CFI and NFI between 0.95 and 0.99 (Bentler and Bonnett 1980). Confirming item-factor structure supports the premise that constructs are independent. If there is not adequate fit, the factor structure will be examined and a new proposed measurement model examined. Once the final model is established a new model will be estimated with the Crowne-Marlowe measure of social desirability. Key informants may respond in a socially desirable way and thus, observed relationships could be due to a socially desirable response. The social desirability measure will be used to control for this response and further reinforce the model’s veracity.
Research Question 3: The relationship between current organizational and management metrics and past injury and illness claim experiences will be examined after controlling for a range of covariates. The primary predictor variables will be the OSH program evaluation scales and the relationship of injury and illness rates with these measures will be examined. As Habeck (1998) noted, past injury and illness experiences may be important drivers of changes in organization and management of OSH. The models will be estimated for both the original metrics and the ones that emerge from the CFA. For the OSH program evaluation scales, multivariate linear regression will be used with predictor variables representing the injury and illness experience in the prior five years. It is expected that firms with poor experiences may respond by establishing better organizational and management programs. Significance tests will be tests of coefficients and overall model fit. For the single scale measure of employee relations, multiple linear regression will be used. Significance tests will be tests of coefficients (t-tests) and tests for overall model fit. In each case, model estimation will proceed first with covariate selection, then estimating unadjusted effects and then adjusted effects. Some covariates will be highly correlated with injury and illness rates and care will be taken in both covariate selection and how models are built. Depending on the CFA results, a single multivariate linear regression model may be estimated. One key dimension of this analysis will be to learn how historical patterns of injuries and illnesses are important for the current organizational and management measures and the most efficient method to statistically capture the patterns. This information will be used in model building for Research Questions 4 and 4a. All the above analyses will be conducted using sampling weights to provide population estimates.
Research Questions 4a & 4b: The relationship between current organizational and management metrics and injury and illness rates will be examined in the following two years after controlling for a range of covariates. The analysis will start with multiple Poisson regression with firm size as offset, but it is expected that the outcome will be over dispersed and the best estimation procedure will be a negative binomial regression. The relationship between 4 dependent variables and the OSH program evaluation scales will be modeled, so significance levels will be adjusted accordingly. Again, significance tests will be tests of coefficients (t-tests) and overall model fit. Covariates will first be identified, and then unadjusted and adjusted models will be estimated. In adjusted models, researchers will first adjust for covariates and then adjust for past injury and illness experience. The adjustment for past injury and illness rates could be considered an over-specification of the model. However, counter-intuitive prospective relationships between organizational metrics and claims rates could be due to an organization recently putting new programs and practices in place as a response to prior claims experiences. If this is the case, the counter-intuitive relationship would be expected to disappear after adjustment. Additionally, information on the relationship between past OBWC program participation (e.g. Safety Councils, Drug Free Safety Program, and onsite loss control visit frequency) will be used as indicators of past organizational behavior. The intention is to examine the effects of the original scales and the final proposed scales. All these analyses will be done using the sampling weights to provide population estimates. These analyses will contribute important information to the assessment of the best organizational and management metrics that predict injury and illness rates. The research team will consider the predictive relationships when making recommendations for benchmarking.
Research Question 5: To answer the question of how participants differ from non-participants, researchers will compare firm size, geographic location, sector and the range of available OBWC data to ascertain whether there is a statistically significant difference (using appropriate statistical tests) between responders and non-responders. Depending on the scope and magnitude of the problem, logistic regression models will be developed with a responder/non-responder outcome. Based on the results of these analyses, a sampling statistician will consider post-sampling weights to account for any observed differential non-response that would potentially bias population estimates. The most likely problem would be differential non-response by firm size with small firms less likely to respond. Researchers do not want population estimates to be estimates of large establishments. Therefore, post-sampling weights would need to be introduced. Researchers also plan further to explore reasons for non-participation in a 10% sample of non-participants. Using interview data, researchers will compile answers for reasons for non-participation and use this in developing recommendations to produce scientifically credible benchmarking reports. This information will also be used to compare how participants differ from non-participants.
Research Question 6: To address whether the position of the person in the organization providing the information affects the content of the information provided, researchers will use standard measures of inter-rater agreement including the intra-class correlation coefficient with observers treated as a random effect (Shrout and Fleiss, 1979) and Lin’s Concordance correlation coefficient (Lin, 1989). Statistical decision rules for the ICC are defined above. Ossman and Amick (2004) showed strong agreement between a key management informant and an injured worker in unionized workplaces but also observed firm size did not affect agreement. Ossman and Amick (2004) suggested that unless an informant experiences the safety system or the disability management system then the informant is not likely to be an accurate reporter. However, there is little literature on which to base hypotheses. The operating methodological hypothesis in the proposed study is that it does not matter who you talk with; that there is high concordance/agreement. If true, this makes the implementation of a state wide benchmarking system less complicated. However, there may be differences and recommendations that emerge from the proposed methodological research for a retail and wholesale industry-wide benchmarking effort may need to take this in to account. If it is concluded from the methodological sub-study that there are differential reliabilities in measurement associated with position in the organization, a reliability-based regression will be conducted. Researchers will have data from a single informant from almost all participating organizations and thus will at best either control for the variation introduced by informant or use the informant information to weight the information. The quantitative data will be augmented with qualitative data obtained from interviews on why answers may differ. This information will be used to develop recommendations on who best to ask in an organization. It is expected this may vary by firm size and sector.
Research Question 7: To address the question whether past participation in OBWC programs (e.g. Safety Councils, Drug Free Safety Program, and onsite loss control visits) predict current organizational and management metrics, the research team will develop a series of indicator variables for program participation. Depending on the nature of the administrative data the analysis will follow the same strategy used to answer research question 3 above. Researchers will also determine if there are subgroups of firms that adopt OBWC programs in the years of data collection. This will provide the opportunity to examine if the adoption of new OBWC programs will lead to changes in the organizational and management metrics. Results may need to be considered by subgroups created by sector, geography or firm size. All the above analyses will be done using sampling weights to provide population estimates.
Recruitment
The initial contact information (firm phone number and/or email address) was gathered for the 10,929 firms in the targeted sample (Attachment J) using OBWC administrative data. In the event contact information is absent or incorrect, researchers are supplementing the contact information utilizing a web search. A Microsoft Access Database is being to maintain the recruitment effort and is being secured with a password. A new unique 8-digit identifier is assigned to each firm in the sampling frame to link responder data to the associated firm. A script is used for initial communication with the firms when telephone contact is necessary to obtain participant data.
Participants from each firm are being contacted using the developed contact list via email or telephone by a Research Coordinator. Recruitment emails to prospective respondents are being sent in a series of cohort waves to effectively manage communication. The respondent asked to complete the survey should be the individual in the organization most knowledgeable about both OSH and disability management policies and practices. If multiple contacts are given within the firm, the informant is being randomly selected from a list of potential respondents until one of the contacts agrees to participate in the survey. An on-line tracking sheet is being used for each firm, which is pre-populated with the firm name, firm number and address/location (example: UTHSC, 111111, 7000 Fannin Street, Houston, TX 77030). Responders have a unique identifier code assigned which cross references their given firm. If completed through online administration, participants are assigned a unique 5-digit code to access the survey. Only the Research Coordinator is responsible for maintaining a database linking responder access codes to firm identifier numbers. This database is stored in the most secure partition of the UTSPH server.
The recruitment process follows a protocol that Amick (2000) used in an earlier study that resulted in a 70% firm-level response rate. If an email address is unavailable, a script is followed when making contact with a given participant. Following initial contact, participants are asked by the Research Coordinator if he or she is the contact person provided. In some cases, (particularly for smaller organizations), the individual most knowledgeable about the OSH practices of the organization is used as a respondent.
“Passive non-responders” (participants with whom the Research Coordinator was unable to make initial contact) are followed up with a total of 10 attempts. The 10 contact attempts occur at 1 week intervals through combination of e-mail and telephone calls which give a brief overview of the study and provide a link to the survey website. Ten attempts by the Research Coordinator failing to make contact deem the given participant a “passive non-responder”. The Research Coordinator cycles through contacting the targeted sample of firms until a completed survey is returned, the firm has refused to participate, or the firm has not responded to ten requests (any combination of voice/ email contacts). Replacement participants are then be recruited from the same firm if feasible or researchers sample another matching firm (same employment size and NAICS) to complete the survey.
Firms identified in year 1 are being followed over year 2 and the same informants are being contacted. If participants have changed roles researchers seek a new individual most knowledgeable about safety and health within the given firm. If firms are no longer in business, researchers resample new firms within strata. New participants who are sampled complete the informed consent. The collection procedures utilized in the year 2 survey again follow the protocol for Amick (2000). As before, if no response is received from potential participants 3 days after the follow-up recruitment e-mail a Research Coordinator attempts to contact the participant a total of 10 tries through a combination of scripted e-mails and phone calls to the firm. Each stresses the importance of their participation and remind participants to click the URL link in their email directed to the Qualtrics website. If no response is received 1 week after the final e-mail reminder, the Research Coordinator deems the participant “unreachable”. Once again the Year 2 survey is easy for the respondent to complete and administered using several options (web portal, hard copy forms, or phone interviews). Participants are again assigned a new unique identifier code with which to access the survey. The approximate time to complete the online survey for Year 2 (Attachment H-2) is once again 12 minutes.
For the nested study of who is the best informant, 60 firms and respondents are being recruited via a question item from the initial survey asking their interest in participating in the “Who is the Best Informant Interview”. Firms are being selected to maximize the generalizability of the findings (e.g., having small and large firms). Only respondents of the initial survey who work at a firm with 100 or more employees are being contacted. The “index participants” (defined as those individuals who completed the initial survey) are being contacted via telephone to answer initial questions about the demographics of their organization. “Index participants” are being asked for names and emails of five to seven individuals responsible for OSH who are being contacted to take the survey. The names and emails of the participants are being managed and contacted by the Research Coordinator only. Each responder are being assigned a unique identifier code with which to access the survey. The Research Coordinator then emails or calls the referred individuals for recruitment and are being provided the Qualtrics survey link directing participants to Who is the Best Informant Survey (same as Attachment H-1). Participants are being asked to complete the survey within one week’s time. An informed consent (Attachment G) are being obtained before survey administration. If the participant prefers, the survey is administered via telephone or hard copy which may be sent through regular mail.
Data Management, Security and Confidentiality
The study is collecting sensitive data (workers’ compensation records, self-reported safety and health program assessments) and maintain personal identifiers (the recruitment database contains the respondent name, firm phone number of respondent, firm address and firm email address of the respondent). All data are maintained such that it is identified with an assigned number, and stored in locked file cabinets and on secured computers, accessible only by password. The identification sheets and consent forms are being kept separate in locked file cabinets and are available only to authorized NIOSH and contractor personnel. Questionnaires are being administered using several options (self-administered secure web portal, self-administered hard copy forms, and telephonic interviews). The respondents are being strongly encouraged to use the self-administered web-based format of the survey. For those respondents lacking internet connections or those who do not wish to complete a web-based survey, a hard copy format is next offered. An interview option is offered as a last resort for those respondents who do not find the web-based or hard copy formats acceptable. The online survey design complies with applicable 508 requirements (http://www.hhs.gov/od/508policy) to accommodate individuals with disabilities.
The confidentiality of all data collected is protected to the extent legally possible, as covered by the Privacy Act of 1974, Title 5, United States Code, Section 522 (a). The method of handling the information complies with the Freedom of Information Act and the Privacy Act of 1974. Disclosure under the Privacy Act System is permitted: to private contractors assisting NIOSH; to collaborating researchers under certain limited circumstances to conduct further investigations; to the Department of Justice in the event of litigation; and to a congressional office assisting individuals in obtaining their records. Records management practices adhere to all applicable federal, Health and Human Services (HHS), Centers for Disease Control (CDC), and NIOSH IT security policies and procedures [Security Requirements for Federal Information Technology Resources, January 2010; Health and Human Services Acquisition Regulation (HHSAR), Clause 352.239-72]. For example, data are stored on encrypted CDs, flash drives, and/or ftp sites according to applicable Federal Information Processing Standards Publications (FIPS PUBS, see http://www.itl.nist.gov/fipspubs).
Use of Results
Results of the study (in de-identified and aggregated form) will be disseminated in the scientific literature and in educational materials through NIOSH and OBWC channels (website, publications).
Notification
All OBWC firms will receive general notification via OBWC’s website of new NIOSH and OBWC publications resulting from this study. Respondents may print their own survey responses, and firms will be provided aggregated reports for comparison. No firm will receive custom reports since all data will be aggregated according to industry sub-sectors.
Risks and Benefits
The study presents very minimal risks to participants. No individuals or participant firms will be identified in published materials. In reference to vulnerable populations, pregnant women may be among questionnaire respondents. Children (16 years or older) will be excluded from this study. No individuals or participant firms will receive any benefits directly related to participation in the data collection. An overall benefit is that an assessment of the effectiveness of OSH programs will allow firms to focus on evidence-based practices, policies, and procedures that have the greatest impact to eliminate or reduce injuries/illnesses.
Informed Consent- Questionnaire Data Collection
Participation in the questionnaire data collection of this NIOSH study is completely voluntary and involves minimal risks. The informed consent form describes the potential benefits and risks of participation in the study (Attachment G). The grade level for the consent process has been estimated as a 13th grade based on the Simple Measure of Gobbledygook (SMOG) formula (McLaughlin, 1969). The target respondents for this questionnaire study typically are safety/ health specialists at a WRT firm. Since these positions normally require at least an associate’s degree, the grade level of the consent form is justified.
To minimize the collection of personal information, researchers have received a waiver of documentation of informed consent. For online surveys, the respondent is asked to read the consent form (Attachment G) and acknowledges consent by clicking a button online. For hard copy surveys, the respondents are being provided a hard copy of the informed consent form and asked to read the form prior to completing the survey. A returned completed form acknowledges consent. For phone interviews, the respondent is read the informed consent form and verbally acknowledges consent.
Timeline
This study will be conducted over four years.
Year 1: The survey to assess OSH program elements using the target instruments was been pilot-tested. An Information Collection Request (ICR) was submitted to the Office of Management and Budget (OMB).
Year 2: The OSH program evaluation survey was administered to a stratified sample of Ohio WRT firms for the first year of study.
Year 3: The OSH program evaluation survey is being administered to a stratified sample of Ohio WRT firms for the second year of study. A nested survey of multiple respondents at the same firm is also being conducted.
Year 4: The analysis of the survey study data will be completed.
Several methods (described below) are being utilized to maximize survey response rate in this study.
Online Surveys: In order to maximize efficiency and reduce burden, a web-based survey is being used for the majority (estimated 95%) of all data collection. Web-based surveys have gained increasing acceptance as a research tool as they offer many advantages, including:
On-line surveys create efficiencies because respondents complete them during a much shorter window of time than other survey modes, and at a substantially reduced cost
On-line surveys create time efficiencies (i.e., less time to complete the survey because it can be programmed to efficiently guide respondents through skip patterns so that they are not asked questions that do not apply to them or have to spend time navigating through complex instructions);
Respondents potentially have the option of answering questions in a private setting where they feel comfortable and at ease (e.g., at home);
Respondents can complete the survey within their own time schedule, and can exit the survey at any time and resume the survey where they ended;
Previous research (Catalano et al 2006) suggests that workers in some industries prefer completing an online survey when given a choice between a web survey and a paper survey.
Brief Surveys: Surveys have been designed to be as brief as possible. The time burden is estimated to be on average 12 minutes, up to a maximum of 20 minutes. It is estimated that the total maximum time burden for each participant to complete online surveys over the course of the 2 year survey study is 40 minutes.
Proven Recruitment Process: The extensive recruitment process described above follows the protocol that Amick (2000) used in an earlier study that resulted in a 70% firm-level response rate. Methods To Deal With Non-Response
As described above, “passive non-responders” (participants with whom the Research Coordinator was unable to make initial contact) are being followed up with a total of 10 attempts. The 10 contact attempts occur at 1 week intervals through combination of e-mail and telephone calls which give a brief overview of the study and provide a link to the survey website. Ten attempts by the Research Coordinator failing to make contact deem the given participant a “passive non-responder”. The contractor cycles through contacting the targeted sample of firms until a completed survey is returned, the firm has refused to participate, or the firm has not responded to 10 requests (any combination of voice/ email contacts). Replacement participants are being recruited from the same firm if feasible or researchers sample another matching firm (same employment size and NAICS) to complete the survey.
Once a person has agreed to complete a survey, the participant are able to complete the survey immediately or at a later time. Additional contact attempts occur at 1 week intervals through combination of e-mails (which give a brief overview of the study and provide a link to the survey website) and telephone calls to prompt the completion of surveys. Ten attempts by the Research Coordinator failing to make contact deem the given participant a “passive non-responder”. Replacement participants are recruited from the same firm if feasible or researchers sample another matching firm (same employment size and NAICS) to complete the survey.
Estimates of time burden and usability for all data collection forms are based on pilot testing conducted at NIOSH. This testing was conducted with 9 safety/health contacts from OBWC-insured WRT firms using an online version of the survey. This group represents the intended industry type and size of firm that is expected to participate in the actual study. The online version of the survey is being used by 95% of the actual respondents. Based on pilot-testing, it is estimated that the OSH Program Evaluation Survey (Attachments H-1 and H-2) requires on average 12 minutes (up to a maximum of 20 minutes ) per data collection.
NIOSH, OBWC and NIOSH contractors (University of Texas, UT) co-designed data collection and data analysis plans. NIOSH contractors (UT) are performing the data collection, and analyzing the data. Below is a summary of individual roles on this project.
Name |
Job Title |
Division |
Contact Information |
Roles on Project |
Steve Wurzelbacher, Ph.D. |
Research Industrial Hygienist |
Division of Surveillance Hazard Evaluation and Field Studies (DSHEFS) |
513.841.4322 |
Project Officer:
Co-designed data collection and data analysis plans
|
Steve Bertke, Ph.D. |
Statistician |
Division of Surveillance Hazard Evaluation and Field Studies (DSHEFS) |
513.841.4493 |
Co-designed data collection and data analysis plans
|
Alysha Meyers, Ph.D. |
Epidemiologist |
Division of Surveillance Hazard Evaluation and Field Studies (DSHEFS) |
513.841.4208 |
Co-designed data collection and data analysis plans
|
The Ohio of Bureau of Workers Compensation (OBWC) also helped design the data collection. Below is a summary of individual OBWC staff roles on this project
Name |
Job Title |
Division |
Contact Information |
Roles on Project |
Mike Lampl, M.S. |
Ergonomics Technical Advisor |
Division of Safety and Health |
614.995.1203 |
Co-designed data collection
|
Abe Tarawneh, Ph.D. |
Superintendent |
Division of Safety and Health |
614.466.0384
|
Supervising OBWC role on overall project |
The University of Texas co-developed the data collection and data analysis plans and is conducting the actual data collection and analysis.
Name |
Job Title |
Contact Information |
Roles on Project |
Ben Amick, Ph.D.
|
Research Psychologist |
|
-Co-designed data collection and data analysis plans
-Data collection -Data analysis
|
David Gimeno Ph.D. |
Public Health Scientist |
210-562-5511 |
-Co-designed data collection and data analysis plans
-Data collection -Data analysis
|
Akbar-Khanzadeh F. &Wagner O.D. (2001). Safety and health program assessment in relation to the number and type of safety and health violations. AIHA J, 62(5), 605-10.
Amick
III BC, Habeck RV, Hunt A, Fossel AH, Chapin A, Keller RB, Katz JN.
Measuring the impact of organizational behaviors on work disability
prevention and management. J Occup Rehabil 2000;10(1):21-38.
Amick
III BC, Habeck RV, Ossmann J, Fossel AH, Keller R, Katz JN.
Predictors of successful work role functioning after carpal tunnel
release surgery. J Occup Environ Med 2004 May; 46(5):490-500.
Bentler
PM. Comparative fit indexes in structural models. Psychol Bull
1990;107(2): 238-246.
Bentler PM, Bonnett DG.
Significance tests and goodness-of-fit in the analysis of covariance
structures. Psychol Bull 1980;88: 588-606.
Bollen KA.
Structural equations with latent variables. New York: John Wiley &
Sons, 1989.
Bollen
KA, Lennox R. Conventional wisdom on measurement: a structural
equation perspective. Psychol Bull 1991;110(2):305-314.
Bollen
KA, Lennox R. Conventional wisdom on measurement: a structural
equation perspective. Psychol Bull 1991;110(2):305-314.
Browne MW, Cudeck R. Alternative ways of assessing model fit. Sociol Methods Res 1992;21:230-258.
Bureau of Labor Statistics, United States Department of Labor, Washington, DC. http://www.bls.gov/iif/oshdef.htm. Accessed 4/28/10.
Cronbach
LJ. Coefficient alpha and the internal structure of tests.
Psychometrika 1951;16:297- 334.
Cullen KL, Williams RM,
Shannon HS, Wessel J, Amick III BC, Westmoreland MG. Workplace
disability management practices in Ontario educational facilities. J
Occup Rehabil 2005;15(3):417-433.
Fernandez-Muniz, B, Montes-Peon, JM, Vazquez-Ordas, CJ. Safety management system: Development and validation of a multidimensional scale. Journal of Loss Prevention in the Process Industries 2007;20:52-68.
Franche RL, Severin, CN, Hogg-Johnson, S, Lee, H, Cote, P, Krause, N. A multivariate analysis of factors associated with early offer and acceptance of a work accommodation following an occupational musculoskeletal injury. Journal of Occupational and Environmental Medicine 2009;51: 969 - 983.
Habeck RV, Hunt HA, VanTol B. Workplace factors associated with preventing and managing work disability. Rehabil Couns Bull 1998;42(2):98-143.
Harrell F. Regression coefficients and scoring rules. J Clin Epidemiol. 1996 Jul;49(7):819.
Hu L, Bentler PB. Cutoff criteria for fit indices in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modelling 1999;6:1-55.
Hunt HA, Habeck RV, VanTol B, Scully, SM. Disability Prevention Among Michigan Employers. Final Report submitted to the Michigan Department of Labor (Upjohn Institute Technical Report No. 93-004). Kalamazoo, MI: W.E. Upjohn Institute for Employment Research; 1993.
Katz
JN, Amick III BC, Keller R, Fossel AH, Ossmann J, Soucie V, Losina
E. Determinants of work absence following surgery for carpal tunnel
syndrome. Am J Ind Med 2005;47:120-130.
Kraemer HC. The
small sample nonnull properties of Kendall’s coefficient of
concordance for normal populations. J Am Stat Assoc
1976;71(355):608-613.
Kraemer HC, Korner AF. Statistical
alternatives in assessing reliability, consistency, and individual
differences in quantitative measures: Application to behavioral
measures of neonates. Psychol Bull 1976;83(5):914-921.
Lin
L. A concordance correlation coefficient to evaluate
reproducibility. Biometrics 1989;45:255–268.
Lohr
SL. Sampling: design and analysis. Toronto: Duxbry Press;
1999.
Mansfield R, Smitherman H, Weems W. Data Accuracy
and Utility Project Report. Developed for the Office of Cooperative
Programs, United States, Department of Labor, Occupational Safety
and Health Administration, September 2002.
Marsden PV,
Cook CR. Measuring organizational structures and environments. Am
Behav Sci 1994; 37(7):891.
McLaughlin GH. SMOG grading: A new readability formula. Journal of Reading, 12 (8) 639-646. (1969)
Norman GR, Streiner DL. Biostatistics: The Bare Essentials. St. Louis: Mosby; 1994.
Occupational Safety and Health Administration, Code of Federal Regulations. Available at http://osha.gov/pls/oshaweb/owadisp.show_document?p_table=STANDARDS&p_id=9765
Ossmann J, Amick III BC, Habeck RV, Hunt A, Ramamurthhy G, Soucie V, Katz JN. Management and employee agreement on reports of organizational policies and practices important in return to work following carpal tunnel surgery. J Occup Rehabil 2005;15(1):17-26.
Park
RM, Bushnell PT, Bailer AJ, Collins JW, Stayner LT. Impact of
publicly sponsored interventions on musculoskeletal injury claims in
nursing homes. Am J Ind Med. 2009 Sep;52(9):683-97.
Reynolds
W. Development of reliable and valid short forms of the
Marlowe-Crowne social desirability scale. J Clin Psychol 1982
;38:119-125.
Shannon
H, Walters V, Lewchuk W, Richarson J, Moran LA, Haines T, Verma D.
Workplace organizational correlates of lost-time accident rates in
manufacturing. AJIM 1996;29:258-268.
Shrout PE, Fleiss
JL. Intraclass correlations: Uses in assessing rater reliability.
Psychol Bull 1979;86:420-428.
Steiger
JH, Lind JC. Statistically based tests for the number of common
factors. Psychometric Society Annual Meeting, 1980.
Streiner
DL. Being inconsistent about consistency: when coefficient alpha
does and doesn’t matter. J Pers Assess 2003; 80: (3): 217–22.
Streiner
DL, Norman GR. Reliability. In: Health measurement scales: a
practical guide to their development and use. 2nd edition. New York:
Oxford University Press, 1995:104-27.
Tompa E.
Productivity and health: empirical evidence and policy implications.
In: Sharpe A, St-Hillaire F, Banting K, editors. The review of
economic performance and social progress: towards a social
understanding of productivity. Montreal: The Institute for Research
on Public Policy; Ottawa: Centre for the Study of Living Standards;
2002. p. 181-202.
Tompa E, Fang M. Occupational health
and safety incentives: The impact of experience rating and firm size
dynamics. Saf Sci 2008.
Tucker LR, Lewis C. A reliability
coefficient for maximum likelihood factor analysis. Psychometrika
1973;38:1-10.
Ware JE, Harris WJ, Gandek B, Rogers BW, Reese PR. MAP-R: multitrait/multi-item analysis program; revised for Windows. Version 1.0 user’s guide. Boston, MA: Health Assessment Lab; 1997.
Williams
RM, Westmorland MG, Shannon H, Rasheed F, Amick III BC. Disability
management practices in education, hotel/motel and health care
workplaces. Am J Ind Med 2005;47:217-226.
Williams RM,
Westmorland MG, Shannon HS, Amick III BC. Disability management
practices in Ontario health care workplaces. J Occup Rehabil
2007;17(1):153-165.
World Heath Organization. (2008).
http://www.who.int/occupational_health/en/.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Wurzelbacher, Steven J. (CDC/NIOSH/DSHEFS) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |