App H external review comments responses

Appendix H

Comments on study protocol from external reviewers (with NIOSH responses)

Standard Electronic External Review Form

for Division of Safety Research Intramural Projects

A. IDENTIFICATION

Name of Project Officer : [Claire Dye]

Title of Proposed Project: [Underreporting of Occupational Illnesses and Injuries by Workers]

Name of Reviewer: [Kate Newman]

Telephone Number of Reviewer: [202-691-6162]

Fax Number of Reviewer: [202-691-6196]

E-mail address of Reviewer: [[email protected]]

******************************************************************************

B. CRITIQUE

1. Significance:

Does this study address an important problem in occupational safety? If the aims of the project are achieved, how will scientific knowledge be advanced? What will be the effect or impact of this study on the DSR mission to reduce worker injuries?

YES, this study addresses and important problem in occupational safety and health. If its aims are achieved, they will expand our knowledge and understanding of the circumstances of injuries and illnesses treated in emergency rooms, factors that may lead to underreporting, and differences between specific groups of workers in their decisions to seek treatment and to notify their employers. In the longer run, this expanded understanding will lead to better measures of the scope and impact of worker injuries and illnesses and, in turn, to better allocation of resources devoted to making work safer.

The official counts and rates of occupational injuries and illnesses are currently developed using a survey of a sample of employers (the Bureau of Labor Statistics (BLS) Survey of Occupational Injuries and Illnesses). By its design, it excludes the self-employed, workers on small farms, and Federal government employees. It also excludes any workplace injury or illness employers do not record and report to the Bureau of Labor Statistics. This latter category is likely to include injuries and illnesses that were not reported to employers as well as those the employer decides not to record for whatever reason. These limitations raise questions about the completeness of the counts and rates decision makers (governmental policy makers, businesses, workers and their advocates) rely upon.

The NEISS-WORK provides a unique mechanism for understanding a wide range of issues related to the completeness of counts and rates by obtaining information directly from workers who were injured or became ill and visited the emergency room. Other national surveys include the population as a whole and, while any injury or illness at work should be made avoidable, such injuries and illnesses are rare in a population that includes entire households or all workers. The NEISS-WORK already captures workplace injury and illness and provides the opportunity to ask follow-back questions of just those workers. There is simply no more efficient way to obtain a representative National sample of this population.

2. Approach:

Are the scientific framework, design (including the composition of the study population), methods, and analyses adequately developed, well integrated, and appropriate to the aims of the project? Does the project officer acknowledge potential problem areas including feasibility, and consider alternative tactics?

This study, particularly the sampling and weighting strategies, are quite sound and reflect recent developments in survey estimation tools and procedures. The oversampling of the self-employed and workers on small farms is likely to provide reliable data from these worker whose injuries and illnesses are not measured in the BLS survey. The questionnaire is also well-designed and the use of skip patterns will ensure each respondent is asked the questions related to their individual situation. This is critically important because respondent burden affects the accuracy and completeness of the responses obtained.

I regret the exclusion of another population of great interest—day laborers, but the means of getting the relevant information really would add tremendously to the length of the interview and burden on respondents. The project officer addresses this difficulty in some detail in Appendix A. I found that discussion persuasive.

3. Innovation:

Where needed, does the project employ novel concepts, approaches or methods? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies?

The study itself is innovative—it will provide the first national estimates of workplace injury and illness by interviewing from workers themselves and provide critical data to understand factors that may lead to underreporting by specific groups of workers or type of injury or illness. Such information is critical to understand, and develop strategies to mitigate, conditions that may be undermining the completeness of the Nation’s official data on occupational injuries and illnesses. The study protocol is quite complete and sophisticated in its sampling strategy, using an appropriate initial design along with procedures to modify sampling if the sample yields an insufficient number of workers willing to participate in the study. While not new, per se, this sampling protocol is more elegant and efficient than many population and health surveys. The power analysis demonstrates its effectiveness for the issues addressed by the study. Variable reduction is an effective way to collect a wide variety of data and then use it effectively in analysis. All these will significantly enhance the quality of the study’s analysis and results and, therefore, its utility in designing means to make the Nation’s understanding of workplace injuries and illnesses more complete.

4. Project Officer (Investigator):

Is the project officer appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the project officer and other researchers (if any)? Please do not include descriptive biographical information unless important to the evaluation of merit. For new or less experienced NIOSH staff, note if the level of supervision appears adequate.

I do not know the qualifications of the investigator. I am familiar with the quality of other NEISS-Work projects and the supervisors of this work. In addition, my experience with the two contractors involved in the project—Westat and Research Triangle Institute—is quite positive. Given these factors, I expect the project officer either has or will be provided the skills she needs to execute this protocol.

5. Environment:

Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of unique features of the scientific environment or employ useful collaborative arrangements? Please do not include a description of available facilities or equipment unless important to the evaluation of merit.

The integrity and analytical environment of the NIOSH Division of Safety Research is exceptional. The protocol itself already demonstrates the important contribution gained from collaboration.

6. Overall Evaluation:

In one paragraph, briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of the application in terms of the five review criteria. Recommend a score reflecting the overall impact of the project on the field of occupational safety and health, weighting the review criteria as you feel appropriate for each application. An application does not need to be strong in all categories to be judged likely to have a major impact and, thus, deserve a high merit rating. For example, an investigator may propose to carry out important work that by its nature is not innovative, but is essential to move a field forward.

This is a critically important and well-designed project. It comes as part of a response to Congressional concern that the information on which policy makers rely to address the safety and health and the Nation’s workers be complete and reliable. It is uniquely able to will fill a gap in the Nation’s understanding of the safety and health risk to workers. The high level of interest and attention to this issue will serve to maximize the visibility and utility of the study results. The only regret I have is that the sample cannot be larger and that the sample and length of the questionnaire have already led to the elimination of day-workers in the study population. Let me repeat, however, that I understand and agree with the conclusion reached by the investigator and described in Appendix A. At present there are many important questions for which NIOSH (and the occupational safety and health community) have to address.

I am not familiar with how NIOSH “scores” project protocols. I would give it the highest score.

******************************************************************************

C. OTHER CONSIDERATIONS

7. Gender, Minority, and Children Inclusion (As Relevant)

Children are explicitly excluded for the study population. The protocol includes measures to ensure representativeness by gender and minority status. The exclusion of day-labors, many of whom are Hispanic, is unfortunate but necessary. The protocol includes analysis by gender and ethnicity.

8. Human Subjects Note that NIOSH projects involving human subjects must obtain review and approval from the NIOSH Human Subjects Review Board.

Clearly, this research involves contact and interview of human subjects (injured or ill workers).

9. Researcher Hazards

There do not seem to be particular hazards to the researchers involved.

10. Other

(Please type any other comments here)

Standard Electronic External Review Form

for Division of Safety Research Intramural Projects

A. IDENTIFICATION

Name of Project Officer : [Claire Dye]

Title of Proposed Project: [Underreporting of Occupational Illnesses and Injuries by Workers]

Name of Reviewer: [Ken Rosenman]

Telephone Number of Reviewer: [xxx xxx xxxx]

Fax Number of Reviewer: [xxx xxx xxxx]

E-mail address of Reviewer: [[email protected]]

******************************************************************************

B. CRITIQUE

1. Significance:

Does this study address an important problem in occupational safety? If the aims of the project are achieved, how will scientific knowledge be advanced? What will be the effect or impact of this study on the DSR mission to reduce worker injuries?

This study addresses a very important issue. The basic premise of public health is to have data that allow the determination of the magnitude and characteristics of the problem. This study addresses the undercount in the current national statistics. Achievement of the aims of this study will allow better understanding of reasons for the undercount which will lead to the design of better methods to obtain accurate data. The results of this study should have an important impact on directing future DSR activity.

2. Approach:

Are the scientific framework, design (including the composition of the study population), methods, and analyses adequately developed, well integrated, and appropriate to the aims of the project? Does the project officer acknowledge potential problem areas including feasibility, and consider alternative tactics?

The statistical sampling methodology and data analysis are well developed. Information is lacking on the methodology to try and maximize participation of potential respondents. How soon after the injury will people be contacted? How many letters will they receive? How many telephone calls will be attempted? What time of day will calls be made? If only a spouse can be contacted, will the questionnaire be administered to the spouse to at least obtain partial information?

The exclusion of certain populations of particular interest are not well justified. Children are defined as less than 21. Why start at age 20? Why not 18?

The five reasons provided to exclude day laborers are not at all convincing. Day laborers is a population of particular interest and there are no other studies with access to ED data that are addressing this group. The protocol provides extensive methodology on dealing with the nonresponders in general and to exclude day laborers based on concern with their participation rate is not justified.

The draft questionnaire appears awkward, some of which will be presumably addressed after the cognitive testing. However, questions like did your union encourage or discourage you from reporting should be coupled with the questions if employers or others encourage or discourage. Cognitive testing on nine people is not likely to address all these issues. Also, the investigators should consider putting some of the more important questions at the beginning of the questionnaire in case respondents only partially complete a relatively long questionnaire.

One major way medical providers know an injury is work-related is that an employer representative brings the injured to the ED. I don’t see this addressed in the questionnaire.

3. Innovation:

Where needed, does the project employ novel concepts, approaches or methods? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies?

Use of the NEISS data for follow back is innovative. It would have been more innovative if NIOSH had arranged to match individual patient names from NEISS with individual names in the BLS annual survey. Perhaps this is something that could be done in the future if actual employee names and addresses are captured in the questionnaire.

4. Project Officer (Investigator):

Is the project officer appropriately trained and well suited to carry out this work? Is the work proposed appropriate to the experience level of the project officer and other researchers (if any)? Please do not include descriptive biographical information unless important to the evaluation of merit. For new or less experienced NIOSH staff, note if the level of supervision appears adequate.

No information has been provided to me about Clair Dye so I cannot answer this question.

5. Environment:

Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of unique features of the scientific environment or employ useful collaborative arrangements? Please do not include a description of available facilities or equipment unless important to the evaluation of merit.

The NEISS data set is unique and this is an excellent environment to conduct this study.

6. Overall Evaluation:

In one paragraph, briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of the application in terms of the five review criteria. Recommend a score reflecting the overall impact of the project on the field of occupational safety and health, weighting the review criteria as you feel appropriate for each application. An application does not need to be strong in all categories to be judged likely to have a major impact and, thus, deserve a high merit rating. For example, an investigator may propose to carry out important work that by its nature is not innovative, but is essential to move a field forward.

The strengths of this study are the issue it addresses, the NEISS data set it uses, and the statistical strengths of the methodology. Weaknesses include the exclusion of children and day laborers, the lack of description of how to ensure an adequate response rate, and omissions in the questionnaire. All the weaknesses can be addressed. I highly recommend that this study proceed. If matching could be done with the BLS annual survey in the future, that would make this study highly innovative.

******************************************************************************

C. OTHER CONSIDERATIONS

7. Gender, Minority, and Children Inclusion (As Relevant)

No rationale for excluding children 18 and 19 years of age is provided. Rationale for excluding minorities (i.e. day laborers) is provided but is not convincing.

8. Human Subjects Note that NIOSH projects involving human subjects must obtain review and approval from the NIOSH Human Subjects Review Board.

There are no issues with obtaining Human Subject approval.

9. Researcher Hazards

None.

10. Other

None.

Response to Peer Review by Ken Rosenman

1. The statistical sampling methodology and data analysis are well developed. Information is lacking on the methodology to try and maximize participation of potential respondents. How soon after the injury will people be contacted? How many letters will they receive? How many telephone calls will be attempted? What time of day will calls be made? If only a spouse can be contacted, will the questionnaire be administered to the spouse to at least obtain partial information?

RESPONSE: To clarify the sampling methodology, we have added additional information to the project document as follows:

Final Respondent Selection, Identification, and Contact. Based on the sample design requirements for final data collection, CPSC will select potential respondents weekly from incoming routine NEISS-Work case data. Prescreening using the basic NEISS-Work data elements will be used to restrict the potential respondents to those individuals most likely to meet the respondent definition (e.g., ages <20 and >64 and volunteers will be excluded). CPSC will then contact the participating hospital and request patient contact information. Individuals identified with potentially viable contact information will be sent one letter notifying them of the interview study and giving them the opportunity to “Opt Out.” Contact information for individuals who do not opt out, or who fail to respond to the letter within ten days, will be provided to a third-party contractor who will conduct the interviews. Contact information will be provided by the CPSC approximately three weeks after the date of treatment. At no time will NIOSH have the individual identifiers or contact information for the potential respondents to the final interview survey.

The telephone interviewers are required to make at least ten attempts to contact potential respondents. The contact attempts are made at varying, but reasonable, hours of the day and on varying days of the week. When no personal contact is made after a number of attempts, the interview is set aside and contact attempts are made at a later date as time permits to maximize the response rate while minimizing recall bias issues. Interviewers are trained to be considerate of respondents and their families, leaving a minimal number of messages or speaking with the respondent or another individual of the residence to arrange a convenient interview time. Messages include a toll-free response number so that the respondent may call at their convenience. When no personal contact is made, no message system is available, and there is no indicator of an incorrect number, the interviewer typically spreads their call attempts over a longer time period and commonly makes more than 10 contacts. Due to privacy concerns, the questionnaire will only be administered to the individual treated in the ED. The interviewers comply with CPSC contract requirements as approved by OMB.

2. The exclusion of certain populations of particular interest are not well justified. Children are defined as less than 21. Why start at age 20? Why not 18?

RESPONSE: To explain our reasons for excluding individuals aged 19 or younger, we have amended the text as follows: “Because of variations in the age of majority across states and the added complication of obtaining parental or guardian consent for a very small number of cases, respondents will be aged 20-64 years.”

3. The five reasons provided to exclude day laborers are not at all convincing. Day laborers is a population of particular interest and there are no other studies with access to ED data that are addressing this group. The protocol provides extensive methodology on dealing with the nonresponders in general and to exclude day laborers based on concern with their participation rate is not justified.

RESPONSE: While we agree that excluding day laborers based solely on concern with their participation rate is not justified, we found during questionnaire testing that many of the topics would not be applicable to day laborers (e.g., job security, union membership, and instructions for reporting work-related injuries or illnesses). Additionally, many of the questions that were relevant to day laborers would have had to be rewritten in order to be applicable to a very small day laborer worker population. This would have resulted in a substantially different questionnaire for day laborers. Thus, due to the small number of day laborers in NEISS-Work, combined with the inability to pool their data with the rest of respondents, the results obtained for day laborers would most likely not be reportable. Therefore, we will proceed with our initial decision to exclude day laborers from survey participation.

4. The draft questionnaire appears awkward, some of which will be presumably addressed after the cognitive testing. However, questions like did your union encourage or discourage you from reporting should be coupled with the questions if employers or others encourage or discourage. Cognitive testing on nine people is not likely to address all these issues. Also, the investigators should consider putting some of the more important questions at the beginning of the questionnaire in case respondents only partially complete a relatively long questionnaire.

RESPONSE: We appreciate these comments and will take them into consideration when making further revisions to the questionnaire. We also acknowledge that cognitive testing on more than nine respondents would garner additional information on potential problems. However, due to the OMB Paperwork Reduction Act, cognitive testing on greater than nine individuals would require OMB approval and would add one to one and a half years to the study timeline and, by extension, increase the cost of the project. We do not have the resources available to expand the project. We will, however, pre-screen potential cognitive interview respondents to ensure that they represent our populations of interest.

Also, the questionnaire was structured to capture the most important information up front. There are some instances where very important information appears at the end, but this frequently happened because the information was also deemed sensitive and risky to try to collect up front. There are also some instances where questions are located so as to maintain the logical flow of the questionnaire.

5. One major way medical providers know an injury is work-related is that an employer representative brings the injured to the ED. I don’t see this addressed in the questionnaire.

RESPONSE: We do ask respondents whether a co-worker, family member, or someone else told the ER staff how the injury or illness occurred. Thus, if the co-worker identified him- or herself as a co-worker to ED staff and explained that the injury was work-related, the respondent should respond “yes” to this question. If the co-worker did not provide ED staff with information on how the injury or illness occurred, the assumption of work-relatedness based on the co-worker’s presence could not be made as the health problem could have occurred during a non-work-related activity at which both workers were present (e.g., carpooling to work or going out to dinner after the work day has ended).

6. Use of the NEISS data for follow back is innovative. It would have been more innovative if NIOSH had arranged to match individual patient names from NEISS with individual names in the BLS annual survey. Perhaps this is something that could be done in the future if actual employee names and addresses are captured in the questionnaire.

RESPONSE: Comparison of respondent names in NEISS-Work with individual names in the BLS annual survey or the Survey of Occupational Injuries and Illness (SOII) is not possible in this study for several reasons. First, the SOII data come only from the 230,000 private employers that are selected to provide their OSHA logs to the Department of Labor for a given year. Thus, because there are no industry restrictions for NEISS-Work, many potential respondents in NEISS-Work may not work for one of the sampled employers. Conversely, many of the SOII sampled employers may not be in the same locale as a hospital in the NEISS-Work sample. Second, because we will first attempt contact with potential respondents approximately a month after their ED visit, the logs for these individuals probably will not have been entered into the SOII data. Third, we collect information on a wide range of injuries and illnesses, some of which are not reportable on the OSHA logs. Finally, matching cases from both BLS SOII and NEISS-Work would be prohibitive in terms of both time and money, and the current project timeline and budget do not allow for this.

7. No information has been provided to me about Clair Dye so I cannot answer this question.

RESPONSE: In the email that was sent by Jim Collins, Claire Dye was incorrectly identified as the project officer. The project officer is in fact Dr. Larry Jackson.

8. No rationale for excluding children 18 and 19 years of age is provided. Rationale for excluding minorities (i.e. day laborers) is provided but is not convincing.

RESPONSE: Please see our response to #3, above.

Standard Electronic External Review Form

for Division of Safety Research Intramural Projects

A. IDENTIFICATION

Name of Project Officer : [Claire Dye]

Title of Proposed Project: [Underreporting of Occupational Illnesses and Injuries by Workers]

Name of Reviewer: [Santosh K. Verma]

Telephone Number of Reviewer: [508 497 0213]

Fax Number of Reviewer: [508 435 3456]

E-mail address of Reviewer: [[email protected]]

******************************************************************************

B. CRITIQUE

1. Significance:

Does this study address an important problem in occupational safety? If the aims of the project are achieved, how will scientific knowledge be advanced? What will be the effect or impact of this study on the DSR mission to reduce worker injuries?

Accurate counting of occupational injuries and illnesses is critical for proper allocation of resources and prevention of occupational injuries and diseases. The Bureau of Labor Statistics is primarily responsible for the collection and reporting of statistics on occupational injuries and illnesses and uses an employer-based surveillance system for the annual Survey of Occupational Injuries and Illnesses (SOII). However, some studies have pointed to undercounting of injuries and illnesses in the SOII data, which also excludes some important employment groups such as the self-employed. It is important to evaluate other surveillance systems to fully understand the burden of occupational injuries and illnesses in the U.S.

The proposal aims to evaluate an emergency department-based surveillance system (NEISS-Work) and follow-up surveys to estimate acute occupational injuries and illnesses by employment status and prevalence of chronic occupational injuries and illnesses. If the aims of the project are achieved, a better understanding of undercounting in SOII data, the occupational injury experience of groups not included in SOII data, such as the self-employed, and the prevalence of chronic occupational injuries and illness in the United States workers will result. This knowledge will help DSR prioritize their injury research program and occupational injury prevention efforts.

2. Approach:

Are the scientific framework, design (including the composition of the study population), methods, and analyses adequately developed, well integrated, and appropriate to the aims of the project? Does the project officer acknowledge potential problem areas including feasibility, and consider alternative tactics?

In 2008, the Congressional Committee on Education and Labor released a report titled, “Hidden Tragedy: Underreporting of Workplace Injuries and Illnesses” (The Committee on Education and Labor, 2008). Based in part on the report’s results, Congress allocated funds for NIOSH to undertake a study on the underreporting of worker injury data, particularly focusing on the self-employed population and undercounting of chronic occupational injuries and illnesses in the U.S. Four specific aims are proposed. The first two aims relate to the estimation of acute occupational injuries and illnesses by employment status and the next two aims relate to the understanding and estimation of chronic occupational injuries and illnesses. A follow-up telephone interview study using NEISS-Work is proposed to achieve these aims.

Summary of the problem and Background are well developed and the design and analysis plans are very well described, including development of the survey materials, sample design and identification of respondents, appropriate case weighting and variable reduction. Major strengths of the proposal are nationally representative survey, number of injuries in each subgroup high enough to produce reliable national estimates, and the ability to contact injured workers.

The first two aims relate to the estimation of acute occupational injuries and illnesses by employment status with a special emphasis on the self-employed. One of the primary motivations of this proposal is the undercounting of occupational injuries in SOII data. However, comparison of injury rates from NEISS-Work and SOII and estimation of the magnitude of undercounting is not proposed. For example, by what magnitude would the national burden of injuries and illnesses increase due to the inclusion of self-employed workers?

The proposal cites a study indicating that only about one third of all medically treated occupational injuries and illnesses are treated in an emergency department (ED). Some injuries are less likely to be treated in the ED than others, such as less severe and slow developing injuries/illnesses. These types of injuries/illnesses are also more likely to be underreported by employers, and both employer-based surveillance systems and ED-based surveillance systems may suffer from a similar potential bias. Expansion of NEISS-Work to include other venues where occupational injuries and illnesses are treated and use of population-based surveillance systems, such as the National Health Interview Survey, may also be explored to fully understand the burden of acute and chronic occupational injuries and illnesses in U.S. workers.

Specific aims 3 and 4 relate to understanding and estimation of chronic occupational injuries and illnesses. Only a very small proportion of chronic injuries and illnesses may be treated in EDs, and it is unclear how and whether they represent the overall burden of chronic occupational injuries and illnesses. Secondly, the accuracy of self-reports of occupational origins of a chronic illness or an injury may be limited due to long induction periods typical of chronic diseases and injuries and the resultant long recall periods required to accurately recount exposures. The proposal also seeks to examine the history of chronic injury and illness among those who presented to the ED with an occupational injury or illness. This approach has two major limitations – first, workers who continued working with a chronic illness and presented to ED with an acute work-related injury may be a very special sub-population and results may not be generalizable. Second, workers with an acute occupational injury may recall their past medical history differently from those without one and there is a high potential for recall bias. Due to these limitations, the reviewer is not certain to what extent the results of the analyses regarding the extent of chronic injuries and illnesses in U.S. workers will be valid and generalizable.

The proposal includes formal cognitive testing of the English questionnaire but not the Spanish version. Back translation of the Spanish version may be considered to ensure equivalency between the English and the Spanish versions. Education level of the respondent selected for cognitive testing should also be taken into account and respondents from different strata of education levels might be selected.

The proposal describes a sample design plan in detail to ensure that the resulting sample for the interview will be representative of a national workforce, and this is a strength of the proposal. The validity of the selected sample can also be tested by comparing the rate of injury from NEISS-Work and the selected sample for the variables that are available in both the datasets, such as gender. An overall completion rate of 40 percent for has been assumed, which is somewhat low and has a potential to introduce selection bias. Although analysis plan includes nonresponse adjustments, efforts to minimize non-response rate, such as a small incentive, may be considered. Expected/acceptable length of time from the ED visit to the interview survey may also be reported in the proposal.

The analysis plan is described in detail. However, the proposal does not define the main outcome of interest and whether it is rate of injury/illnesses or the prevalence of occupational injury/illness. One benefit of reporting rate of injury is a more straightforward comparison with SOII data. Moreover, rate of injury takes into account number of hours worked (i.e. time at risk), in addition to total number of workers. Since prevalence of occupational injury is low, odds ratios obtained from logistic regression will approximate the prevalence ratio.

3. Innovation:

Where needed, does the project employ novel concepts, approaches or methods? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies?

The proposal is innovative in its use of NEISS-Work to estimate occupational injuries and illnesses particularly in groups excluded in SOII such as self-employed and government workers.

4. Environment:

Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of unique features of the scientific environment or employ useful collaborative arrangements? Please do not include a description of available facilities or equipment unless important to the evaluation of merit.

The environment appears adequate for the proposed project. The sample design and the analysis plan for the project were developed by Westat, a research services company that provides services to the United States government, among other entities and has experience in all aspects of survey design and analysis. Research Triangle Institute (RTI), an institute with many years of experience in survey methodology, will conduct the cognitive testing.

5. Overall Evaluation:

In one paragraph, briefly summarize the most important points of the Critique, addressing the strengths and weaknesses of the application in terms of the five review criteria. Recommend a score reflecting the overall impact of the project on the field of occupational safety and health, weighting the review criteria as you feel appropriate for each application. An application does not need to be strong in all categories to be judged likely to have a major impact and, thus, deserve a high merit rating. For example, an investigator may propose to carry out important work that by its nature is not innovative, but is essential to move a field forward.

This proposal aims to estimate prevalence of acute and chronic occupational injuries and illnesses in the U.S workers using an emergency department-based surveillance system (NEISS-Work) with special emphasis on employment groups excluded from BLS SOII data. This is a significant step forward to accurately measure the national burden of occupational injuries and illnesses. Major strengths of the proposal are nationally representative survey, number of injuries in each subgroup high enough to produce reliable national estimates, the ability to contact injured workers, and detailed design and analysis plans. Its primary weaknesses are the potential for significant bias and lack of generalizability when estimating prevalence of chronic occupational injuries and illnesses, under-representation of certain types of injuries and illnesses due to ED-based surveillance, and potential for selection bias due to low completion rate. Expansion of NEISS-Work to include other venues where occupational injuries and illnesses are treated and use of a population-based surveillance system, such as National Health Interview Survey, may be explored to address some of the weaknesses of the proposal. The proposal’s strengths outweigh its weaknesses.

Score: 1.9

******************************************************************************

C. OTHER CONSIDERATIONS

6. Gender, Minority, and Children Inclusion (As Relevant)

(Please type your comments here)

7. Human Subjects Note that NIOSH projects involving human subjects must obtain review and approval from the NIOSH Human Subjects Review Board.

(Please type your comments here)

8. Researcher Hazards

(Please type your comments here)

9. Other

(Please type any other comments here)

Response to Peer Review by Santosh K. Verma

1. Comparison of injury rates from NEISS-Work and SOII and estimation of the magnitude of undercounting is not proposed. For example, by what magnitude would the national burden of injuries and illnesses increase due to the inclusion of self-employed workers?

Response: Although comparing illness and injury rates across NEISS-Work and SOII is possible, it is not one of the primary aims of this project. Instead, our focus is on characterizing reporting behavior among injured and ill workers, characterizing the chronic aspects of ED-treated occupational injuries and illnesses, and estimating the prevalence of work-related chronic injuries and illnesses among United States workers. In addition, we will collect information on a wider range of injuries and illnesses than are reportable on the OSHA logs, causing our results to not be directly comparable to those obtained by SOII. Finally, comparisons of NEISS-Work and SOII and the estimated undercount related to the exclusion of various types of workers in SOII is already available (e.g., see Leigh, Marcin, & Miller, 2004).

2. Expansion of NEISS-Work to include other venues where occupational injuries and illnesses are treated and use of population-based surveillance systems, such as the National Health Interview Survey, may also be explored to fully understand the burden of acute and chronic occupational injuries and illnesses in U.S. workers.

Response: Expansion of NEISS-Work is currently not possible due to budgetary and logistic restraints. However, we agree that a surveillance system that combines multiple data sources, such as that used for the Census of Fatal Occupational Injuries (CFOI), would most likely result in improved data. Analysis of other data sources, such as the National Health Interview Survey (NHIS), is not within the scope of the proposed project, but may be considered in future projects.

3. Only a very small proportion of chronic injuries and illnesses may be treated in EDs, and it is unclear how and whether they represent the overall burden of chronic occupational injuries and illnesses. Secondly, the accuracy of self-reports of occupational origins of a chronic illness or an injury may be limited due to long induction periods typical of chronic diseases and injuries and the resultant long recall periods required to accurately recount exposures.

Response: We recognize that capturing only those cases treated in a hospital ED limits the generalizations that can be made to the general working population (see p. 17 for our section on “Methodological Limitations”). We have added text to the Limitations section to capture the various problems discussed by the reviewer, including the potential for recall bias. These limitations will also be discussed in future work products.

4. This approach has two major limitations – first, workers who continued working with a chronic illness and presented to ED with an acute work-related injury may be a very special sub-population and results may not be generalizable. Second, workers with an acute occupational injury may recall their past medical history differently from those without one and there is a high potential for recall bias.

Response: The Limitations section already mentions the possibility that the past behavior, knowledge, and perceived reporting barriers of sampled workers may be different than workers who have not experienced an ED-treated occupational injury or illness. We have added text that indicates that workers with chronic illnesses who report to an ED with an acute work-related injury or illness may be a distinct subpopulation of American workers.

5. The reviewer is not certain to what extent the results of the analyses regarding the extent of chronic injuries and illnesses in U.S. workers will be valid and generalizable.

Response: This is a recognized limitation of the present project and is noted in the Limitations section (p. 17).

6. Back translation of the Spanish version may be considered to ensure equivalency between the English and the Spanish versions.

Response: Back translation of the Spanish language version of the questionnaire is planned and text has been added to reflect this (see p. 5).

7. Education level of the respondent selected for cognitive testing should also be taken into account and respondents from different strata of education levels might be selected.

Response: Unfortunately, respondent’s education level is not available during selection of the cognitive interview sample. However, we will select the sample so that it contains a variety of individuals in different hospitals and from different industries. Care was taken during construction of the questionnaire to make it understandable at a lower reading level.

8. The validity of the selected sample can also be tested by comparing the rate of injury from NEISS-Work and the selected sample for the variables that are available in both the datasets, such as gender. An overall completion rate of 40 percent for has been assumed, which is somewhat low and has a potential to introduce selection bias.

Response: Although the current analysis plan will examine the correlation between characteristics of respondents versus non-respondents and use the resulting information to adjust for differential response rates, this provides an alternative way of analyzing non-response and will be taken into consideration during analysis.

9. Although analysis plan includes nonresponse adjustments, efforts to minimize non-response rate, such as a small incentive, may be considered.

Response: Although incentives may help to increase response rates, the project budget does not contain sufficient funds to offer respondents an incentive.

10. The proposal does not define the main outcome of interest and whether it is rate of injury/illnesses or the prevalence of occupational injury/illness.

Response: The study has multiple aims and thus multiple outcomes of interest. As alluded to on page 16, outcomes will include frequency of reporting by individual, job, injury or illness, and other important characteristics; binary outcomes for logistic regression such as whether an occupational injury or illness was reported or whether a workers’ compensation claim was submitted; and prevalence of chronic health problems among workers with ED-treated occupational injuries or illnesses.

Cited Source

Leigh, J.P., Marcin, J.P., & Miller, T.R. (2004). An estimate of the U.S. government’s undercount of nonfatal occupational injuries. J Occup Environ Med, 46:10-18.

8