SUPPORTING STATEMENT
Part A
Pilot Test of the Proposed Workforce Safety Supplemental Item Set
For the Surveys on Patient Safety Culture™
July 28, 2020
Agency for Healthcare Research and Quality (AHRQ)
1. Circumstances that make the collection of information necessary 3
2. How, by Whom, and for What Purpose Information Will Be Used 6
3. Use of Improved Information Technology 7
4. Efforts to Avoid Duplication 7
5. Involvement of Small Businesses 9
6. Consequences if Information Collected Less Frequently 9
8. Federal Register Notice and Outside Consultations 9
9. Payments/Gifts to Respondents 10
10. Assurance of Confidentiality 10
11. Questions of a Sensitive Nature 10
12. Estimates of Annualized Burden Hours and Costs 10
13. Estimates of Annualized Respondent Capital and Maintenance Costs 12
14. Estimates of Annualized Cost to the Government 12
16. Time Schedule, Publication and Analysis Plan 12
17. Exemption for Display of Expiration Date 14
AHRQ’s mission. As described in its 1999 reauthorizing legislation, Congress directed the Agency for Healthcare Research and Quality (AHRQ) to enhance the quality, appropriateness, and effectiveness of health services, as well as access to such services, by establishing a broad base of scientific research and promoting clinical and health systems practice improvements.i The legislation also directed AHRQ to “conduct and support research, evaluations, and training, support demonstration projects, research networks, and multidisciplinary centers, provide technical assistance, and disseminate information on healthcare and on systems for the delivery of such care, including activities with respect to health statistics, surveys, database development, and epidemiology.”ii
Furthermore, AHRQ shall conduct and support research “to provide objective clinical information to healthcare practitioners and other clinicians of healthcare goods or services; identify the causes of preventable healthcare errors and patient injury in healthcare delivery; develop, demonstrate, and evaluate strategies for reducing errors and improving patient safety; and disseminate such effective strategies throughout the healthcare industry”.iii
The safety of healthcare workers has been the focus of much research and regulation in the U.S. In 2011, U.S. hospitals recorded 253,700 work-related injuries and illnesses, a rate of 6.8 work-related injuries and illnesses for every 100 full-time employees, which is almost twice the rate for private industry as a whole.iv In 2013, 34 percent of recorded hospital worker injuries nationwide resulted in days away from work were associated with patient interactions.v Healthcare workers face a wide range of hazards on the job, including sharps injuries, harmful exposures to chemicals and hazardous drugs, back injuries, latex allergy, violence, and stress.vi These healthcare workforce injuries result in numerous indirect and less visible costs including employee turnover, training, overtime, incident investigation time, productivity, and morale.
Given that the healthcare industry is the fastest-growing sector of the U.S. economy, with an estimated 18.3 million workersvii, the safety of the healthcare workforce has major implications for healthcare workers across the spectrum of care, as well as the economic welfare of our country.
Improving the safety of the healthcare system includes the safety of the healthcare workforce, which is a necessary precondition to advancing patient safety. It is therefore a public health issue to both eliminate harm to the healthcare workforce and to patients. Although recent estimates show declines in some patient safety episodes, such as adverse drug events and injuries from falls, there is broad consensus that additional steps must be taken to further protect patients from injury or death.viii In May 2018, the Institute for Healthcare Improvement (IHI) relaunched the patient safety agenda by convening the National Steering Committee for Patient Safety (NSC) to develop a national action plan for reducing harm in the delivery of health care. The NSC was convened to leverage the knowledge of various stakeholders — influential federal agencies, leading health care delivery organizations and associations, patient and family leaders, and respected industry experts — into a set of actionable and effective recommendations.ix
AHRQ is co-leading the NSC, which is co-chaired by Jeffrey Brady, M.D., Director of AHRQ’s Center for Quality Improvement and Patient Safety, and Tejal Gandhi, M.D., Chief Safety and Transformation Officer at Press Ganey. The NSC convened four subcommittees, with workforce safety being one of the four foundational areas:
Culture, Leadership, and Governance—focusing on the role of leaders, governing bodies, and policymakers to establish safety as a core value and promote cultures of safety.
Workforce Safety—focusing on the safety of the healthcare workforce as a necessary precondition to advancing patient safety.
Patient and Family Engagement—focusing on co-designing and co-producing care with patients and families.
Learning Systems—focusing on learning systems within and across health care organizations.
While the safety of the healthcare workforce has been a longstanding concern and is a core component of the NSC’s efforts, the recent emergence of the COVID-19 pandemic in early 2020 has brought the issue of healthcare workforce safety even more to the forefront around the world. Shortages of proper personal protective equipment (PPE) have made healthcare workers more vulnerable to exposure to the virus. In addition, the physical and psychological well-being of healthcare workers is being tested as patient loads have increased and co-workers become infected with COVID-19, contributing significantly to burnout and a decline in the mental health of the healthcare workforce.x
Given the foundational importance of workforce safety as a precondition for patient safety, and renewed attention on the criticality of workforce safety as a result of the ongoing COVID-19 pandemic, AHRQ is undertaking timely work to develop survey items that will help healthcare organizations identify and improve workforce safety. The workforce safety survey items will assess the extent to which the organization’s culture supports workforce safety. The items will be developed as a new supplemental item set that can optionally be administered at the end of the AHRQ Surveys on Patient Safety Culture™ (SOPS®). By developing a supplemental item set, AHRQ will build this new measure of workforce safety upon its existing and highly successful SOPS survey program. SOPS surveys are completed by providers, staff, and administrators within healthcare organizations to assess the extent to which their organizational culture supports patient safety. There are SOPS surveys for hospitals, medical offices, nursing homes, ambulatory surgery centers, and community pharmacies. There are also supplemental items on health information technology patient safety and value and efficiency, with an item set on diagnostic safety currently under development.
Similar to work on previous SOPS survey supplemental item sets, our goal will be to develop a reliable, public-use item set that is limited in scope so it can be used in conjunction with one of the SOPS surveys, or used as a stand-alone supplemental item set. The goal is to develop 12 to15 items across 3 or 4 composite measures rather than developing a full-length survey.
The supplemental item set will be used by healthcare organizations to enable them to assess the organizational culture factors that contribute to workforce safety, and help them identify strengths and areas for improvement to efficiently target resources to improve workforce safety.
Most existing surveys on workforce safety focus on organizational, management, and provider and staff compliance with regulations, and are more like safety checklists. There is currently a void of validated survey items focusing on provider and staff perspectives about aspects of organizational culture that facilitate workforce safety. We have been unable to find instruments that address multiple, relevant areas of workforce safety for use in a variety of healthcare settings, with a variety of respondents, in one, brief instrument, which are the objectives of this research. To achieve these objectives, we propose the following activities:
Conduct cognitive testing – The purpose of cognitive testing is to understand the cognitive processes respondents engage in when answering each draft item on the survey, which will help refine the survey instrument. Because we also seek to see if these supplemental items will be applicable in a variety of healthcare settings, cognitive testing will be conducted with a mix of personnel--including clinicians, nurses, and other types of staff such as administrative and clinical support staff--from hospitals, nursing homes, ambulatory surgery centers, and medical offices.
Cognitive interviews will be conducted with individual respondents to test the feasibility and applicability of the workforce safety supplemental items in all SOPS survey settings, except community pharmacies because the items are not directly applicable to that setting. English cognitive interviews will be conducted by telephone for up to 20 individuals, which includes respondents from each setting. Limited Spanish cognitive testing will also be conducted with up to 5 individuals from hospital settings.
Respondents will complete the draft supplemental items (see Attachment A). The cognitive interview guide found in Attachment B will be used during these interviews.
Feedback obtained from these interviews will be used to refine the supplemental items. The results of the cognitive testing, along with the proposed revisions, will be reviewed by the SOPS Technical Expert Panel and subject matter experts before proceeding with pilot testing.
Conduct pilot test data collection. While ideally the item set would be pilot tested in all applicable settings of care, due to limited funds, and because the SOPS Hospital Survey has the broadest adoption among the SOPS surveys, pilot test data collection will be conducted in the hospital setting only. A pilot test will be done in 25 hospitals, and based on an analysis of results, we will determine which survey items to retain and refine the questionnaire accordingly.
If cognitive testing supports the feasibility and applicability of the items in nursing homes, ambulatory surgery centers, and medical offices, then AHRQ could recommend that these settings administer the workforce safety items for beta testing and internal quality improvement purposes. AHRQ can consider pilot testing in these additional settings at a future date or encourage early adopters in these settings to share their data.
We will plan one data collection effort of the draft supplemental items aimed at assessing the psychometric properties of the items and composite measures. We will assess the variability, reliability, factor structure and construct validity of the draft supplemental items and composite measures, allowing for their further refinement (see Part A, Section 16 for analysis plan description). The draft supplemental items (see Attachment A) will be added to the SOPS Hospital 2.0 survey and be administered to approximately 11,500 clinicians and staff from 25 hospitals to facilitate analysis of the data. A hospital point of contact (POC) will be recruited to publicize the pilot test of the supplemental items and assemble a list of sample clinicians and staff. We have found that, on average, one point of contact will represent approximately 1 hospital; therefore, we estimate that we will be working with approximately 25 hospital points of contact. Instructions for the POCs are included in Attachment C, and Exhibit 2 includes a burden estimate for the POCs’ time in assisting with the pilot test. Clinicians and staff will receive notification of the supplemental item set and reminders via email. The supplemental item set will be administered via web survey. The draft pilot test notification and the weekly follow-up reminder notice are included in Attachment D.
Obtain Technical Expert Panel (TEP) and Subject Matter Expert (SME) feedback. The existing SOPS Technical Expert Panel (TEP) and additional subject matter experts in workforce safety will provide input to guide the development of the workforce safety supplemental item set. The TEP and SMEs will be engaged to provide feedback on survey drafts, and upon completion of the pilot test, to review results from the pilot test to finalize the supplemental items. The TEP and SMEs are discussed in more detail in Section 8b. This activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.
Disseminate the Survey Items. The final supplemental items will be made publicly available through the AHRQ website and we will conduct promotion and dissemination activities to encourage adoption and uptake by healthcare organizations. This dissemination activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.
This work is being conducted by AHRQ through its contractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
The responses from the cognitive testing of the draft workforce safety supplemental items will be used by project staff to test and improve the items and composite measures. Following cognitive testing results, we will revise the survey for pilot testing. Further, the information collected in the pilot test data collection effort will be used to test and improve draft supplemental items. Psychometric analysis will be conducted on the supplemental items data to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items and composite measures. Because the items are being developed to measure specific aspects or composite measures of workforce safety, the factor structure of the items will be evaluated through confirmatory factor analysis. On the basis of the data analyses, items or composite measures may be dropped to create the final supplemental item set.
Hospitals participating in the pilot test data collection effort will receive a report of their hospital-specific results. This feedback report serves as an incentive for participation, and saves the hospitals time and effort to analyze their own results.
The final supplemental item set will be made publicly available for healthcare organizations to assess how their culture supports workforce safety. The supplemental items can be used by healthcare organizations to identify areas for improvement related to their workforce safety culture. Researchers are also likely to use the supplemental items to assess the impact of workforce safety improvement initiatives.
The pilot test data collection will be conducted using a web survey. The majority of hospitals that voluntarily submitted their patient safety culture survey data to the 2018 AHRQ SOPS Hospital Database administered web surveys (83 percent)xi so this is their typical mode of administration. In addition to reducing the burden associated with survey administration (printing and tracking paper surveys), a web-based survey will offer increased security of responses and eliminate data entry expense.
We conducted a review of the literature, searching for staff surveys that measure workforce safety in healthcare settings. In reviewing the literature, we identified a number of topics that contribute to workforce safety: (1) safe patient handling; (2) slips/trips/falls/spills/hazards; (3) personal protective equipment/exposures; (4) sharps injuries; (5) workplace violence, harassment, bullying, disruptive behavior; (6) stress and burnout; (7) leader/management support for workplace health and safety; (8) psychological safety/speaking up; (9) reporting of workplace safety hazards and incidents; (10) training on workplace health and safety. Listed below is information on related surveys that measure various aspects of workforce safety. Among the surveys found, there is less information on leader/management support for workforce safety, workforce training, and reporting of hazards and incidents. As noted below, there is a void for a validated provider and staff survey with a focus on organizational culture, rather than compliance with regulations, that addresses all of these relevant areas for use in a variety of healthcare settings, succinctly captured in one, brief instrument.
The American Nurses Association (ANA) Enterprise launched a Healthy Nurse, Healthy Nation™ Grand Challenge in May 2017xii. Its purpose was to change the nation’s health by improving the health of nurses. Data was gathered through a 99 question survey that comprehensively assessed many aspects of health, safety, and wellness of the nursing workforce. While this survey asks questions that are applicable to nurses and workforce safety, it does not assess perceptions from all staff and providers in healthcare settings. The unique perspectives of other healthcare workers is important to capture in order to get an overall sense of the organizational culture around workforce safety.
Two surveys, the OHS Vulnerability Scalexiii and the Health and Safety Practices Survey of Healthcare Workersxiv, are used to measure, in part, healthcare workers’ compliance with using personal protective equipment (PPE). The OHS Vulnerability Scale assesses occupational health and safety (OHS) in hazard exposure; workplace policies and procedures; worker awareness of hazards and OHS rights and responsibilities; and worker empowerment to participate in injury and illness prevention. The scale is 27 items and includes questions on a variety of topics included in the draft Workforce Safety supplemental item set, however, some of the questions are not culture-focused, but measure the types of health and safety hazards respondents are exposed to in their job.
The Health and Safety Practices Survey of Healthcare Workers consists of one screener survey; one core survey; and seven module surveys, which each focus on a different type of exposure, such as aerosolized medication and surgical smoke. The core survey is 65 items and includes questions about exposure experiences with specific chemical agents and other hazardous materials. Because of the specificity of the questions, the survey is not as widely applicable as the AHRQ workforce safety supplemental items will aim to be.
Additional questions were included in an article by Peterson, et al xv and focused on management commitment to safety, management feedback on safety procedures, coworker safety norms, worker involvement in health and safety issues, and worker training. In a report by the U.S. Bureau of Reclamationxvi, several items were included to measure worker safety perceptions. Some of these questions are similar to questions that may be included in the proposed AHRQ workforce safety supplemental item set, but these items did not include all the topics on workforce safety we intend to capture.
The Sharps Management Surveyxvii includes questions about sharps injuries, but these questions focus on the type of sharp, when the injury occurred, and what the respondent did after they were injured. These questions are at a level of detail that is beyond the intended scope for the AHRQ workforce safety item set.
The Centers for Disease Control and Prevention has a workbook about sharps injury prevention programsxviii, which include a sample set of questions measuring workers’ perceptions of the culture of safety and their occupational exposure to blood and body fluids. Some items may be used as a reference for questions on the proposed item set, but overall, the items in this workbook are either beyond our scope or only cover a small subset of the topics that the AHRQ workforce safety item set will include.
In an article by Kerrxix about sharps injury reporting amongst surgeons, the survey items focused mainly on the specifics of reporting. The AHRQ workforce safety items will be focused on culture, so the Kerr items are not duplicative to those in the proposed item set.
Two surveys, the NDNQI RN Survey with Practice Environment Scalexx and the Safe Patient Handling Staff Assessment Surveyxxi, both measure safe patient handling. The NDNQI survey asked about a specific program, the Safe Patient Handling and Mobility program. The wording for these items is too complex or specific to be used in the AHRQ workforce safety item set. The Safe Patient Handling Staff Assessment Survey is a collection of generally worded items with wide applicability. It includes questions on injury/incident reporting, safe patient handling training, and equipment availability. While these items could be adapted and tailored to the healthcare settings we intend to assess, these surveys were limited in measuring safe patient handling.
There are many publications and surveysxxii,xxiii,xxiv,xxv, xxvi on workplace violence that measure specific incidences of physical, verbal, psychological, or sexual violence. However, none of these are focused on the culture around healthcare workplace violence. Some had questions related to reporting or training, but they did not capture the workforce culture that could impact either of those elements.
The NY State Workplace Violence Prevention Program Guidelinesxxvii included a list of sample questions that did touch on some culture aspects (via yes/no response option), however, the questions were limited to violence only while the proposed item set intends to cover more topics.
The Psychological Safety Scalexxviii, xxix is a 54-item scale used to measure psychological safety. These questions referred to “teams,” rather than healthcare workers. Some of the items in this scale were very similar to items that are included on existing AHRQ Surveys on Patient Safety Culture surveys, and therefore will not be added to the supplemental workforce safety item set.
A scale measuring employee safety voice, perceived coworker support for safety, and perceived organizational support for safety was included in an article by Tucker et alxxx. This scale focused on urban bus drivers and included relevant concepts on organizational support for safety that could be adapted to healthcare settings and incorporated as one component of workforce safety in the proposed item set.
There are several surveys that assess burnout and stress among healthcare workers. We reviewed the Mini Z Burnout Surveyxxxi, Maslach Burnout Inventoryxxxii, Nursing Occupational Stressor Scale (NOSS)xxxiii, Job Stress Scalexxxiv, and American Psychological Association Workplace Surveyxxxv. While all of these surveys measure burnout or stress, they do not include questions on the breadth of topics that the AHRQ workforce safety item set will measure. Questions from most of these surveys have been adapted to be included on the draft workforce safety item set, but none include all of the burnout/stress aspects the item set will cover and also, lack the other topics to be assessed.
In summary, while there are many surveys that assess specific components of workforce safety, we did not discover a brief, validated survey, covering the breadth of areas needed, that could be administered to all providers and staff as a supplement to the AHRQ SOPS surveys for assessing the organizational culture around workforce safety. Therefore this development and data collection effort is not duplicative of existing measures.
It is unlikely that any hospitals participating in the pilot test will be small businesses.
This effort is a one-time data collection.
The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).
The SOPS Technical Expert Panel (TEP) and subject matter experts (SMEs) will be consulted to guide the development of the Workforce Safety supplemental items. The TEP and SMEs will review drafts of the supplemental items and feedback from the cognitive interviews and assist in finalizing the supplemental items. The SOPS TEP contains 18 members from various parts of the health sector covered by the patient safety culture surveys, and there are 5 SMEs listed with expertise in workforce safety (see Attachment E).
Cognitive Interview Respondents. Survey research literature uniformly demonstrates that incentives are an effective means of communicating the importance of the study to the respondent and increasing participation. In a meta-analysis, Mercer, Caporaso, Cantor, and Townsend (2015)xxxvi show that incentives follow a dose-response model – the greater the incentive, the greater the level of respondent participation, to a point. This is equally true for the incentives offered in cognitive testing.
Monetary incentives or honoraria are usually offered to healthcare staff for participation in research. Because of the time pressures in a clinical setting, cognitive testing with clinical staff is particularly challenging.
The amounts proposed for cognitive interview incentives are based on the average incentive amounts typically paid by research recruitment vendors for these positions. As these vendors attest per their experience, healthcare positions, physicians and surgeons particularly, are very difficult to recruit for research participation unless incentives are sufficient. Therefore, to successfully recruit 25 cognitive interview participants for 1-hour cognitive interviews, we propose $300 cash remuneration for each of an estimated 6 clinicians (5 physicians, 1 surgeon), $200 cash remuneration for each of an estimated 10 staff members (8 registered nurses, 1 physician assistant or Nurse Practitioner, 1 manager), $150 cash remuneration for each of an estimated 5 staff members (1 medical assistant, 3 nursing assistants, 1 technician), and $100 cash remuneration for each of the 4 non-clinical staff members (4 medical secretaries). The amount for these incentives totals $4,950.
Pilot Test Sites. No incentive is proposed for organizations or individuals participating in the pilot test. Participating pilot hospitals will have the AHRQ SOPS Hospital Survey 2.0 administered to providers and staff free-of-charge and will receive a customized feedback report that compares their results with the aggregated results from other participating hospitals. We think these benefits are sufficient for hospitals to participate.
Individuals and organizations will be assured limitation on use of certain information under Section 944(c) of the Public Health Service Act, 42 USC 299c-3(c). This law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.
We do not consider survey questions related to the culture of workforce safety as particularly sensitive; however, if during cognitive testing we discover any sensitivities, we will modify or delete these questions accordingly.
Exhibit 1 shows the estimated annualized burden hours for the participants’ time to take part in this research. Cognitive interviews for the supplemental items will be conducted with 25 individuals (approximately 5 physicians, 1 surgeon, 8 registered nurses, 1 physician assistant or Nurse Practitioner, 1 manager, 1 medical assistant, 3 nursing assistants, and 4 medical secretaries) and require approximately one hour to complete.
For the pilot test, the supplemental items will be administered to about 11,500 individuals from 25 hospitals and require 20 minutes to complete. Assuming a response rate of 50 percent, this data collection effort will yield a total of approximately 5,700 completed questionnaires. We expect an estimated 25 POCs, each representing a single hospital, will complete the hospital information form survey (completion is estimated to take about 3 minutes).
We estimate the total annualized burden is 1,940.25 hours.
Exhibit 2 shows the estimated annualized cost burden associated with the participants’ time to take part in this research. The total cost burden is estimated to be $88,793.
Exhibit 1. Estimated annualized burden hours
Form Name/Activity |
Number of respondents/POCs |
Number of responses per respondent |
Hours per response |
Total burden hours |
Cognitive interviews |
25 |
1 |
1 |
25 |
Hospital information survey |
25 |
1 |
3/60 |
1.25 |
Pilot test |
5,742 |
1 |
20/60 |
1,914 |
Total |
5,782 |
na |
na |
Exhibit 2. Estimated annualized cost burden
Form Name/Activity |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Cognitive interviewsa |
25 |
$46.38 |
$1,160 |
Hospital information surveyb |
1.25 |
$75.74 |
$95 |
Pilot testc |
1,914 |
$45.74 |
$87,538 |
Total |
1,940.25 |
na |
$88,793 |
a Based on the weighted average wages for 5 General Internal Medicine Physicians (29-1219; $107.86 for medical offices, $82.30 for hospitals, $94.16 for nursing homes), 1 Surgeon (29-1248; $131.33 for ambulatory surgery centers), 8 Registered Nurses (29-1141; $38.20 for hospitals, $33.14 for nursing homes, $40.73 for ambulatory surgery centers), 1 Physician Assistant or Nurse Practitioner (29-1071; $53.68 or 29-1171; $52.371 for medical offices), 1 Medical Assistant (31-9092; $17.06 for medical offices), 3 Nursing Assistants (31-1131; $14.17 for nursing homes, $17.76 for ambulatory surgery centers), 1 General and Operations Manager (11-1021; $58.10 for medical offices), 1 technician (29-2010; $27.26 for hospitals), and 4 medical secretaries (43-6013; $17.69 for medical offices, $18.61 for hospitals, $19.76 for ambulatory surgery centers);
b Based on the average wages for General and Operational Managers (11-1021; $75.74) in the hospital setting;
c Based on the weighted average wages for 442 General Internal Medicine Physicians (29-1219; $82.30), 3533 Registered Nurses (29-1141; $38.20), 884 Clinical Lab Technologist/Technicians (29-2010, $27.26), 883 General and Operations Managers (11-1021, $75.74) in the hospital setting;
* National Occupational Employment and Wage Estimates in the United States, May 2019, “U.S. Department of Labor, Bureau of Labor Statistics” (available at http://www.bls.gov/oes/current/naics4_621100.htm [for medical office setting], https://www.bls.gov/oes/current/naics4_622100.htm [for hospital setting], https://www.bls.gov/oes/current/naics4_621400.htm [for ambulatory surgery center setting], and https://www.bls.gov/oes/current/naics3_623000.htm [for nursing home setting]
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
Exhibit 3 shows the estimated annualized cost for this project, which is estimated at $300,000. This cost includes $4,950 in incentives to the cognitive interview respondents under Data Collection Activities.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Annualized Cost |
Project Development |
$40,000 |
Data Collection Activities |
$110,000 |
Data Processing and Analysis |
$40,000 |
Publication of Results |
$25,000 |
Project Management |
$35,000 |
Overhead |
$50,000 |
Total |
$300,000 |
Exhibit 4. Estimated Annual cost to AHRQ for Project Oversight
AHRQ Position |
% Time |
Annualized Cost |
GS 15 |
3% |
$4,852 |
GS-14 |
3% |
$4,125 |
GS-13 |
3% |
$3,490 |
Total |
|
$12,467 |
This data collection effort is a new activity.
As soon as OMB approval is received, cognitive testing activities will begin. The estimated time schedule to conduct these activities is shown below:
Complete up to 25 cognitive interviews (2 months)
Pilot test data collection (5 months)
Data analysis, feedback report production, and development of technical reports (6 months)
Final Workforce Safety supplemental item set and development of toolkit materials (1 month)
The final version of the Workforce Safety supplemental item set, technical reports, and accompanying toolkit materials will be made publicly available on the AHRQ website.
This section describes the specific analyses that we will conduct on the pilot test data.
Pilot Test Psychometric Analysis. Psychometric analysis will be conducted to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items. Because the supplemental items are being developed to measure specific aspects or composite measures of culture of workforce safety, the factor structure of the supplemental items will be evaluated through confirmatory factor analysis. On the basis of the data analyses, items or composite measures may be dropped.
Descriptive Statistics
The means, standard deviations, and response frequencies for the supplemental items will be examined to ensure that respondents and hospitals exhibit adequate response variability on the supplemental items. In addition, items will be examined to ensure that there are low rates of missing data (i.e. lower than 20 percent missing response per item). Poorly functioning items will be identified.
Confirmatory Factor Analysis
A confirmatory factor analysis will be conducted to initially examine whether groups of items intended to measure a specific patient safety composite measure are interrelated, ignoring the nesting of respondent data within hospitals. Factor loadings for each item in an a priori composite measure will be considered as having an adequate contribution to a particular composite measure or factor if the strength of the item's relationship to that factor (i.e., its factor loading), is 0.40 or greater.
We will also examine overall model fit indices using standard fit statistics: the chi-square, comparative fit index (CFI), and the standardized root mean square residual (SRMR). For chi-square statistics, lower and non-significant chi-squares indicate good fit. The factor structure is determined to adequately fit the data if the CFI is at least 0.90. A value of zero for the SRMR indicates perfect fit, but a value less than 0.08 is considered a good fit.
Intraclass Correlations (ICCs) and Design Effects
Intraclass correlations (ICCs) will be computed for each composite measure. ICC's determine if substantial variation exists between groups compared to variation within groups. ICCs above 0.05 or 5 percent indicate that the between group variance is greater than expected by chance and imply that nesting in groups does have an effect on the responses of individuals.
Given that ICCs are likely to be influenced when there are many groups with few individuals within the groups (or when there are few groups with many individuals within the groups), we will also examine design effects, which take into account within-group sample size. A design effect of 2 or more implies that group membership or nesting of individuals within groups does have an effect on the responses of the individuals.
Reliability Analysis
Reliability analyses will then be performed on the composite measures to examine whether individuals responded consistently to the items within each composite measure. Internal consistency reliability will be calculated using Cronbach's alpha. The minimum criterion for acceptable reliability is an alpha of at least 0.70.
Intercorrelations
Intercorrelations among the supplemental item set’s patient safety composite measures will also be examined. Intercorrelations will be explored at the individual and hospital levels of analysis. While the composite measures should be correlated since they measure aspects of the workforce safety culture, the intercorrelations should not be extremely high (0.80 or higher) because very high intercorrelations indicate that the composite measures may not be unique enough to be considered separate constructs or measures. While there is no standard criterion for acceptable levels of dimension intercorrelations and construct validity, in general, such correlations should be less than 0.80 for the composite measures to be considered unique and to avoid problems with multicollinearity.
The above analyses will be used to determine which, if any, items and composite measures are functioning poorly and remove them from the survey to derive a final set of items and composite measures with good psychometric properties and reduce the overall length of the final supplemental item set. The Technical Expert Panel and subject matter experts will be informed of the data analysis results and asked to advise Westat on which items to retain or drop (when the psychometric results do not provide enough guidance and decisions can be made on the content value of the items).
The final supplemental item set will be made publicly available on the SOPS pages of the AHRQ website for use by healthcare organizations and researchers.
No exemption is requested.
Attachment A: Draft Workforce Safety Supplemental Item Set
Attachment B: Draft Workforce Safety Cognitive Interview Guide
Attachment C: Hospital Point of Contact (POC) Instructions
Attachment D: Survey Invitation and Reminder Notices
Attachment E: Subject Matter Experts and Technical Expert Panel Members
i Healthcare Research and Quality Act of 1999. Available at https://www.ahrq.gov/policymakers/hrqa99a.html. Last accessed 7/6/2020.
ii See Section 902, (a) (8) of the Healthcare Research and Quality Act of 1999. Available at https://www.ahrq.gov/policymakers/hrqa99a.html. Last accessed 7/6/2020.
iii See Section 912, (b) (2) (A) (ii) (I) and (iii) (II) and (c) (1) (2) and (3) of the Healthcare Research and Quality Act of 1999. Available at http://www.ahrq.gov/policymakers/hrqa99b.html. Last accessed 7/6/2020.
iv Worker Safety in Hospital, Occupational Safety and Health Administration. Available at https://www.osha.gov/dsg/hospitals/index.html. Last accessed 7/14/2020.
v Worker Safety in Hospitals, Safe Patient Handling, Occupational Safety and Health Administration. Available at https://www.osha.gov/dsg/hospitals/patient_handling.html. Last accessed 7/14/2020.
vi Centers for Disease Control/National Institute for Occupational Safety and Health (NIOSH). Available at https://www.cdc.gov/niosh/topics/healthcare/default.html. Last accessed 7/14/2020.
vii Bureau of Labor Statistics. Employed persons by detailed industry, sex, race, and Hispanic or Latino ethnicity, 2019. Available at https://www.bls.gov/cps/cpsaat18.htm. Last accessed 7/14/2020.
viii AHRQ press release May 2018. Available at https://www.ahrq.gov/news/newsroom/press-releases/national-steering-committee.html. Last accessed 7/14/2020.
ix A National Strategy for Reducing Harm. Institute for Healthcare Improvement, Boston, MA. Available at http://www.ihi.org/Engage/Initiatives/National-Steering-Committee-Patient-Safety/Pages/default.aspx. Last accessed 7/14/2020.
x Ehrlich, H., McKenney, M., & Elkbuli, A. (2020). Protecting our healthcare workers during the COVID-19 pandemic. The American Journal of Emergency Medicine, 38(7), 1527–1528 https://doi.org/10.1016/j.ajem.2020.04.024)
xi SOPS Hospital Database. Content last reviewed March 2018. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/sops/databases/hospital/index.html. Last accessed 7/6/2020.
xii American Nurses Association Healthy Nurse, Healthy Nation™. Available at https://www.healthynursehealthynation.org/en/about/survey/. Last accessed 7/14/2020.
xiii Institute for Work & Health (2015). OHS Vulnerability Measure [Survey]. Available at https://www.iwh.on.ca/sites/iwh/files/iwh/tools/ohs_vulnerability_measure_guide_2017.pdf. Last accessed 7/14/2020.
xiv The National Institute for Occupational Safety and Health (NIOSH) (2014). Health and Safety Practices Survey of Healthcare Workers [Survey]. Available at https://www.cdc.gov/niosh/topics/healthcarehsps/aboutstudy.html. Last accessed 7/14/2020.
xv Peterson, K., Rogers, B.M.E., Brosseau, L.M., Payne, J., Conney, J., Joe, L., & Novak, D. (2016). Differences in hospital managers’, unit managers’, and health care workers’ perceptions of the safety climate for respiratory protection. Workplace Health Saf., 64(7), 326-336
xvi U.S. Bureau of Reclamation. (2016). Baseline assessment of employee perceptions of safety culture. https://www.usbr.gov/safety/sap/documents/teamreports/Team14.pdf
xvii Reddy, S., Joseph, V., Liu, Z., & Straker, T. (2018). Characteristics of sharps injuries in anaesthesia providers in New York state: A cross-sectional study. Journal of Clinical and Diagnostic Research, 12(12), 6-10
xviii Centers for Disease Control and Prevention (2008). Workbook for designing, implementing, and evaluating a sharps injury prevention program. Retrieved from https://www.cdc.gov/sharpssafety/pdf/sharpsworkbook_2008.pdf
xix Kerr, H., Stewart, N., Pace, A., & Elsayed, S. (2009). Sharps injury reporting amongst surgeons. Ann R Coll Surg Engl, 91, 430-432
xx Press Ganey (2018). NDNQI RN Survey with Practice Environment Scale© [Survey]. Available at https://www.upstate.edu/nursing/documents/ndnqi_rn_survey_with_pratice_environment_scale.pdf. Last accessed 7/16/2020
xxi Lax, M., Zoeckler, J., Goessling, K., Greetham, S. (2019). New York State’s Safe Patient Handling Law. Occupational Safety and Health Working Group. http://ohccupstate.org/index_htm_files/Safe%20Patient%20Handling%20Survey%20Report%20April%202019%20FINAL.pdf
xxii The American College of Emergency Physicians. (2018). ACEP Emergency Department Violence Poll research results. https://www.emergencyphysicians.org/globalassets/files/pdfs/2018acep-emergency-department-violence-pollresults-2.pdf
xxiii Texas Center for Nursing Workforce Studies. (2016). Workplace Violence Against Nurses Survey (Publication No. 25-14922). Texas Department of State Health Services.
https://cdn.ymaws.com/www.texasnurses.org/resource/resmgr/docs/WPVAN_Report_12142016.pdf
xxiv Appendix 1. SOVES-A questionnaire. (2013). Clinical Medicine, 13(6), 1. https://doi.org/10.7861/clinmedicine.13-6-1
xxv International Labour Office, International Council of Nurses, World Health Organization, Public Services International. (2003). Workplace Violence in the Health Sector Country Case Study – Questionnaire. Available at
https://www.who.int/violence_injury_prevention/violence/interpersonal/en/WVquestionnaire.pdf?ua=1. Last accessed 7/16/20.
xxvi Emergency Nurses Association (2011). Emergency Department Violence Surveillance Study. Institute for Emergency Nursing Research. https://www.ena.org/docs/default-source/resource-library/practice-resources/workplace-violence/2011-emergency-department-violence-surveillance-report.pdf?sfvrsn=5ad81911_4
xxvii New York Department of Labor. (2009). NY State Workplace Violence Prevention Program Guidelines.
xxviii Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350-383
xxix Edmondson, A. C. (2018). The fearless organization: Creating psychological safety in the workplace for learning, innovation, and growth. John Wiley & Sons.
xxx Tucker, S., Chmiel, N., Turner, N., Hershcovis, M. S., & Stride, C. B. (2008). Perceived organizational support for safety and employee safety voice: The mediating role of coworker support for safety. Journal of Occupational Health Psychology, 13(4), 319-330.
xxxi Mini-Z Burnout Survey. Available at https://www.christushealth.org/-/media/files/for-providers/ama-mini-z-burnout-survey.ashx?la=en. Last accessed 7/16/2020.
xxxii Maslach Burnout Inventory™. Available at https://www.mindgarden.com/117-maslach-burnout-inventory. Last accessed 7/14/2020.
xxxiii Chen, Y., Guo, Y.L. Lin, L., Lee, Y., Hu, P., Ho, J. & Shaio, J.S. (2020). Development of the Nurses Occupational Stressor Scale. International Journal of Environmental Research and Public Health., 17, 649.
xxxiv Job Stress Scale. Available at https://www.tandfonline.com/doi/full/10.1080/23311975.2015.1134034. Last accessed 7/14/2020.
xxxv APA Workplace Survey. Available at https://www.apa.org/news/press/releases/phwa/workplace-survey.pdf. Last accessed 7/14/2020.
xxxvi Mercer A., Caporaso, A., Cantor, D., and Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79(1), 105-129.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | AHCPR |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |