Evaluation of the Third Decade of the National Occupational Research Agenda (NORA) Council Effectiveness
Supporting Statement A
Emily J.K. Novicki
NIOSH Program Portfolio and NORA Coordinator
Health Scientist
National Institute for Occupational Safety and Health
Phone: 404-448-2581
Fax: 404-498-2573
September 11, 2019
Table of Contents
Section A: Justification
A1. Circumstances Making the Collection of Information Necessary
A2. Purpose and Use of the Information Collection
A3. Use of Improved Information Technology and Burden Reduction
A4. Efforts to Identify Duplication and Use of Similar Information
A5. Impact on Small Businesses or Other Small Entities
A6. Consequences of Collecting the Information Less Frequently
A7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
A8. Comments in Response to the Federal Register Notice and Efforts to
Consult Outside the Agency
A9. Explanation of Any Payment or Gift to Respondents
A10. Protection of the Privacy and Confidentiality of Information Provided by Respondents
A11. Institutional Review Board (IRB) and Justification for Sensitive Questions
A12. Estimates of Annualized Burden Hours and Costs
A13. Estimates of Other Total Annual Cost Burden to Respondents and
Record Keepers
A14. Annualized Cost to the Government
A15. Explanation for Program Changes or Adjustments
A16. Plans for Tabulation and Publication and Project Time Schedule
A17. Reason(s) Display of OMB Expiration Date is Inappropriate
A18. Exceptions to Certification for Paperwork Reduction Act Submissions
List of Attachments
1. Authorizing Legislation
2. 60-Day Federal Register Notice
3. IRB Determination of Non-Research
4. Council member survey
5. Pre-Survey Email to Potential Respondents
6. First Email with Survey Link
7. Second Email with Survey Link
8. Final Email with Survey Link
Goal of the study: Understand how effectively National Occupational Research Agenda (NORA) councils are fulfilling their three purposes (information sharing, networking, and dissemination/implementation).
Intended use of the resulting data: Determine best practices for NORA councils to improve their effectiveness in the future.
Methods to be used to collect: Qualitative Survey
The subpopulation studied: Members and leaders of NORA councils
How data will be analyzed: Descriptive analysis
This is a new information collection request. The National Institute for Occupational Safety and Health (NIOSH) is responsible for conducting research and making recommendations to prevent worker injury and illness, as authorized in Section 20(a)(1) of the Occupational Safety and Health Act (29 U.S.C. 669) (Attachment 1). In 1995-1996, NIOSH saw an opportunity to enhance its ability to accomplish its mission through partnerships that involved a broad national stakeholder base in occupational safety and health. With stakeholder input, NIOSH developed and launched a decade-long partnership program titled the National Occupational Research Agenda (NORA) in 1996. Participation in NORA includes stakeholders from universities, large and small businesses, professional societies, government agencies, and worker organizations. NORA runs in ten year cycles, with the first decade from 1996-2006, the second from 2006-2016, and the third from 2016-2026.
The structure of NORA has evolved over time, and now, in the third decade, it is organized into ten industry sectors based on major areas of the U.S. economy, and seven health and safety cross-sectors organized according to the major health and safety issues affecting the U.S. working population. The work of the sectors and cross-sectors is managed through a partnership structure of councils. Each council is co-led by a NIOSH co-chair and a co-chair from another organization. Most of the councils consist of approximately 1/3 NIOSH members with the rest from other organizations, including some from other Federal agencies. The corresponding NIOSH sector or cross-sector program provides logistical support for council meetings and activities. Each of the 17 councils develops and maintains an agenda for the decade for its sector. The sector agendas become part of the national agenda for improvements in occupational safety and health through research and partnerships. Representing all stakeholders, the councils use an open process to set research objectives, share information, encourage partnerships, and promote improved workplace practices.
NIOSH is conducting a process evaluation throughout the third decade of NORA. This includes collecting data from NIOSH personnel about NORA council activities on an annual basis. NIOSH will also survey Federal employees who are or were leaders or members of a NORA council during the third decade. In addition, we are seeking permission to open the same survey to current and former council leaders and members who are not Federal employees. This will take no more than 1 year. The web-based survey requests information on council activities, the effectiveness of the council and its processes, and suggestions for improving the effectiveness and impact of NORA councils in the future.
The requested collection of information is necessary for NIOSH to fully assess the efficiency and effectiveness of the NORA councils. Without this data collection, NIOSH’s internal review of NORA would lack critical stakeholder input from its many non-Federal partners.
The study is an internal, process evaluation with a utilization-focused approach. This approach will provide council leaders with evaluation data that will help councils function as effectively as possible and maximize their contributions to improved occupational safety and health. This evaluation focuses on whether council members are satisfied with their experience, and how well councils are achieving their stated purposes of information sharing, networking, and disseminating/implementing evidence-based solutions. The information will also be used by NIOSH to develop best practices for effective NORA councils. The goal is to provide information to help councils function as effectively as possible to maximize their contributions towards the NORA Agendas and to improved occupational safety and health.
The survey results will be summarized and included in a comprehensive report in aggregate and posted on the NORA website. Council co-chairs will receive aggregate data from the survey for their particular council. This level of analysis will not be publicly posted, however.
As the steward of NORA, it is NIOSH’s responsibility to ensure that councils, which are central to the work of NORA, are operating well. In order to do this, it is necessary to collect information from non-Federal participants. With this information, NIOSH will be able to help councils function more effectively and achieve greater occupational safety and health impact. Without this data collection, NIOSH’s internal review of NORA would lack critical stakeholder input from its many non-Federal partners.
This is the first of three planned data collections. A similar survey will be fielded in 2022 and 2025, allowing NIOSH to monitor the effectiveness of councils over the course of the third decade of NORA and make course corrections if needed.
A3. Use of Improved Information Technology and Burden Reduction
100% of the responses will be through Survey Monkey and the internet. We know each respondent has access to the internet, because they routinely receive and respond to NORA sector council communications sent by email. In addition, the process of completing an online survey imposes only a small burden. This process consists of opening the invitation email, clicking on a link to their questionnaire, clicking on responses, having the ability to close the questionnaire and come back to it later and submitting the survey with a click. Such a process reduces the burden on the respondent compared to other survey administration approaches.
Additional efforts in designing the survey and its administration reduce the burden. Reminder
emails will not be sent to those who have already responded. Correspondence about the survey is concise. As a result of conducting a peer review of the instrument, the number of survey items was reduced by 30% to the minimum required to achieve the purpose of the review. Most responses are multiple-choice. For the six questions with narrative responses, there is no lower or upper limit on response length.
NORA is a partnership program organized by NIOSH. There is no other review of NORA ongoing or planned within NIOSH. No other entity has identified itself as collecting information to review the National Occupational Research Agenda program during NORA sector council meetings, other NORA discussions or in the literature known to NIOSH. Some individual NORA sector councils have internal discussions periodically about how to improve their effectiveness and impact, but those discussions do not provide the type of information NIOSH needs to determine best practices and help all councils improve their effectiveness.
This data collection is intended for volunteer members of NORA sector councils. Some may be owners or employees of a small business, but the questions are only about their volunteer activities. The number of questions has been limited to the absolute minimum for the intended use of the data.
This is a one-time information collection. The survey instrument will be modified for the FY 2022 collection based on the findings.
This request fully complies with the regulation 5 CFR 1320.5.
A 60-day Federal Register notice was published in the Federal Register on April 19, 2018, vol. 83, No. 76, pp. 17415-16 (see att 2). CDC did not receive public comments related to this notice.
Respondents will not receive any payments or gifts for participating.
NIOSH’s Information Systems Security Officer has reviewed the submission and determined that Privacy Act does not apply. The evaluators will collect the names and business email addresses of council leaders and members who will be asked to complete the survey. Personal identification information (i.e., respondent names) will not be collected in the survey instrument and the unit of analysis is the council, not the individual.
It is possible that due to the small number of council leaders, it may be possible for the NIOSH analyst to guess the identity of individual respondents based on their answers. Steps will be taken to make this very unlikely. Survey Monkey will not associate the email address of the respondent with the individual response data. Identifiers will be removed before quoting narrative text in draft reports.
The introduction to the survey says the following regarding privacy and security of the information submitted:
Your participation is completely voluntary and there will be no adverse effects for you if you choose not to participate. You can discontinue participation at any time and choose whether to complete the survey at a later date. Your previous responses will be saved. Your responses will be secured and not disclosed unless otherwise compelled by law. All survey results will be reported in aggregate form, except for some narrative information that may be summarized, or quoted in whole or in part. All identifiers will be removed from narrative text. Completing and submitting this survey will imply your consent.
Submitted information will be secured in a password-protected Survey Monkey account that is included in a subscription paid by NIOSH. After the information is downloaded in an Excel file for analysis, the file will be password protected by the PI and stored only in government file space. The passwords will only be available to her supervisors.
IRB Approval
The requested information collection has been determined to be exempt from IRB review. See Attachment 3 for the IRB determination of non-research.
Sensitive Questions
None of the questions request sensitive information.
A12. Estimates of Annualized Burden Hours and Costs
A 17 item survey (Attachment 4) has been developed and will be sent to all NORA sector council members or leaders. Some members are NIOSH or other Federal employees. Only those who are not Federal employees are included in the burden estimates. As of December 2017, there are approximately 380 non-Federal employees serving as members of NORA councils. Historically, councils grow over time, so this number is likely to grow between now and when the survey is fielded. We are estimating 425 potential respondents, based on an estimated 5% growth rate per year. This data collection follows the approach utilized in the second decade review of NORA, which also had a survey of NORA council members (OMB ID 0920-1052, exp. 3/31/2018) and achieved a 53% response rate. Therefore, we estimate a total of 225 responses for this survey. A pilot test of the questionnaire was conducted by asking 5 NIOSH employees who are members of NORA councils to complete the questionnaire and provide feedback. Respondents to the pilot test estimated the questionnaire requires approximately 12 minutes to complete. The total estimated annualized burden hours is 45 hours. The table below provides a summary of the total burden hours.
Estimate of Annualized Burden Hours
Type of Respondent
|
Form |
No. of respondents |
No. of responses per respondent |
Average burden per response in hours |
Total Burden Hours |
Non-federal NORA Council members or leaders |
Council survey |
225 |
1 |
12/60 |
45 |
Total |
|
|
|
|
45 |
The following table uses the Bureau of Labor Statistics Occupational Employment and Wages results for May 2018. The respondents are typically senior occupational safety and health professionals who have moved into program management positions. The estimated hourly wage rate used is the median for the category Natural Sciences Managers (11-9121).
Type of Respondents |
Form Name |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Non-federal NORA Sector Council members or leaders |
Council Survey |
45 |
$59.55 |
$2,680 |
|
Total |
$2,680 |
||
There are no capital or start-up costs for collection of information in this project.
There is no cost to respondents for operation or maintenance in this project. Respondents are not asked or expected to purchase any services.
Costs for conducting the survey are summarized in the table below. The request is for one year, so all costs are automatically annualized costs. There will be no new overhead, support staff, or construction required for the survey administration and data analysis.
Table A1. Annualized cost to the Federal Government
Personnel--1 GS-12, 30% time |
$22,278 |
Survey Monkey subscription |
$ 408 |
Total of Annualized estimate of federal cost |
$22,686 |
This is a new data collection.
Project Time Schedule |
|
Activity |
Time Schedule |
Pre-survey email sent to respondents |
1 week after OMB approval |
First email with survey link |
2 weeks after OMB approval |
Second email with survey link |
4 weeks after OMB approval |
Data collection ends |
5 weeks after OMB approval |
Data downloaded and validation completed |
9 weeks after OMB approval |
Data analyses completed |
15 weeks after OMB approval |
NIOSH review of draft report completed |
20 weeks after OMB approval |
Report reviewed and revisions incorporated |
29 weeks after OMB approval |
Final document prepared and released to the public |
36 weeks after OMB approval |
All quantitative material in this study, including the council survey, will be organized in a table format in Microsoft Excel for data analysis. Nominal survey data such as demographic information and multiple choice items will be depicted through frequency counts.
Qualitative data will be compiled and analyzed using qualitative analysis software. Data will be analyzed using qualitative content analysis (QCA). This approach involves exploring multiple pieces of qualitative information (e.g., narrative) to develop general themes of the data, determined by research questions or study objectives. A pre-determined code book, based on the research questions, will be developed for standardized analysis of qualitative responses.
Both the quantitative and qualitative survey data will be summarized overall and by each of the 10 NORA sectors. Qualitative narrative will be highlighted for each sector, featuring impact stories and other accomplishments.
The display of the OMB expiration date is not inappropriate.
There are no exceptions to the certification.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |