Supporting STatement PART A
A.1. Circumstances that Make Data Collection Necessary
Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.
The National Science Foundation (NSF) seeks approval to collect ongoing information on the perceptions and experiences of individuals who have submitted proposals to NSF (applicants) and/or served as reviewers for NSF proposals between FY 2018 and FY 2020 and between FY 2020 and FY 2022. This information will be gathered biennially through the Merit Review Survey, a Web-based survey instrument. The primary objective of the survey is to assess applicant and reviewer perceptions of the quality of the merit review process. Another aim is to gather information on the experiences of key subpopulations and how those perceptions and experiences have changed over time. The survey is necessary to support NSF’s stated performance goal to “improve the quality of written reviews of NSF proposals” as outlined in their FY 2018 through FY 2021 Budget Requests to Congress, and to “[create] a diverse and inclusive science and engineering enterprise” as emphasized in the National Science Board’s Vision 2030.1
Background
NSF is an independent Federal agency tasked with promoting the progress of science; advancing the national health, prosperity, and welfare; and securing the national defense. NSF funds research and education in science and engineering by conferring limited-term grants. This funding goes to specific research proposals judged the most promising by a rigorous merit review process. Of 41,000 proposals submitted in FY 2019, 11,300 were selected for funding.2
Proposals for funding are submitted to NSF and evaluated in a fair, competitive, transparent, and rigorous system of merit review. On average, NSF proposals are reviewed by three to five independent reviewers who are experts in the field and do not work at NSF. External reviewers provide expert advice about the merits of proposals that inform NSF’s decision-making process to ensure the best projects make it to the funding stage. All proposals submitted to NSF are evaluated against two merit review criteria approved by the National Science Board (NSB): intellectual merit and broader impacts. NSF defines intellectual merit as the potential to advance knowledge, and broader impacts as the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. Additional review criteria beyond intellectual merit and broader impacts may be added to announcements or solicitations that speak to specific program goals and objectives.3
In alignment with the strategic goal outlined in its 2018–2022 Strategic Plan to “ensure that NSF’s… merit review process is of high quality and integrity”, NSF strives to continuously improve the merit review process to promote fairness, transparency, effectiveness, and efficiency in decision making.4 Core strategies to achieve this goal include allowing for open access to data about the merit review process and increasing the usefulness of those data. NSF has engaged in such continuous improvement activities, and efforts to monitor them, for many years. In 2007, NSF’s Impact of Proposal and Award Management Mechanisms working group launched the “NSF 2007 Proposer Survey” (OMB # 3145-0157). In 2015, 2017, and 2019, NSF’s Office of Integrative Activities conducted three iterations of the “Assessment of Investigator and Reviewer Experiences Survey” (OMB # 3145-0215). The proposed 2021 and 2023 iterations of the Merit Review Survey will continue this work and gauge whether satisfaction levels have changed with the merit review process since the prior survey iteration.
A.2. Purpose and Use of the Information
Indicate how, by whom, how frequently, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.
The data collected using the instrument included in this ICR (Appendix A) will be used internally by NSF for the following purposes:
Assess applicant and reviewer perceptions of, and satisfaction with, various aspects of the merit review process.
Document the time burden the merit review process places on reviewers and applicants.
Examine applicant and reviewer perceptions of the quality of reviews and of proposals.
Assess the changes in applicant and reviewer perceptions of burden, satisfaction, and quality between the 2019 and 2021 surveys and the 2021 and 2023 surveys.
Examine the variation of applicant and reviewer perception of satisfaction, burden, and quality by key population subgroups, including race/ethnicity, gender, and disability.
Describe the extent to which NSF’s reviewer orientation video is correlated with awareness of different types of cognitive biases and the use of strategies to reduce cognitive bias and to provide constructive feedback.
Describe the extent to which the elimination of annual proposal deadlines affected reviewer and applicant burden, perceptions of proposal and review quality, and satisfaction with the merit review process.
Describe applicants and reviewers experiences with student support programs as well as what NSF application and funding support is associated with the receipt of financial support from NSF as an undergraduate or graduate student.
The information will be collected biennially using a Web-based survey from applicants who have submitted proposals and reviewers who have participated in the merit review process. The first data collection will be conducted in Fall 2021 will survey applicants who submitted proposals and individuals who served as reviewers between FY 2018 and FY 2020. The second data collection will survey individuals who were applicants or reviewers between FY 2020 and FY 2022 and will be conducted in Fall 2023.
NSF will use data from the survey to inform continuous improvements to the fairness, effectiveness, and efficiency of the merit review process. NSF has used data on quality metrics to inform the effectiveness of a reviewer orientation designed to improve review quality.5 In addition, several directorates have transitioned from piloting rolling submissions (those with no annual proposal deadline) to implementing this fully “to increase efficiency in operations”, a topic that was informed by the 2015 and 2017 Merit Review Surveys.6
The 2021 Merit Review Survey will further assess these findings by looking at respondent perceptions of quality by key demographic subpopulations. Beginning with the 2021 survey, data will be analyzed by subgroups of critical interest to NSF, including race/ethnicity, gender, and disability status to understand how the experiences of applicants and reviewers may vary across groups and how they can be improved. Analyses across NSF directorates will also be conducted to understand variation in applicant and reviewer experiences within NSF with findings reported back to directorates to inform ongoing directorate-level policy decisions.
A.3. Use of Information Technology and Burden Reduction
Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.
The Merit Review Survey will be administered via the web and is expected to take 20 minutes to complete. Using contact information that NSF has on file for applicants and reviewers, potential respondents will be emailed a unique survey link to access the secure web survey. The web survey will be consistent with the attached paper version, including the OMB approval number, once granted. This method offers the most efficient means of reaching applicants and reviewers. In addition, programmed skip patterns will allow for respondents to only answer questions relevant to them and their experiences. The survey will allow respondents to break off and resume completing the survey, as needed, and to complete the survey at their convenience.
A.4. Efforts to Identify Duplication and Use of Similar Information
Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purpose described in item 2 above.
The most recent data on the merit review process were collected two years ago. In 2007, NSF administered the “NSF 2007 Proposer Survey,” and in 2015, 2017, and 2019 NSF administered similar iterations of the “Assessment of Investigator and Reviewer Experiences Survey” which assessed levels of satisfaction among applicants and reviewers and areas for improvement. Conducting this survey in 2021 and 2023 will allow NSF to compare responses to these prior iterations and determine whether applicant and reviewer satisfaction and experiences have changed. This information cannot be obtained through other data sources. While NSF has access to administrative data on the merit review process, such as the numbers of proposals received and granted and the characteristics of applicants and reviewers, conducting the survey is the only way to fulfill key research objectives such as assessing applicant and reviewer perceptions and opinions of the process, documenting the burden associated with the review process, describing perceptions of new strategies to reduce cognitive biases and burden, and documenting applicant and reviewer experiences with student support programs.
A.5. Impacts on Small Businesses or Other Small Entities
If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.
NSF has determined that the requirements for this information collection do not adversely impact small businesses or other small entities. All new data is collected directly from applicants and reviewers.
A.6. Consequences of Collecting the Information Less Frequently
Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
The information collection proposed for this study consists of a biennial survey anticipated to take approximately 20 minutes to complete. This data collection will be conducted in separate efforts in 2021 and 2023. If these data are not collected, NSF will not have critical information that can be used to refine the process including participant satisfaction with the merit review process, variation of experiences across critical population subgroups, and an assessment of whether satisfaction, quality, and fairness have improved and burden has reduced over time.
A.7. Special Circumstances Relating to the Guideline of 5 CFR 1320.5
Explain any special circumstances that would cause an information collection to be conducted in a manner:
Requiring respondents to report information to the agency more often than quarterly
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it
Requiring respondents to submit more than an original and two copies of any document
Requiring respondents to retain records other than health, medical, government contract, grant-in-aid, or tax records for more than 3 years
In connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use
Requiring respondents to submit proprietary trade secret or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.
There are no special circumstances for the proposed data collection. The collection of information will be conducted in a manner consistent with the guidelines in 5.CFR 1320.5.
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency
If applicable, provide a copy and identify the date and page number of publications in the Federal Register of the agency’s notice, soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments.
Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and recordkeeping, disclosure, or reporting form, and on the data elements to be recorded, disclosed, or reported.
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on October 15, 2020 (Vol. 85, No. 200, page 65436). No public comments were received.
Consultations Outside of the Agency
NSF consulted with RIVA Solutions and Insight Policy Research on the survey design, administration, and analysis of the 2021 Merit Review Survey.7 Many of the survey items were taken from the “NSF 2007 Proposer Survey” (OMB Control # 3145-0157) and the 2015, 2017, and 2019 iterations of the “Assessment of Investigator and Reviewer Experiences Survey” (OMB # 3145-0215). Revisions to survey items underwent extensive review by NSF staff.
A.9. Explanation of Any Payment or Gift to Respondents
Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.
No payment or gift will be offered to survey respondents.
A.10. Assurance of Confidentiality Provided to Respondents
Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.
Participants in this study will be subject to safeguards as provided by the Privacy Act of 1974 (5 USC 552a), which requires the safeguarding of individuals against invasion of privacy. The Privacy Act also provides for the confidential treatment of records maintained by a Federal agency and retrieved by either the individual’s name or some other personal identifier. The use of these data will be allowed under the Privacy Act Systems of Record Notice, NSF-50 and NSF-51, which have been published in the Federal Register. In addition, any reporting of data in aggregate form will be consistent with OD 18-10 interim guidance.
Potential respondents will be assured that their responses will be kept confidential and reported in a manner that will not identify individual respondents. This confidentiality statement will be provided in the prenotification, invitation, and reminder emails as well as in the “Introduction” section of the web survey. These communications will also make clear to potential respondents that their participation is voluntary.
Several specific measures will be taken to protect respondent privacy. The survey contractor will use Qualtrics web survey platform to collect the data. Potential respondents will receive a unique link to access the survey so that they can only enter and view their own data. Qualtrics’ servers are protected by high-end firewall systems and scans are performed regularly to ensure that any vulnerabilities are quickly found and patched. Application penetration tests are performed annually by an independent third-party. All services have quick failover points and redundant hardware, with backups performed daily. Access to systems is restricted to specific individuals who have a need-to-know such information and who are bound by confidentiality obligations. Access is monitored and audited for compliance. Qualtrics uses Transport Layer Security (TLS) encryption (also known as HTTPS) for all transmitted data. Surveys may be protected with passwords and services are hosted by trusted data centers that are independently audited using the industry standard SSAE-18 method. In addition, Qualtrics has certification for ISO 27001 and HITRUST and is FedRAMP Authorized. FedRAMP is the standard of U.S. government security compliance, with over 300 controls based on the highly regarded NIST 800-53 that requires constant monitoring and periodic independent assessments.
The study contractor collecting and analyzing the survey data has established data security plans for handling all data during survey execution and processing. The information technology and security infrastructure that will be used in this data collection is aligned with the Federal Information Security Modernization Act Moderate Baseline standard. The study team will implement those security controls in the National Institute of Standards and Technology (NIST) SP 800-53 Revision 4 and NIST SP 800-171 Revision 1 related to protecting the confidentiality, integrity, and availability of controlled unclassified Information and other sensitive datasets. Any information that could identify a respondent, such as name and email address will be stored separately from the survey data file. Furthermore, the link between any response and any individual will be secured in an encrypted and restricted computer file and pursuant to applicable NSF regulations.
A.11. Justification for Sensitive Questions
Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
None of the questions in the survey instrument are of a sensitive nature. Survey questions are limited to asking respondents to report their behavior (i.e., past participation in proposal submission and/or reviews, number of hours spent on a proposal) and their perceptions of their experiences (i.e., fairness of review process, quality of reviews) as relates to the merit review process. In addition, there are limited demographic questions (race, ethnicity, gender, disability, and years since highest degree granted), all of which can be skipped by the respondent, if desired. Respondents will also be notified in outreach communications and at the beginning of the web survey that their participation is voluntary and that responses will be kept confidential and used only in an aggregate form. As part of the consent process, respondents will be informed that they may choose to not answer any specific questions, and as noted in A.10 above, that responses will be treated as private.
A.12. Estimates of Hour Burden Including Annualized Hourly Costs
Provide estimates of the hour burden of the collection of information. The statement should:
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. If this request for approval covers more than one form, provide separate hour-burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories.
It is estimated the survey will require approximately 20 minutes (on average) to complete. The anticipated universe size for each survey cycle is 87,000 individuals, which includes all applicants who submitted proposals and all reviewers between FY 2018 and FY 2020 (for the 2021 survey) and between FY 2020 and FY 2022 (for the 2023 survey). The 2015, 2017, and 2019 iterations of the “Assessment of Investigator and Reviewer Experiences Survey” (OMB # 3145-0215) garnered 30 percent, 36 percent, and 30 percent response rates, respectively. Beginning with the 2021 iteration, the Merit Review Survey will employ enhanced outreach strategies intended to improve response rate, including: 1) capitalization of existing NSF communication platforms to convey legitimacy and importance of the survey; 2) increased advanced confirmation of email address quality by third-party vendors; and 3) use of secondary email addresses, when available, to contact nonresponsive universe members. In addition, the 2021 survey instrument has been shortened from prior iterations to drop items that are not necessary for analyses to streamline questions. Demographic questions have also been revised to align with NSF-wide standards, providing more inclusive response options to increase item response rate. Based on the response rate associated with prior iterations of the survey and these changes to outreach efforts, the estimated survey response rate for each the 2021 and 2023 survey rounds is 40 percent. Thus, we project the total burden is 23,200 hours; this is a respondent burden of 11,600 hours per survey year (2021 and 2023).
Based on 2019 merit review survey data, we expect that most survey respondents will be working at an academic institution, likely in a teaching and/or research capacity. 8 Therefore, for the purpose of burden estimates, we have used the annual mean wage for postsecondary teachers from Bureau of Labor Statistics, which is $79,540.9 Assuming a 40-hour workweek over the course of 52 weeks annually, the hourly wage for this occupation is approximately $38.00. Therefore, the overall cost to survey respondents for each survey year (2021 and 2023) would be approximately $440,800 (11,600 burden hours x $38.00 per hour), as shown in table A.12.1 below.
Table A.12.1. Estimate of Respondent Burden and Cost by Year
Year |
Number of Respondents |
Number of Responses per Respondent |
Average Burden per Response (hours) |
Total Burden Hoursa |
Average Hourly Wage |
Total Cost |
2021 |
34,800 |
1 |
0.33333 |
11,600 |
$38 |
$440,800 |
2022 |
0 |
0 |
0 |
0 |
0 |
$0 |
2023 |
34,800 |
1 |
0.33333 |
11,600 |
$38 |
$440,800 |
Total |
69,600 |
1 |
0.33333 |
23,200 |
$38 |
A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information, (do not include the cost of any hour burden shown in items 12 and 14). The cost estimates should be split into two components: a) a total capital and start-up cost component annualized over its expected useful life, and b) a total operation and maintenance and purchase of services component.
These information collection activities do not place any additional costs on respondents or record keepers.
A.14. Annualized Cost to Federal Government
Provide estimates of annualized cost to the Federal government. Also, provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.
The estimated annualized cost of this study to the Federal Government to conduct the 2021 and 2023 Merit Review Survey $339,433. This total includes costs associated with the study design, instrument development, survey programming, information collection, analysis, reporting, and presentation/publication of the results.
A.15. Explanation for Program Changes or Adjustments
Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.
This is a new submission. There is no request for program changes or adjustments.
A.16. Plans for Tabulation and Publication and Project Time Schedule
For collections of information whose results are planned to be published, outline plans for tabulation and publication.
The survey data obtained from the instrument included in this IRC will be used to describe applicant and reviewer satisfaction and experiences and identify trends over time. The data will be summarized using basic descriptive methods. Data preparation will include data cleaning, variable construction, computing descriptive statistics, nonresponse bias analyses, and weighting calculation. To facilitate analysis of each data source we will create variables to address the study’s research questions. These analyses will focus on examining applicant and reviewer burden, perception of proposal and review quality and fairness, and overall satisfaction with the merit review process. Analyses will also examine changes in these findings from the prior survey cycle. For example, to assess reviewer and applicant experiences with and perceptions of the merit review process, the study contractor will conduct descriptive analyses including the calculation of frequencies for categorical variables and means for continuous and discrete variables. To assess whether experience varies by race/ethnicity, gender, disability status, early career status, or reviewer type subgroups (obtained through NSF administrative data), the study contractor will conduct a variety of statistical tests depending on the type of variable (e.g., conduct z-tests for each continuous variable, chi-square tests for each binary or nominal variable, and Mann-Whitney tests for each ordinal variable). Findings will be displayed in predetermined table shells which will display calculation of frequencies and means for descriptive analyses and results of statistical analyses.
The study contractor will also use administrative data provided by NSF to identify how many proposal reviews were completed for NSF, how many proposals were submitted to NSF, the applicant funding success rate (number of proposals awarded/number of proposals submitted), and how much funding applicants have obtained from NSF to date. Administrative data will be combined with the survey data collected to examine the relationship between the NSF application and funding outcomes and self-reported receipt of financial support from NSF as an undergraduate or graduate student. Prior to conducting these analyses, administrative data may also be used to populate demographic data for nonresponse items in the annual survey in order to conduct more robust subgroup analyses.
Survey data will be tabulated in SAS with results presented in tabular form appropriate to the data type with selected applicant or reviewer characteristics. A final report combining the survey results with administrative data on proposal submissions and reviews from NSF records will be submitted to NSF. The schedule for data collection, analysis, and reporting appears in table A.16.1 below.
Table A.16.1
2021 Merit Review Survey Project Time Schedule
Activity |
Expected Activity Period |
Develop and test web-based data collection instrument |
September 2020 – July 2021 |
Conduct survey |
August – November 2021 |
Analysis of survey data |
November 2021 – March 2022 |
Submit final report to NSF |
April 2022 |
Table A.16.2
2023 Merit Review Survey Project Time Schedule
Activity |
Expected Activity Period |
Develop and test web-based data collection instrument |
September 2022 – July 2023 |
Conduct survey |
August – October 2023 |
Analysis of survey data |
October 2023 – February 2024 |
Submit final report to NSF |
March 2024 |
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
The agency plans to display the expiration date of OMB approval on all forms/questionnaires associated with this information collection.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
Explain each exception to the certification statement identified in Item 19 “Certification for Paperwork Reduction Act.”
There are no exceptions to the certification statement.
1 Source: https://nsf.gov/about/budget/index.jsp
3 Source: https://www.nsf.gov/nsb/publications/2017/nsb201726.pdf
5 Source: https://www.nsf.gov/about/budget/fy2021/pdf/fy2021budget.pdf (see “Performance and management, p. 17”)
6 Source: https://www.nsf.gov/about/budget/fy2021/pdf/fy2021budget.pdf (see “Integrative Activities, p. 4”)
7 Source: Comparable work on the 2023 Merit Review Survey will be conducted under a separate contract solicitation.
8 The 2019 Merit Review Survey data found that nearly 90 percent of respondents were employed by an institution of higher education and nearly 90 percent of those individuals worked in a teaching capacity.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Marietta Bowman |
| File Modified | 0000-00-00 |
| File Created | 2021-02-04 |