REQUEST FOR CLEARANCE: SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT REQUEST FOR:
ENGINEERING INDUSTRIAL INNOVATIONS AND PARTNERSHIPS (IIP) PROGRAM MONITORING DATA COLLECTIONS
National Science Foundation
4201 Wilson Boulevard
Arlington, VA 22230
The National Science Foundation (NSF) is the primary federal agency supporting research at the frontiers of knowledge, across all fields of science and engineering (S&E) research and all levels of S&E education (NSF, "Empowering the Nation: Through Discovery and Innovation," NSF Strategic Plan for Fiscal Years (FY) 2011-2016). NSF awards grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and other eligible institutions, and provides graduate fellowships to individuals in all parts of the United States [1].
NSF provides nearly 20 percent of federal funding for basic research to academic institutions [2]. Within NSF, the Directorate for Engineering (ENG) has primary responsibility for promoting the progress of engineering in the United States in order to enable the Nation’s capacity to perform. Its investments in engineering research and education aim to build and strengthen a national capacity for innovation that can lead over time to the creation of new shared wealth and a better quality of life. Most NSF programs in engineering are funded through the Directorate for Engineering, which also sponsors the NSF's Industrial and Innovation Partnerships (IIP) Division. To these ends, NSF-ENG provides support for research and implementation activities that may meet national needs. While scientists seek to discover what is not yet known, engineers apply fundamental science to design and develop new devices and engineered systems to solve societal problems. NSF-ENG also focuses on broadening participation in engineering research and careers, particularly among those individuals traditionally underrepresented and underemployed in the STEM workforce, including but not limited to, women, persons with disabilities, and racial and ethnic minorities.
This request seeks approval for a group of information collections intended to monitor outputs, short-term, intermediate and long-term outcomes of NSF-ENG investments in research and innovation in the Division of Industrial Innovation and Partnerships (IIP). IIP serves the entire foundation by fostering partnerships to advance technological innovation and plays an important role in the public-private innovation partnership enterprise by investing in science and engineering research across all disciplines that have the potential for high impact in meeting national and societal needs. IIP focuses on leveraging federal, small business, industrial, university, state and community college resources. Genuine partnerships between academe and industry are an important aspect of IIP programs and should facilitate the types of infrastructure that can sustain and nurture the spread of innovative activity.
Innovation infrastructures educate and train human capital for research enterprise and the entrepreneurial aspects of innovation; develop social networks characterized by shared commitment and trust; and build a base of operational support without which sustainable partnerships cannot exist. This support includes a diversified base of private investment, a physical place to provide a context for incubation, technical, management, and administrative support, laboratories, communications services, and reliable sources of capital. One end of the innovation spectrum within the division includes unsolicited research proposals generated by the academic community. On the other end of the innovation spectrum, IIP supports small business research proposals aimed at pursuing opportunities to commercialize products and services.
IIP is home to the two congressionally mandated small business research programs, the Small Business Innovation Research (SBIR) program and the Small Business Technology Transfer (STTR) program. IIP also manages the Partnerships for Innovation: Accelerating Innovation Research (PFI:AIR) as well as the Partnerships for Innovation: Building Innovation Capacity (PFI:BIC) program, which stimulate innovation by building partnerships across the scientific, engineering, and business community. In addition, the IIP leverages industrial support through the Industry/University Cooperative Research Centers (I/UCRC) program. The division also actively participates in NSF-wide programs, such as the Grants Opportunities for Academic Liaison with Industry (GOALI) program. Another NSF-wide program in which IIP actively participates is the Innovation Corps program (I-Corps), which equips scientists with the entrepreneurial tools needed to transform discoveries with commercial realization potential into innovative technologies [3].
These survey questionnaires, individually tailored for different programs, will provide essential information for program monitoring purposes. Data collected by ENG IIP program monitoring collections will be used for program planning, management, and evaluation. Summaries of monitoring data are used to respond to queries from Congress, the public, NSF's external merit reviewers who serve as advisors, including Committees of Visitors (COVs), and NSF's Office of the Inspector General. These data are needed for effective administration, program and project monitoring, evaluation, and for measuring attainment of NSF's program and strategic goals, as identified by the President's Accountable Government Initiative, the Government Performance and Results Act (GPRA) Modernization Act of 2010, and NSF's Strategic Plan.
The seven (7) program-specific collections included in this request (see attachments 2-8) are designed to assist in management of specific programs and to serve as data resources for current and future program evaluations. As such, expected outcomes could vary according to the nature of the program funding, field of study, and other program characteristics.
Office |
Programs |
Industrial Innovation and Partnerships (IIP) |
Grant Opportunities for Academic Liaison with Industry (GOALI) |
Innovation Corps (I-Corps) |
|
Partnerships For Innovation: Accelerating Innovation Research (PFI:AIR) |
|
Partnerships For Innovation: building Innovation Capacity (PFI:BIC) |
|
Small Business Innovation Research (SBIR) |
The NSF Directorate of Engineering is responsible for analyzing and evaluating engineering research and development activities in NSF’s ENG and ENG co-funded portfolios.
Data from NSF administrative databases can be incorporated with information gathered through initiative-specific, division-specific, and program-specific data collections. NSF-ENG uses these data for monitoring, managing, and evaluating its investment in engineering programs, initiatives, and activities.
Attached is a crosswalk that details the topics each of the questions address and how they vary between collections. These questions are based on the current understanding of data management needs and potential use in evaluation activities, including the development of metrics associated with several program-level logic models and program or agency progress toward the NSF strategic goals in coordination with ENG and NSF efforts.
Recent developments within NSF-ENG are expected to lead to improvements in how data from these collections are used. A comprehensive evaluation plan [4] that specifically addresses the use of monitoring data was presented to the ENG Advisory Committee in April 2012. The ENG plan establishes evaluation and assessment as one of its core strategies to accomplish its strategic goals, as demonstrated by its commitment to:
“Learn through assessment and evaluation of NSF/ENG programs, processes and outcomes; continually improve them; and employ outcomes to inform planning, policies, and procedures.”
To fulfill its planning and management responsibilities, and to answer queries from Congress, OMB, and NSF management, ENG needs current and standardized information about the short and long-term outcomes of projects in NSF’s engineering portfolio. This information is specifically important to support studies and evaluations by NSF-ENG, and studies by other NSF organizational units. Collection of these data has several purposes, including:
Providing a source of information on the outcomes of the research investments in terms of advancements in science, benefits to the institutions, researchers, students and society and educational and/or career impact on participants/students in NSF-funded projects, in compliance with Foundation responsibilities to monitor scientific and technical resources enabling NSF to monitor the effectiveness of NSF-sponsored projects and identify outputs of projects funded under NSF awards for management and for reporting to the Administration and Congress, especially under the GPRA Modernization Act of 2010 [5], 5 U.S.C. 306 [6] and 39 U.S.C. 2801-2805 [7], and under the President’s Accountable Government Initiative [8], and Performance Improvement Guidance as represented by OMB’s guidance to agencies (M-10-24) [9].
The collections have been assembled for these IIP programs based on their logic model (see appendices 2-6), theory of change or management needs. The data collected under these surveys is focused on initiative-specific, division-specific, and program-specific quantitative and qualitative indicators. The question items elicit participants’ details and activities, outputs (i.e., the accomplishments of program grantees (projects) in terms of specific program objectives), outcomes and impacts (i.e., the longitudinal accomplishments (post-award) accomplishments of program grantees). These descriptive data collections provide essential information for documenting progress toward NSF’s major performance goals, as described in NSF’s Strategic Plan. (The Foundation’s FY 2014-2018 Strategic Plan describes three strategic goals: Transform the Frontiers of Science and Engineering, Stimulate Innovation and Address Societal Needs through Research and Education, and Excel as a Federal Science Agency [10].
The information collected under this request is required for effective program administration, program and project monitoring, evaluation, and for measuring attainment of NSF’s program and strategic goals as laid out in NSF’s Strategic Plan. This section describes how the data to be collected under the new clearance authority will be used for internal program management and administration; as a data source for NSF’s performance assessment activities, including Committees of Visitors and Directorate and Office Advisory Committees (ACs); for documenting the attainment of NSF’s program and strategic goals in accordance with the President’s Accountable Government Initiative and GPRA reporting; and as a foundation for the rigorous research required to evaluate the effectiveness of engineering research programs. Few NSF-ENG programs regularly conduct a variety of data collection activities that include routine program monitoring, program evaluations, and education- related data collections from federally-funded Institutions of Higher Education. The primary objective of this clearance is to allow IIP programs in NSF-ENG to collect outcome and output data from grantees, their partners and students, which will enable the evaluation of the impact of its investments in engineering research over time.
The impact of investments in scientific research is hard to quantify and measure due to the unique characteristics of scientific activity. One factor is that fundamental research findings take a long time to mature and do not immediately impact society. The only way to elicit long-term outcome information is by utilizing longitudinal monitoring collections that are voluntary in nature (given that the collection will occur beyond the life of the award). In guidance from the Director of OMB, M-10-32 [11], the need for rigorous evaluations and the objectives of program evaluations were clearly outlined, including the use of evaluation resources. The collection of data by these monitoring questionnaires contributes to the formal evaluation of the corresponding NSF-ENG programs by providing indicators of outcome achievement from each project, during and after the life of the award. In this regard, the OMB guidance provides a rationale for the questionnaires covered under this request and the activities implemented on behalf of the development of this request.
“Improving and coordinating the use of existing evaluation resources. In addition to the voluntary evaluation initiative, agencies should continue to carefully assess, report on, and allocate the base funds and resources that the agencies have for conducting evaluation. Agencies are encouraged to share information beyond what is requested in guidance and consult with OMB’s Resource Management Offices (RMOs) to coordinate and improve the design, implementation, and utilization of evaluations.”
This directive reinforces the need for NSF-ENG to collect information about its programs to improve program evaluation processes.
Another central use of the ENG IIP Program Monitoring Clearance data is to document attainment of NSF’s program and strategic goals and to report on the attainment of these goals. NSF’s performance assessment is guided by three elements: the GPRA Modernization Act of 2010 [5], the President’s Accountable Government Initiative [8], and NSF’s Strategic Plan [10]. The Foundation’s FY 2014-2018 Strategic Plan describes three strategic goals: Transform the Frontiers of Science and Engineering, Stimulate Innovation and Address Societal Needs through Research and Education, and Excel as a Federal Science Agency [10].
“NSF Evaluation Initiative: The Administration has emphasized the need to use evidence-based assessment methodology in budget, management, and policy decisions to make government work effectively. In response, NSF has launched the NSF Evaluation Initiative to expand the agency’s capability in the area of evidence-based evaluation. The effort will be staffed to carry out an agency-wide coordination of data gathering and development of evaluation capabilities, and to conduct pilots for selected programs.” – NSF Strategic Plan for 2014-2018, p. 16.
“Means for Carrying Out Core Strategies:…Develop, where appropriate, quantitative or evidence-based evaluation of outcomes.” – NSF Strategic Plan for 2014-2018, p. 14.
NSF-ENG programs should serve strategic goals for the Directorate. Information that enables ENG to report on progress toward these goals could be collected by the questionnaires to be cleared under this request.
Finally, a key measure of NSF’s success in achieving its goals is the effectiveness of its engineering research and education programs. NSF is committed to implementing program evaluation in accordance with the President’s Accountable Government Initiative [8]. While the questionnaires used to collect data under the ENG IIP Program Monitoring Clearance will play a role in this work, it is understood that they are not evaluative studies. NSF does conduct program-level management reviews to ensure that programs are administered properly and in accordance with federal guidelines and agency missions. Going forward, NSF-ENG will emphasize the use of monitoring data in future evaluation activities, creating a foundation for the kind of evaluation the President’s Accountable Government Initiative requires of federal agencies. For example, in order to conduct program-level or portfolio-level evaluations, quasi-experimental evaluation research studies on engineering awards require evaluators to identify individual-level and organizational-level or project-level control and treatment groups or comparison groups. NSF-funded contract or grantee researchers and evaluators could use the data to identify control, comparison, or treatment groups for NSF’s engineering portfolio using some of the descriptive data gathered through this request to conduct well-designed, rigorous research and portfolio evaluation studies.
All of the collections included under this clearance request use Web-based data collection systems to minimize data duplication and respondent burden. In some cases, program officers may call the respondent for follow-up or clarification, and enter data gathered in these conversations into the web-based data collection system. NSF-ENG favors Web-based systems because they facilitate respondents’ data entry across computer platforms. One innovative feature of many of the individual Web systems is the thorough reviewing and editing of all submitted data for completeness, validity, and consistency. Editing and validation are performed as data are entered. Most invalid data cannot be entered into the system, and questionable or incomplete entries are called to respondents’ attention before they are submitted to NSF.
ENG IIP Program Monitoring Clearance Web-based data collection systems will employ user-friendly features such as automated tabulation, data entry with custom controls such as checkboxes, data verification with error messages for easy online correction, standard menus, and predefined charts and graphics. All of these features facilitate the reporting process, provide useful and rapid feedback to the data providers, and reduce burden.
All collections in the ENG IIP Program Monitoring Clearance comply with Section 508, the 1998 amendment to the Federal Rehabilitation Act, which mandates that the electronic and information technology used by federal agencies be made accessible to all people with disabilities
The ENG IIP Program Monitoring Clearance does not duplicate efforts undertaken by the Foundation, other federal agencies, or other data collection agents. For example, NSF grants require the submission of annual and final project reports in accordance with OMB 3145-0058. Recipients of NSF grants, such as principal investigators (PIs), must create and submit annual and final project reports using research.gov. The introduction of the new annual and final reports based on the RPPR format improved the submission of project information, but does not change the need for additional data that monitoring systems provide on a program-specific basis. Data collected under the ENG IIP Program Monitoring Clearance is mainly intended to monitor investigators and their students with questions and timeframes of two types. (1) questions not found in the RPPR during the life of the award. These are unique and not available in either the NSF annual or final reporting system. And, (2) on a longitudinal basis, after the life of the award and after the reporting requirements have been fulfilled in RPPR. These questions might be unique and not available at any time in RPPR or could be a question that was asked during the RPPR period but it is not asked anywhere else after the end of the performance period. These monitoring systems will continue to collect updated information requested by the RPPR after the life of the award.
In the ENG IIP Program Monitoring Clearance, the IIP Division will collect information from small businesses. These businesses are partners to current and/or former awardees or are run/owned by current and/or former awardees. The only impact of this data collection on the business will be the time required for respondents to complete the survey or have a conversation with data collectors.
Data collected for the ENG IIP Program Monitoring Clearance will be used to manage programs, monitor projects, inform project and program evaluations, coordinate with federal and non-federal partners, provide Congress with information about government-supported activities, and report for GPRA and other requirements.
If additional information were not collected, NSF would be unable to document outputs and outcomes of its programs that occur many years after the award is made given the nature of scientific discovery. It would be unable to meet its accountability requirements or assess the degree to which projects and programs are meeting their goals over time.
All data collections will comply with 5 CFR 1320.6. All collections under the ENG IIP Program Monitoring Clearance could ask respondents for data annually, or occasionally at shorter intervals for some data collected during the life of the award, or at longer intervals for post-award monitoring. Because many of the potential outcomes and impacts of investments in engineering research are realized sometimes years after the award is made (particularly for fundamental research investments), it is necessary to capture some of these outcomes and impacts via post-award monitoring. Post-award data collections are voluntary. Analysis of non-response bias will be implemented every time inferences about a program are to be drawn from the data. Post-award monitoring systems, including the frequency of post-award data collection are tailored to the expected timeline of outcomes and impacts from each individual program. In most cases, we expect to collect post-award data at 1-year, 3-year, and 5-year intervals, with some post-award systems capturing a 4th data point at 10 years post-award. These collections for programs in the IIP division, expect that important indicators for outcomes and impacts may become apparent at shorter intervals post-award, and in these cases, data collection intervals could be more frequent, but will extend over a shorter total period post-award.
The notice inviting comments on the Engineering IIP Program Monitoring Data Collections Clearance (OMB 3145-NEW) was published in the Federal Register February 19, 2014, Volume 79, Number 33, pages 9485 – 9488. No comments were received.
When developing collection instruments, NSF-ENG routinely consults with research and evaluation experts, PIs, and educators affected by NSF-ENG investments. The purpose of these consultations is to assess the relevance, availability, and clarity of items. As suggested by OMB guidelines, these consultations also enable NSF-ENG staff to obtain a reliable estimate of the respondent burden generated by new instruments. When a new collection is developed or when an existing collection is modified to add new instruments, each instrument is pretested with nine or fewer individuals and revised following debriefings with participating respondents.
Consultations have included knowledgeable outsiders such as representatives of NSF-ENG contractors such as VentureWell responsible for technical and evaluation tasks and fellows who work at the Foundation as guests under programs such as the Einstein Fellows Program or the American Association for the Advancement of Science - Science & Technology Policy Fellows Program.
There are no plans to provide incentives to respondents because the value of program and project monitoring surveys is of value to the respondents as well as NSF. Program monitoring can be used by projects as a foundation for project-level evaluation.
Respondents will be informed that any information on specific individuals is maintained in accordance with the Privacy Act of 1974. Every data collection instrument will display both OMB and Privacy Act notices.
Respondents will be told that data collected are available to NSF officials and staff, evaluation contractors, and the contractors hired to manage the data and data collection software. Data will be processed according to federal and state privacy statutes. The system will limit access to personally identifiable information to authorized users. Data submitted will be used in accordance with criteria established by NSF for monitoring research and education grants and in response to Public Law 99-383 and 42 USC 1885c.
The information requested through NSF monitoring systems may be disclosed to qualified researchers and contractors for evaluation purposes and to a federal agency, court, or party in court or federal administrative proceedings, if the government is a party.
Some of the proposed question items in the ENG IIP Program Monitoring Clearance request information from respondents, including name and email address (see table below). These data are collected in order to monitor the award sites and evaluate the success of the award programs. Information of this nature is also used to track recipients of funding and training. Responses to all items of a sensitive nature are voluntary. Respondents may choose not to provide information that they deem as privileged. Any individual-level data that are collected will be provided only to program staff and consultants conducting studies using the data as authorized by NSF. Any public reporting of data will be in aggregate form, and all personal identifiers will be removed.
Post-award monitoring data collections are voluntary and respondents will clearly be given the choice of non-response.
The table below shows which questions of a sensitive nature are included in the IIP collections.
Collection Title |
Name |
Email Address |
Grant Opportunities for Academic Liaison with Industry (GOALI) |
X |
X |
Innovation Corps (I-Corps) |
X |
X |
Innovation Corps (I-Corps) Pre-Course Survey |
X |
X |
Innovation Corps (I-Corps) Post-Course Survey |
X |
X |
Partnerships For Innovation: Accelerating Innovation Research (PFI:AIR) |
X |
X |
Partnerships For Innovation: building Innovation Capacity (PFI:BIC) |
X |
X |
Small Business Innovation Research (SBIR) |
X |
X |
While output and short-term outcome monitoring can happen yearly, long-term outcomes will only be monitored at longer intervals (potentially 1, 3, 5, and 10 years after the end of the award). The table below shows the ideal scenario for a hypothetical year, in which all respondents in all the collections participate in the collection.
For the IIP division, many awards are made in translational research, such that we might expect a shorter and more condensed timeline of outcomes and impacts. In these cases, it is necessary to collect very limited monitoring data from awardees at short intervals (e.g. annually) during the life of the award. The data collected could serve two purposes: 1) to provide formative feedback to awardees on possible mid-course corrections as challenges arise to assist them in optimizing the impact of their NSF funding, and 2) to provide trend data to NSF program staff such that they might be able to identify patterns that indicate or suggest characteristics of successful awardees. Identification of trends and patterns can allow NSF to eventually provide training or resources to future awardees to enhance these characteristics of success. If data collections are deemed to occur more frequently, the burden to awardees will be limited to no more than 20 minutes of the respondents’ time in each instance.
Collection Title |
No. of Respondents |
Annual No. of Hours/Respondent |
Annual Hour Burden |
Grant Opportunities for Academic Liaison with Industry (GOALI) |
200 |
2 |
400 |
Innovation Corps (I-Corps) Longitudinal Collection |
800 |
.25 |
200 |
Innovation Corps (I-Corps) Pre-Course Survey Questionnaire |
150 |
.25 |
38 |
Innovation Corps (I-Corps) Post-Course Survey Questionnaire |
150 |
.25 |
38 |
Partnerships For Innovation: Accelerating Innovation Research (PFI:AIR) |
200 |
2 |
400 |
Partnerships For Innovation: building Innovation Capacity (PFI:BIC) |
30 |
2 |
60 |
Small Business Innovation Research (SBIR) |
1,100 |
2 |
2200 |
Total |
2,630 |
8.75 |
3,336 |
As shown in Table 3 below, the annual response burden for the collections under this request is 3,335 hours. This collection will happen at 1, 3, and 5, and 10 years after the award so the total number of hours is 3,335 x 4 = 13340 hours/10 year period, an average of 1,334 hours per year.
For life-of-award monitoring, the data collection burden to awardees will be limited to no more than 120 minutes of the respondents’ time in each instance, but will most likely average 60 minutes of the respondents’ time in each instance.
The respondents are PIs, partners or students. For some programs, (I-Corps) the burden already includes a response from 3 members of the team in the pre and post course surveys. For all other, one PI or assignee per award completes the questionnaire.
The following table shows the annualized estimate of costs to PI/program coordinator respondents, who are generally university professors. This estimated hourly rate is based on a report from the American Association of University Professors, “Annual Report on the Economic Status of the Profession, 2013-14,” Academe, March–April 2014, Survey Report Table 4. According to this report [12], the average salary of an associate professor across all types of doctoral-granting institutions (public, private-independent, religiously affiliated) was $86,293. When divided by the number of standard annual work hours (2,080), this calculates to approximately $41 per hour.
When at full implementation, in a year in which all programs monitor their investments and all respondents participate in the collection, the overall annualized cost to the respondents is estimated to be $136.736. Since these data collections will not take place every year, the average cost to the respondent in a ten year period is $53,9151.
Respondent Type |
No. of Respondents |
Total Burden Hours |
Average Hourly Rate |
Estimated Annual Cost |
PIs, Assignees, Partners or Students (IIP Division) |
2,630 |
3336 |
$41 |
$136,736 |
Not applicable
Estimated costs include ~$500,000 to develop one monitoring system for each of the 6 divisions and offices, ~$300,000/year to maintain each of these systems by a contractor, and 30% of full-time salary for each of two (2) Evaluation & Assessment staff who are salaried at ~$100,000/year.
Not applicable: this is a new collection.
Like many agencies, NSF no longer relies on formal (i.e., traditional) publication methods and publication formats. News media advisories, notices of funding opportunities for colleges and universities, and results from survey collections are all examples of the types of publications that NSF regularly publishes without putting ink to paper.
For content authored by NSF or by a third party at NSF’s request, the agency rarely uses paper to publish the information. NSF publishes most documents electronically using the agency’s Web site, from requests for proposals to evaluation or statistical reports, using an archive called an On-Line Document System (ODS).
Like NSF itself, the scope of publication plans and practices by the ENG IIP Program Monitoring Clearance has a dual nature. Some individual collections contribute to formal products (e.g., analytical reports) that can be published by NSF’s ODS.
Most of what the ENG IIP Program Monitoring Clearance will collect, however, will not be published as a stand-alone product, because the data will be used as an input to how NSF manages, documents, evaluates, and measures its performance as an agency. NSF’s GPRA Performance Report or an individual division’s annual report to the NSF Director could use information from the collection to report to Congress. This is an annual cycle.
The data collection efforts included under this request will likely be administered by third-party contractors that will deliver (1) analytical reports, (2) the raw data from the collections, or (3) both. Third parties are contractually forbidden from publishing results unless NSF has made a specific exception. In short, all products of the collections are the property of NSF. After the products are delivered, NSF determines whether the quality of the products deserves publication verbatim by NSF; i.e., NSF typically is the exclusive publisher of the information collected by the collections. Often it is only after seeing the quality of the information the collection delivers that NSF decides the format (raw or analytical) and manner (in the ODS or simply a page on the NSF Web site) in which to publish.
At this time, NSF plans to produce a summary or descriptive report every year after completion of the data collections.
Not applicable
No exceptions apply.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Griswold, Shannon |
| File Modified | 0000-00-00 |
| File Created | 2021-01-25 |