NPRA OMB Package Part A (112817)

OMB Clearance Package
NSF Survey of Nonprofit Research Activities (NPRA)

Supporting Statement
for
2017 Survey of Nonprofit Research Activities
and
Debriefing Interviews

Submitted by:
National Center for Science and Engineering Statistics
National Science Foundation
Arlington, VA

November 28, 2017

Table of Contents
A.

B.

Justification ........................................................................................................................1
A.1

Need for Data Collection and Legislative Authorization ........................................1

A.2

How, by Whom, and for What Purpose the Information Is to Be Used ..................2

A.3

Use of Automated, Electronic, Mechanical, or Other Technological Collection
Techniques ...............................................................................................................3

A.4

Efforts to Avoid Duplication ...................................................................................3

A.5

Collection of Data From Small Businesses or Other Small Entities .......................4

A.6

Consequences of Less Frequent Data Collection.....................................................4

A.7

Special Circumstances .............................................................................................5

A.8

Federal Register Notice and Consultation with Persons Outside the Agency .........5

A.9

Payments to Respondents ........................................................................................7

A.10

Assurance of Confidentiality ...................................................................................7

A.11

Sensitive Questions ..................................................................................................9

A.12

Estimates of Response Burden.................................................................................9

A.13

Estimate for the Total Annual Cost Burden to Respondents or Recordkeepers ....11

A.14

Annualized Estimates of Cost to the Federal Government ....................................11

A.15

Changes in Burden .................................................................................................11

A.16

Plans for Tabulation and Publication .....................................................................12

A.17

Approval to Not Display Expiration Date .............................................................13

A.18

Exceptions to the Certification Statement..............................................................13

Collection of Information Employing Statistical Methods...........................................14
B.1.

Respondent Universe .............................................................................................14

B.2

Statistical Methodology .........................................................................................16

B.3

Methods to Maximize Response Rates and to Deal with Issues of Nonresponse .22

B.4

Tests of Procedures and Methods ..........................................................................24

B.5.

Individuals Consulted on Technical and Statistical Aspects of Design .................24

Bibliography .................................................................................................................................25
Attachments ..................................................................................................................................26

Attachments
A. FY 2016 NSF Survey of Nonprofit Research Activities
i. A-1: FY 2016 NSF Survey of Nonprofit Research Activities
ii. A-2: FY 2016 NSF Survey of Nonprofit Hospital Research Activities
B. Comment Letter from Andrew Reamer
C. Debriefing Interview Protocols


Debriefing Protocol for Respondents



Debriefing Protocol for Non-respondents

D. Debriefing Interview Correspondence
E. NCCS Core File Description
F. Likely Performer and Funder Sources
G. Propensity Models for Performer and Funder Stratification
H. Communications with Nonprofit Organizations of Unknown Eligibility
I. Communications with Nonprofit Organizations of Known Eligibility

A.

Justification

The National Center for Science and Engineering Statistics (NCSES) of the National Science
Foundation (NSF) serves as a clearinghouse for the collection, interpretation, analysis, and
dissemination of objective data on science, engineering, technology, and research and
development (R&D) for use by policymakers, practitioners, researchers, and the public. One
component of this activity is NCSES’s sponsorship of the Survey of Nonprofit Research Activities
(NPRA), which is planned to collect information annually on research- and development-related
(R&D) activities performed or funded by nonprofit organizations in the United States.
This revision of a currently approved collection clearance request is for two projects in the
implementation of the Survey of Nonprofit Research Activities: (1) three survey cycles for the
Fiscal Years (FY) 2016, 2017, and 2018 each with approximately 6,500 nonprofit organizations,
and (2) Debriefing Interviews of 20 respondents and 20 non-respondents per year. The debriefing
interviews with respondents will be conducted on a rolling basis approximately two weeks after
survey completion and with non-respondents approximately two weeks after the final deadline.
The NPRA survey will be conducted in the following two phases: (1) Screener Phase and (2)
Survey Phase. During the Screener Phase, NCSES will contact organizations for which we have
no information about their R&D activities. These organizations will receive a brief screener asking
whether the organization performed or funded research in FY 2016. The Survey Phase will include
all organizations believed to have performed or funded research, organizations that self-identified
as performers or funders of research during the Screener Phase, and organizations that did not
respond during the Screener Phase. During the Survey Phase, organizations will receive mail, email, and telephone requests to complete the survey by providing their organization’s expenditures
for R&D activities.
A.1

Need for Data Collection and Legislative Authorization

The nonprofit sector is one of four major sectors ([1] business, [2] government, [3] higher
education, and [4] nonprofit organizations) of the economy that perform and/or fund R&D.
Historically, NCSES has combined nonprofit sector data with the other three sectors’ data to
estimate total national R&D expenditures via the annual National Patterns of R&D Resources
report. The other sectors are surveyed annually; however, prior to fielding the FY 2015 NPRA
Pilot Survey last year, NCSES last collected R&D data from nonprofit organizations nearly 20
years ago. Since then the National Patterns of R&D Resources has relied on statistical modeling
based on those survey results to continue estimation of this sector's R&D expenditures.
Once viewed as a small contributor to the U.S. economy as a whole, the nonprofit sector has been
one of the most consistent in national growth since the 1970s. Nonprofit sector revenue has grown
53% (by approximately $457 billion) from 1997 to 2007, compared to private sector growth of
32%, as Salamon (2015) presented from his research in the Committee on National Statistics
workshop, Measuring Research and Development Expenditures in the U.S. Nonprofit Sector:
Conceptual and Design Issues,
1

Approval to conduct the NPRA Survey is being requested under the America COMPETES
Reauthorization Act of 2010 §505, codified in the National Science Foundation Act of 1950 (NSF
Act), as amended, at 42 U.S.C. §1862p, that under paragraph “b” directs the Foundation through
NCSES to
1) collect, acquire, analyze, report, and disseminate statistical data related to the science and
engineering enterprise in the U.S. and other nations that is relevant and useful to
practitioners, researchers, policymakers, and the public, including statistical data on
(a) research and development trends;
(b) the science and engineering workforce;
(c) U.S. competitiveness in science, engineering, technology, and research and
development; and
(d) the condition and progress of U.S. STEM (Science, Technology, Engineering and
Math) education.
The NPRA Survey will provide essential data on the R&D activities performed and funded by
nonprofit organizations in the U.S. The results of the survey will enable NCSES to produce valid
and reliable estimates of nonprofit R&D expenditures in the U.S.
A.2

How, by Whom, and for What Purpose the Information Is to Be Used

The FY 2015 NPRA pilot data collection yielded vital information about the sample frame
methodology, the survey methodology, and development of a weighting strategy that addressed
unit non-response. Ultimately, the results from the FY 2016 NPRA survey will provide the first
updated, valid, and reliable estimates of nonprofit R&D activity in the U.S. since the late 1990s,
as well as a better understanding of the scope and nature of R&D in the nonprofit sector.
Federal Uses
NCSES will use the results of the survey to report updated, valid, and reliable estimates of U.S.
nonprofit R&D in National Patterns of R&D Resources. The data will also be incorporated into
the National Science Board’s biennial report, Science and Engineering Indicators (SEI). The SEI
report is mandated by Congress (42 U.S.C. 1863, Section 4(j)), as follows:
"The Board shall render to the President and Congress, no later than January 15 of each
even numbered year, a report on indicators of the state of science and engineering in the
United States.”
Other federal agencies that may be interested in the NPRA data include the Bureau of Economic
Analysis, Office of Science and Technology Policy, Congressional Budget Office, and
Congressional Research Service. Policy officials and R&D proponents can use the data to explore
questions about economic growth in the U.S. overall, as well as the nonprofit R&D sector
specifically. These data will help federal agencies develop long-range plans and policies for R&D
funding opportunities and the nonprofit sector as a whole.
2

International Uses
NCSES expects the Organization for Economic Cooperation and Development (OECD) will
request that NCSES provide NPRA Survey data for use in its periodic publications and for
international comparisons of R&D efforts. NCSES anticipates statistical agencies in other
countries that collect similar data, such as Statistics Canada, also will request NPRA data.
Professional Associations, Educational Centers, and Other Uses
NCSES anticipates professional associations will use data from the NPRA Survey to enhance
awareness of their member organizations’ R&D. Likely users in this category include, but are not
limited to, the Science Philanthropy Alliance, Association of Independent Research Institutes,
Health Research Alliance, Consortium of Social Science Associations, American Educational
Research Association, Urban Institute’s Center on Nonprofits and Philanthropy, Association for
Research on Nonprofit Organizations and Voluntary Action, Washington State University Social
and Economic Sciences Research Center, National Academy of Sciences, American Hospital
Association, and Council on Foundations.
A.3

Use of Automated, Electronic, Mechanical, or Other Technological
Collection Techniques

The NPRA survey will primarily be a web-based data collection effort. The website will include a
link to download a PDF and print a paper version of the questionnaire (see Attachment A). The
web instrument simplifies certain aspects of the survey process for the respondents, such as
automatically skipping questions that do not apply based on previous answers, performing
arithmetical operations, and edit checks that will help to limit certain types of respondent errors
(such as providing totals that do not match a total previously reported). The web version of the
survey will have a real-time monitoring system that allows NCSES to monitor data, response
status, and comments from respondents. From the perspective of the respondents, a web version
of the survey is more convenient and simplifies the survey (e.g., by automatically calculating
totals). NCSES benefits from the use of the web version by receiving improved data quality and
timely responses.
A.4

Efforts to Avoid Duplication

The NPRA Survey collects essential information on the financial resources allocated to R&D by
nonprofit organizations. There are no other statistical sources of comprehensive national data for
this information. The IRS collects information from nonprofits on IRS Form 990 (Return of
Organization Exempt from Income Tax) and IRS Form 990-PF (Return of Private Foundation).
However, these forms do not specifically collect R&D expenditures and other information
included in the NPRA Survey.
The potential exists for some overlap with nonprofit hospitals affiliated with universities and
colleges where R&D expenditure data may be collected through the NSF Higher Education
Research and Development (HERD) Survey and state-affiliated nonprofits where data may be
collected in the NCSES/Census State Government Research and Development Survey. Once the
3

sample is drawn for NPRA, NCSES will compare it with the institutions surveyed through HERD
and the State R&D Surveys and remove any duplication. NCSES will also ask the sampled
organizations to inform us if they are part of a university to eliminate any component units of
universities not already removed during the initial sample de-duplication. The NPRA Survey
includes a new question (not asked in the NPRA Pilot Survey) that asks, “Is your organization
affiliated or considered part of another for-profit, nonprofit, or higher education organization?” If
the respondent replies yes, they are asked to provide the name of the organization with which their
organization is affiliated. Throughout the fielding period, NCSES will monitor the responses to
this question and follow up as necessary to eliminate duplication.
A.5

Collection of Data From Small Businesses or Other Small Entities

This survey of nonprofit organizations may have an impact on small entities. However, the survey
was intentionally designed to be brief to reduce burden and to be easy to complete. Also, the NPRA
Survey will be offered over the web, which has branching logic that reduces the number of
questions that need responses for entities engaged in only funding or only performing R&D. To
boost the response rate, a personalized survey booklet will be mailed at two different intervals
during data collection to non-respondents including small entities.
Representatives from the smaller nonprofits interviewed during cognitive testing indicated
minimal difficulty providing the requested financial information. Some modifications were made
as a result of the debrief interviews to mitigate issues encountered by the small entities (refer to
A.8 Consultation with Persons Outside the Agency for additional information).
A.6

Consequences of Less Frequent Data Collection

The current request is for three cycles of the NPRA Survey. The survey was piloted in 2016 with
3,640 nonprofit organizations. Prior to the NPRA Pilot Survey, no similar project had been
conducted in 20 years. During that time, NCSES extrapolated the R&D expenditures for the
nonprofit sector using statistical modeling. Because the number of nonprofit organizations filing
IRS nonprofit Form 990 (EZ, PF) has increased by over 20% in this timeframe, NCSES believes
that the statistical modeling based on the previous version of the survey is out of date and may be
producing imprecise estimates for R&D expenditures. Conducting a new survey of nonprofit
research activities will provide more accurate national estimates of R&D and better support for
economic policy decisions.
The current request is for three years, which will enable NCSES to conduct annual surveys in
2017, 2018, and 2019, each with approximately 6,500 nonprofit organizations. Based on the
findings from the NPRA Pilot Survey and from the last survey in the late 1990s, the number of
nonprofit organizations performing or funding research activities is estimated to be a very small
proportion of the overall population of nonprofits. Given the large and heterogeneous population,
accurate and complete identification of all research performers and funders is an extremely
challenging task. Each survey iteration will help us identify research performers and funders not
previously surveyed and eliminate organizations that do not conduct or fund research and further

4

refine our list of known performers and funders. This frame refinement will make the survey
administration more efficient, less costly, and more accurate in future cycles.
A.7

Special Circumstances

Information collection will be conducted in a manner consistent with Office of Management and
Budget (OMB) guidelines, and there are no special circumstances.
A.8

Federal Register Notice and Consultation with Persons Outside the Agency

Federal Register Notice
On May 4, 2017, NCSES published a notice in the Federal Register (82 FR 20921 https://www.federalregister.gov/d/2017-09044) inviting the general public and other federal
agencies to comment on NCSES’s plans to submit this request. NCSES received one request for a
copy of the questionnaire and the OMB supporting statement from Andrew Reamer, Research
Professor in the George Washington Institute of Public Policy at George Washington University
(Attachment B). The questionnaire was provided to him, and the supporting statement will be
provided upon submission to OMB.
Consultation With Persons Outside the Agency
Many people and organizations have been consulted to develop the questionnaire. In 2014, the
Committee for National Statistics convened a steering committee on measuring R&D expenditures
in the U.S. nonprofit sector. A workshop was held in June 2014 that included experts in survey
methodology and nonprofit R&D. Potential data users who attended the workshops included
academics, government employees, and other stakeholders.
During the summer of 2014, NCSES conducted 23 exploratory interviews with representatives of
nonprofit organizations in Atlanta, GA; Austin, TX; Boston, MA; and Seattle, WA. During these
interviews, staff gathered information about how nonprofits are organized; the types of research
they performed or funded; their familiarity with the terms “research” and “development”; types of
financial data they maintain that are specific to R&D; benefits and barriers to participating in an
NCSES-sponsored survey; and possible topics to include in a new survey.
In September 2014, an expert panel was convened to help NCSES shape the content and coverage
of data related to R&D in the nonprofit sector. Information was garnered to gain a better
understanding of the similarities and differences among nonprofit sectors, learn about the needs of
data users, and obtain advice on priorities and strategies for content to be included in the survey.
Using the information from the Committee on National Statistics, exploratory interviews, and the
expert panel, NCSES began the NPRA Pilot Survey efforts including developing the NPRA pilot
instrument and accompanying communications. In the summer of 2015, 28 cognitive tests were
conducted with representatives of nonprofit organizations that perform and/or fund R&D in the
Washington DC Metropolitan area; Los Angeles, CA; and Chicago, IL. Participants in the

5

cognitive interviews were asked to provide feedback on the prenotification letter, a handout that
would be included with the letter, and the survey instrument.
In the spring/summer of 2016, two rounds of usability testing were conducted with nonprofits
before the NPRA Pilot Survey was launched. Testing was completed remotely with participants
controlling wireframe prototypes (Round 1, n=8) or the online web survey (Round 2, n=9) in a
browser on the facilitator’s computer.
The NPRA Pilot Survey was conducted from September 12, 2016 through February 10, 2017.
Afterward, debriefing interviews were conducted with 37 organizations—16 with organizations
that submitted a survey and 21 with non-respondents, including four organizations that answered
a few questions but did not submit a survey. Additional cognitive testing was completed with eight
nonprofit organizations to test revisions made to the FY 2016 NPRA Survey. Copies of the reports
from these activities are available upon request and were the basis for modifying the FY 2016
NPRA Survey. The NSF 2016 NPRA Pilot Survey evaluation report is available on request from
the NCSES Project Officer.
Changes from the NPRA Pilot to the FY 2016 NPRA Survey include
1) Modifications to mitigate duplication and ensure the correct organizations respond:
a. confirming each organization's Employee Identification Number (EIN)
b. asking if the organization is affiliated or considered part of another for-profit,
nonprofit, or higher education organization
2) Modifications to improve the consistency and accuracy of question responses:
a. adding information which clarifies the types of employees to include in the
organization’s calculation of full-time equivalents
b. clarifying that work performed by contract employees should be included in
Question 6 (intramural R&D performance) and not in Question 7 (R&D funded by
the nonprofit but performed by others)
c. providing space for additional detail or clarification about responses to some
questions
3) Modifications to encourage participation in the survey:
a. allowing organizations to provide a brief description of research they perform or
fund
b. allowing organizations to opt out of having their organization’s data released by
name

6

Adding a new question asking organizations to categorize research expenditures performed and/or
funded by type of research (basic research, applied research, and experimental development) and
by source (federal and nonfederal). Additionally, due to the lower than average response rates from
the hospital sector to the NPRA Pilot Survey, NCSES identified a need to modify the survey
communication to increase salience for nonprofit hospitals. Beginning with the FY 2016 NPRA
Survey, nonprofit hospitals will receive a customized version of the survey and of communications
with language associated with hospitals (see Attachment A-2).
A.9

Payments to Respondents

No payments will be made to NPRA Survey respondents.
A.10 Assurance of Confidentiality
No items on the NPRA Survey are deemed confidential. It is NCSES’s intention to publish detailed
results of the full implementation of the NPRA Survey with tables showing R&D funding and
expenditures for the nonprofit organizations surveyed. Organizations will be able to opt out of having
their organization-level data published on the NCSES website. However, their data will be included
in published aggregate estimates and also be used to prepare estimates of R&D expenditures in the
National Patterns of R&D Resources. Examples of the tables that will be produced are presented in
Exhibits A.10.1 and A.10.2.

7

Exhibit A.10.1: Example of a Table for Public Dissemination of the Data
with Organizational Data
Table XX. Total research performance expenditures at nonprofit organizations,
by organization and Federal/Nonfederal source of funds: FY 2016
All research
performance

Federal
government

Nonfederal
sources

11,111,111

1,111,111

1,111,111

Organization A
Organization B
Organization C
Organization D
Organization E

0,000,000
0,000,000
0,000,000
0,000,000
0,000,000

0,000,000
0,000,000
0,000,000
0,000,000
0,000,000

0,000,000
0,000,000
0,000,000
0,000,000
0,000,000

Organization F
Organization G
Organization H
…

0,000,000
0,000,000
0,000,000
…

0,000,000
0,000,000
0,000,000
…

0,000,000
0,000,000
0,000,000
…

Organization Z

0,000,000

0,000,000

0,000,000

000,000

000,000

000,000

Organization
All nonprofit organizations

All other surveyed
nonprofit organizationsa
a

Expenditures included under All other surveyed nonprofit organizations are
for those organizations that did not want their data reported by name.
SOURCE: National Science Foundation, National Center for Science and
Engineering Statistics, Survey of Nonprofit Research Activities, FY 2016.

Exhibit A.10.2: Example of a Table for Public Dissemination of the Data
with Aggregate Data
Table XX. Total research performance expenditures at nonprofit
organizations, by source: FY 2016
All research
performance

Source
All sources

11,111,111

Federal government
State and local government

1,111,111
1,111,111

Business

1,111,111

Foundations

1,111,111

Universities
Other nonprofits

1,111,111
1,111,111

Internal funds

1,111,111

Individual donors

1,111,111

Other sources

1,111,111

SOURCE: National Science Foundation, National Center for Science and
Engineering Statistics, Survey of Nonprofit Research Activities, FY 2016.

8

A.11 Sensitive Questions
The NPRA Survey has no sensitive questions. During the cognitive interviews, usability testing,
and debriefing interviews, no respondents said that they found the questions to be sensitive. NPRA
is a voluntary survey, and no organizations are required to respond to it. The voluntary aspect of
the survey is stated clearly in the introduction.
A.12 Estimates of Response Burden
The response rate for the NPRA Pilot Survey was 43.4%1. Of these organizations, 16.4% (n=255)
reported that they performed, and/or funded R&D activities. The average reported burden hours
for those that performed or funded R&D activities was 4 hours. The reported burden was 20
minutes for those who reported no R&D activity.
On the basis of changes in our sampling and contact procedures, NCSES expects to improve upon
the response rate to the NPRA Pilot Survey to achieve a 60% response for the full implementation
of the NPRA Survey. See section B.3 for a discussion of our plans for nonresponse bias analysis.
To achieve this response rate, NCSES will conduct the full implementation of the NPRA Survey
for approximately 6,500 organizations in two phases: (1) a screener for those organizations for
which NCSES has no information concerning their R&D activities (referred to as “Unknowns”)
and (2) a survey for organizations identified as likely research performers and/or funders, for those
that self-identify via the screener as a performer and/or funder of research (referred to as
“Knowns”), and for those that do not complete the screener.
Phase I: Screener
The screener will be sent to an estimated 4,100 nonprofit organizations for which NCSES has no
indication of research performance or funding. It is estimated that the screener will take
approximately 10 minutes to review, complete, and return by mail to our survey contractor.
NCSES anticipates around 70% (2,870) of these organizations will return the screener. The
expected burden for Phase I is 478 (2,870 organization x 10 minutes) hours.
Phase II: Survey
In Phase II, nonprofit organizations will be asked to complete the NPRA Survey instrument. These
organizations will fall into one of three groups:




1

Organizations identified at the start of the study as likely performers or funders of research
(Knowns)
Organizations that responded to the screener as performing or funding research
(Screened-Ins), and
Organizations that did not respond to the screener (Unknowns)

1,555 nonprofits provided partial or complete data out of 3,587 eligible organizations.

9

Upon response, these three groups will get collapsed into two groups: a) organizations that perform
and/or fund research, and b) organizations that neither perform nor fund research. It is anticipated
that it will take organizations that perform and/or fund research approximately 4 hours to respond
to the survey. The estimated burden for organizations that neither perform nor fund research is 20
minutes.
a) Organizations that Perform and/or Fund Research

It is estimated that this group contains approximately 1,222 nonprofit organizations (922
organizations from the Knowns group, 241 from the Screen-Ins group, and 59 from the Unknowns
group). These estimates are based on response rates and eligibility rates from the NPRA Pilot
Survey. The estimated burden for this group is 4,888 hours (1,222 organizations x 4 hours).
Based on the NPRA Pilot Survey, the Knowns group, which will include approximately 2,400
organizations, is expected to have a response rate of approximately 60% (1,440). Of the responding
organizations, approximately 64% (922) will report performing or funding research activities.
The NPRA Pilot Survey indicated that approximately 12% (344) of the 2,800 organizations that
respond to the screener in Phase 1 will self-identify as research performers or funders. These
organizations will be invited to participate in Phase II. Of those organizations, it is anticipated that
70% (241) will participate in the survey.
Lastly, based on the NPRA Pilot Survey, the Unknowns group, which will include approximately
1,230 non-respondents to Phase 1, is anticipated to have a response rate of 40% (492). Of those
responding organizations, 12% (59) will report that they perform and/or fund research.

b) Organizations that Neither Perform Nor Fund Research

It is estimated that this group contains approximately 951 nonprofit organizations (518 from the
Knowns group, 433 from the Unknowns group). These estimates are based on response rates and
eligibility rates from the NPRA Pilot Survey. The estimated burden for this group is 317 hours
(951 organizations x 20 minutes).
The NPRA Pilot Survey indicated that approximately 36% (518) of the responding organizations
in the Knowns group and approximately 86% (433) of the responding organizations in the
Unknowns group will not actually have performed or funded research.
Debriefing Interviews
After the survey fielding period, the NCSES survey contractor will conduct debriefings of 20
respondents and 20 non-respondents (see Attachments C and D, Debriefing Interview Protocols
and Debriefing Interview Correspondence, respectively). Debriefing interviews for respondents
will last no more than 60 minutes per organization; interviews with non-respondents should be
approximately 30 minutes. NCSES estimates a total of 30 burden hours for the debriefing
interviews.

10

Total Burden
The combined burden for the survey and debriefing interviews is estimated at 5,713 hours. A
summary of the FY 2016 NPRA Survey burden estimates is included in the following table. At an
estimated cost of $79.31 per hour (based on the Bureau of Labor Statistics May 2016 average
hourly wages for “Top Executives” within NAICS Sector 54 - Professional, Scientific, and
Technical Services),2 Table A.12.1 shows the annual burden estimates for the NPRA Survey and
debriefing interviews.
Table A.12.1: Combined Annual Burden Estimates for Phases I and II and Debriefing Interviews
Category

Respondent

Estimated #

Total Burden

Total Cost

Cost per

Burden

of Responses

Hours

Burden

Respondent
(average)

Phase I
NPRA screener

10 minutes

2,870

478

$

37,910

$13.21

20 minutes

951

317

$

25,141

$26.44

4 hours

1,222

4,888

$

387,667

$317.24

1 hour

20

20

$

1,586

$79.31

30 minutes

20

10

$

793

$39.66

5,083

5,713

$

453,097

Phase II
Organizations that neither
perform nor fund research
Organizations that
perform and/or fund
research
Debriefing Interviews
Respondents interviews
Non-respondent interviews
Total Annual Burden

A.13 Estimate for the Total Annual Cost Burden to Respondents or
Recordkeepers
Not applicable. Respondents are not expected to incur any capital and start-up costs, or systems
maintenance costs in responding.
A.14 Annualized Estimates of Cost to the Federal Government
The total estimated cost to the federal government for the NPRA Survey for FYs 2016, 2017, and
2018 is approximately $6,899,043 over a period of 63 months, for an annualized cost of
$1,314,103.
A.15 Changes in Burden
The burden rate for the NPRA Pilot Survey was based on the cognitive interviews described earlier
and was estimated to be 10 hours for those who performed and/or funded R&D activities and 1
hour or less from organizations that did not perform or fund. The NPRA Pilot Survey included a
question that asked the respondent the total amount of staff time it took their organization to
2

Available at http://www.bls.gov/oes/current/naics3_541000.htm. Accessed on April 26, 2017.

11

complete the questionnaire. Respondents were further instructed to include the time spent reading
the instructions, working on questions, and obtaining information, as well as time spent by all
employees collecting and providing the information. An analysis of the data revealed that the
survey is far less burdensome than anticipated.
The addition of the Phase I screener in the FY 2016 NPRA Survey will allow those organizations
that do not perform or fund research to select out of the larger survey. Reviewing the materials and
returning the screener will take an estimated 10 minutes. For the NPRA FY 2016 Survey
instrument, NCSES estimates a burden of 4 hours for organizations that perform and/or fund
research, and 20 minutes for those organizations that do not perform or fund R&D based on the
NPRA Pilot Survey responses.
A.16 Plans for Tabulation and Publication
NCSES plans to tabulate and publish the results in aggregate and individual form. During the
debriefing interviews with organizations that (1) participated in the NPRA Pilot Survey and (2)
performed and/or funded R&D, participants were asked how they would feel seeing their research
performing and/or funding expenditures shared in data tables on NCSES’s website. Draft data
tables showing both aggregate and organization level data were provided as examples to clarify
the request. Approximately half (n = 8) had no problem having their data published. The other half
(n = 8) said that they would need to consult their president, executive director, or chief operating
officer for approval to complete and submit the survey. One organization mentioned that its legal
team would also have to review the survey before submission. Based on this feedback, NCSES
drafted a new final question (number 24) that allows all organizations to let us know their
preferences for public release of their data. The question specifies that if they choose to opt out of
organization level data release, their data will still be included in published aggregate estimates and
will also be used to prepare estimates of R&D expenditures in the National Patterns of R&D
Resources.
Timeline
The schedule for data collection, report completion, and publication of statistical tables is shown
below in Table A.16.1.
Table A.16.1: Proposed Schedule of Activities
Activity
Date*
Phase I: Data Collection Begins (Screener)
January 2018
Phase II: Data Collection Begins (Survey)
February 2018
Data Collection Closes
July 2018
Summary and Debrief Report
September 2018
Publication of Statistical Tables
March 2019
*Dates are tentative and based on OMB approval.

12

A.17 Approval to Not Display Expiration Date
Not applicable. The survey will display the expiration date for OMB approval of the information
collection.
A.18 Exceptions to the Certification Statement
Not applicable. No exceptions to the certification statement are being sought.

13