NPRA Supporting Statement - Section A

OMB Clearance Package
NSF Survey of Nonprofit Research Activities (NPRA)

SF-83-I Supporting Statement
for
2016 Pilot Survey of Nonprofit Research Activities
and
Debriefing Interviews

Submitted by:
National Center for Science and Engineering Statistics
National Science Foundation
Arlington, VA

March 22, 2016

Table of Contents
A.

B.

Justification ................................................................................................................... 1
A.1

Need for Data Collection and Legislative Authorization ........................................ 1

A.2

How, by Whom, and for What Purpose the Information Is to Be Used.................. 2

A.3

Use of Automated, Electronic, Mechanical or Other Technological Collection
Techniques .......................................................................................................... 3

A.4

Efforts to Avoid Duplication .................................................................................. 3

A.5

Collection of Data From Small Businesses or Other Small Entities ...................... 4

A.6

Consequences of Less Frequent Data Collection................................................. 4

A.7

Special Circumstances......................................................................................... 4

A.8

Federal Register Notice and Consultation with Persons Outside the Agency ....... 5

A.9

Payments to Respondents ................................................................................... 6

A.10

Assurance of Confidentiality ................................................................................. 6

A.11

Sensitive Questions ............................................................................................. 6

A.12

Estimates of Response Burden ............................................................................ 6

A.13

Estimate for the Total Annual Cost Burden to Respondents or Recordkeepers .... 7

A.14

Annualized Estimates of Cost to the Federal Government ................................... 7

A.15

Changes in Burden .............................................................................................. 7

A.16

Plans for Tabulation and Publication .................................................................... 7

A.17

Approval to Not Display Expiration Date .............................................................. 7

A.18

Exceptions to the Certification Statement ............................................................. 8

Collection of Information Employing Statistical Methods .......................................... 8
B.1.

Respondent Universe .......................................................................................... 8

B.2

Statistical Methodology ...................................................................................... 11

B.3

Methods to Maximize Response Rates and to Deal with Issues of Nonresponse ............................................................................................................ 22

B.4.

Tests of Procedures and Methods ..................................................................... 24

B.5.

Individuals Consulted on Technical and Statistical Aspects of Design................ 24

Bibliography ............................................................................................................................ 25
Attachments ............................................................................................................................ 26

Table of Contents i

Attachments
A.

2016 Pilot Survey of Nonprofit Research Activities

B.

Pilot Survey Correspondence

C.

Comment Letter from Andrew Reamer, George Washington University

D.

Debriefing Interview Protocols

E.

Debriefing Interview Correspondence

F.

NCCS Core File Description

G.

Likely Performer and Funder Sources

H.

Propensity Models for Performer and Funder Stratification

Table of Contents ii

A.

Justification

The National Center for Science and Engineering Statistics (NCSES) of the National Science
Foundation (NSF) serves as a clearinghouse for the collection, interpretation, analysis, and
dissemination of objective data on science, engineering, technology, and research and
development (R&D) for use by policymakers, practitioners, researchers, and the public. One
component of this activity is NCSES’s sponsorship of the Nonprofit Research Activities Survey
(NPRA), which is planned to collect information periodically on research- and developmentrelated activities performed or funded by nonprofit organizations in the United States.
This clearance request is for two projects in the design of the Nonprofit Research Activities
Survey: (1) the Fiscal Year (FY) 2016 NPRA Pilot Survey of 3,640 nonprofit organizations, and
(2) Debriefing Interviews of 20 respondents and 20 non-respondents. The interviews will be
conducted at the conclusion of the pilot test to inform its evaluation.
A.1

Need for Data Collection and Legislative Authorization

The nonprofit sector is one of four major sectors ([1] business, [2] government, [3] higher
education, and [4] nonprofit organizations) of the economy that perform and/or fund R&D.
Historically, NSF has combined nonprofit sector data with the other three sectors’ data to estimate
total national R&D expenditures via the annual National Patterns of R&D Resources report. The
other sectors are surveyed annually; however, it has been more than 18 years since NSF last
collected R&D data from nonprofit organizations.
While in the past the nonprofit sector has been viewed as a small contributor to the U.S. economy
as a whole, it has actually been one of the most consistent in its growth since the 1970s. As Salamon
(2015) presented from his research in the Committee on National Statistics workshop, Measuring
Research and Development Expenditures in the U.S. Nonprofit Sector: Conceptual and Design
Issues, nonprofit sector revenue has grown 53% (by approximately $457 billion) from 1997 to
2007, compared to private sector growth of 32%. Although a majority of this growth occurred in
the nonprofit health field (53%), other fields such as education, research, and social services also
grew by 50% or more.
The primary objective of the new survey is to fill data gaps in the National Patterns of R&D
Resources in such a way that the data are compatible with data collected on the other sectors of
the U.S. economy and appropriate for international comparisons. Since the last survey of research
and development activity in the nonprofit sector took place more than 18 years ago, and because
of the more than 21% growth of nonprofit organizations that registered with the IRS between 2001
and 2011, it is important that a new survey of nonprofit R&D be fielded to update existing national
estimates for the nonprofit sector. Before that can take place, the survey must first be pilot tested
to evaluate the quality and effectiveness of the frame, questionnaire, and contact strategies.
Approval to conduct the NPRA Pilot Survey is being requested under the America COMPETES
Reauthorization Act of 2010 §505, codified in the National Science Foundation Act of 1950 (NSF

1

Act), as amended, at 42 U.S.C. §1862, that under paragraph “b” directs the Foundation through
NCSES to
1) collect, acquire, analyze, report, and disseminate statistical data related to the science and
engineering enterprise in the U.S. and other nations that is relevant and useful to
practitioners, researchers, policymakers, and the public, including statistical data on
(a) research and development trends;
(b) the science and engineering workforce;
(c) U.S. competitiveness in science, engineering, technology, and research and
development.
The NPRA Survey pilot will provide essential data on the research and development activities
performed and funded by nonprofit organizations in the U.S. The pilot survey will allow NCSES
to better understand the current range of nonprofits and help determine the most effective data
collection questions and sampling protocols necessary to gather the nonprofit R&D expenditure
data. The results of the pilot survey will be another step toward producing valid and reliable
estimates of nonprofit R&D activity in the U.S. that will be collected during the full
implementation of the survey.
A.2

How, by Whom, and for What Purpose the Information Is to Be Used

The proposed pilot data collection will yield vital information through the sample frame
methodology, the survey methodology, and the subsequent evaluation of the pilot. These insights
will help determine the feasibility of moving forward with full implementation of the survey.
Ultimately, the results from the full implementation of the survey will provide the first updated,
valid, and reliable estimates of nonprofit R&D activity in the U.S. since the late 1990s, as well as
a better understanding of the scope and nature of R&D in the nonprofit sector.
Federal Uses
We anticipate the pilot data collection will interest a number of federal agencies, including the
Bureau of Economic Analysis, the Office of Science and Technology Policy, the Congressional
Budget Office and the Congressional Research Service. Policy officials and R&D proponents will
be interested in the data collected during the pilot to see whether it answers questions about
economic growth in the U.S. overall and specifically in the nonprofit R&D sector.
While the pilot data collection will only be useful for federal agencies in determining the utility of
the data, the full implementation of the NPRA Survey planned for 2017 will provide a key
understanding of the R&D activities and expenditures in the nonprofit sector. These data will help
federal agencies develop long-range plans and policies for R&D funding opportunities and the
nonprofit sector as a whole.

2

Professional Associations Uses
Data from the pilot will not be reported publically, but the pilot survey methodology will be
reported, and we believe that professional associations may review the methodological results to
advocate for a broader survey sample and more extensive surveying of their constituents. With the
full implementation, these organizations will probably use the results to enhance awareness of their
member organizations’ R&D. Likely users in this category include, but are not limited to, the
Science Philanthropy Alliance, Association of Independent Research Institutes, Health Research
Alliance, Consortium of Social Science Associations, American Educational Research
Association, Center on Nonprofits and Philanthropy, Association for Research on Nonprofit
Organizations and Voluntary Action, Social & Economic Sciences Research Center, and the
Council on Foundations.
International Uses
While we will not be releasing the results of the pilot study to the public, we would expect the
Organization for Economic Cooperation and Development (OECD) to be interested in our
methodological findings. We anticipate a request from OECD for NCSES to provide NPRA
Survey data from the full implementation for use in its periodic publications and for international
comparisons of R&D efforts.
A.3

Use of Automated, Electronic, Mechanical or Other Technological
Collection Techniques

The NPRA Pilot Survey will be a web-based data collection effort. The website will include a link
to download a PDF and print a paper version of the questionnaire (see Attachment A). The web
instrument simplifies certain aspects of the survey process for the respondents, such as
automatically skipping questions that do not apply based on previous answers, performing
arithmetical operations, and edit checks that will help to limit certain types of respondent errors
(such as providing totals that do not match a total previously reported). The web version of the
survey will have a real-time monitoring system that allows NCSES to monitor data, response
status, and comments from respondents. From the perspective of the respondents, a web version
of the survey is more convenient and simplifies the survey (e.g., by automatically calculating
totals). NCSES benefits from the use of the web version by receiving improved data quality.
The primary form of communication and response with nonprofit organizations will be electronic.
With the exception of a prenotification letter and mailed reminder [postcard/letter], all other
correspondence will be sent electronically or organizations will be contacted by telephone (see
Attachment B).
A.4

Efforts to Avoid Duplication

The data collected through NPRA have not been collected in over 18 years and are not available
through administrative data. IRS Form 990, Return of Organization Exempt from Income Tax,
IRS Form 990-EZ, Short Form Return of Organization Exempt from Income Tax, and IRS Form
990-PF, Return of Private Foundation collect information from nonprofit organizations. However,
3

these forms do not specifically collect R&D expenditures and other information included in the
NPRA Survey.
There is the potential for some overlap with nonprofit hospitals affiliated with universities and
colleges where data may be collected through the NSF Higher Education Research and
Development (HERD) survey. Once the sample is drawn for NPRA, we will compare it with the
institutions surveyed through HERD and remove any duplication. We will also ask the sampled
organizations to let us know if they are part of a university or other higher education institution in
order to eliminate any component units of universities not easily identified and removed during
the initial sample de-duplication (Attachment B, Contacts 1 and 2).
A.5

Collection of Data From Small Businesses or Other Small Entities

This pilot survey of nonprofit organizations may have an impact on small entities choosing to
provide the requested data. We intentionally designed the survey to be brief to reduce the burden
and to be easy to complete. The survey will be offered over the web, which participants in
exploratory interviews indicated was their preferred mode. A web-based instrument also allows
branching logic that reduces the number of questions that need responses for entities engaged in
only funding or only performing R&D. In addition, representatives from the smaller nonprofits
interviewed during cognitive testing indicated minimal difficulty providing the requested financial
information. We will pay close attention during the debriefing interviews to any difficulties small
organization participants may have had completing the survey.
A.6

Consequences of Less Frequent Data Collection

The current request is for the pilot study of a new survey effort on research and development
activities of the U.S. nonprofit economic sector. No similar project has been conducted in the past
18 years. During that time, NSF has extrapolated the R&D expenditures for the nonprofit sector
using statistical modeling. Because the number of nonprofit organizations filing IRS nonprofit
Form 990 (EZ, PF) has increased by over 20% in this timeframe, we believe that the statistical
modeling based on the previous version of the survey is out of date and, therefore, may be
producing imprecise estimates for R&D expenditures. Conducting a new survey of nonprofit
research activities will provide more accurate national estimates of R&D and provide better
support for economic policy decisions. While the current request is for a pilot study of 3,640
nonprofit organizations and 40 debriefing interviews, this pilot is the first stage of being able to
more accurately gauge research and development in the nonprofit sector. If this pilot is deemed
successful, a separate request will be submitted for the full implementation of the survey.
A.7

Special Circumstances

This information collection will be conducted in a manner consistent with Office of Management
and Budget (OMB) guidelines and there are no special circumstances.

4

A.8

Federal Register Notice and Consultation with Persons Outside the Agency

Federal Register Notice
On January 11, 2016, NSF published a notice in the Federal Register (81 FR 1233 https://federalregister.gov/a/2016-229) inviting the general public and other federal agencies to
comment on NSF’s plans to submit this request. One comment came from Andrew Reamer,
Research Professor in the George Washington Institute of Public Policy at George Washington
University (Attachment C). He expressed support for the survey and wanted a copy of the
questionnaire and methodology. The questionnaire was provided to him immediately and he
agreed to wait until the methodological summary was finalized before receiving a copy.
Consultation With Persons Outside the Agency
Many people and organizations have been consulted in order to develop the questionnaire. In 2014,
the Committee for National Statistics convened a steering committee on measuring R&D
expenditures in the U.S. nonprofit sector. Two workshops were held in February and June 2014
that included experts in survey methodology and nonprofit R&D. During the February workshop,
NCSES presented background information about the 1996–1997 survey, the tasks and timeline for
the new NPRA Survey, and information about the target population and sample design for the
survey. The definition of R&D expenditures and its relevance for a nonprofit sector survey were
discussed. In June, the second workshop focused on providing a profile of the nonprofit sector in
the U.S. (including short presentations by nonprofits), understanding R&D within this sector,
approaches to survey and sample design, suggestions about the question design and survey
implementation. Potential data users who observed the workshops included academics,
government employees, and other stakeholders.
During the summer of 2014, NCSES (through its contractor) conducted 23 exploratory interviews
with representatives of nonprofit organizations in Atlanta, GA; Austin, TX; Boston, MA; and Seattle,
WA. During these interviews, staff gathered information about how nonprofits are organized, the
types of work they performed or funded, their familiarity with the terms “research” and
“development,” types of financial data they maintain that are specific to R&D, benefits and barriers
to participating in an NSF-sponsored survey, and possible topics to include in a new survey.
In September 2014, the contractor convened an expert panel to help NCSES shape the content and
coverage of data related to R&D in the nonprofit sector. Information was garnered to gain a better
understanding of the similarities and differences among nonprofit sectors, learn about the needs of
data users, and obtain advice on priorities and strategies for content to be included in the survey.
Using the information from the Committee on National Statistics, exploratory interviews, and the
expert panel, NCSES and its contractor developed the NPRA instrument and accompanying
communications. In the summer of 2015, 28 cognitive tests were conducted with representatives of
nonprofit organizations that perform and/or fund R&D. Participants in the cognitive interviews were
asked to provide feedback on the prenotification letter, a handout that will be included with the letter,
and the survey instrument. These materials were tested in an iterative fashion over 4 months.

5

A.9

Payments to Respondents

No payments will be made to NPRA Pilot Survey respondents.
A.10 Assurance of Confidentiality
No items on the NPRA Survey are deemed confidential. While data from the pilot survey will not
be published, it is NSF’s intention to publish detailed results of the full implementation of the NPRA
Survey with tables showing R&D funding and expenditures for the nonprofit organizations surveyed.
A.11 Sensitive Questions
The NPRA Survey has no sensitive questions. During the cognitive interviews, no respondents
said that they found the questions to be sensitive. NPRA is a voluntary survey, and no persons are
required to respond to it. The voluntary aspect of the survey is stated clearly in the introduction.
A.12 Estimates of Response Burden
The FY 2015 Pilot Survey of Nonprofit Research Activities will collect data from 3,640 nonprofits.
The expected response rate for the pilot survey is 65% or 2,365 organizations. Of those, we expect
approximately 400 to be organizations that fund and/or perform research. On the basis of the
cognitive interviews, we estimate that the survey will require approximately 10 hours (on average)
to review the materials and complete the questionnaire for organizations that perform or fund
research. For those that do not perform or fund research, we expect the survey to take less than 1
hour. Therefore, we estimate the total response burden hours for the survey to be 5,965 (4,000
hours for the 400 performers and funders; 1,965 hours for the remainder of the sample).
In addition, debriefings of 20 respondents and 20 non-respondents will be conducted (see
Attachments D and E, Debriefing Interview Protocols and Debriefing Interview Correspondence,
respectively). Debriefing interviews for respondents will last no more than 60 minutes per
organization; interviews with non-respondents should be approximately 30 minutes. We estimate
a total of 30 burden hours for the debriefing interviews. The combined burden for the survey and
debriefing interviews is estimated at 5,995 hours (5,965 hours for the questionnaire and 30 hours
for the debriefing interviews).
During the exploratory interviews and cognitive tests, we were consistently told that the
prenotification letter and e-mail invitation should be sent to the President/Chief Executive
Officer/Executive Director of each nonprofit organization. At an estimated cost of $76.12 per hour
(based on the Bureau of Labor Statistics May 2014 average hourly wages for “Top Executives”
within NAICS Sector 54 - Professional, Scientific, and Technical Services),1 Table A.12.1 shows
the annual burden estimates for the pilot survey and debriefing interviews.

1

Cost per hour is based on BLS 2014 average hourly wages for “Top Executives” in NAICS Sector 54. Estimate is $76.12 per hour. Available at
http://www.bls.gov/oes/current/naics3_541000.htm. Accessed on November 16, 2015.

6

Table A.12.1: Annual Burden Estimates for 2016 Pilot Survey and Debriefing Interviews
Category
Survey NPO
Performer/ Funder
Survey NPO NonPerformer/ Funder
Respondent Debriefing
Interviews
Non-respondent
Debriefing Interviews

Respondent
Burden
(hours)

Estimated # of
Responses

Total Burden
Hours

Total Cost
Burden

Cost per
Respondent
(average)

10

400

4,000

$304,480

$761.20

1

1,965

1,965

$149,576

$76.12

1

20

20

$1,522

$76.12

0.5

20

10

$761

$38.06

2,405

5,995

$456,339

Total Annual Burden

A.13 Estimate for the Total Annual Cost Burden to Respondents or
Recordkeepers
Not applicable. Respondents are not expected to incur any capital and start-up costs, or systems
maintenance costs in responding.
A.14 Annualized Estimates of Cost to the Federal Government
The total estimated cost to the federal government for the NPRA Pilot Survey is approximately
$1,893,614 over a period of 36 months, for an annualized cost of $789,944.
A.15 Changes in Burden
Not applicable. The NPRA Pilot Survey is new because the nonprofit sector and the survey itself
have changed substantially since the last survey was conducted 18 years ago.
A.16 Plans for Tabulation and Publication
This pilot survey is being conducted (1) to test the survey procedures across different types of
nonprofit organizations (including the quality of the sampling frame); (2) to gain additional
knowledge about the eligibility rates in different classes of nonprofits; and (3) to test the survey
instrument. Because this is a pilot survey, a report that focuses on an evaluation of the survey
methodology will be prepared. While there are no plans to tabulate and publish the results either
in aggregate or individual form, NCSES may use the data to improve the model to derive our
national estimates of nonprofit research spending in National Patterns of R&D Resources, as well
as prepare papers, articles and/or presentations on the pilot survey methodology.
A.17 Approval to Not Display Expiration Date
Not applicable. The survey will display the expiration date for OMB approval of the information
collection.
A.18 Exceptions to the Certification Statement
Not applicable. No exceptions to the certification statement are being sought.
7