NPRA OMB Package Part B (112817)

B.   Collection  of  Information  Employing  Statistical  Methods    
B.1     Respondent  Universe  
Target  Population  and  Sampling  Frame  
The target population for the NPRA Survey is nonprofit organizations that perform or fund
research and development activities. An organization is nonprofit if categorized by the Internal
Revenue Service (IRS) as 501(c)(3) public charity, 501(c)(3) private foundation, or other exempt
organization [e.g., 501(c)(4), 501(c)(5), 501(c)(6)]. As recorded on the IRS Exempt Organizations
Business Master File (December 2016), there are 1.6 million tax-exempt organizations in the
United States. Of those, nearly 1.2 million filed an information return with the IRS in the past 24
months. Certain nonprofit organizations are not required to file an information return (e.g.,
churches), but those that are required must file Form 990, 990-N, 990-EZ, or 990-PF, according
to their organization type and financial size (see Table B.1.1).
Table B.1.1: Criteria for Filing IRS Exempt Organization Forms
Type of Exempt Organization
Public Charity and Other Exempt Organizations:
Gross receipts normally ≤ $50,000
Gross receipts < $200,000, and Total assets < $500,000
Gross receipts ≥ $200,000, or Total assets ≥ $500,000
Private Foundations

Required to file
990-N
990-EZ
990
990-PF

Small organizations, those with gross receipts under $50,000 are allowed to file Form 990-N (“epostcard”), which does not require the organization to report financial data. Nearly half of the
filing organizations filed Form 990-N. These organizations were excluded from the frame due to
their relatively small size and lack of financial data. The remaining organizations filed Forms 990,
990-EZ, or 990-PF, which require financial reporting.
The financial information on Forms 990, 990-EZ, and 990-PF is captured on the National Center
for Charitable Statistics’ (NCCS) Core Financial Files. NCSES will use the circa 2013 versions of
the core files to construct the sampling frame for the NPRA Survey. The 2013 Core Financial Files
are the latest available at the time of the sample selection for the NPRA Survey. These files are
labeled “circa” 2013 because the data may be from different fiscal years. The due date for 990s is
rolling based on the 15th day of the fifth month after the end of the organization’s taxable year.
When the 2013 core file was constructed, most organizations had filed for their 2013 fiscal year;
however, a few organizations had filed only for FY 2012 or had already filed for FY 2014. Further,
if an organization did not file a return, but is presumed to be still active, NCCS uses the form from
the latest filing. As shown in Table B.1.2, three separate core files are available.

14

Table B.1.2: NCCS Core Financial Files and Number of Records
Core File
501(c)(3) public charities that file Form 990 or 990-EZ
Other exempt organizations that file Form 990 or 990-EZ
501(c)(3) private foundations (Form 990-PF)
Total

Number of Records
382,401
140,219
95,992
618,612

The core files include a total of 618,612 organizations with unique employee identification
numbers (EIN) that NCSES will use as the sampling frame. Of those, 95,992 represent private
foundations of all sizes that complete Form 990-PF. The other 522,620 represent public charities
and other exempt organizations that filed Form 990 or 990-EZ. Typically, these organizations
have gross receipts over $50,000, because those with gross receipts under $50,000 are allowed to
file Form 990-N and are not recorded on the core files.
Frame  Coverage  
Using the core financial files as a sampling frame excludes most public charities and other exempt
organizations with gross receipts under $50,000 (note some file Form 990 or Form 990-EZ).1 While
organizations with gross receipts under $50,000 represent 70% of the public charities and other
exempt organizations, they represent approximately 0.1% of the total gross receipts generated by
public charities and other exempt organizations.2 Exclusion of these small organizations is a
limitation because they will not be covered by the sampling frame (i.e., undercoverage).
Alternatively, the IRS Exempt Organization Business Master File (EO/BMF) includes all exempt
organizations regardless of financial size. However, the financial variables on the EO/BMF are
limited to gross receipts, revenue, and total assets. The additional financial information available
on the core files will be used in developing a propensity score model to improve the sample
stratification (see Attachment E. NCCS Core File Description for a list of the variables on the core
files). Although exclusion of these small organizations is a limitation in coverage, NCSES believes
the likelihood that these organizations are performing and/or funding R&D is small and the impact
on the estimate of the total research expenditures will be negligible.
Another source of undercoverage is from new organizations that have not filed a Form 990, 990-EZ
or 990-PF. However, these coverage errors are likely to be small due to an increase of only 1.1% of
organizations between the 2014 and 2015 EO/BMF files.3
Frame  Exclusions  
The core files include some organizations that are not in the scope of this study, such as
organizations outside the United States, educational institutions, churches, and government

1

Based on the filing requirements code, 126,377 organizations on the core file are not required to file a Form 990, 990-EZ, or 990-PF.

2

Based on gross receipts on the 2013 Exempt Organization Business Master File. For zero-filers, most of which are allowed to file Form 990-N
and for which NCSES has no record of income, NCSES assumed the maximum threshold ($50,000) to calculate the gross receipts percent. The
percentage was 0.8% with this maximum threshold assumption.

3 The number of organizations filing Form 990, 990-EZ, or 990-PF within 24 months was 601,415 in September 2015 and 593,715 in September
2014.

15

organizations. These organizations are excluded from the sampling frame based on the criteria
presented in table B.1.3.
Table B.1.3: Summary of Organizations Excluded From the Frame
Nonprofit
Organizations
618,612

Total Organizations on NCCS Core Files
Exclusions
1) IRS subsection code: Exclude organizations with subsection code not equal to 3, 4, 5, or 64
2) Foundation code: Exclude organizations with foundation code 10=Church or
14=Government
3) State: Exclude organizations outside the U. S.

57,282
6,613
1,321

4) North American Industry Classification System (NAICS): Exclude organizations with
NAICS code 92=Public Administration
5) “Out of Scope” based on NCCS Coding5
6) Educational exclusions

14
32
49,568

7) Inactive organizations (based on pilot)

51

Response  Rate    
Response rates will be calculated using the American Association for Public Opinion Research
standard definitions.6 The NPRA Pilot Survey achieved a screening response rate of 43.4% based
on AAPOR’s RR1. The cooperation rate for answering the amount spent on research was 88.0%
for performers and 85.6% for funders. See section B.3 for a discussion of the methods NCSES
plans to use to maximize response rates for the survey.
B.2     Statistical  Methodology  
Identifying  Likely  Performers  and  Likely  Funders  of  R&D  
Organizations performing and/or funding R&D will be rare and difficult to target, one of the
greatest challenges for this survey. NCSES will use two strategies to increase the sampling
efficiency of locating performers and funders: (1) frame truncation, and (2) stratification. Both the
frame truncation and stratification use the core financial information and classification codes to
identify organizations that are more likely to perform or fund research.

4

Most exclusions were based on the following 501(c) subsection codes: 02 - Title holding corporations for a tax-exempt organization; 07- Social
and recreational clubs; 08- Fraternal beneficiary societies and associations; 09 - Voluntary employees' beneficiary associations; 10 - Domestic
fraternal societies; 12 - Benevolent life insurance associations, mutual ditch or irrigation companies, etc.; 13 - Cemetery companies, providing
burial and incidental activities for members; 14 - State-chartered credit unions, etc.; and 19 - Post or organization of war veterans. The full list
of the 501(c) subsection codes is available on the data dictionary for the NCCS Core Financial files, available at
http://nccsweb.urban.org/PubApps/showDD.php#Core%20Data.

5

NCCS assigns a “G” identifying organizations that are considered government entities and “N” identifying organizations that have a physical
IRS contact address, but nearly all program operations are conducted/focused outside the U.S. A total of 33 cases had a code of N or G. One
organization coded as G was retained as a hospital: Christiana Care Health Services. The remaining 32 were eliminated.

6

The American Association for Public Opinion Research (AAPOR), 2015 Standard Definitions: Final dispositions of case codes and outcome
rates for survey, 8th edition, AAPOR (2015).

16

To identify the core financial variables related to performing and/or funding research and evaluate
the impact of frame truncation, NCSES identified a set of “likely performers” and a set of “likely
funders” on the frame. The likely performers and likely funders are a subset of organizations
identified from auxiliary sources that strongly indicate that they are performing or funding
research. The list was then updated based on responses from the NPRA Pilot Survey.
Organizations that stated that they performed or funded were included; organizations that stated
that they did not perform or fund were excluded.
This process resulted in 1,747 likely performers and 1,240 likely funders. The auxiliary sources
and the process for matching to the frame are described in Attachment F. Likely Performer and
Funder Sources. These counts reduced to 1,680 and 1,216, respectively, after the frame
exclusions.
Organizations not identified as a likely performer or a likely funder will be referred to as
“unknown.” The unknown organizations will be a mix of those performing or funding research
(but not flagged via the auxiliary sources), as well as those not conducting any R&D. Nearly all of
the organizations flagged as a likely performer and/or funder are expected to be conducting R&D.
By comparing the organizations identified as likely performers or funders with organizations of
unknown status, NCSES will identify characteristics associated with R&D and use this information
to stratify the organizations into likelihood-based strata.
Table B.2.1 presents the likely performer and likely funder organizations by form type. There were
168 organizations classified as both a likely performer and a likely funder. Most (151) filed Form
990 while two filed 990-EZ, and 15 filed 990-PF. A large majority of the likely performers filed
Form 990. The likely funders filed either Form 990 (35%) or Form 990-PF (64%).
Table B.2.1: Likely Performers and Likely Funders by Form Type
Nonprofit
Organizations

Likely
Performer

Both

Likely
Funder

Likely Performer
or Funder

Form 990

233,484

1,409

151

370

1,930

Form 990-EZ

176,948

71

2

7

80

Form 990-PF

93,299

32

15

671

718

Total

503,731

1,512

168

1,048

2,728

Likely Performers

1,680

Likely Funders

1,216

Financial  Truncation  
To increase the efficiency of reaching organizations that perform or fund research, NCSES will
impose a financial threshold. Table B.2.2 presents the mean revenue, assets, and expenses for the
likely performers and the likely funders, as well as those not identified as either.

17

Table B.2.2: Mean Revenue, Assets, and Expenses
Mean
Likely Performer (n=1,680)

Likely Funder (n=1,216)

Not likely Performer or Funder (n=501,033)

Total Revenue

$107,092,455

Total Assets

$174,924,720

Total Expenses

$99,380,581

Total Revenue

$53,520,896

Total Assets

$297,838,957

Total Expenses

$34,912,864

Total Revenue

$2,934,023

Total Assets

$5,282,247

Total Expenses

$2,643,318

For the NPRA Pilot Survey, NCSES examined cut-offs based on revenue, assets, and expenses. For
each variable, NCSES evaluated the percentage of organizations remaining in the frame and the
percentage of likely performers or funders remaining in the sample. The goal was to retain
organizations that are likely to be performing or funding research, while eliminating organizations
that are unlikely to be performing or funding research. The evaluation resulted in the following
financial requirements:
Form 990 or 990-EZ - Includes organizations with at least $500,000 in expenses. At this
financial threshold, the frame includes 85% of the likely performers/funders and 24% of
all organizations. The 99,943 organizations that meet these criteria represent 97% of total
revenue, 98% of total expenses, and 95% of total assets.
Form 990-PF - Includes organizations with at least $2,750,000 in total assets. At this
financial threshold, the frame includes 88% of the likely funders and 19% of all
organizations. The 17,827 organizations that meet this threshold represent 90% of total
revenue, 83% of total expenses, and 94% of total assets.
Final  Stratification  
After financial truncation, the frame includes 99,943 Form 990 and Form 990-EZ organizations
and 17,827 Form 990-PF organizations. Table B.2.3 includes the number of organizations meeting
the financial threshold by form type.

18

Table B.2.3: Number of Organizations Meeting the Financial Truncation Threshold
All Organizations
Form 990
Form 990-EZ
Form 990-PF

99,869
74
17,827

Likely Performers/ Funders
1,705
1
718

To increase the sampling efficiency, NCSES will stratify the organizations based on frame
variables associated with R&D. The stratifying variables will be codes available on the frame, such
as National Taxonomy of Exempt Entities (NTEE) codes (e.g., hospitals, research institutes), as
well as a propensity score measuring the likelihood that the organization is performing or funding
research.
The propensity score is developed from logistic regression models, with likely performer or funder
as the outcome, and codes and financial characteristics from the core files as the predictor
variables. The propensity scores are used to classify organizations into density strata, where
organizations with high propensity scores will be classified into the high-density stratum, and the
organizations with low propensity scores will be classified into the low-density stratum. Because
NCSES expects the percentage of performers or funders in the high-density stratum to be higher
than the low-density stratum, NCSES will oversample the high-density stratum to increase the
likelihood of sampling a performer or funder.
Table B.2.4 presents the final stratification.
Table B.2.4: Final Stratification
Stratum
1   Likely Performer and Funder
2   Likely Performer (not Funder)
3   Likely funder (not Performer)
4   Hospitals
5   Research Organizations7
6 Form 990 organizations
a.   High density (top decile of
propensity scores)
b.   Medium density (deciles 8, 9)
c.   Low density (deciles 1–7)
7 Form 990-PF organizations
a.   High density (top decile of
propensity scores)
b.   Medium density (deciles 7–9)
c.   Low density (deciles 1–6)

7

Size Stratification
1
2
49
33
588
267
492
201
1,367
634
742
726

Total
3
23
140
96
421
290

4
20
94
57
342
146

5
19
76
44
291
65

6
20
75
42
255
46

164
1,240
932
3,310
2,015

3556
10,468
37,349

1655
4,187
19,171

670
1,848
6,961

362
866
3,283

201
399
1,191

146
227
409

6590
17,995
68,364

582
2,160
4,929

368
1,652
3,228

235
741
1,480

125
358
570

79
126
231

55
64
103

1,444
5,101
10,541

Research organizations are defined as 1) NTEE major group = H-Health - Disease Specific (research) or U-Science and Technology; 2)
Common Code = 05-Research Institutes and/or Public Policy Analysis; and 3) Non-Private Foundation Reason Code = 05-Medical Research.

19

Sample  Allocation  
To determine the sample allocation for the study, NCSES used the estimated percentage of
performers and funders from the NPRA Pilot Survey. NCSES used the estimated percentages for
two aspects of the allocation. First, NCSES estimated the stratum population sizes for performers
and funders. Second, NCSES estimated the relative cost per performer/funder survey in each
stratum. All costs are scaled to the expected performer/funder surveys in the Research
Organization stratum. Organizations identified as likely performers or funders (Strata 1–3) will be
selected with certainty. The organizations that reported performing and/or funding research are
included in Strata 1–3. The estimated population sizes and cost ratios are presented in Table B.2.5.
Table B.2.5: Stratification and Sample Allocation Assumptions

Stratum

Total on
Frame
(N)

Assumed
Perf %

Assumed
Eligible
Perform

Assumed
Fund %

Assumed
Eligible
Fund

Relative
Cost

164

64%

105

75%

123

-

1

Likely Funders and Performers

2

Likely Performers (not Funders)

1,240

61%

752

39%

488

-

3

Likely Funders (not Performers)

932

8%

76

50%

466

-

4

Hospitals

3,310

9%

284

6%

189

4.79

5

Research Organizations

2,015

38%

764

31%

615

1.00

6a

Form 990 Organizations

6,590

19%

1,267

12%

811

2.17

6b

Form 990 Organizations

17,995

6%

1,038

4%

692

7.12

6c

Form 990 Organizations

68,364

5%

3,097

2%

1327

10.57

7a

Form 990-PF Organizations

1,444

5%

77

12%

166

4.07

7b

Form 990-PF Organizations

5,101

3%

135

12%

587

4.83

7c

Form 990-PF Organizations

10,541

4%

373

9%

933

5.52

117,696

7,968

6,397

All likely performers and likely funders (Strata 1-3) will be selected with certainty. The
remaining strata are allocated sample based on the relative cost per performer/funder survey and
the estimated performer population size, and variance of total expenses for each size stratum
The overall sample size was determined to obtain a CV of 1.5% of mean expenses for performing
organizations. Based on the stratification, sample allocation, and assumptions described above,
this requires a sample size of 730 performing organizations. Based on 60% screening response
rate, the expected incidences from the NPRA Pilot Survey and valid responses to the performance
expenditure amount for 95% of organizations, the required sample size is sample size 6,219
organizations. The resulting design included some strata with a single unit selected. NCSES
increased all stratum sizes to a minimum of 15. This resulted in an overall sample size of 6,437.

20

Sample  Selection  
The sample will be a systematic (1-in-k) sample of organizations within each stratum. Organizations
will be selected with equal probability within strata. The organizations will be implicitly stratified
by size before selecting the systematic sample to ensure the sample is proportionally distributed by
size. The organizations in each stratum will be sorted by total expenditures.
Estimation  
Weighting  

Estimates from the NPRA Survey will be weighted to be representative of nonprofit organizations
in scope for the survey. These weights will apply to all estimates derived from the survey data,
such as total expenditures on R&D. The weighting process will begin with the computation of
sampling (base) weights for all selected organizations. The sampling weight is calculated as the
number of organizations in the frame (𝑁" ) divided by the number of organizations selected (𝑚" )
in stratum ℎ, or the reciprocal of the probability of selection, 𝑊1
  =
  𝑁" /𝑚" . The base weight
corrects for differential sampling rates for the strata.
NCSES will then adjust for nonresponse within each stratum. The stratum nonresponse adjustment
will be a ratio adjustment where the respondents (𝑟) will be weighted to account for the nonrespondents (𝑛𝑟). The base weight is used to calculate the adjustment factor,
𝑓1 = (

./0. 𝑊1)

(

. 𝑊1).

The base weight is adjusted by the nonresponse factor, 𝑊2
  =
 𝑊1
  ∗
 𝑓1.
NCSES will do a second nonresponse adjustment to increase the sample representativeness relative
to the data on the frame (i.e., classification codes and financial variables). This will be based on a
propensity score adjustment. For this adjustment, first, NCSES will build a logistic regression
model with survey response as the outcome (1=respond, 0=no response). The outcome will be
modeled based on the frame data. Respondents and non-respondents will be grouped into quintiles
on the basis of their response propensity score. Within each quintile, the respondents (𝑟) will be
weighted to account for the non-respondents (𝑛𝑟), 𝑓2 = ( ./0. 𝑊2) ( . 𝑊2).
The nonresponse adjusted weight is 𝑊3
  =
 𝑊2
  ∗
 𝑓2. The nonresponse adjustment will be
conducted for Form 990 and Form 990-PF separately.
Finally, the weights will be calibrated to match the total number of nonprofit organizations on the
frame. NCSES will use generalized regression calibration (GREG) to match the frame based on
total assets, total expenses and total organizations in each stratum.
Variance  Estimation  

NCSES will use successive difference replication (SDR) (Fay & Train, 1995) a replication
variance method based on a variance estimator for systematic samples presented by Wolter (1985).

21

B.3     Methods  to  Maximize  Response  Rates  and  to  Deal  with  Issues  of  
Nonresponse  
The methodology for the NPRA Survey incorporates features that promote increased response
rates, including the addition of a screener phase to the survey, modifications to the number and
style of communications, and real-time monitoring of survey comments and errors. Utilizing these
strategies should help us engage nonprofit organizations and quickly intervene with respondents
who have questions or issues to improve responses and response rates.
Addition  of  a  Screener  Phase  
Based on our experience with the NPRA Pilot Survey, NCSES found that many of the nonprofit
organizations selected to complete the survey do not perform or fund research. Therefore, NCSES
will conduct a screener so that organizations that do not perform or fund research activities can
quickly respond and then be removed from the survey. As a result, the survey fielding period will
be divided into two phases, Phase I—Screener and Phase II—Survey. Before the beginning of
Phase I, the sample will be divided into two groups: (1) nonprofits strongly believed to currently
perform and/or fund R&D (Knowns) and (2) nonprofits for which NCSES has no information
concerning their R&D activities (Unknowns).
Nonprofits in the Unknown group will begin with the Phase I screener which will identify R&D
performers and funders. Nonprofits in the Known group will be excluded from Phase I because
they have already been identified as likely performers or funders of research in the past, based on
information from the NPRA Pilot Survey or the NCSES Survey of Federal Science and
Engineering Support to Universities, Colleges, and Nonprofit Institutions.
Communications  
Our methodology uses multiple modes of contact (e-mail, mail, telephone); tailoring and varying
the message of contact attempts; short and to-the-point communications; clear and detailed
instructions for responding to the survey; and easy survey access for multiple respondents within
the organization. These features are all associated with increased response rates (Dillman, Smyth,
& Christian, 2008).
NCSES will use several key components of the communications to demonstrate the authenticity
of the survey. First, the Prenotification Letters will be signed by the NCSES Director, John
Gawalt. Other mailed communications and several e-mails will be signed by the NCSES Project
Officer, Ronda Britt. Second, while NCSES already hosts a web page on its site that provides some
information about the survey (https://www.nsf.gov/statistics/srvynpra/), for the full
implementation, information will be added, including the fielding periods, survey contractor’s
name, PDF copies of the survey instruments that can be downloaded, as well as the name and
telephone number of the NCSES Project Officer.

22

Real-­Time  Monitoring  of  Survey  Comments  and  Issues  
In addition to providing more communication with the organizations and demonstrating more
evidence of the authenticity of the survey, NCSES will monitor responses in real time using our
data collection monitoring system. NCSES can identify organizations that have errors in their
survey (which would prevent them from submitting), and proactively contact the organization to
offer assistance or answer questions about the item(s) at issue. This real-time monitoring should
help us increase survey completion and alleviate delays or nonresponse due to questions
organizations may have.
Nonresponse  Analysis  
We will conduct a nonresponse analysis and it will inform the weighting adjustments to correct
for a sample that is disproportionate from the population.
The nonresponse analysis will include four components:
1)   Response rates: During survey fielding, response rates for key subgroups will be
calculated and monitored throughout each phase of the data collection. Subgroups will be
defined based on the frame variables, including by NTEE code and organization size.
Differential response rates across key subgroups will guide data collection activities toward
increasing response where it is low. NPRA Pilot Survey response rates for hospitals have
led to a modified data collection protocol to increase response from these organizations.
2)   Early/Late Responders: The NPRA survey is a multi-mode data collection protocol
designed to maximize response rates. In addition to monitoring response rates by phase,
NCSES will monitor the research expenditures by phase. NCSES will compare the research
expenditures between early responders and those responding later in the survey (i.e. those
more difficult to reach and/or reluctant responders) to determine any systematic differences
in response.
3)   Administrative Data: The NPRA frame contains hundreds of data items from
administrative data collected by the IRS (Forms 990 and 990-PF.) These data include key
correlates to research expenditures, including total expenditures, total assets, as well as
classification codes such as NTEE and NAICS codes. Using the frame data, NCSES will
conduct a logistic regression with survey response as the outcome (1=respond, 0=no
response). The resulting response propensity scores provide a measure of
representativeness based on the frame data, which will be quantified in the form of an Rindicator as described by Schouten et al (2009).
4)   Debriefing Interviews: Forty Debriefing Interviews will be conducted. Twenty will be
conducted with organizations that have completed the questionnaire approximately two
weeks after submission and will focus on the survey content and process. The other 20
interviews will be conducted with organizations that chose not to respond at the end of data
collection to examine why they decided not to respond. Specific issues that will be covered

23

are general attitudes toward the survey; whether the organization received the contacts,
and, if not, which modes did they not receive; whether the contacts were sent to the right
person; and the frequency of attempts. Information from the Debriefing Interviews will be
used to refine the data collection strategy for the next cycle of the survey.
B.4  

Tests  of  Procedures  and  Methods  

No methodological tests are planned for the survey. NCSES conducted the NPRA pilot survey in
2016 to test the feasibility of the questions, sampling procedures, and data collection procedures.
The methodology and results of this pilot survey are described in the evaluation report which is
available upon request from the NCSES Project Officer.
B.5  

Individuals  Consulted  on  Technical  and  Statistical  Aspects  of  Design  

The NPRA Survey will be conducted by NSF's National Center for Science and Engineering
Statistics. ICF is the contractor in charge of data collection. The name, title, affiliation, and
telephone numbers for those consulting on the project are below.
John Jankowski
Program Director, R&D Statistics
National Science Foundation
(703) 292-7781

Rebecca L. Morrison
Survey Statistician
National Science Foundation
(703) 292-7794

Ronda Britt
Project Officer, NPRA Survey
Survey Statistician
National Science Foundation
(703) 292-7765

Randal ZuWallack
Senior Statistician
ICF
(802) 264-3724

Samson Adeshiyan
Chief Statistician
National Science Foundation
(703) 292-7765

Dr. William Bryan Higgins
Senior Methodologist
ICF
(703) 934-3498

Jock Black
Mathematical Statistician
National Science Foundation
(703) 292-7802

Adam Lee
Statistician
ICF
(301) 572-0814

Michael Gibbons
Survey Statistician
National Science Foundation
(703) 292-4590

Arlen Rosenthal
Project Manager
ICF
(301) 572-0222

24

Bibliography  
Cochran, WG, Sampling Techniques, John Wiley & Sons (1977).
Dillman DA, Smyth, JD, Christian, LM, Internet, Phone, Mail, and Mixed-Mode Surveys: The
Tailored Design Method, 4th Edition, John Wiley & Sons (2014).
Fay RE, Train GF, Aspects of survey and model-based postcensal estimation of income and
poverty characteristics for states and counties, Proceedings of the Section on Government
Statistics, pp. 154–159, American Statistical Association (1995).
Pettijohn SL, The nonprofit sector in brief: Public charities, giving, and volunteering, 2013,
http://www.urban.org/research/publication/nonprofit-sector-brief-public-charities-givingand-volunteering-2013 (2013).
Salamon LM, A profile of the nonprofit sector in the United States. In National Research
Council (C. House, H. Rhodes, & E. Sinha, Rapporteurs, Committee on National
Statistics, Division of Behavioral and Social Sciences and Education), Measuring
Research and Development Expenditures in the U.S. Nonprofit Sector: Conceptual and
Design Issues, Summary of a Workshop, pp. 9–26, The National Academies Press (2015).
Schouten B, Cobben F, Bethlehem J, Indicators for the representativeness of survey response.
Survey Methodology, 101–113 (2009).
Wolter KM, Introduction to Variance Estimation, Springer-Verlag (1985).

25

Attachments  
A.   FY 2016 NSF Survey of Nonprofit Research Activities
i.   A-1: FY 2016 NSF Survey of Nonprofit Research Activities
ii.   A-2: FY 2016 NSF Survey of Nonprofit Hospital Research Activities
B.   Comment Letter from Andrew Reamer
C.   Debriefing Interview Protocols
a.   Debriefing Protocol for Respondents
b.   Debriefing Protocol for Non-respondents
D.   Debriefing Interview Correspondence
E.   NCCS Core File Description
F.   Likely Performer and Funder Sources
G.   Propensity Models for Performer and Funder Stratification
H.   Communications with Nonprofit Organizations of Unknown Eligibility
I.   Communications with Nonprofit Organizations of Known Eligibility

26