NPRA Respondent Debriefing Interviews

Attachment C. Debriefing Interview Protocols
i: Debriefing Protocol for Respondents
ii. Debriefing Protocol for Non-respondents

i: Debriefing Protocol for Respondents

Debriefing Protocol for Respondents

Organization Name: ___________________________________________
Organization ID: __________________
Performer:

 Yes

 No

Funder:

 Yes

 No

Completed IRS Form  990  990-PF
Completed Screener:

 Yes  No

Completed survey:  Online  Paper  Phone
Organization contact NPRA help desk by:
E-mail to [email protected]

 Yes

 No

E-mail to NSF (Ms. Britt)

 Yes

 No

Phone to NPRA Help Line

 Yes

 No

Phone to NSF (Ms. Britt)

 Yes

 No

Be sure you have a PDF of the organization’s survey in case you need to e-mail it to the
participant.
Review Excel file of errors that were triggered when the organization completed the survey.
This information is needed when asking Q13.

Thank you for completing the FY 2016 Survey of Nonprofit Research [Hospital] Activities and
for agreeing to participate in this follow-up interview. The information you provide today will
help us to better understand your perspective about completing the survey and will help us make
changes to the survey before we conduct the survey in the winter of 2019.
If organization completed survey via web: I’ll be showing you the web survey your
organization [] completed. Is that okay?
If yes: Great! I’m going to send you an e-mail right now (repeat e-mail address) with a link to
www.join.me that will allow you to share my screen. You should not have to download anything.
Please let me know when you receive the e-mail.

Page | Ci-1

Received the e-mail: Okay, please go ahead and click on the link. Let me know what you see.
Skip to A below
Did not receive e-mail: Let me repeat your e-mail address. The e-mail you receive will come
from . The subject says: NPRA Debrief Interview. Would you mind checking
your spam or junk mail folder?
If the participant is unable or unwilling to login into www.join.me: Since we are not having
luck with the computer, I’m going to e-mail you a copy of your organization’s completed survey.
Is that okay? You should see an e-mail coming from in a minute.
It will have a PDF attached. Please let me know when you get it.
If the organization submitted a paper survey: [If you would prefer,] I will e-mail you a PDF
of the completed survey. Would that be okay?
If still no: Would you prefer to reschedule the interview? OR Is there someone else you
think I should talk to?
A. Before we begin our interview, I want to review some information with you.







Your participation in this interview is voluntary.
The interview should last about 1 hour.
There is another person on the telephone with me who is taking notes. Her name is
.
If appropriate: , from NSF, is also listening to our discussion today. Is that
okay? If no, NSF staff member will hang up the phone and end his/her join.me session.
I would like to audio record this interview so that I get all of your comments. Is that okay
with you?
Before we continue with the interview, I need your consent to be interviewed. I’m going
to [join.me: read you the consent form and you can read along with me] and then I’ll ask
for your verbal consent]. OR [non join.me: I will read the consent information to you,
then ask for your verbal consent.]

Page | Ci-2

The National Science Foundation designs, conducts, and sponsors surveys on science,
engineering, and technology. In order to produce the best information possible, the National
Science Foundation routinely evaluates its surveys.
You have volunteered to take part in a study to improve one of the National Science
Foundation’s surveys. In order to have a complete record of your comments, your interview
session will be audio recorded. We plan to use the recording to improve the survey. NSF and
ICF staff directly involved in this research project will have access to the recording.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for
this study is XXXX-XXXX.
I have volunteered to participate in this study, and I give permission for my recording to be
used for the purposes stated above.
If you consent to participate in this interview, please state your name and that you consent.
Do you have any questions before we start?
I would like to start with a few general questions. Later we will talk about some of the questions
in the survey that I’ll show you [online OR ask you to open and refer to the PDF].
First, let’s talk for a couple of minutes about participating in this survey and then about your
experience completing the survey online/on paper.
1. What was your initial reaction to the National Science Foundation’s request for you to
complete this survey?
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
2. Why did your organization decide to complete the survey?
_____________________________________________________________________
_____________________________________________________________________
Ask Q3-Q7 if the respondent used a mobile device. Otherwise skip to Q8.
3. We noticed you registered and possibly completed some of the questions on an
iPad/iPhone/Android/other mobile device, which surprised us. Can you tell me why you
chose to do that?
____________________________________________________________________
____________________________________________________________________

Page | Ci-3

4. How much of the survey did you complete using your mobile device?
____________________________________________________________________
_____________________________________________________________________

5. Can you describe your experience using your mobile device to interact with the online data
collection system?
____________________________________________________________________
____________________________________________________________________
6. What did you like about interacting with the survey on your device?
_____________________________________________________________________
____________________________________________________________________

7. What did you not like about interacting with the survey on that device?
____________________________________________________________________
____________________________________________________________________
ALL RESPONDENTS
8. How easy or difficult was it for you to create a password and log into the survey?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
8a. For “somewhat difficult” or “very difficult” ask: What were some of the
difficulty/difficulties you had?
____________________________________________________________________
____________________________________________________________________

Page | Ci-4

Ask Q 9, 9a, 10 and 11 only if the organization completed the survey online:
9. Did you download a blank PDF copy of the survey?
 Yes – ask 9a
 No – skip to 10
9a. (If Yes) What did you do with the PDF of the survey that you downloaded? (e.g., gather
and enter information on the paper copy before entering data online, send to Board/President
for approval, send to other staff to complete sections)
_____________________________________________________________________
_____________________________________________________________________
10. What was your approach to completing the survey? Did you:
 Gather all the information needed and entered it yourself online?
 Asked others to log in and complete sections of the survey? If yes:
o Did you use the online system to invite other members of your staff to
complete certain sections of the survey:
 Yes
 No
 Asked another staff member to gather the information and enter it into the system
 Other, specify _________________________________________________________
11. Ask only if respondent completed the entire survey: Were you aware that you could allow
another person to access the online survey so that the individual could assist you in
completing the survey?
 Yes
 No

Page | Ci-5

ALL RESPONDENTS
12. How easy or difficult was it to navigate through the paper survey/online survey system?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Ask Q13 only if organization completed the survey online and the Excel file indicates the
organization made one or more errors. If survey was not completed online or the organization
did not make any errors, skip to Q14
13. I noticed that you encountered an error message(s) when you were completing the survey.
Do you remember seeing any error messages while working on the questionnaire?
 No
 Yes. If yes ask: Do you happen to remember what the message(s) were telling you?
_________________________________________________________________
_____________________________________________________________________
Do you remember how you felt when you saw the message(s)? Were you able to solve the
problem(s)?
_____________________________________________________________________
_____________________________________________________________________
14. Thinking about the overall experience of completing the survey, how positive or negative
was your experience?
 Very positive
 Somewhat positive
 Neutral
 Somewhat negative
 Very negative

Page | Ci-6

Interviewer, if our records indicate this nonprofit called or e-mailed us, ask the following
question, otherwise skip to Q16.
15. Our records indicate that you or your staff our survey helpdesk.
How satisfied were you with the help you received?
 Very satisfied
 Somewhat satisfied
 Neutral
 Somewhat dissatisfied
 Very dissatisfied
For the questions that follow, we would like to focus on the questions and content of the
survey itself. Interviewer, for participants who can see your screen, say: First, I’d like to point
to a couple of the directions. Can you see this? . For participants looking at
a PDF say: Go ahead and open the PDF I sent you. Let’s look at the instructions at the top of
page 4 and the ones at the top of page 6.
16. Were the instructions provided in the survey generally clear or confusing?
 Clear
 Confusing. If confusing, ask: Can you describe what was difficult to understand?
_____________________________________________________________________
_____________________________________________________________________

17. Were there any terms in the instructions that were not defined and should be defined?
 No
 Yes. If yes ask: Which one(s)?
_____________________________________________________________________
_____________________________________________________________________

Page | Ci-7

18. The survey included a definition of research, what it includes, and examples of what may
meet and most likely does not meet the criteria for research. Do you happen to remember
reading and/or referring to this information? Show this section of the survey on the screen or
refer participant to page 4 in the PDF.
 Yes
 No
18a. Was this information helpful or not helpful? Is there anything you would suggest we add
or remove from the definition and guidance page?
_____________________________________________________________________
_____________________________________________________________________

[Interviewer, be sure to note if the nonprofit is a performer, funder, or both performer and
funder and skip the questions that do not apply.]
Now let’s talk about some of the specific questions in the questionnaire.
We’ll go through the questionnaire sequentially.
19. (Q2) We put your organization’s EIN in Q2 and asked if it was correct.
If organization checked yes, ask: Did you verify the number before you checked yes, or did
you not verify it?
 Verified EIN
 Did not verify EIN
If organization checked no, ask: What did you do when you saw the incorrect EIN?
_____________________________________________________________________
_____________________________________________________________________
20. (Q3) How easy or difficult was it to answer Question 3?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

Page | Ci-8

21. (Q4) You reported your organization’s total expenses for 2016 were . Do you
remember if this was the same amount reported on your IRS Form (990/990-PF), or was it
different?
 Same
 Different: Please explain. __________________________________________
 Don’t remember
22. (Q6) How did you obtain or calculate the number of full-time equivalent employees (FTEs)
that were employed in your organization in FY 2016?
____________________________________________________________________
____________________________________________________________________

23. How easy or difficult was it to obtain or calculate this figure? (probe to see if the
example/formula was helpful)
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

23a. Did you include or exclude:
Contract employees

 Included

 Excluded

 NA

24. We noticed that you did not respond to Questions 1…5 can you please explain why you did
not respond?
Survey Q1
_____________________________________________________________________
_____________________________________________________________________
Survey Q2
_____________________________________________________________________
_____________________________________________________________________
Page | Ci-9

Survey Q3
_____________________________________________________________________
_____________________________________________________________________
Survey Q4
_____________________________________________________________________
_____________________________________________________________________

Survey Q5
_____________________________________________________________________
_____________________________________________________________________

Survey Q6
_____________________________________________________________________
_____________________________________________________________________
If organization does not perform research (Q7=No), skip to Q36 for questions about the
funding research section of the survey
25. (Q7) This question asks if your organization performed research in FY 2016. How did you
choose your answer to this question? _______________________________________
____________________________________________________________________
25a. Can you briefly describe some of the research projects you thought of when you
answered Question 7?
_____________________________________________________________________
_____________________________________________________________________

Page | Ci-10

26. You indicated on Question 7 that your organization performed research in FY 2016. One of
the instructions was to include research performed by your contract employees. Was this
instruction clear or confusing?
 Clear
 Confusing. If confusing ask: What difficulties did you have?
_____________________________________________________________________
_____________________________________________________________________

27. Similarly, we asked you to exclude research performed outside your organization by
subcontractors or sub-awardees in Q7 but to include that research in Q8. Was this instruction
clear or confusing?
 Clear
 Confusing. If confusing ask: What difficulties did you have?
_____________________________________________________________________
_____________________________________________________________________
Interviewer: Ask debrief Questions 28−35 ONLY if the organization performed research
(i.e., responded yes to Q7).
The next section of the survey asked several questions on the performance of research by
your employees and contract employees. For the next few minutes, we will focus on this
part of the survey.
28. (Q9) How easy or difficult was it to provide the total amount your organization spent on
research performed within your organization in FY 2016?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

Page | Ci-11

Interviewer, if respondent replied “Somewhat difficult” or “Very difficult,” ask:
Can you tell me why Q9 was difficult for you to provide the total amount your organization
spent on research?
_____________________________________________________________________
_____________________________________________________________________
28a. Was the amount you reported——an estimate or the actual amount your
organization spent on research performed within your organization?
 Estimate
 Actual
28b. How confident are you that the answer provided is the correct answer?
 Very confident
 Somewhat confident
 Not at all confident

29. (Q10) We asked you to indicate the sources of funding for the research your organization
performed in FY 2016 (e.g., federal government, state & local government, businesses,
foundations, universities, other nonprofits, internal funds, individual donors). How easy or
difficult was the process of allocating the total funds you reported in Q9 by source?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________

Page | Ci-12

29a. If NPO checked “Individual donors” ask respondent: I see that one of your sources of
funding is individual donors. Can you tell me a little more about the individual donor(s)?
____________________________________________________________________
____________________________________________________________________
29b. [If applicable] (Refer to Q9) You indicated on your survey that you received funding
from entities other than the federal government. Do you know if any of these other amounts
you reported were subcontract awards on federal projects?
 No
 Yes, please describe.
__________________________________________________________________________
__________________________________________________________________________

30. (Q11) How easy or difficult was it for you to allocate the total FY 2016 research performed
within your organization in the fields we provided on Question 10?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
30a. How easy or difficult was it to match the sum total of all fields in Question 11 to the
total you reported in Question 9? (In FY 2016, how much did your organization spend on
research performed within your organization?)
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Page | Ci-13

Ask 30b only if respondent completed the survey online and if you are showing their
survey.
30b. Did you click on any of the

symbols to see examples of subfields?

 Yes – ask 30c
 No – why not? _______________________________________________ Go to Q32
30c. How helpful were the drop-down menus of fields in each of the major categories?
 Very helpful
 Somewhat helpful
 Not at all helpful
 Didn’t notice them
Please explain.
_____________________________________________________________________
_____________________________________________________________________
31. (Q12) Question 12 asked how much of the research your organization performed was for
basic research, applied research, and experimental development. We also asked you to report
the totals the sources of research expenditures (Federal and Nonfederal).
31a. How easy or difficult was it for you to report the total amounts of basic research,
applied research, and experimental development?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this
difficult?
_____________________________________________________________
_____________________________________________________________

Page | Ci-14

If survey was completed online ask: Did you notice that you could have selected
unavailable?
 Yes
 No
31b. How easy or difficult was it for you to report the amounts of basic research, applied
research, and experimental development by Federal and Nonfederal Sources?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
31c. Were the examples of basic research, applied research, and experimental development
helpful or unhelpful?
 Helpful
 Neutral
 Not helpful -- Why? -________________________________________________

32. (Q13) The survey included a question asking for the approximate number of paid full-time
equivalents (FTEs) who worked on research activities in FY 2016. We asked you to include
the number of FTE Researchers, and FTE Technicians and other support personnel. How did
you obtain or calculate the number of FTEs in each category?
____________________________________________________________________
____________________________________________________________________

Page | Ci-15

32a. How confident are you that you provided an accurate count of FTE researchers and FTE
technicians and other support personnel?
 Very confident
 Somewhat confident
 Not at all confident
Interviewer, if respondent was only somewhat or not at all confident, ask: Why was this
difficult?
____________________________________________________________________
____________________________________________________________________
33. (Q14) We also asked if any contract employees worked on your research activities in
FY2016. How easy or difficult was it for you to answer this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

Interviewer, if respondent was only somewhat or very difficult, ask: What suggestions can
you offer for making this question easier to understand?
____________________________________________________________________
____________________________________________________________________
34. (Q15) Question 15 asked if you had any volunteers work on your research activities in FY
2016. How easy or difficult was it for you to answer the question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Page | Ci-16

Interviewer, if respondent was only somewhat or very difficult, ask: What suggestions can
you offer for making this question or the categories easier to understand?
____________________________________________________________________
____________________________________________________________________
[Enumerate the ones skipped]:
35. We noticed that you did not respond to Questions 8…14. Can you please explain why you did
not respond?
Survey Q7
_____________________________________________________________________
_____________________________________________________________________
Survey Q9
_____________________________________________________________________
_____________________________________________________________________
Survey Q10
_____________________________________________________________________
_____________________________________________________________________
Survey Q11
_____________________________________________________________________
_____________________________________________________________________
Survey Q12
_____________________________________________________________________
_____________________________________________________________________
Survey Q13
_____________________________________________________________________
_____________________________________________________________________
Survey Q14
_____________________________________________________________________
_____________________________________________________________________

Page | Ci-17

Survey Q15
_____________________________________________________________________
_____________________________________________________________________
If the organization does NOT fund research, skip to Q43.
Funder only (yes to Q8): Begin with Q36 and continue through the remaining questions. If
the organization is a performer and funder, ask questions if time permits. Be sure to ask Q4346.
The next section of the survey asked several questions about funding research activities of
other organizations—either through contracts, grants, subcontracts, or other sub-awards.
For the next few minutes, we will focus on this part of the survey.
36. Let’s begin by looking at Q8, considering the guidance above, did your organization fund
research performed outside your organization in FY 2016? How easy or difficult was it for
you to answer this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
36a. Can you briefly describe some of the research projects you thought of when you
answered Question 8?

37. (Q16) We asked you how much funding your organization provided to others in FY 2016 to
perform research activities. In addition to providing a total amount, we asked for subtotals
for the amount funded via (a) research grants or contracts, and (b) subcontracts and subawards. How did you go about answering this question?
____________________________________________________________________
____________________________________________________________________

Page | Ci-18

37a. How easy or difficult was it for your organization to answer this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
__________________________________________________________________
__________________________________________________________________
38. (Q17) In Question 17 we asked you to tell us how much your organization provided to (a)
universities or other educational entities, (b) other nonprofit organizations, (c) businesses,
and/or (d) other entities. How easy or difficult was it to separate the amounts you awarded to
these types of organizations?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
__________________________________________________________________
__________________________________________________________________
39. (Q18) In Question 18, how easy or difficult was it for you to indicate how much research
funding your organization provided to others in FY 2016 using the fields listed?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

Page | Ci-19

Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
39a. How easy or difficult was it to match the sum total of all fields in Question 18 to the
total you reported in Question 16? (In FY 2016, how much funding did your organization
provide to others to perform research activities?)
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Ask 39b only if respondent completed the survey online and if you are showing their
survey.
39b. Did you click on any of the

symbols to see examples of subfields?

 Yes – ask 39c
 No – why not? _______________________________________________ Go to Q40
39c. How helpful were the drop-down menus of fields in each of the major categories?
 Very helpful
 Somewhat helpful
 Not at all helpful
 Didn’t notice them

Page | Ci-20

40. (Q19) Question 19 asked you to identify the different sources of funding for the total
reported in Question 16. How easy or difficult was this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
40a. [Q19, If applicable] You indicated on your survey that you received funding from
entities other than the federal government. Do you know if any of these other amounts you
reported were subcontract awards on federal projects?
 No
 Yes, please describe.
__________________________________________________________________________
__________________________________________________________________________
41. (Q20) Question 20 asked how much of the research your organization performed was for
basic research, applied research, and experimental development. We also asked you to report
the totals the sources of research expenditures (Federal and Nonfederal).
41a. How easy or difficult was it for you to report the total amounts of basic research,
applied research, and experimental development?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

Page | Ci-21

Interviewer, if respondent says somewhat or very difficult, ask: Why was this
difficult?
_____________________________________________________________
_____________________________________________________________
If survey was completed online ask: Did you notice that you could have selected
unavailable?
 Yes
 No
41b. How easy or difficult was it for you to report the amounts of basic research, applied
research, and experimental development by Federal and Nonfederal Sources?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
41c. Were the examples of basic research, applied research, and experimental development
helpful or unhelpful?
 Helpful
 Neutral
 Not helpful -- Why? -________________________________________________

Page | Ci-22

[Enumerate the ones skipped]:
42. We noticed that you did not respond to Questions 8, 16…20, can you please explain why you
did not respond?
Survey Q8
_____________________________________________________________________
_____________________________________________________________________
Survey Q16
_____________________________________________________________________
_____________________________________________________________________
Survey Q17
_____________________________________________________________________
_____________________________________________________________________
Survey Q18
_____________________________________________________________________
_____________________________________________________________________
Survey Q19
_____________________________________________________________________
_____________________________________________________________________
Survey Q20
_____________________________________________________________________
_____________________________________________________________________

Page | Ci-23

Ask all respondents:
Ask only if organization answered Q21.
43. I see that you provided some additional information about some of the research your
organizations [performs and/or funds]. Is this something you were able to cut and paste from
another document?
 Yes – if yes, What type of document did you copy this from? For example, your
annual report, IRS Form 990/990-PF, something else?
 No. So you wrote something specifically for this survey?  Yes  No
44. We included space for you to write a comment on several of the questions. As appropriate
ask 44a or 44b.
44a. I noticed that that you wrote some comments. Was it helpful to be able to provide
additional information about your response?
 Yes.
 No. Why not? __________________________________________________

44b. I noticed that you did not write any comments. Why not? ________________________
______________________________________________________________________

45. (Q24) Let’s take a look at the last survey item – it’s Question 24. How easy or difficult was it
for you to answer this question? Would you say:
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

Page | Ci-24

Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
__________________________________________________________________________

45a. Did you need to consult with anyone else before you answered this question?
 Yes. If yes, what is the title of the person(s) ___________________________
 No
45b. Two examples were included to give you an idea of what a table with organizational
level may look like and another example of what an aggregated table may look like. How
helpful or unhelpful were these examples?
 Helpful
 Not helpful
46. Before we conclude this interview, I’d like to ask you if there is anything else you would like
to share regarding your experience completing the survey. Is there anything else we haven’t
discussed that you’d like to see improved or changed before the next survey?
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
On behalf of the National Science Foundation, thank you very much for your time and
assistance.

Page | Ci-25