NPRA Respondent Debriefing Interviews

Attachment D
Debriefing Interview Protocols

Protocol for Debriefing Interview with Respondents
Nonprofit Research Activities Pilot Survey

Organization ID: __________________
Performer:
Funder:

 Yes
 Yes

 No
 No

Thank you for completing the Pilot of the Nonprofit Research Activities Survey and for agreeing
to participate in this follow-up interview. The information you provide today will help us to
better understand your perspective about completing the survey and will help us make changes to
the survey before we fully implement it next year. Before we begin our interview, I want to
review some information with you.







Your participation in this interview is voluntary.
The interview should last about 1 hour.
There is another person on the telephone with me who is taking notes. Her name is
.
If appropriate: , from NSF, is also listening to our discussion today. Is that
okay? If no, NSF staff member will hang up the phone and end his/her join.me session.
I would like to audio record this interview so that I get all of your comments. Is that okay
with you?
Before we continue with the interview, I need your consent to be interviewed. I’m going
to read it to you, then ask for your verbal consent.

The National Science Foundation designs, conducts, and sponsors surveys on science,
engineering, and technology. In order to produce the best information possible, the National
Science Foundation routinely evaluates its surveys.
You have volunteered to take part in a study to improve one of the National Science
Foundation’s surveys. In order to have a complete record of your comments, your interview
session will be audio taped. We plan to use the audio tape to improve the survey. NSF and ICF
staff directly involved in this research project will have access to the tape. In addition, a
transcription service may have access to the audio tape, to provide us with a written version of
your comments.
This study is authorized by law (42 U.S.C. 1862 Section 3.a.6). The OMB control number for
this study is XXXX-XXXX.
I have volunteered to participate in this study, and I give permission for my tape to be used for
the purposes stated above.
If you consent to participate in this interview, please state you name and that you consent.

D-1

Do you have any questions before we start?
As mentioned when she spoke to you, I would like to share my computer screen with
you. I’ll be showing you the web survey completed. Is that okay?
If yes: Great! I’m going to send you an e-mail right now (repeat e-mail address) with a link to
www.join.me. You should not have to download anything. Please let me know when you receive
the e-mail.
Received the e-mail: Okay, please go ahead and click on the link. Let me know what you see.
Did not receive e-mail: Would you mind checking your spam or junk mail folder?
If respondent still does not find it: Let me repeat your e-mail address. The e-mail you
receive will come from . The subject says: NPRA Debrief Interview.
If the participant is unable to login into www.join.me: Since we are not having luck with the
computer, I’m going to e-mail you a copy of completed survey. Is that
okay? You should see an e-mail coming from in a minute. It will
have a PDF attached. Please let me know when you get it.
If no, it’s not okay to go online: If you would prefer, I can e-mail you a PDF of the
completed survey. Would that be okay?
If still no: Would you prefer to reschedule the interview? OR Is there someone else you
think I should talk to?
Assuming participant is able to get online and sees the interviewer’s screen or the PDF …
I would like to start with a few general questions, then later we will talk about some of the
questions in the survey that I’ll show you [online OR ask you to open and refer to the PDF].
First, let’s talk for a couple of minutes about participating in this survey and then your
experience completing the survey online. Do you have any questions before we begin?

D-2

1. What was your initial reaction to the National Science Foundation’s request for you to
complete this survey?
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
2. Why did your organization decide to complete the survey?
_____________________________________________________________________
_____________________________________________________________________
The next couple questions are about your experience completing the survey online.
3. Did you download a PDF copy of the survey?
 Yes
 No
4. Did you gather all the information needed and enter it into the online system yourself?
 Yes (Skip to question Q6)
 No
5. Did you use the online system to invite other members of your staff to complete certain
sections of the survey?
 Yes
 No
6. How easy or difficult was it to navigate through the survey?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

D-3

7. Did you encounter any error messages while working on the questionnaire?
 No
 Yes. If yes ask: Do you happen to remember what the messages were telling you?
____________________________________________________________________
_____________________________________________________________________
What was your reaction to getting these messages? Were you able to solve the problem(s)?
_____________________________________________________________________
_____________________________________________________________________
8. How positive or negative was your experience answering the questions online?
 Very positive
 Somewhat positive
 Neutral
 Somewhat negative
 Very negative
Interviewer, if our records indicate this nonprofit called or e-mailed us, ask the following
question, otherwise, skip to Q10.
9. Our records indicate that you or your staff our survey helpdesk.
How satisfied were you with the help you received?
 Very satisfied
 Somewhat satisfied
 Neutral
 Somewhat dissatisfied
 Very dissatisfied
For the questions that follow, we would like to focus on the questions and content of the
survey itself. Interviewer, for participants who can see your screen, say: First, I’d like to point
to a couple of the directions. Can you see this? . For participants looking at
a PDF say: Go ahead and open the PDF I sent you. Let’s look at the instructions at the top of
page 3 and the ones at the top of page 4.

D-4

10. Were the instructions provided in the survey generally clear or confusing?
 Clear
 Confusing. If confusing, ask: Can you describe what was difficult to understand?
_____________________________________________________________________
_____________________________________________________________________

11. Were there any terms that were not defined and should be defined?
 No
 Yes. If yes ask: Which one(s)?
_____________________________________________________________________
_____________________________________________________________________
12. The survey included a definition of research, what it includes, and examples of what may
meet and most likely does not meet the criteria for research. Do you happen to remember
reading and/or referring to this information? Show this section of the survey on the screen or
refer participant to page 3 in the PDF.
 Yes
 No
Was this information helpful? Is there anything you would suggest we add or remove from
the definition and guidance page?
_____________________________________________________________________
_____________________________________________________________________

D-5

[Interviewer, be sure to note if the nonprofit is a performer, funder, or both performer and
funder and skip the questions that do not apply.]
Now let’s talk about some of the specific questions in the questionnaires.
We’ll go through the questionnaire sequentially. I’ll show them to you on my screen OR You
can look at the questions in your survey.
13. (Q2) You reported your organization’s total expenses for FY 2015 were . Do you
remember if this was the same amount reported on your IRS Form (990/990-EZ/990-PF), or
was it different?
 Same
 Different: Please explain.
 Don’t remember
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
14. (Q4) How easy or difficult was it to provide the number of full-time equivalent employees
(FTEs) that were employed in your organization in FY 2015?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Did you include or exclude consultants, part-time employees, and temporary, or seasonal
employees?
_____________________________________________________________________
_____________________________________________________________________

D-6

15. [Q5, if applicable] When you completed the survey, you indicated that your organization
performed research in FY 2015. One of the instructions was to include consultants. Was this
instruction clear or confusing?
 Clear
 Confusing. If confusing ask: What difficulties did you have?
_____________________________________________________________________
_____________________________________________________________________
Similarly, we asked you to exclude research performed outside your organization by
subcontractors or sub-awardees in Q5 but to identify those relationships in Q6. Was this
instruction clear or confusing?
 Clear
 Confusing. If confusing ask: What difficulties did you have?
_____________________________________________________________________
_____________________________________________________________________
[Enumerate the ones skipped]:
16. We noticed that you did not respond to Questions 1…6, can you please explain
why you did not respond?
Survey Q1
_____________________________________________________________________
_____________________________________________________________________
Survey Q2
_____________________________________________________________________
_____________________________________________________________________
Survey Q3
_____________________________________________________________________
_____________________________________________________________________
Survey Q4
_____________________________________________________________________
_____________________________________________________________________

D-7

Survey Q5
_____________________________________________________________________
_____________________________________________________________________
Survey Q6
_____________________________________________________________________
_____________________________________________________________________
Interviewer: Ask questions 17–20 ONLY if the organization performed research
(i.e., responded yes to Q5).
The next section of the survey asked several questions on the performance of research by
your employees or consultants. For the next few minutes, we will focus on this part of the
survey.
17. (Q7) How easy or difficult was it to provide the total amount your organization spent on
research performed by your employees or consultants in FY 2015?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent replied “Somewhat difficult” or “Very difficult,” ask:
Were any of the following reasons why it was difficult for you to provide the total amount
your organization spent on research?
A. Excluding capital expenditures
 Yes
 No
B. Separating research and other expenditures
 Yes
 No

D-8

C. Other reasons why the question was difficult to answer.
____________________________________________________________________
Was the amount you reported——an estimate or the actual amount your
organization spent on research performed by your employees or consultants?
 Estimate
 Actual
How confident are you that the answer provided is the correct answer?
 Very confident
 Somewhat confident
 Not at all confident

18. (Q8) We asked you to indicate the sources of funding for the research your organization
performed in FY 2015 (e.g., federal government, universities, foundations, other nonprofits).
How easy or difficult was the process of allocating the total funds you reported in Q7 by
source?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________

D-9

[If applicable] You indicated on your survey that you received funding from entities other
than the federal government. Were any of these other amounts you reported actually
subcontract awards on federal projects?
 No
 Yes, please describe.
__________________________________________________________________________
__________________________________________________________________________

19. (Q9) How easy or difficult was it for you to allocate the total FY 2015 research performed by
your staff or consultants in the fields we provided?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
How easy or difficult was it to match the sum total of all fields in Question 9 to the total you
reported in Question 7? (In FY 2015, how much did your organization spend on research
performed by your employees or consultants?)
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

D-10

How helpful were the drop-down menus of fields in each of the major categories?
 Very helpful
 Somewhat helpful
 Not at all helpful
Please explain.
_____________________________________________________________________
_____________________________________________________________________
20. (Q10) The survey included a question asking for the approximate number of paid full-time
equivalents (FTEs) who worked on research activities in FY 2015. We asked you to include
the number of FTE Researchers, and FTE Technicians and other support personnel. How did
your organization decide which category to put your paid full-time FTEs?
____________________________________________________________________
____________________________________________________________________
How confident are you that you provided the correct number of FTE researchers and FTE
technicians and other support personnel?
 Very confident
 Somewhat confident
 Not at all confident
Interviewer, if there’s confusion, ask: What suggestions can you offer for making this
question or the categories easier to understand?
____________________________________________________________________
____________________________________________________________________
[Enumerate the ones skipped]:
We noticed that you did not respond to Questions 7…11. Can you please explain why you did not
respond?
Survey Q7
_____________________________________________________________________
_____________________________________________________________________

D-11

Survey Q8
_____________________________________________________________________
_____________________________________________________________________
Survey Q9
_____________________________________________________________________
_____________________________________________________________________
Survey Q10
_____________________________________________________________________
_____________________________________________________________________
Survey Q11
_____________________________________________________________________
_____________________________________________________________________
Interviewer, ask questions 20–24 only if the organization funded research (i.e., responded yes
to Q6). If the organization is a performer and funder, ask questions if time permits.
The next section of the survey asked several questions about funding research activities of
other organizations—either through contracts, grants, subcontracts, or other sub-awards.
For the next few minutes, we will focus on this part of the survey.
21. Let’s begin by looking at Q6, considering the guidance above, did your organization fund
research performed outside your organization in FY 2015? How easy or difficult was it for
you to answer this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult

D-12

22. (Q12) We asked you how much funding your organization provided to others in FY 2015 to
perform research activities. In addition to providing a total amount, we asked for subtotals
for the amount funded via (a) research grants or contracts, and (b) subcontracts and subawards. How did you go about answering this question?
____________________________________________________________________
____________________________________________________________________

How easy or difficult was it for your organization to answer this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
__________________________________________________________________
__________________________________________________________________
23. (Q13) In Question 13 we asked you to tell us how much your organization provided to (a)
universities or other educational entities, (b) other nonprofit organizations, (c) businesses,
and/or (d) other entities. How easy or difficult was it to separate the amounts you awarded to
these types of organizations?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
__________________________________________________________________
__________________________________________________________________

D-13

24. (Q14) How easy or difficult was it for you to indicate how much research funding your
organization provided to others in FY 2015 using the fields listed?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
How easy or difficult was it to match the sum total of all fields in question 14 to the total you
reported in questions 12? (In FY 2015, how much funding did your organization provide to
others to perform research activities?)
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
How helpful were the drop-down menus of fields in each of the major categories?
 Very helpful
 Somewhat helpful
 Not helpful
.

D-14

25. (Q15) Question 15 asked you to identify the different sources of funding for the total
reported in question 12. How easy or difficult was this question?
 Very easy
 Somewhat easy
 Neutral
 Somewhat difficult
 Very difficult
Interviewer, if respondent says somewhat or very difficult, ask: Why was this difficult?
____________________________________________________________________
____________________________________________________________________
[If applicable] You indicated on your survey that you received funding from entities other
than the federal government. Were any of these other amounts you reported actually
subcontract awards on federal projects?
 No
 Yes, please describe.
__________________________________________________________________________
__________________________________________________________________________
Next, I’d like to give you a hypothetical example. Please tell me how you would have
reported your organization’s funding in this situation.
Your organization received $25 million from the federal government. Of this amount, your
organization awarded a university $5 million to be paid out evenly over the next 5 years
starting in FY 2015. How would your organization have reported this?
Interviewer, if the organization only reported internal funds as its source, ask: Let me give
you a hypothetical example. If your organization provided a $5 million dollar grant to
another organization in FY 2015 to be paid out equally over 5 years, starting in FY 2015,
what amount would you report in this question?
____________________________________________________________________
____________________________________________________________________

D-15

[Enumerate the ones skipped]:
We noticed that you did not respond to Questions 11…15, can you please explain why you did
not respond?
Survey Q13
_____________________________________________________________________
_____________________________________________________________________
Survey Q14
_____________________________________________________________________
_____________________________________________________________________
Survey Q15
_____________________________________________________________________
_____________________________________________________________________
26. Before we conclude this interview, I’d like to ask you if there is anything else you would like
to share regarding your experience completing the survey. Is there anything else we haven’t
discussed that you’d like to see improved or changed before the next survey?
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
_____________________________________________________________________
On behalf of the National Science Foundation thank you for your time and assistance.

D-16