NCSES - State Government R&D Survey Cognitive Interview Protocol

State Government R&D Survey

Cognitive Interview Protocol

Draft

January 10, 2020

Research Objectives

• Is the requested information on agency personnel conducting R&D at state agencies available?

• How do respondents interpret the questions as presented?

• Approximately how much additional burden will be required by asking the new questions?

Expected Length of Interview

60 minutes

Materials Needed

• 2-3 copies of the 2019 (completed) questionnaire, with new questions added between questions 9 and 10

• Consent forms

• Digital recorder

• Business cards

Introduction

• People in the room, and the roles they play in completing the SGRD Survey

• Purpose: new questions have been proposed for the SGRD. Talking to SGRD respondents in a number of states, in agencies with different missions.

• Some questions may seem odd, redundant, because talking to wide variety of people.

• Testing the questionnaire, not the respondent. No right or wrong answers.

• Meeting structure:

  • General and specific questions about the questions.

• Permission to record? Get signed consent form.

General Probes

• Tell me more about that.

• I want to make sure I understand. I think you said [paraphrase]. Is that right?

• What terms do you use to describe this?

• Can you tell me how you would arrive at your answer? What would you include? What would you exclude? What records would you use?

• Do your records contain this information? What do these records look like? What are they called?

• Would you use book figures for would you have to estimate to report for this item?

• Would you have to manipulate the book figure to get an answer? How?

• Would you need help from other coworkers to get this data?

Background

• Can you give me a high-level overview of this agency’s mission?

• Now give me some information about the research and development activities that this agency performs or funds. What sort of research and development projects is this agency involved with?

R&D Activities

• Would you please take a moment to read through the material under the headings “What is R&D” and “What is NOT R&D”?

  • What is your reaction to these lists?

  • Does this match your own definition of R&D, or is it different in some way?

  • Does your agency do this kind of work, or not?

  • Are you involved in this work, or not?

• Let’s take a look at the table under “Examples of R&D” – it lists the five criteria for R&D, and gives examples of when a project may or may not be R&D. Please take a moment to read through it.

  • Are the examples provided useful to you, or not?

  • Can you give me an example of a R&D project that your agency is involved with that fits the criteria for novel? Creative? Uncertain outcome? Systematic? Useful to others?

• Now let’s look at “Examples of classifying R&D from four activities.” Please take a moment to read through it.

  • Is this information helpful to you, or not?

  • How would you summarize the information here? In your own words, what makes something R&D vs. not R&D?

Question 3 (R&D Activities)

Let’s turn ahead a few pages to Question 3.

• You recently answered these questions in this way [show prior response]. I’ll give you a minute to read through this. Please pay special attention to Line(s) [a, b, or f, whichever had a Yes response].

• I know it’s been a while since you submitted this data, but is there any chance you could tell me what you were thinking about as you answered this question?

Question 4 (Expenditures for R&D Performed Internally)

• This is how you recently answered these questions [show prior response].

• Can you tell me how you went about answering this question?

  • What records did you consult to answer this question?

  • Did you get help from others in your agency to answer this question?

• Can you give me some examples of these types of expenditures?

• What did the term “internal R&D performance” mean to you? Do you use this term yourself? If not, what term or terms do you use to describe this type of activity?

• What does “state funds used for internal R&D” mean to you? Can you give me an example of this?

Question 9 (Expenditures for R&D Performed Internally by Type of R&D)

We’re going to skip over several questions now. Let’s move to question 9.

• This is how you answered the question recently [show prior response].

• Can you tell me how you went about answering this question?

  • Did you use records to come up with an answer, or did you estimate?

    • If estimate, can you tell me how you went about estimating?

    • If records, what records did you consult to answer this question?

  • Did you get help from others in your agency to answer this question?

• How easy or difficult was it to provide a response for this question?

• Do you remember whether or not you referred to the examples of basic research, applied research, and experimental development?

• Does your agency classify its R&D work in these three buckets (basic research, applied research, experimental development) or not?

  • If not, does your agency classify its R&D projects some other way? Can you tell us about it?

• Were there any projects or expenditures that you were unsure how to classify? What made it hard? How did you decide what to do with it/them?

New Questions about Employee Counts and FTEs

Now I’m going to give you a draft of two new questions we are thinking about adding to the survey. We’d like to get your feedback on it. I’ll give you a moment to read through it.

• What are your overall impressions of these questions?

• What sort of employees would your agency consider to be “researchers”?

• How about “R&D technicians and equivalent staff”?

• How about “R&D support staff”?

• Does your agency classify its own employees in the way the questions ask? In other words, can you identify researchers vs. R&D technicians and equivalent staff vs. R&D support staff, or not?

  • How would you go about determining which employees fit in each category?

    • Would employee occupation classifications map to any of these categories? Would occupation codes exclude some employees from being included in these categories?

  • Would your agency’s records have this information? How easy or difficult is it to get this information from the records?

• How easy or difficult would it be to obtain the numbers of employees in these three categories (rather than a headcount for the whole agency) – very difficult, somewhat difficult, neither easy nor difficult, somewhat easy, or very easy?

• Does your agency have any part-time employees that would fit into any of these three categories (researchers, R&D technicians/equivalent staff, R&D support staff)?

• Do you happen to know if your agency calculates the number of full-time equivalent employees it has? Who has access to those records?

• For the FTE question, would you estimate the figures, or would you look at records?

  • (If records) Where are those records located? Do you have access to them yourself, or would you need to work with someone else to get this information? Who would that other person(s) be?

  • (If estimate) How would you go about estimating the number of FTEs in these three types of jobs? What process would you use? Would this be something you estimate in your head, or would you use a tool of some sort? What tool(s)?

  • How easy or difficult would it be to obtain this information only for those employees in these three categories (rather than the number of FTEs for the whole agency) – very difficult, somewhat difficult, neither easy nor difficult, somewhat easy, or very easy?

• Would you be able to answer these two questions yourself, or would you have to consult with others?

  • If others, who would you consult with?

  • If others, how would you go about getting information on this topic from that person/those people? (Phone call? E-mail? Share the questionnaire or not?)

• What would happen if you were able to come up with a figure for “total R&D employees” or “Total R&D FTEs,” but couldn’t break it down into these three categories? How would you answer the questions?

• Is one question easier or harder than the other? Which one? How come?

• If difficult, how could we make these questions easier for you to answer?

• Approximately how long would it take you to answer these two questions, including the time it takes to gather information, talk to anybody you’d need to talk to, etc.?

• We were thinking about putting these new questions between questions 9 and 10.

  • Do you think this is a good place or a bad place to put them?

  • Is there another location that makes sense to you, or that you think would be better?

Wrap Up

We’ve spent our time today focused on only part of the questionnaire.

• Were there other parts of questionnaire you wanted to discuss?

• Did you have any feedback to share on your experiences with completing the questionnaire?

• Was there anything else you wanted to tell us about the State Government R&D Survey?

Thank you for your time and feedback.

State Government R&D Cognitive Interview Protocol

Page 5 of 5