State Government Research and Development (SGRD)

Survey of State Government Research and Development (SGRD)

Usability Testing (Centurion)

Draft Protocol

June 18, 2015

(This protocol is a guide – the questions presented here will not necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Part A – Introduction

1. Introduce observers and their background.

2. Purpose of visit:

Let me start by telling you what we will be doing today. The Survey of State Government Research and Development (SGRD) is conducted by the U.S. Census Bureau on behalf of the National Science Foundation (NSF). The SGRD is a biennial survey that collects data on state research and development expenditures from state government departments, agencies, commissions, public authorities, and dependent entities with research and development activities. We are in the process of updating our electronic reporting system for this survey and we would like to get your feedback in order to make revisions before it is formally released this Fall.

3. Permission to audio-tape discussion? Have respondent (R) sign consent form.

Before we get started: I'd like to audio tape this interview, so I don't have to rely on my memory later. This session is confidential. Only persons connected with this project will have access to your tape. If that's all right with you, please sign this consent form. It also tells you about the confidentiality of this session.

Part B – Background Information

1. Respondent Background

• What are your roles and responsibilities here at this agency?

Part C – Testing

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)

General Observations:

• Scrolling: Make careful note of how respondent scrolls through long items.

• Instructions: Make careful note of how respondent treats instructions and examples throughout the survey.

• Navigation: Make careful note of how respondent uses the navigation buttons and pages through the survey.

• Error: Make note if respondent triggers any errors. If not, have them trigger one purposely.

As you go through the electronic version of the survey I would like you to complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. There are no right or wrong answers. If you need to get information from records or others within the agency in order to answer a question, just let us know what you would need to do. If you do not have the information available at this time, please feel free to insert any number into the boxes you would normally report in.

Don’t hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.

1. Graphic

At this time I am going to share a document with you that we have created to help complete the questionnaire. We are thinking this graphic will be attached to the email you receive asking you to complete the survey.

• Could you describe to me in your own words what this graphic is telling you?

• How likely or unlikely would you be to use a graphic like this?

  • Could you tell me a bit more about that?

• Do the icons we used seem to fit, or not fit with the text?

• Where would be the best location for a graphic like this?

• What are your overall reactions to this graphic?

• Note if the respondent returns to this graphic while completing the questionnaire.

At this time, I am going to provide you with the Web address and the login information to begin the survey. When you receive the request to participate this information will be provided to you via email.

2. Login Screen

• What are your overall reactions to this screen? What information are you looking for on this screen?

• Is there any other information you would like to see while logging in?

• What are your overall reactions to the process for logging into the system?

3. Contact Information (Question 1)

• Let’s suppose your address is incorrect. Please update the street address to “123 Main Street.”

• What are your overall reactions to this page?

• What information stood out for you on this page?

4. Reporting Period (Question 2)

• Note any difficulties with this item.

• What are your overall reactions to this page?

• Are there any other types of information that would be helpful for you to have on this screen?

5. R&D Activities (Question 3)

• Note respondent’s interaction with the instructions. (Did R read them? Skip them? Skim them?)

• What are your reactions to this page?

• What does R&D mean to you?

  • What types of R&D activities does this agency engage in or fund?

• How would you go about answering this item?

  • Could you walk me through your answers?

**Note: The following three questions (Internal R&D Expenditures, External R&D Expenditures & Total Expenditures) will appear on one Web screen. Respondents will be asked questions about this screen after they have completed the entire page.

6. R&D Expenditures - Internal R&D Expenditures (Question 4))

• Can you explain in your own words what this question is asking you for?

• If you had questions about definitions or the survey items, where would you go?

7. R&D Expenditures - External R&D Expenditures (Question 5))

• Can you explain in your own words what this question is asking you for?

• Would you have any difficulty reporting for this question?

  • Could you tell me a bit more about that?

8. R&D Expenditures - Summary (Question 6)

• Note how respondents interact with this table.

• What are your reactions to this table? Where are the figures coming from?

• What would you do with this information, if anything?

• Let’s say you needed to correct some information. How would you do that?

• Note whether respondent attempts to correct in internal/external questions, or in summary table.

• What did you think of the pre-filled items?

  • Is this something that you would review in any way? (If yes) Could you tell me a bit more about that?

• What do you think about the amount of scrolling on this page?

• Do you have any recommendations for how we could improve the layout of the items on this screen?

• Is there any other information you would like to see on this screen?

9. Total R&D Expenditures by Function (Question 7)

• What are your overall reactions to this screen?

• Let’s say you needed to return to the question where you reported all of your R&D expenditures, how would you do this?

• Note: Respondent reactions to the error message.

• What would you do if you got this message?

  • Do you have any recommendations for how we could improve this message?

• How would you return back to the question you were about to answer?

10. Total R&D Expenditures by Agency (Question 8)

• What are your reactions to this screen?

• What is the amount listed within the question referring to?

• Let’s say you wanted to add other agencies, how would you do that?

  • What if you wanted to add a second one? How would you do that?

11. Total Internal R&D Expenditures by Type of Work (Question 9)

• Note how respondents interact with this question.

• Note how respondents interact with the examples (i.e., skip, skim, read)

• Note if respondents click on links to examples.

• What are your reactions to the items that are grayed out?

• Could you tell me what this question asking in your own words?

• Would you be reporting for 2014, 2015, or both?

• What information, if any, stands out to you?

• Could you tell me about the examples that are listed below the questions?

  • What do the blue links indicate to you?

12. Total External R&D Expenditures - Type of Entity (Question 10)

• What are your reactions to this screen?

• Could you tell me what the dollar amounts within the question is referring to?

  • Could you tell me a bit more about that?

13. Total R&D Expenditures - Construction, Land, and Facilities (Question 11)

• What are your reactions to this screen?

• How would you answer this question if it did not apply to your agency? (e.g., enter 0, leave it blank)

14. Review Your Responses

• Did you expect to come to this screen when you completed the form or was this unexpected?

  • Where would you have expected to go?

  • (If no) What would you like to see?

• Take a look at the information here. Did you miss any items? How can you tell?

  • Was this clear?

  • How do you correct an item?

• Please try to return to item 10 (R&D by activity) or 13 (external R&D by type of entity), make a correction, then get back to this review screen. What are your thoughts on this process?

• What other types of review, if any, would you be interested in performing?

• How will you know if the form is completed?

• After you’re done reviewing this screen, where would you go next?

• Let’s say you wanted to set this aside for a couple of days before submitting your data. How do you go about doing that?

15. Submission Confirmation

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• What would you do after arriving to this page?

• Please try saving a copy of this information.

• Is there anything else you would like to see on this screen?

• Let’s say you wanted to print a copy of the form, how would you do that?

  • Is this a feature you would use?

  • (If yes) Could you tell me a bit more about how you would use it?

Part D – Wrap Up

• Now that you have gotten a good feel for the online form, what is your overall impression of it?

• About how long would it take to complete this form online?

• How do you collect the information we are requesting on the form?

  • Would you contact others at your agency to complete these questions?

    • (If yes) Who would you contact?

    • (If yes) How would you share this information with them? Would you create a spreadsheet, worksheet, or something else to share with them?

      • About how long would this take?

      • What information would they send back?

      • What format would the data come back in?

      • What other steps would you take to incorporate the data into the form, if any?

• Would you use a copy of the form to help you collect your answers?

  • (If yes) Would you prefer to use an electronic copy, a printed copy, or something else? (If something else) Could you tell me a bit more about that?

    • How would you get a copy of the form to use?

• How did you feel about the amount of information that was provided on each page? Was it too much information, too little information, or about right?

• How did you feel about the amount of scrolling? Too much, too little, about right?

• Did you have any issues with the navigation on the site?

• What are some of the things that you liked the most about the online instrument?

• What are some of the things that you liked the least about the online instrument?

• Do you have any suggestions for how to make this Web questionnaire easier for you to use?

Thank you for your time and input!

6

Survey of State Government Research and Development (SGRD)

Usability Testing Draft Protocol (Centurion)

June 2015