Various Government Pretesting Activities

Quarterly Summary of State and Local Government Tax Revenue (Q-Tax)

(Centurion) Usability Testing Draft Protocol

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Materials needed:

• Web URL, user ids and passwords

• Emails (with user id and password for WebEx)

• Consent form returned via fax/email

• notebook, pen

Part A– Background Information

1. Respondent Background

• What is your role here at this government entity?

• Have you completed any other government surveys? If so, which ones?

• What benefits would there be to reporting electronically to a survey like Q-Tax? Would there be any drawbacks?

2. Electronic Reporting Background (if reported to other government surveys)

• Did you report electronically to any of those surveys?

• What made you choose to report electronically?

• What was your overall experience? Did anything stand out as being very helpful? Was there anything that you did not like?

Now I’d like to show you an electronic draft of the Q-Tax form. Here is the web-site address for accessing the survey (show the respondent the URL, survey key (if applicable), test User ID and test password. Before R has arrived on the webpage turn control over to R.) Next, I am going to have you enter the web address into your browser. Then I will let you control the mouse so I can view what you are seeing and where you are clicking.

I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do.

Don’t hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.

Part B – Testing

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)

General Observations:

• Scrolling: Make careful note of how respondent scrolls through long items.

• Instructions: Make careful note of how respondent treats the examples at the top of some of the items.

• Navigation: Make careful note of how respondent uses the navigation buttons.

• Error: Make note if respondent triggers any errors, if not have them trigger one purposely.

1. Login Screen

• Do you think that the information given on the Login Screen was sufficient?

• What are your overall reactions to the process for logging into the system?

• Is there any other information you’d like to see on any of the screens you saw while logging in?

• Where would you find the due date?

2. General Instructions

• What are your overall reactions to this page?

• What information stands out to you?

• What do you find most important?

• What do you find least important?

• What is the icon at the bottom of the screen? What do you expect will happen?

• Where would you go to begin the form? Was this clear? Why or why not?

3. Quarter Specific Menu

• Can you describe to me what you see on this page?

• Where do you need to go in order to start completing your form?

• Was this clear? If not, what could be done to make the start of the form clearer?

• If you had questions about definitions or the survey items where would you go?

• Is there anything else that you would need at this point in order to start completing the form? Anything else that should be included on this screen?

• Did you notice the toolbar on this screen? What do you think each of the options (Main Menu, FAQs, About Survey, Instructions, Print/Review Form, Attach Data, Logout) will do?

• Did you notice the dark purple bar at the bottom of the screen? What do each of these headings mean to you?

• Are there any other types of information that would be helpful for you to have? Would you look at this information before completing the form or when you needed it?

4. Main Menu

• Can you describe to me what you see on this page?

• Where do you need to go in order to start completing your form?

• Was this clear? If not, what could be done to make the start of the form clearer?

• What do you expect the two icons at the bottom of the screen do?

  • Was this clear? Why or why not?

  • Would you use either of these? When? Why or why not?

5. Address Verification (Q1)

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

6. Taxes (Q2)

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that does not belong?

• Did you notice this message at the top of the page? What did this mean to you? What is this message referring to?

• Did you notice the bulleted items in part A? What are they telling you?

  • Is there anything we can do to help them become more noticeable?

• Did you notice the underlined word in part B? What does it mean?

  • Is there anything we can do to help this stand out to you?

• Did you notice the underlined word in part C? What does it mean?

  • Is there anything we can do to help this stand out to you?

7. Trigger an Edit

• (If the R hasn’t triggered an edit at this point, have them do so on purpose)

• Is it clear to you based on the messages you are receiving on the screen that there is an issue? Can you explain to me what the problem(s) are?

• Is there a better way that we could notify you about the problem(s) on this screen?

• What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?

• What other information should we provide to you about your problems and how to correct them?

8. Remarks (Q3)

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Did you notice the bulleted items? What are they referring to?

• Would you have enough space to provide an answer?

• How easy or difficult is it to use this response box?

9. Navigation Questions

• If you needed to return to a prior section, how would you do this?

• Did you have any problems navigating back to a prior section/screen?

• Do you have any recommendation for improving the navigation within the site?

10. Contact Information (Q4)

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there any information you would like to provide here that we do not provide a space for?

• Which items would you complete? Which items are required? Was this noticeable? Is there anything we can do to help make this more obvious?

11. Data Review

• What are your first thoughts when seeing this screen?

• Can you explain to me what is expected of you? What is this screen telling you?

• (If there are edits) How will you go about fixing this/these problem(s)?

  • Where would you go?

  • When you have finished fixing the problems, what do you need to do next?

• How will you submit the form?

• What do you think of the title of this screen? Is it a logical title?

• What other types of review would you be interested in performing prior to submission?

• If you were not ready to submit your data at this point, what would you do?

• If you logged out of the survey, do you think your data will be saved when you return?

  • How do you feel about this?

12. Submission Confirmation

• Is there any other information that you would need to have for your completion?

• Would you save a copy of this information for your records? How would you do that? Which icon would you select?

13. Attach Data

• What is your initial reaction to this page?

• Where are your eyes drawn?

• What information stands out to you?

• What is the information at the top of the page describing to you?

• What are your reactions to the request for a contact person?

  • Is this the same person listed in the previous section?

  • How do you feel about that?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Could you explain to me in your own words how this screen works?

• Is this something that you would be using? Why or why not?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• Is there anything we could do to help make this screen more helpful?

Part C – Wrap Up

• Now that you have gone through this form electronically, would you prefer to do this survey on paper or electronically?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

• Would you complete this on paper before completing online?

• Do you think that you would complete the online form in one sitting?

• If you needed to contact someone with a question about the survey, what would you do?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?

• Did you have any issues with the navigation on the site?

• What are some of the things that you liked the most about the online form?

• What are some of the things that you liked the least about the online form?

• Are there any other options that we should provide in the online form?

• Do you have any suggestions for how to make this online form easier for you to use?

Thank participant for their time and input!

Q-Tax Usability Testing via WebEx Draft Protocol

July 2013

5