Various Economic Area Pretesting Activities

2011 Government Units Survey

Usability Testing Protocol

(This protocol is a guide – the questions presented here may not be asked exactly as worded in the protocol or in this order. It also is important to note that not all questions will be asked in every interview.)

Background Information

Respondent Background

• Can you tell me a little about your role at this government?

• How long have you worked in your current position?

• Have you completed forms (i.e., surveys, censuses) from the Census Bureau for [government name]?

  • If so, which ones?

  • Have you ever filled any of those forms out online?

  • (If yes) How did you find that process? Easy? Difficult?

  • Were their things that you liked about filling it out online? Things you didn’t like?

Filling out the form

Ok, what I’d like you to do now is go ahead and fill out the survey. (Have respondent access the website address). Go ahead and work through the survey. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do. If at any point something isn’t clear or you have questions about something, please ask and we’ll talk about it.

I’ll periodically stop you and ask you questions abut some of the pages and get your feedback on them.

• Note respondent’s behavior as they go through the questionnaire:

  • Use of previous and next buttons

  • Answering all questions

  • Responding to error messages.

  • Scrolling

Navigating the Questionnaire

Logging in

First, I’d like you to walk me through how you would log into the survey. (R will have an ID and password combination)

• How easy or difficult was it to login?

• How did you decide what information to use?

• How easy or difficult is it to understand the login page.

• What do you think of the overall layout of this page?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Dashboard/Main Menu

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Where would you go to start filling out the form?

  • Is it clear what you need to do to start?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• Where would you go if you wanted to find out more information about the security of providing this information online?

  • How easy or difficult was finding that information?

• Where would you go if you wanted to find out more information about this survey?

  • How easy o difficult was finding that information?

• Would you look at this information before completing the form or when you needed it?

Changing password:

• Pretend that you have decided to change your password. How would you go about doing that? (Note how the respondent goes through the process)

• How easy or difficult was finding that information?

• How easy or difficult were the steps you needed to take?

• Do you think this feature is a desirable feature to have?

• Given what steps you need to take, would you be more likely or less likely to change your password?

Item 1: Government ceased existence

• What is your initial reaction to this page?

• What do you think of the overall layout?

• How would you go about answering this question?

• How would you get to the next question? (See if respondent finds and clicks “Next” button)

Items 2-7: Background information

(Give respondent time to look over the page)

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• (If respondent doesn’t change address information) Suppose you wanted to update the address information. How would you do that?

• If you had a question about how to answer Item 2, where would you go to find the phone number to call? (If they can’t find info on the page) Did you see any place on this page where you could find that information?

Items 8-11: Debt

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

(Note how respondents answer Item 8 – there are only two response options, but some governments require voter approval for certain kinds of debt, prompting a “sometimes” response)

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• Suppose you wanted to go back to Item 7 and double check your answer. How would you do that from this page?

Item 12: Impose Fees

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• What do you think would happen if you didn’t specify anything in the “other” category in part Q? (Have respondent leave it blank after marking “yes” and hit “Next”)

  • What did you expect to see?

  • Is this error message helpful?

  • Does it help you figure out what you need to do to fix the problem?

  • What do you think will happen if you don’t fix the error?

(Have R try to move on to next page – leave error)

• Is that what you expected to happen?

Item 13: Levy Taxes

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Items 14-15 Retirement/Healthcare

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Items 16-36: Government Activities

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• What do you think about the amount of information on this screen?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• What do you think would happen if you accidentally skipped a question on this page? (Have respondent leave Q20 blank after marking “yes” and hit “Next”)

  • What did you expect to see?

  • Is this error message helpful?

  • Does it help you figure out what you need to do to fix the problem? (Have R move on to next page)

Items 37-44: Government Activities Con’t

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• What do you think about the amount of information on this screen?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Items 45-47: Judicial/Legal Services

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• What do you think about the amount of information on this screen?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Items 48-51: Government Finance

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• What do you think about the amount of information on this screen?

• Is it clear what information you are supposed to provide?

• Did you notice the zeros after the amount fields? Was this helpful?

• (If R doesn’t use commas) I noticed you didn’t use commas. Did you think you could enter commas in the answer blank?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

(Have respondent enter an invalid dollar amount that has incorrect comma usage: 1,00,000)

• Is this error message helpful?

• Does it help you to know what to do to fix the problem?

(Have R move on without fixing the problem.)

Remarks/Contact Information

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Do you think you would use the remarks section? (If yes) What information do you think you would put there?

Review your responses

• What is your initial reaction to this page?

• Is it clear what this screen is telling you? (Note: there will be an indication that there is a problem with one of the sections)

  • Do you find this information helpful?

  • Does it help you figure out what you need to do?

• How would you go about correcting the errors? (Have R try and correct problems)

  • (When R has fixed problems) What do you need to do next? How will you submit the form?

• If you weren’t ready to submit your data at this point, what would you do?

• What other types of review would you be interested in performing prior to submission?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Completion Certificate

• What is your initial reaction to this page?

• If you wanted to print out a copy of your answers to this form for your own records, what would you do?

  • Do you find this feature helpful?

  • Would you save a copy of this information for your records?

• Is there any other information that you would need to have for your records?

Wrap Up

• Now that you have gotten to see the online form, do you think you would complete this form online?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

  • Would you complete this on paper before completing online?

• There were numbers presented in the items that corresponded to the numbers on the paper form. Was this helpful?

• Do you think that you would complete the online form in one sitting?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?

  • Did you have any issues with the navigation on the site?

• What, if anything, did you like the most about the online form? Least?

• Do you have any suggestions for how to make this online form easier for you to use?