Various Governments Area Pretesting Activities

ASPEP 2012 Usability Testing – DRAFT Protocol

2012 Census of Government Survey of Public Employment & Payroll

Usability Testing Protocol

(This protocol is a guide – the questions presented here may not be asked exactly as worded in the protocol or in this order. It also is important to note that not all questions will be asked in every interview.)

Introduction

Introduce observers and their background

Purpose of visit:

Let me start by telling you a little about what we will be doing today. The Annual Survey of Public Employment & Payroll collects information on the number of full-time and part-time employees and payroll amounts for various types of governments

The survey will be mailed out in April. People can respond in one of two ways: by filling out and mailing back a paper form, or by filling out and submitting the survey online. One of our goals is to assess how well online respondents are able to navigate the questionnaire and fill out the online form. We need your feedback on how the online questionnaire is working.

Permission to audio-tape discussion? Have R sign consent form.

Before we get started, I'd like to audio tape this interview so I don't have to rely on my memory later. The recordings will only be heard by those working specifically on this project. If that's all right with you, please sign this consent form.

Background Information

Respondent Background

• Can you tell me a little about your role at this government?

• How long have you worked in your current position?

• Have you completed forms (i.e., surveys, censuses) from the Census Bureau for [government name]?

  • If so, which ones?

  • Have you ever filled any of those forms out online?

  • (If yes) How did you find that process? Easy? Difficult?

  • Were there things that you liked about filling it out online? Things you didn’t like?

Filling out the form

Ok, what I’d like you to do now is go ahead and fill out the survey. (Have respondent access the website address). Go ahead and work through the survey. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do. If at any point something isn’t clear or you have questions about something, please ask and we’ll talk about it.

I’ll periodically stop you and ask you questions abut some of the pages and get your feedback on them.

• Note respondent’s behavior as they go through the questionnaire:

  • Use of previous and next buttons

  • Answering all questions

  • Responding to error messages.

  • Scrolling

Navigating the Questionnaire

Login screen

First, I’d like you to walk me through how you would log into the survey. (R will have an ID and password combination)

• How easy or difficult was it to login?

• How did you decide what information to use?

• How easy or difficult is it to understand the login page?

• What do you think of the overall layout of this page?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

General Instructions page

• What is your initial reaction to this page?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Main Menu

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Where would you go to start filling out the form?

  • Is it clear what you need to do to start?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• Where would you go if you wanted to find out more information about the security of providing this information online?

  • How easy or difficult was finding that information?

• Where would you go if you wanted to find out more information about this survey?

  • How easy or difficult was finding that information?

• Would you look at this information before completing the form or when you needed it?

• (Review data not active until all sections attempted. Watch for reaction. Probe if necessary.)

Address Verification

• What do you think of the overall layout of this page?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• (If respondent doesn’t change address information) Suppose you wanted to update the address information. How would you do that?

Full-Time Employees Standard Weekly Hours

(Give respondent time to look over the page)

• What is your initial reaction to this page?

• What do you think of the overall layout?

• How would you go about answering this question?

• How would you get to the next question? (See if respondent finds and clicks “Save and Continue” button)

Employees, Payroll, and Part-Time Hours

(Give respondent time to look over the page)

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• If you had a question about how to answer this item, where would you go to find the phone number to call? (If they can’t find info on the page) Did you see any place on this page where you could find that information?

• (For E-2, E-4, E-5, E-6,E-7 and E-9) Did you notice that the total is at the end, rather than the beginning, like on the paper form? What is your reaction to that?

• (For E-4, sometimes skip whole sections if small. Watch to see how they react. Probe if necessary.)

• (For E-6, watch how they react to the two totals screens. Probe if necessary.)

• (For E-7, watch how they react to different sectioning on screens. Probe if necessary.)

• (For E-9, see either A or B depending on type of government.) Did you notice that you didn’t see the other categories that are on the paper form? What is your reaction to that?

Remarks

• What is your initial reaction to this page?

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Do you think you would use the remarks section? (If yes) What information do you think you would put there?

• Suppose you wanted to go back to a previous item and double check your answer. How would you do that from this page?

Certification

• What is your initial reaction to this page?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• Please leave the phone number blank, so we can see what happens on the next screen.

Data Review

• What is your initial reaction to this page?

• Is it clear what this screen is telling you? (Note: there will be an indication that there is a problem with one of the sections)

  • Do you find this information helpful?

  • Does it help you figure out what you need to do?

• How would you go about correcting the errors? (Have R try and correct problems)

  • (When R has fixed problems) What do you need to do next? How will you submit the form?

• How would you go about submitting your data?

• What do you expect to happen after you click the “Submit Data” button?

• If you weren’t ready to submit your data at this point, what would you do?

• What other types of review would you be interested in performing prior to submission?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Completion Certificate

• What is your initial reaction to this page?

• If you wanted to print out a copy of your answers to this form for your own records, what would you do?

  • Do you find this feature helpful?

  • Would you save a copy of this information for your records?

• Is there any other information that you would like to have for your records?

Attaching Files

• If you wanted to attach a file and send it to the Census Bureau, how would you do that?

• What sort of description would you enter?

• What else would you like to see here?

Wrap Up

• Now that you have gotten to see the online form, do you think you would complete this form online?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

  • Would you complete this on paper before completing online?

• There were numbers presented in the items that corresponded to the numbers on the paper form. Was this helpful?

• Do you think that you would complete the online form in one sitting?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, hard?

  • Did you have any issues with the navigation on the site?

• What, if anything, did you like the most about the online form? Least?

• Do you have any suggestions for how to make this online form easier for you to use?

Thank you so much for your assistance today.