2017 SU Instrument Usability Testing –
Protocol
Research Questions
eCorrespondence:
Is the navigational path of the site logical to respondents?
Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons?
Are respondents able to discern the functions of individual screens?
What additional features or information do respondents want in the eCorrespondence prototype?
Single-unit (SU) Instrument:
Does the flow of the instrument support respondents’ informational and decision-making needs with regard to reporting options and instrument functionality? If not, what other information do they need, and what are the optimal locations and formats for presenting such information?
Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons? If not, what terms do respondents suggest?
Are respondents able to discern the functions of individual screens? If not, what are the sources of confusion, and how do they suggest making them clearer?
What features do respondents typically look for when completing the survey, and does our instrument meet their expectations?
What additional features do respondents want in the instrument?
Reporting Guide Prototype:
Would respondents use the worksheets, and if so, how? E.g., reviewing question requirements during course of gathering data, writing down responses, writing down procedural notes, sending to data providers, receiving data from providers, retaining for records, etc.
How do respondents react to the format and content of our downloadable questions/instructions document mockup?
Do respondents have any suggestions for changing content or visual design?
Are there any other formats we should consider for presenting sufficient reporting instructions? E.g., writeable PDF.
Materials
eCorrespondence Flyer with dummy authentication code
Testing URL for 2016 eCorrespondence which links to SU Reporting Instrument
Printed copy of worksheet prototype
Consent forms
Recorder, batteries
Procedures
Have respondent create an eCorrespondence account and link a survey ID
Have respondents access and walk through the test instrument
Review paper mockups of worksheet
Introduction
Reason for visit – To evaluate the prototype collection system and reporting tools for the Report of Organization (and Annual Survey of Manufactures, if R has manufacturing locations).
Introduction of Census employees
Introduction of company employee(s)
Audio recording
Before we get started: I'd like to audio record this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. If that's all right with you, please sign this consent form. It also tells you about the confidentiality of this session.
Respondent Information
How long have you been with the company?
What is your title/role?
What are your major responsibilities?
Are you responsible for other government surveys? If so, which ones?
eCorrespondence
(If respondent did not read the letter ask them to read the portion explaining how to set up a MyCensus account)
Now let’s take a look at a new system we are developing to act as a hub for Census Bureau surveys. Please keep in mind that it is still a work in progress, so we may come across some things that don’t work as intended.
Please register and create an account. You will have to confirm your email before being allowed to login.
Go through registration process. Note any difficulties the respondent has, complaints about security questions, and whether the respondent would use personal or business information for the security questions.
General probes:
Is this information you would be able to provide?
Are the instructions clear or not clear?
Was in clear from the flyer that you would need to provide this information?
Based on the flyer, is there any other information you would need to successfully create an account?
From the information provided to you so far, is it clear whether this is a personal or business account?
(Have respondent login to eCorrespondence) This is the login screen you would come to after creating an account. Please login, and keep in mind this is a test account for a fake company. I will ask you to click through the pages, and ask you questions about them as we go along.
Survey Dashboard:
What do you think is the purpose of this page?
How would you proceed once you got to this page?
Is there anything that stands out on this page?
The flyer describes linking a survey ID to your account. How would expect this process to work?
Please link this survey to the account according to the directions in the flyer.
Did that work the way you expected it to?
Were the instructions in the flyer useful or not useful?
Were the instructions on the page helpful or not helpful?
The flyer mentions the authentication code is “one-time use only,” what does that mean to you?
Is there any other information you would like to see on this page?
This website allows you to delegate surveys to other people, please locate that function on the website.
How would you expect the delegate feature to function?
How would you use it?
Are the instructions useful or not useful?
Is there any other information you would like to see on this page?
Login/Landing Page
Now let’s take a look at the new prototype collection system. Please begin the 2016 Report of Organization. (Ask the respondent to click the “Start” button if they have trouble beginning the survey.)
What do you notice about this screen?
Is the information on this screen useful or not useful?
What other information, if any, would you like to see on this page?
Introductory Screens (Overview/Reporting Options)
General probes
In your own words, what would you say is the purpose of this page?
What do you notice about this screen?
How would you use this screen?
In your opinion, is the information on this page helpful or not helpful?
Is there other information you think should be included?
Reporting Steps
How well does this page describe the steps you took to complete the [Report of Organization and/or Annual Survey of Manufactures]?
Do you have any suggestions for additional information for this page?
Reporting Options
Do you have any suggestions for additional information for this page?
Is it clear or unclear that there are multiple ways for companies to report?
When do you think one mode may be preferable than another?
Do you have any suggestions for additional information for these introductory screens?
Dashboard
What do you notice about this screen?
How would you use this screen?
What would you do when you got to this screen?
Do you recall reading any of the text above the table?
(As needed) What does [each line] mean to you?
In your opinion, what is the purpose of the table?
How would you use it? (Use the following probes as needed):
Please tell me in your own words what each column header means to you.
Do you recall seeing the black triangles in the column headers? In your opinion, what are they for, if anything?
Do you recall seeing the white boxes in the column headers? In your opinion, what are they for, if anything?
Which column headers do you need in your reporting process?
Are there any columns you do not need?
Are there any items missing that you would prefer to see on this table?
Do you recall seeing the white buttons along the bottom?
What would you expect to happen if you clicked on them?
Do you recall seeing the PDF icons on the left?
What would you expect to happen if you clicked on one of them?
Do you recall seeing the blue and green boxes on the left?
What would you expect to happen if you click on one of them?
Please click on one now.
Form view page
What do you notice about this screen?
How would you use this screen?
Did you notice the answer fields? Is the expected format of the answer clear or not clear?
Is there anything else you would want to see on this kind of screen?
Did you notice there aren’t any question numbers?
Would you need question numbers at any point in your reporting process?
Additional Pages
Add Locations
If you needed to add a new location that your company had opened, how would you go about it?
If respondent doesn’t click, have them click on ‘Add Location’
How would you use this page?
What do these labels (form numbers) mean to you?
Please try adding some locations.
Are the instructions clear or unclear?
Where would you expect to find these locations, now that they have been added?
Have respondent return to Dashboard and locate new locations.
Delete Locations
Now let’s try to delete a location.
When would you expect to use this function?
How would you expect this function to work?
Please try to delete a location.
Are the instructions clear or unclear?
Did that work as you expected it to?
Export/Import from Spreadsheet
Is it clear why these buttons are greyed out?
When do you think these functions are made available?
What, if any, additional information would you need about these buttons?
If needed: When/Where would you expect to see this information?
What would you do if you saw these functions unavailable?
Would it be useful for you to see the questions in a spreadsheet?
Do you think you would use these functions if they were available to you?
Notifications
Do you recall seeing an orange mail button?
How would you expect to use this button?
Please try to use this function.
Did that work as you expected it to?
What information would expect to receive?
What, if any, additional information would you need about this feature?
Summary Report (additional probes):
If tab is available:
In your opinion, what is the purpose of this page?
Do you find this kind of information helpful or not helpful?
Do you think you would you use this information? If so, how?
Are there any additional types of information that you would like to see?
Are there any other features we should consider?
If tab is grey:
Is it clear why these buttons are greyed out?
When do you think these functions are made available?
Do you or do you not think any further additional action is required?
Review
This is a placeholder for Step 2 - Review probes.
Submit
This is a placeholder for Step 3 - Submit probes.
Worksheet Prototype
Next, I’d like to show you the question preview prototype. For the rest of the interview, I will refer to it as a ‘worksheet.’ Please take a moment to look through it.
What are your initial impressions of the worksheet?
Do you think you would use this worksheet?
If yes:
How would you use it? (If necessary: reviewing question requirements during course of gathering data, writing down responses, writing down procedural notes, sending to data providers, receiving data from providers, retaining for records, etc.)
If no: Why do you think you would not use it?
Now, I’d like to ask you some questions about specific sections of the worksheet.
Cover page:
What do you notice about the front page?
Do you find the information useful, or not useful?
Is there any information you would get rid of?
Is there any other information that you would like to see on this page?
Do you recall reading the ‘Do not mail’ message? What did you think about it?
Overview section:
Is there anything you notice about this section?
Do you find the information to be useful or not useful?
What information, if any, would you get rid of?
Is there any other information you’d like to see in this section?
Item Specific Instructions:
What do you think about the way the questions and instructions are laid out?
Taking the entire worksheet into consideration:
Do you have any suggestions for making the instructions more useful, or easier to use?
If you were designing such a document, what would it look like? What kinds of information would it include?
I’ve been referring to this document as a worksheet, what would you call a document such as this?
Wrap-up
Are there any other comments or suggestions you would like to make about anything we have gone over today?
Thank the respondent for their time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Michael Brennan |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |