2017 MU Instrument Usability Testing – Round 1
Protocol
Research Questions
eCorrespondence Prototype:
Can respondents access the test site?
Is the navigational path through the screens of the site logical to respondents?
Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons?
Are respondents able to discern the functions of individual screens?
What additional features or information do respondents want in the MyCensus prototype?
Multi-unit (MU) Instrument Screen Mockups:
Does the flow of the instrument screens support respondents’ informational and decision-making needs with regard to reporting options and instrument functionality? If not, what other information do they need, and what are the optimal locations and formats for presenting such information?
Do button labels and descriptions make sense to respondents and inform respondents of the correct functions of the buttons? If not, what terms do respondents suggest?
Are respondents able to discern the functions of individual screens? If not, what are the sources of confusion, and how they suggest making them clearer?
What features do respondents typically look for when completing the survey, and does our instrument meet their expectations?
What additional features do respondents want in the instrument?
Spreadsheet Prototype:
Do respondents understand the intended use of the spreadsheet template?
Is the visual design of the spreadsheet template effective in promoting readability of questions and instructions?
Would respondents use the spreadsheet, and if so, how? E.g., VLOOKUP, copy/paste, send to other respondents to gather data, etc.
What additional features or information do respondents want in the spreadsheet template?
Reporting Guide Prototype:
Would respondents use the worksheets, and if so, how? E.g., reviewing question requirements during course of gathering data, writing down responses, writing down procedural notes, sending to data providers, receiving data from providers, retaining for records, etc.
How do respondents react to the format and content of our downloadable questions/instructions document mockup?
Do respondents have any suggestions for changing content or visual design?
Are there any other formats we should consider for presenting sufficient reporting instructions? E.g., writeable PDF.
Pressure Sealed Envelopes
How do respondents react to receiving a pressure-sealed envelope?
Are respondents familiar with pressure-sealed envelopes? Do they receive this type of mail from other government sources, and if so, which ones?
Do they perceive this kind of mailing to be a legitimate request from the Census Bureau?
Do they have any problems opening such an envelope?
Do they have any problems finding the information they need – information about the survey, legal information, due date, log-in information, etc.?
Do they anticipate any problems with the way such a mailing would be handled and processed in intra-company mail systems?
Materials
Testing URL for 2016 MU electronic reporting instrument, IDs/PWs
Printed screen mockups of 2016 MU electronic reporting instrument
Printed copy of spreadsheet prototype
Printed copy of worksheet prototype
Testing URL for eCorrespondence, IDs/PWs
Printed copy of eCorrespondence screenshots (as back-up)
Consent forms
Procedures
Have respondents access the eCorrespondence site using a browser of their choice and complete process of setting up an account.
Have respondents access the COS/ASM test instrument site. Review additional paper mockup screens as needed to provide context and get additional feedback.
Review paper mockups of spreadsheet and worksheet.
Review pressure-sealed envelope sample.
Introduction
Reason for visit – To evaluate the prototype collection system and reporting tools for the Report of Organization (and Annual Survey of Manufactures, if R has manufacturing locations).
Introduction of Census employees
Introduction of company employee(s)
Audio recording
Before we get started: I'd like to audio record this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. If that's all right with you, please sign this consent form. It also tells you about the confidentiality of this session.
Respondent Information
How long have you been with the company?
What is your title/role?
What are your major responsibilities?
Are you responsible for other government surveys? If so, which ones?
eCorrespondence Prototype
Now let’s take a look at a new system we are developing to act as a hub for Census Bureau surveys. Please keep in mind that it is still a work in progress, so we may come across some things that don’t work as intended.
(Give URL) Please open a browser of your choice, go to this page. I will ask you to click through the pages, and ask you questions about them as we go along.
General probes for each page:
What do you think is the purpose of this page?
How would you use it?
Are the instructions useful or not useful?
Is there any other information you would like to see on this page?
How do you think would you link a survey to your account?
How would you expect the delegate feature to function?
Overall, how do you think you would use this website?
Are there any other features you would like to see in a website such as this?
Landing Page
Now let’s take a look at the new prototype collection system.
Please open a browser of your choice, go to this page. I will ask you to click through the pages, and ask you questions about them as we go along.
(Note which browser and operating system are being used)
What do you notice about this screen?
Is the information on this screen useful or not useful?
What other information, if any, would you like to see on this page?
Introductory Screens (Overview/Reporting Options/Reporting Unit)
General probes
In your own words, what would you say is the purpose of this page?
What do you notice about this screen?
How would you use this screen?
In your opinion, is the information on this page helpful or not helpful?
Is there other information you think should be included?
Reporting Steps
How well does this page describe the steps you took to complete the [Report of Organization and/or Annual Survey of Manufactures]?
Do you have any suggestions for additional information for this page?
Reporting Options
Do you have any suggestions for additional information for this page?
(If R used Surveyor previously) Were you aware of these options in the current system?
Reporting Unit
(Ask before general probes) What does each bullet mean to you?
In your opinion, is this information clear or not clear?
Do you have any suggestions for additional information for this page?
Dashboard
What do you notice about this screen?
How would you use this screen?
What would you do when you got to this screen?
Do recall reading any of the text above the table?
(As needed) What does [each line] mean to you?
In your opinion, what is the purpose of the table?
How would you use it? (Use the following probes as needed):
Please tell me in your own words what each column header means to you.
Do you recall seeing the black triangles in the column headers? In your opinion, what are they for, if anything?
Do you recall seeing the white boxes in the column headers? In your opinion, what are they for, if anything?
Do you recall seeing the green buttons along the bottom?
What would you expect to happen if you clicked on them?
Do you recall seeing the red-and-white icons on the left?
What would you expect to happen if you clicked on one of them?
Do you recall seeing the blue text “View All” in the fourth column?
How would you expect to use that, if at all?
Do you recall seeing the blue and green boxes on the left?
What would you expect to happen if you click on one of them?
Please click on one now.
Form view page
What do you notice about this screen?
How would you use this screen?
Did you notice the answer field? Is the expected format of the answer clear or not clear?
Is there anything else you would want to see on this kind of screen?
Additional Pages
Export wizard
All locations vs. Full survey spreadsheet:
In your opinion, what is the difference between these two options?
How would you expect to use the ‘All Locations’ spreadsheet?
How would you expect to use the ‘Full Survey’ spreadsheet?
Spreadsheet type:
In your opinion, what is the difference between these two options?
Are you familiar with standard excel spreadsheets and comma separated values spreadsheets?
Which are you more likely to choose?
Select spreadsheet/PDF page:
What do these options mean to you?
Which ones if any would you select, and what files you would expect to receive if you chose them?
Summary Report (additional probes):
In your opinion, what is the purpose of this page?
Do you find this kind of information helpful or not helpful?
Do you think you would you use this information? If so, how?
Are there any additional types of information that you would like to see?
Are there any other features we should consider?
Return to Dashboard
(If R used Surveyor-Workbook View or said they would expect to write in this table) The current reporting system has a workbook that allows you to enter or change information directly, but this new version does not allow that. If you were not able to enter or change data directly in this table, would that affect your ability to report? If so, how?
Spreadsheet Prototype
Now I’d like to show you the Excel spreadsheet template prototype.
Do you think you would use this spreadsheet?
If yes: What would be your reasons for using it?
How would you use it?
Optional probes: VLOOKUP, copy/paste, send to other respondents to gather data, etc.
If no: What would be some reasons why you wouldn’t use it?
In your opinion, what is the purpose of the text in the column headers?
Is there any other information you would like to see in this spreadsheet or other features we should consider?
Worksheet Prototype
Next, I’d like to show you the question preview prototype. For the rest of the interview, I will refer to it as a ‘worksheet.’ Please take a moment to look through it.
What are your initial impressions of the worksheet?
Do you think you would use this worksheet?
Now, I’d like to ask you some questions about specific sections of the worksheet.
Cover page:
What do you notice about the front page?
Do you find the information useful, or not useful?
Is there any information you would get rid of?
Is there any other information that you would like to see on this page?
Do you recall reading the ‘Do not mail’ message? What did you think about it?
Overview section:
Is there anything you notice about this section?
Do you find the information to be useful or not useful?
What information, if any, would you get rid of?
Is there any other information you’d like to see in this section?
Item Specific Instructions:
What do you think about the way the questions and instructions are laid out?
Taking the entire worksheet into consideration:
Do you have any suggestions for making the instructions more useful, or easier to use?
If you were designing such a document, what would it look like? What kinds of information would it include?
Pressure Sealed Envelope
This is a new envelope style we are considering using for mailing our letters to notify people about the survey. (Give respondent sample envelope) Please open this for me.
(After opening) What are your impressions of this piece of mail?
Have you ever received mail in this kind of envelope? If so, what kinds of mail like this have you received?
Have you ever received mail like this from any government agencies? If so, which ones?
If the Census Bureau sent your company a piece of mail like this, how would it be handled?
Assuming we printed your name as the recipient, do you think there would be any problems as far as it getting to you?
What would you think if something like this landed in your inbox? What would you do with it?
In your opinion, does an envelope like this look legitimate, or does it look like junk mail?
Wrap-up
Are there any other comments or suggestions you would like to make about anything we have gone over today?
Thank the respondent for their time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Aryn Hernandez (CENSUS/ESMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |