MEPS (Centurion) Usability Testing Draft Protocol
(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)
Can you tell me a little about your company/government organization? What products or services does it provide?
What is your role in the organization?
What has been your overall experience with completing the MEPS-IC?
Have you completed any other government surveys? If so, which ones?
Did you report electronically to any of those surveys?
What made you choose to report electronically?
What was your overall experience? Did anything stand out as being very helpful? Was there anything that you didn’t like?
What benefits would there be to reporting electronically to a survey like the MEPS-IC? What drawback?
Now I’d like to show you an electronic draft of the MEPS-IC form. Here is the web-site address for accessing the survey (show the respondent the URL, survey key, test username and test password).
I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do.
Don’t hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We’ll try our best to answer your questions.
(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)
3. Login Screen
Do you think that the information given on the Login Screen was sufficient?
What are your overall reactions to the process for logging into the system?
Is there any other information you’d like to see on any of the screens you saw while logging in?
4. Main Menu
Can you describe to me what each of the columns under “survey tasks” is showing you (i.e., Form, plan name, status, view as PDF)?
Where do you need to go in order to start filling completing your form?
Was this clear? If not, what could be done to make the start of the form clearer?
If you had questions about definitions or the survey items where would you go?
Is there anything else that you would need at this point in order to start completing the form? Anything else that should be included on this screen?
Did you notice the toolbar on this screen? What do you think each of the options will do?
What would you expect to see under the ‘Information’ option on the toolbar? After looking at the options, do you think that this is the best label? If not, what would be more descriptive?
Are there any other types of information that would be helpful for you to have? Would you look at this information before completing the form or when you needed it?
For the Tools option, do you think that Tools is an effective label for the options given? If not, what other suggestions do you have? Can you think of any other options that would be helpful under Tools?
5. Business information: Section A – Number of Plans – Screen 1 (provide health plans)
If you needed assistance with understanding this question, where would you go? Which online resource would you consult?
Did you notice the help link on this screen? Would you need this information?
6. Business information: Section A – Number of Plans – Screen 2 (number of health plans)
Is the definition for number of health plans clear?
7. Business information: Section C – Employment Characteristics – Screen 3 (number of employees, eligibility and enrollment, etc.)
What do you think about the amount of information being asked on this screen? Do you feel that it is too little, too much, or a comfortable amount?
The different questions on the screen are divided by lines, is this an effective way for divide up the questions on the screen?
There are options for help offered in many different areas of the screen. Is this helpful? What do you think about that design?
Do you have any other recommendations for how we could improve the layout of the questions on this screen?
8. Business information: Section C – Employment Characteristic – and Section D – Business Characteristics Screen 4 (% women, %$ earned, fringe benefits, etc.)
What do you think about the amount of information on this screen?
On this screen there are several questions as well as two different sections. Do you have any recommendations for how we could improve the way we lay out the changes from question to question and section to section on this screen?
Do you have any recommendations for how we designed the tables on this screen? Do you think that it was helpful to have this information being asked in a table format, or would another layout be more helpful?
2nd question, Section D – what do you think the difference is between the help link and the definitions link for this question?
2nd question, Section D – was the use of all caps and italics helpful in this question.
9. Business information: Section E – General Health Coverage Characteristics – Screen 5 (premium separate from comp healthplan premium, etc.)
Did you read the instructions under the question? If not, is there anything that we could do to make it easier to read these?
What do you think of the way the question is designed? Do you have any suggestions for improving it?
10. Business information: Section E – General Health Coverage Characteristics – Screen 6 and 7 (if yes for 6) (waiting period and for how long)
What do you think about the navigation options offered at the bottom of the screen? Do they make sense to you?
Do you have any suggestions for improving them?
11. Business information: Section E – General Health Coverage Characteristics – Screen 8 (spousal coverage, not electing coverage)
If you needed to return to the main menu, how would you do that? Please go back to the main menu.
Now that you are on the main menu, how would you return to the question you were working on?
12. Business information: Section F – Retiree Health Coverage Characteristics – Screen 9 (retiree cov. other than cobra)
What would you expect to get if you clicked on the link for COBRA or state continuation of benefits law? Is this information helpful to you? Is there a better way that we could provide this information to you?
Do you have any other suggestions for how we could improve the layout of this question?
13. Business information: Section F – Retiree Health Coverage Characteristics – Screen 10 (retiree cov. other than cobra) and Screen 11 if yes to Screen 10)
What do you think of the way the questions on this screen are divided? Is it helpful? Do you have any suggestions for how it could be improved?
Did you read the exclude statement above the question? Were you more likely to read the exclude/include information when it was above the question or below the question?
14. Business information: Section F – Retiree Health Coverage Characteristics, Under/Over age 65 – Screens 12 and 13, if yes to items in screen 11.
(if they got the under 65 and over 65 screens) – what was the difference between these two screens? Was it obvious to you? If not, how could we make it clearer?
What do you think about the amount of questions that we are asking on these screen(s)?
On this screen there are questions asking for dollar figures and percentages. Were the symbols next to the answer spaces for these questions helpful in distinguishing what was expected? Is there anything that we could do to improve this?
15. Business information: Section F – Retiree Health Coverage Characteristics, New retirees – Screens 13, 14
Which locations did you answer this question for? Did you notice the instructions above the questions for ALL LOCATIONS?
Did you read the other instructions above the question? If not, how could we make it more prominent?
16. General navigation -- TBD
Did you have any problems navigating back to a prior section/screen? Do you have any recommendation for improving the navigation within the site?
17. Form-Specific section -- TBD
(If the R hasn’t triggered an edit at this point, have them do so on purpose)
Is it clear to you based on the messages you are receiving on the screen that there is an issue? Can you explain to me what the problem(s) are?
Is there a better way that we could notify you about the problem(s) on this screen?
What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?
What other information should we provide to you about your problems and how to correct them?
18. Review your responses
What are your first thoughts when seeing this screen?
Can you explain to me what is expected of you? What is this screen telling you?
(If there are edits) How will you go about fixing this/these problem(s)? When you have finished fixing the problems, what do you need to do next? How will you submit the form?
What do you think of the title of this screen? Is it a logical title?
What other types of review would you be interested in performing prior to submission?
If you weren’t ready to submit your data at this point, what would you do?
19. Completion Certificate
What additional information would be helpful for you on this screen?
Would you save a copy of this confirmation information for your records?
20. Main Menu (after submission)
Would you save a copy of your data for your records?
Where would you need to go on this site in order to do that?
Now that you have gone through this form electronically, would you prefer to do this survey on paper or electronically?
Do you think that you would complete the online form in one sitting?
If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?
Did you have any issues with the navigation on the site?
What are some of the things that you liked the most about the online form?
What are some of the things that you liked the least about the online form?
Are there any other options that we should provide in the online form?
Do you have any suggestions for how to make this online form easier for you to use?
March 9, 2010
| File Type | application/msword |
| File Title | Quarterly Financial Report: Long Form |
| Author | oneil302 |
| Last Modified By | demai001 |
| File Modified | 2010-03-22 |
| File Created | 2010-03-22 |