Attachment III: Interviewer Protocol/Test Overview
Background. Before we begin, let me give you some background about what you will be doing today. We are trying to improve a website that companies use to report work-related injuries and illnesses. Our purpose today is to find out how easy or difficult the website is to use and to identify ways of improving it. To do this, I’m going to ask you to act as if you were a respondent in the Survey of Occupational Injuries and Illnesses for a fictional company, and your job is to report information about any injuries and illnesses that employees in this company have had over the past year. I’ll provide you with all the information you’ll need to complete the SOII survey for this fictional company, so you’ll just be taking that information and trying to use the website to enter it in. We’ll run through a few different tasks to examine different sections of the website, and I’ll also ask you some questions as we go along to get your feedback.
So, again, our purpose here today is to evaluate how people use this website. Our purpose is not to evaluate you. If the website is easy to use, that’s great. But if you have any difficulties using it, we assume that it is a problem with the way the site is design, not with the user. We’ll take what we learn from this study to work on design changes that we hope will make it easier to use the website in the future. Okay? Any questions?
[If not] Alright, to get us started, I want you to assume that you are an employee of a fictional company called Ski Solutions, Inc. For the purposes of this study, imagine that Ski Solutions is a small, independently-owned company that operates three ski slopes in Pennsylvania. The company runs and maintains the ski lifts for the slopes, operates a small ski lodge with a snack shop, and is open for business 5 months out of the year during the peak ski season. I’d like you to imagine that you are the Safety Manager for Ski Solutions, and that your company has been selected to participate in the Survey of Occupational Injuries and Illnesses. You’ll be reporting for the company using a BLS website called the Internet Data Collection Facility or IDCF.
We’ll go through several tasks using the survey website, as I mentioned, and you’ll be using this packet of information to help you with each task. [Hand the packet to the participant.] On each page of the packet there is a brief description of the task and some information about the company that you’ll need to complete the task. We’ll go through the tasks one at a time – you’ll start by first reading the task description. Once you’ve reviewed the material, I’ll launch the SOII website so you can begin entering the company information, and when you’re done with each task we’ll have a few brief questions about your experience using the site.
It’s important that you read through the materials carefully before we bring up the website because we’ll be using an eye-tracker today to get a better sense of where people are looking on the screen when they fill out the survey, and we’d like you to try to focus on the website and not be looking back and forth to the packet too much. Our eye-tracking monitor is basically like any other computer monitor so it shouldn’t be too intrusive while you’re working through each task. [Show participant the Tobii monitor.]
Think aloud instructions. Finally, as you complete each task, I would like you to think out loud to explain what you are doing and why. What I’m going to ask you to do is tell me what you’re thinking as you try to fill out the information on the SOII website. It may feel strange at first, but it’s a valuable way for us to learn about your experience as you move through the website. If you forget, I may remind you to think aloud as you work during the survey.
Do you have any questions for me before we begin?
Attachment IV: Tasks and Debriefing Protocol
Task 1: Enter Information about Number of Company Employees and Total Hours Worked
For the purposes of this task and those that follow, you will be reporting for the calendar year 2011.
Over the course of the 2011 ski season, Ski Solutions had an average of 15 employees.
The total hours worked by all employees for 2011 was 12,000 hours.
2011 was a typical year for Ski Solutions. There was nothing unusual that affected the number of employees it hired or the hours those employees worked.
In this task, you will be asked to report the annual average number of employees and the total hours worked by all employees for 2011.
When you are ready to launch the website and begin entering in the company information, please let us know. (And, please don’t forget to think out loud as you go through the task!)
Post-task debriefing probes:
How difficult was it for you to enter in information about the company’s employees and hours?
Was there any language on the website that confused you?
Were there any features of the website layout that you found confusing or frustrating, or which you think could be improved?
[IF NOT MENTIONED OR OBSERVED PREVIOUSLY] Did you notice the information about Average Hours Worked per Employee? What was your reaction to this – did it make sense? Can you tell me in your own words what this means?
Task 2: Enter Information about Number of Employees and Total Hours Worked II
For this task, let’s assume that the company had the same average number of employees as you saw in the previous task (i.e., 15 employees), but because some of those employees worked part-time, you’ll need to do a bit more to calculate the total hours worked in 2011. The SOII website offers respondents a worksheet to estimate the total hours worked by all employees, so we’d like you to use the worksheet and the following information to enter in a new total hours worked estimate.
In 2011, Ski Solutions paid its employees twice a month, so over the five months that the company was open for business, it had a total of 10 pay periods.
In a typical pay period, the company employed 10 full-time employees and 5 part-time employees. (So, the annual average number of employees working for the company in 2011 was still 15.)
Full-time employees worked 80 hours per pay period (40 hours per week)
Part-time employees worked 40 hours per pay period (20 hours per week)
Again, 2011 was a typical year for Ski Solutions. There was nothing unusual that affected the number of employees it hired or the hours those employees worked.
For this task, please report the annual average number of employees and use the worksheet to estimate the total hours worked by all employees for 2011.
When you are ready to launch the website and begin entering in the company information, please let us know. (Don’t forget to think out loud as you go through the task!)
Post-task debriefing probes:
How difficult was it for you to use the worksheet to calculate the total number of hours?
Was there any language on the worksheet that confused you?
Were there any features of the worksheet layout that you found confusing or frustrating, or which you think could be improved?
Task 3: Enter Case Details – I
In 2011, Ski Solutions had two work-related incidents that resulted in employee injuries. In both cases, the injuries were suffered by members of the ski lift maintenance crew as they worked to repair faulty equipment. The injuries were not severe, but both employees had to miss two days of work before they were sufficiently recovered. There were no other accidents, illnesses, or other incidents at Ski Solutions that resulted in days away from work or job transfers/restrictions in 2011.
For this task, please use the website to begin reporting the cases with days away from work.
Please note: the website may asks you to refer to specific OSHA forms as you fill out the survey, but we will not be using those forms for this study. Simply use the information provided in this task description to fill out the relevant fields of the survey form.
For this task, you will be asked to enter specific details about the first injury cases. Please try to do so – just make up answers for any questions on the page that require a response (e.g., about the injured employee’s name, the date of injury, etc.). The specific answers you give are not important – we are only interested in how easy or difficult it is for people to navigate through the website and provide the information necessary for completing the survey.
When you are ready to begin entering information about the first of the 2 injury cases that resulted in days away from work, please let us know. (Again, please think out loud as you go through the task.)
Post-task debriefing probes:
How difficult was it for you to enter in information about the company’s work-related injury cases?
Was there any language on these pages that confused you?
Were there any features of the layout that you found confusing or frustrating, or which you think could be improved?
[Note any comments Rs make about edits, but do not probe about them in any detail until after the completion of the next task.]
Task 4: Enter Case Details - II
In this task, you’ll report details about the 2nd case that resulted in days away from work. Recall that the injury kept the employee away from work for 2 days. Again, for this task you will not be using the OSHA/SOII forms – you should make up answers for any questions on the page that require a response. We only are interested in how easy or difficult it is for people to navigate through the website and provide the information necessary for completing the survey.
When you are ready to launch the website and begin entering information about the second of the 2 injury cases, please let us know. (Again, please think out loud as you go through the task.)
General Debriefing questions. Now I have a few general questions to ask you about your experience completing the survey. Non-scripted questions will be added based on testing observations and additional post-testing questions will be added based on tasks.
What would you say was the most difficult step involved in completing the survey? If you personally didn’t have a problem, what do you think would be the most difficult step for others?
Were there any places where the instructions could be improved or were needed?
What are your general impressions of the online survey?
You may have noticed that at various times when you were filling out the survey the website showed messages asking you to confirm or correct an answer that you provided.
What were your reactions to the messages generally?
Let’s take a closer look at the individual messages and get your reactions to each of them. How about the messages that appeared on the first screen (show respondent screen/message)
Did this make sense to you? Can you tell me in your own words what it was asking you to do?
[If edit was triggered] Did you notice this when you were filling out the form?
If not, why not?
If so, what was your reaction – did you find it helpful or not?
Now let’s look at look at the messages that appeared on the detailed case entry screen (show respondent screen/message)
Did you notice this when you were filling out the form?
If not, why not?
If so, what was your reaction – did you find it helpful or not?
Now let’s look at a couple of alternative ways these messages could have been presented (show respondent screen/message, demonstrate edits)
What are your reactions to this message format?
Do you think you would have been more or less likely to notice the message with this format or the other format you saw?
Have you seen this kind of format/this approach used before?
Which approach/format do you prefer?
[If not obvious from previous answer] Can you tell me more about that – why do you prefer __________?
[Show 2nd alternative format for Detailed Cases page] On this task we presented the messages a bit differently?
What are your reactions to this format/approach?
Do you think you would have been more or less likely to notice the message with this format or the other format you saw?
Have you seen this kind of format/this approach used before?
Which approach/format do you prefer – of the three seen?
[If not obvious from previous answer] Can you tell me more about that – why do you prefer __________?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | kincaid_n |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |