Note to Reviewer SOII IDCF Usability Testing

20

July 6, 2012

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Scott Fricker Office of Survey Methods Research
SUBJECT:	Submission of Materials for the SOII IDCF Usability Testing

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is 25 hours.

If there are any questions regarding this project, please contact Scott Fricker at

202-691-7390.

1. Introduction and Purpose

The purpose of this study is to evaluate the usability of the Survey of Occupational Injuries and Illness (SOII) web-based instrument on the Internet Data Collection Facility (IDCF). Specifically, the SOII program managers are interested in exploring alternative designs for Section 1 (Establishment Information page), Section 3 (summary page on Cases with Days Away from Work), and the detailed Case reporting page. Previous evaluations of the SOII IDCF instrument (expert reviews, respondent debriefings, usability tests) suggest that respondents may:

• Have difficultly understanding what they are supposed to enter in the “Total hours worked by all employees” field and in using the optional worksheet that accompanies this field (Section 1).

• Be confused and/or frustrated by the way information about Average Hours Worked per Employee is derived and presented on the screen (Section 1).

• Misunderstand the purpose of the summary page (Section 3) due to the way case information is presented and the number and type of buttons on the page.

• Miss or have negative reactions to error message that appear on the detailed Case Reporting page.

The program is in the process of making changes to the layout/functionality of these pages and the placement and formatting of error messages (the changes are being made to a test instrument and not to the production survey). This study will evaluate those changes through common usability metrics (e.g., task success rates, time to completion, etc.), eye-tracking analyses, and respondent think-alouds and debriefings, in an attempt to identify any confusion or problems participants encounter as they complete the web survey.

2. Participants

The study will recruit 20 individuals from the Office of Survey Methods Research (OSMR) participant database. The study will not recruit actual SOII respondents given the objectives, time constraints, and evaluation methods used. Efforts will be made to select participants who are experienced computer/Internet users, with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process. Should a participant with a disability be selected, we will make necessary accommodations to allow them to participate.

3. Methodology

Participants will be sent an advance letter containing a reminder of the date and time of their appointment, directions to BLS, and examples of the SOII and OSHA forms to familiarize them with the general SOII reporting requirements and format (Attachments I and II). During the in-house sessions, half of the participants will work with a test instrument resembling the current SOII design and half will be given an alternative SOII test instrument. The alternative test instrument still is in development so screenshots are not available, but the following changes will be evaluated:

• Screen 1 – Change look-and-feel of Average Hours Worked per Employee field. This field currently is calculated and displayed based on respondents’ answers to previous questions (i.e., annual average number of employees, total hours work by all employees), and looks like a button that respondents can click on. The field label also is somewhat ambiguous. The alternative design will use more descriptive language (e.g., “Based on your responses above, we estimate that your average hours worked per employee is XX.”).

• Screen 3 – For instances where there are multiple cases with days away from work, present a case-summary table containing the specified number of cases (e.g., Case 1, Case 2, Case 3, …), and allow respondents to begin reporting detailed case information by clicking on a hyperlink within the table or by clicking an Enter Data Now button. Once respondents have finished entering the detailed information for the first case, the alternative instrument will automatically take the respondent to the next detailed case reporting page. Currently, the case summary page indicates how many cases respondents reported, but does break these out in tabular form. Additionally, respondents are brought back to the summary page after each detail case report has been completed, and this often confuses respondents.

• Detailed case reporting page – The current edit validation occurs only when respondents submit data on this page, and the resulting error message is placed at the top of the page. The alternative design will use both inline validation (immediate error messages that appear next to the field when an edit is triggered) and an augmented, deferred validation (error messages presented at the top of the page after submission, but with hyperlinks to the tripped items).

Each Section (i.e., SOII survey web page) will be treated as a separate respondent task. Prior to each task, respondents will read a narrative description of the task that contains the information necessary to complete the task (Attachment IV). Respondents will be encouraged to think aloud during the task. For the third task (reporting Detailed Case information), the scenario will intentionally trigger error message in two fields – one ‘above the fold’ (requiring no scrolling) and one ‘below the fold’ (requiring users to scroll down to see it).

Quantitative measures will assess success rates on predefined tasks (e.g., entering required SOII information, navigating correctly, and submitting the data) and efficiency (e.g., time to complete tasks). Participants also will be debriefed after testing sessions to obtain qualitative feedback about the IDCF designs. For example, qualitative measures will include ratings of ease of use, intuitiveness of the design, and open-ended feedback about layout, formatting, aesthetics, etc.

4. Burden Hours

We estimate that each test session will take a maximum of 60 minutes; in addition, respondents will be asked to review study-related materials for a maximum of 15 minutes prior to the study. Thus, this results in an estimated maximum of 25 burden hours (75 minutes x 20 participants).

5. Data Confidentiality

Lab participants will be informed of the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of SOII IDCF instrument. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. OMB Control Number: 1220-0141. Prior to the interview, participants will be given a consent form to read and sign (Attachment V).

6. Attachments

1. Attachment I: Advance Letter

2. Attachment II: SOII and OSHA Forms (separate file)

3. Attachment III: Interviewer Protocol/Test Overview

4. Attachment IV: Tasks and Debriefing Protocol

5. Attachment V: Consent form

6. Attachment VI: SOII IDCF Screenshots

Attachment I: Advance Letter

Dear ,

Thank you for agreeing to participate in this study for the Bureau of Labor Statistics (BLS). As we explained on the phone, the BLS is looking to improve the Survey of Occupational Injuries and Illness’s (SOII) web survey. Your participation in our study will help us evaluate the survey instrument, and identify any changes that should be made to make the website as easy as possible for respondents to use.

You are scheduled to come in to the BLS usability lab on Day/Date at Time (map and directions are provided on the back of this letter). During your session, you will be asked to fill out the web SOII survey based on fictional information we will provide about a company’s injuries and illnesses, and then to discuss your reactions about the survey with the study researcher. We anticipate that the entire session will last approximately 45 – 60 minutes.

In preparation for the study, we have attached examples of the 2011 SOII form, as well as forms from the Occupational Safety and Health Administration (OSHA) that companies use to help them fill out the SOII survey. You do not need to fill these out, but please take 10 or 15 minutes to review the forms before you come into your session to get a sense of the types of questions that are asked on the SOII survey.

If you have any questions, or need to reschedule, please contact Scott Fricker at 202-691-7390 or [email protected].

Thank you in advance for your help with this study!

Scott S. Fricker, PhD

Office of Survey Methods Research

Bureau of Labor Statistics

Washington DC 20212

202-691-7390

[email protected]

DIRECTIONS TO THE USABILITY LAB

BUREAU OF LABOR STATISTICS

MAP

PSB

Union

Station

Metro exit to Amtrak

Visitors’

Entrance

BLS is located in the Postal Square Building (“PSB” on map) at 2 Massachusetts Ave., N.E. in the District of Columbia. This building sits between First St. N.E. and North Capitol Street and also houses the U.S. Postal Museum. BLS is most easily accessible by the Metro Red Line. There is a Metro stop at Union Station just across First Street, N.E. from BLS. (There a parking garage behind Union Station and there are many public garages and parking lots within a few blocks of BLS. However, we strongly recommend taking Metro.)

You must have a picture ID to be admitted to BLS, and must enter through the Visitors’ Entrance, which is located on First St. N.E. directly across from the First Street exit from Metro. As you leave the Metro train, use the escalators designated for Amtrak at the north end of the platform. When you reach the turnstile level of Metro, you will see an exit to your left leading to a crosswalk marked with white paint stripes. This crosswalk leads directly to the Visitors’ Entrance to BLS. . The heavy double arrow shows this relationship on the map above. Your name will be on a list at the guard station inside the Visitors’ Entrance. The guards will ask you to step through a metal detector, and your purse or other hand-carried property will go through an X-ray screening like the airport uses.

The Usability Laboratory is in Suite 1950, which is accessible only by elevator from the second floor. An escalator leads from the lobby to that floor. Go to the top of the “up” escalator and turn right. At the end of the corridor, turn left. Look for the elevator to 1950 along the right-hand wall after the left turn. Take it down, and call the number posted on the wall in the downstairs lobby.

Attachment II: SOII and OSHA Forms (see separate file)

Attachment III: Interviewer Protocol/Test Overview

• Background. Before we begin, let me give you some background about what you will be doing today. We are trying to improve a website that companies use to report work-related injuries and illnesses. Our purpose today is to find out how easy or difficult the website is to use and to identify ways of improving it. To do this, I’m going to ask you to act as if you were a respondent in the Survey of Occupational Injuries and Illnesses for a fictional company, and your job is to report information about any injuries and illnesses that employees in this company have had over the past year. I’ll provide you with all the information you’ll need to complete the SOII survey for this fictional company, so you’ll just be taking that information and trying to use the website to enter it in. We’ll run through a few different tasks to examine different sections of the website, and I’ll also ask you some questions as we go along to get your feedback.

So, again, our purpose here today is to evaluate how people use this website. Our purpose is not to evaluate you. If the website is easy to use, that’s great. But if you have any difficulties using it, we assume that it is a problem with the way the site is design, not with the user. We’ll take what we learn from this study to work on design changes that we hope will make it easier to use the website in the future. Okay? Any questions?

• [If not] Alright, to get us started, I want you to assume that you are an employee of a fictional company called Ski Solutions, Inc. For the purposes of this study, imagine that Ski Solutions is a small, independently-owned company that operates three ski slopes in Pennsylvania. The company runs and maintains the ski lifts for the slopes, operates a small ski lodge with a snack shop, and is open for business 5 months out of the year during the peak ski season. I’d like you to imagine that you are the Safety Manager for Ski Solutions, and that your company has been selected to participate in the Survey of Occupational Injuries and Illnesses. You’ll be reporting for the company using a BLS website called the Internet Data Collection Facility or IDCF.

We’ll go through several tasks using the survey website, as I mentioned, and you’ll be using this packet of information to help you with each task. [Hand the packet to the participant.] On each page of the packet there is a brief description of the task and some information about the company that you’ll need to complete the task. We’ll go through the tasks one at a time – you’ll start by first reading the task description. Once you’ve reviewed the material, I’ll launch the SOII website so you can begin entering the company information, and when you’re done with each task we’ll have a few brief questions about your experience using the site.

It’s important that you read through the materials carefully before we bring up the website because we’ll be using an eye-tracker today to get a better sense of where people are looking on the screen when they fill out the survey, and we’d like you to try to focus on the website and not be looking back and forth to the packet too much. Our eye-tracking monitor is basically like any other computer monitor so it shouldn’t be too intrusive while you’re working through each task. [Show participant the Tobii monitor.]

• Think aloud instructions. Finally, as you complete each task, I would like you to think out loud to explain what you are doing and why. What I’m going to ask you to do is tell me what you’re thinking as you try to fill out the information on the SOII website. It may feel strange at first, but it’s a valuable way for us to learn about your experience as you move through the website. If you forget, I may remind you to think aloud as you work during the survey.

• Do you have any questions for me before we begin?

Attachment IV: Tasks and Debriefing Protocol

• Task 1: Enter Information about Number of Company Employees and Total Hours Worked

For the purposes of this task and those that follow, you will be reporting for the calendar year 2011.

Over the course of the 2011 ski season, Ski Solutions had an average of 15 employees.

The total hours worked by all employees for 2011 was 12,000 hours.

2011 was a typical year for Ski Solutions. There was nothing unusual that affected the number of employees it hired or the hours those employees worked.

In this task, you will be asked to report the annual average number of employees and the total hours worked by all employees for 2011.

When you are ready to launch the website and begin entering in the company information, please let us know. (And, please don’t forget to think out loud as you go through the task!)

Post-task debriefing probes:

• How difficult was it for you to enter in information about the company’s employees and hours?

• Was there any language on the website that confused you?

• Were there any features of the website layout that you found confusing or frustrating, or which you think could be improved?

• [IF NOT MENTIONED OR OBSERVED PREVIOUSLY] Did you notice the information about Average Hours Worked per Employee? What was your reaction to this – did it make sense? Can you tell me in your own words what this means?

• Task 2: Enter Information about Number of Employees and Total Hours Worked II

For this task, let’s assume that the company had the same average number of employees as you saw in the previous task (i.e., 15 employees), but because some of those employees worked part-time, you’ll need to do a bit more to calculate the total hours worked in 2011. The SOII website offers respondents a worksheet to estimate the total hours worked by all employees, so we’d like you to use the worksheet and the following information to enter in a new total hours worked estimate.

In 2011, Ski Solutions paid its employees twice a month, so over the five months that the company was open for business, it had a total of 10 pay periods.

In a typical pay period, the company employed 10 full-time employees and 5 part-time employees. (So, the annual average number of employees working for the company in 2011 was still 15.)

• Full-time employees worked 80 hours per pay period (40 hours per week)

• Part-time employees worked 40 hours per pay period (20 hours per week)

Again, 2011 was a typical year for Ski Solutions. There was nothing unusual that affected the number of employees it hired or the hours those employees worked.

For this task, please report the annual average number of employees and use the worksheet to estimate the total hours worked by all employees for 2011.

When you are ready to launch the website and begin entering in the company information, please let us know. (Don’t forget to think out loud as you go through the task!)

Post-task debriefing probes:

• How difficult was it for you to use the worksheet to calculate the total number of hours?

• Was there any language on the worksheet that confused you?

• Were there any features of the worksheet layout that you found confusing or frustrating, or which you think could be improved?

• Task 3: Enter Case Details – I

In 2011, Ski Solutions had two work-related incidents that resulted in employee injuries. In both cases, the injuries were suffered by members of the ski lift maintenance crew as they worked to repair faulty equipment. The injuries were not severe, but both employees had to miss two days of work before they were sufficiently recovered. There were no other accidents, illnesses, or other incidents at Ski Solutions that resulted in days away from work or job transfers/restrictions in 2011.

For this task, please use the website to begin reporting the cases with days away from work.

Please note: the website may asks you to refer to specific OSHA forms as you fill out the survey, but we will not be using those forms for this study. Simply use the information provided in this task description to fill out the relevant fields of the survey form.

For this task, you will be asked to enter specific details about the first injury cases. Please try to do so – just make up answers for any questions on the page that require a response (e.g., about the injured employee’s name, the date of injury, etc.). The specific answers you give are not important – we are only interested in how easy or difficult it is for people to navigate through the website and provide the information necessary for completing the survey.

When you are ready to begin entering information about the first of the 2 injury cases that resulted in days away from work, please let us know. (Again, please think out loud as you go through the task.)

Post-task debriefing probes:

• How difficult was it for you to enter in information about the company’s work-related injury cases?

• Was there any language on these pages that confused you?

• Were there any features of the layout that you found confusing or frustrating, or which you think could be improved?

• [Note any comments Rs make about edits, but do not probe about them in any detail until after the completion of the next task.]

• Task 4: Enter Case Details - II

In this task, you’ll report details about the 2^nd case that resulted in days away from work. Recall that the injury kept the employee away from work for 2 days. Again, for this task you will not be using the OSHA/SOII forms – you should make up answers for any questions on the page that require a response. We only are interested in how easy or difficult it is for people to navigate through the website and provide the information necessary for completing the survey.

When you are ready to launch the website and begin entering information about the second of the 2 injury cases, please let us know. (Again, please think out loud as you go through the task.)

• General Debriefing questions. Now I have a few general questions to ask you about your experience completing the survey. Non-scripted questions will be added based on testing observations and additional post-testing questions will be added based on tasks.

1. What would you say was the most difficult step involved in completing the survey? If you personally didn’t have a problem, what do you think would be the most difficult step for others?

2. Were there any places where the instructions could be improved or were needed?

3. What are your general impressions of the online survey?

4. You may have noticed that at various times when you were filling out the survey the website showed messages asking you to confirm or correct an answer that you provided.

   1. What were your reactions to the messages generally?

   2. Let’s take a closer look at the individual messages and get your reactions to each of them. How about the messages that appeared on the first screen (show respondent screen/message)

      1. Did this make sense to you? Can you tell me in your own words what it was asking you to do?

      2. [If edit was triggered] Did you notice this when you were filling out the form?

         1. If not, why not?

         2. If so, what was your reaction – did you find it helpful or not?

   3. Now let’s look at look at the messages that appeared on the detailed case entry screen (show respondent screen/message)

      1. Did you notice this when you were filling out the form?

         1. If not, why not?

         2. If so, what was your reaction – did you find it helpful or not?

   4. Now let’s look at a couple of alternative ways these messages could have been presented (show respondent screen/message, demonstrate edits)

      1. What are your reactions to this message format?

         1. Do you think you would have been more or less likely to notice the message with this format or the other format you saw?

         2. Have you seen this kind of format/this approach used before?

         3. Which approach/format do you prefer?

            1. [If not obvious from previous answer] Can you tell me more about that – why do you prefer __________?

      2. [Show 2^nd alternative format for Detailed Cases page] On this task we presented the messages a bit differently?

         1. What are your reactions to this format/approach?

         2. Do you think you would have been more or less likely to notice the message with this format or the other format you saw?

         3. Have you seen this kind of format/this approach used before?

         4. Which approach/format do you prefer – of the three seen?

            1. [If not obvious from previous answer] Can you tell me more about that – why do you prefer __________?

Attachment V: Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 60 minutes to participate in this research (ranging from 45 minutes to 75 minutes).

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02-28-2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Attachment VI – SOII IDCF Screenshots

Section 1: Establishment Information Page

Detailed Case Reporting Screenshot

20