Note to Reviewer - SOII Usability Testing

June 24, 2013

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Office of Survey Methods Research
SUBJECT:	Submission of Materials for the SOII IDCF Usability Testing

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is 20 hours.

If there are any questions regarding this project, please contact Jennifer Edgar at

202-691-7528

1. Introduction and Purpose

The purpose of this study is to evaluate the usability of the Survey of Occupational Injuries and Illnesses (SOII) web-based instrument on the Internet Data Collection Facility (IDCF). Specifically, the SOII program managers are interested in exploring alternative designs for the Add Establishments pages and Section 3 (summary page on Cases with Days Away from Work).

Previous evaluations of the SOII IDCF instrument (expert reviews, respondent debriefings, usability tests) suggest that respondents may experience confusion about the difference between “adding cases” (Section 2) and adding case data (Section 3). Additionally, SOII is looking to see if combining all the ‘add establishment’ functionality onto one page, rather than having it on three separate pages, improves the usability for respondents.

This study will evaluate those changes through common usability metrics (e.g., task success rates, time to completion, etc.), and respondent think-alouds and debriefings, in an attempt to identify any confusion or problems participants encounter as they complete the web survey.

2. Participants

The study will recruit up to 20 individuals from the Office of Survey Methods Research (OSMR) participant database. The study will not recruit actual SOII respondents given the objectives, time constraints, and evaluation methods used. Efforts will be made to select participants who are experienced computer/Internet users, with varying levels of age, gender, and education based on self-reported information provided during the initial recruitment process. Should a participant with a disability be selected, we will make necessary accommodations to allow them to participate.

3. Methodology

Participants will be sent an advance letter containing a reminder of the date and time of their appointment, directions to BLS, and examples of the SOII and OSHA forms to familiarize them with the general SOII reporting requirements and format (Attachments I and II). During the in-house sessions, half of the participants will work with a test instrument resembling the current SOII design and half will be given an alternative SOII test instrument. The alternative test instrument will evaluate the following:

• Combining the add establishment functionality on one page

• Reformatting Section 3 to make it easier for respondents to enter the correct number of cases

These test changes are further described in Attachment III.

Each change will be treated as a separate respondent task. Before beginning, participants will be given an introduction to the study purpose and goals (Attachment IV). Prior to each task, respondents will be read a description of the task and given all the information necessary to complete the task (Attachments V, VI and VII). Respondents will be encouraged to think aloud during the task.

Quantitative measures will assess success rates on predefined tasks (e.g., entering required SOII information, navigating correctly, and submitting the data) and efficiency (e.g., time to complete tasks). Participants also will be debriefed after testing sessions to obtain qualitative feedback about the designs.

4. Burden Hours

We estimate that each test session will take a maximum of 45 minutes; in addition, respondents will be asked to review study-related materials for a maximum of 15 minutes prior to the study. Thus, this results in an estimated maximum of 20 burden hours (60 minutes x 20 participants).

5. Data Confidentiality

Lab participants will be informed of the voluntary nature of the study, and that the study will be used for internal purposes to improve the design of the SOII IDCF instrument. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice. . Prior to the interview, participants will be given a consent form to read and sign (Attachment VIII).

6. Attachments

1. Attachment I: Advance Letter

2. Attachment II: Blank OSHA (separate file)

3. Attachment III: Screen Shots of proposed changes

4. Attachment IV: Interviewer Protocol/Test Overview

5. Attachment V: Tasks and Debriefing Protocol

6. Attachment VI: OSHA Log Task 2 (separate file)

7. Attachment VII: OSHA Log Task 3 (separate file)

8. Attachment VIII: Consent form

Attachment I: Advance Letter

Dear ,

Thank you for agreeing to participate in this study for the Bureau of Labor Statistics (BLS). As we explained on the phone, the BLS is looking to improve the Survey of Occupational Injuries and Illnesses (SOII) web survey. Your participation in our study will help us evaluate the survey instrument, and identify any changes that should be made to make the website as easy as possible for respondents to use.

You are scheduled to come in to the BLS usability lab on Day/Date at Time (map and directions are provided on the back of this letter). During your session, you will be asked to fill out the web SOII survey based on fictional information we will provide about a company’s injuries and illnesses, and then to discuss your reactions about the survey with the study researcher. We anticipate that the entire session will last approximately 45 – 60 minutes.

In preparation for the study, we have attached examples of a form from the Occupational Safety and Health Administration (OSHA) that companies use to help them fill out the SOII survey. You do not need to fill this out, but please take 10 or 15 minutes to review the form before you come into your session to get a sense of the types of questions that are asked on the SOII survey.

If you have any questions, or need to reschedule, please contact Jennifer Edgar at 202-691-7528 or [email protected]

Thank you in advance for your help with this study!

Jennifer Edgar, PhD

Office of Survey Methods Research

Bureau of Labor Statistics

Washington DC 20212

202-691-7528

[email protected]

DIRECTIONS TO THE USABILITY LAB

BUREAU OF LABOR STATISTICS

MAP

PSB

Union

Station

Metro exit to Amtrak

Visitors’

Entrance

BLS is located in the Postal Square Building (“PSB” on map) at 2 Massachusetts Ave., N.E. in the District of Columbia. This building sits between First St. N.E. and North Capitol Street and also houses the U.S. Postal Museum. BLS is most easily accessible by the Metro Red Line. There is a Metro stop at Union Station just across First Street, N.E. from BLS. (There is a parking garage behind Union Station and there are many public garages and parking lots within a few blocks of BLS. However, we strongly recommend taking Metro.)

You must have a picture ID to be admitted to BLS, and must enter through the Visitors’ Entrance, which is located on First St. N.E. directly across from the First Street exit from Metro. As you leave the Metro train, use the escalators designated for Amtrak at the north end of the platform. When you reach the turnstile level of Metro, you will see an exit to your left leading to a crosswalk marked with white paint stripes. This crosswalk leads directly to the Visitors’ Entrance to BLS. . The heavy double arrow shows this relationship on the map above. Your name will be on a list at the guard station inside the Visitors’ Entrance. The guards will ask you to step through a metal detector, and your purse or other hand-carried property will go through an X-ray screening like the airport uses.

Attachment II: OSHA Form (see separate file)

Attachment III: Screen Shots of proposed changes

Change 1: Add Establishments

Description: Currently, respondents are asked if they received more than one survey to report for in the current year, and a yes response brings them to an ‘add establishment’ page. After adding the establishments on this page, they are then taken to a summary page with information about all the establishments currently associated with their account. The new approach starts with the summary page (Screen 1), and allows the respondent to add establishments directly on this page using the ‘add establishment’ button. A ‘remove’ button will also be provided if an establishment was added by mistake (Screen 2).

Screen 1: New Add Establishment Page

Screen 2: New Add Establishment Page (as an establishment is entered)

Screen A1: Current Add Establishment Page (before an additional establishment is entered)

Screen A2: Current Add Establishment Page (as an additional establishment is entered)

Change 2: Enter Case Data

Description: Currently, after reporting the number of cases in Section 2, respondents are taken to Section 3 where they are asked to enter additional data for cases requiring days away from work. This is accomplished by clicking the “Enter Case Data” button, however some users have been found to click the “Continue” arrow instead and skip this step completely. The goal of the change is to make it clear to respondents that they need to enter data for their cases before moving on to Section 4. There are two paths respondents can follow, and both will be tested. The first (Screen 3) will be shown when respondents indicate in Section 2 that they have cases requiring additional details. The second (Screen 4) will be shown when respondents indicate that they do not have cases requiring additional details.

Screen 3: New Enter Case Data Page (with cases to enter)

Screen 4: New Enter Case Data Page (without cases to enter)

Screen A3: Current Enter Case Data page (with cases to enter)

Screen A4: Current Enter Case Data Page (without cases to enter)

Screen A5: Current Enter Case Data Page (after cases have been entered)

Attachment IV: Interviewer Protocol/Test Overview

• Background. Before we begin, let me give you some background about what you will be doing today. We are trying to improve a website that companies use to report work-related injuries and illnesses. Our purpose today is to find out how easy or difficult the website is to use and to identify ways of improving it. To do this, I’m going to ask you to act as if you were a respondent in the Survey of Occupational Injuries and Illnesses (SOII) for a fictional company, and your job is to report information about any injuries and illnesses that employees in this company have had over the past year. I’ll provide you with all the information you’ll need to complete the SOII survey for this fictional company, so you’ll just be taking that information and trying to use the website to enter it in. We’ll run through a few different tasks to examine different sections of the website, and I’ll also ask you some questions as we go along to get your feedback.

So, again, our purpose here today is to evaluate how people use this website. Our purpose is not to evaluate you. If the website is easy to use, that’s great. But if you have any difficulties using it, we assume that it is a problem with the way the site is designed, not with the user. We’ll take what we learn from this study to work on design changes that we hope will make it easier to use the website in the future. Okay? Any questions?

• [If not] Alright, to get us started, I want you to assume that you are an employee of a fictional company called Ski Solutions, Inc. For the purposes of this study, imagine that Ski Solutions is a small, independently-owned company that operates three ski slopes in Alaska. I’d like you to imagine that you are the Safety Manager for Ski Solutions, and that your company has been selected to participate in the Survey of Occupational Injuries and Illnesses. You’ll be reporting for the company using a BLS website called the Internet Data Collection Facility or IDCF.

We’ll go through several tasks using the survey website, as I mentioned, and you’ll be using this packet of information to help you with each task. [Hand the packet to the participant.] On each page of the packet there is a brief description of the task and some information about the company that you’ll need to complete the task. We’ll go through the tasks one at a time – you’ll start by first reading the task description. Once you’ve reviewed the material, I’ll launch the SOII website so you can begin entering the company information, and when you’re done with each task we’ll have a few brief questions about your experience using the site.

• Think aloud instructions. Finally, as you complete each task, I would like you to think out loud to explain what you are doing and why. What I’m going to ask you to do is tell me what you’re thinking as you try to fill out the information on the SOII website. It may feel strange at first, but it’s a valuable way for us to learn about your experience as you move through the website. If you forget, I may remind you to think aloud as you work during the survey.

• Do you have any questions for me before we begin?

Attachment V: Tasks and Debriefing Protocol – Test Group

Task 1: Add Two Establishments

For the purposes of this task and those that follow, you will be reporting for the calendar year 2012.

You have received three mailings indicating the surveys that you need to report for. You have just logged in to the IDCF website using the account for establishment ID 02-88811107-0.  In this task, you need to make sure all three surveys are entered in the system.  Please use the survey forms to do that, and don’t forget to think out loud as you go through the task!

Now you want to report for establishment number 02-888111007-0, please show me how you would do that.

Post-task debriefing probes:

• (Show Screen 1)

• How difficult was it for you to add establishments?

• Was there any language on the website that confused you?

• Were there any features of the website layout that you found confusing or frustrating, or which you think could be improved?

• How difficult was it for you to locate your establishment ID on your survey mailing?

Task 2: Enter Case Data (establishment with cases)

Now, for one of your establishments you need to enter additional data. You have been provided copies of the forms that employers are required to use to keep track of workplace injuries and illnesses. Please use the information that has been entered in these forms to complete Sections 2 and 3 of the survey. If you have any questions as you work please let me know, and remember to talk out loud as you work.

Debriefing probes:

• (Show Screen 3)

• How easy or difficult was it to indicate that you needed to enter information about a case?

• On this screen, what did you expect the “Enter Case Data 1” button to do?

• What about that “Continue” button, what did you think that would do?

• Does the number of rows in the table mean anything to you?

  • How do you think the computer decided how many rows to show?

• What if you had an additional case to add, what would you do?

• Was there anything confusing about this page?

Task 3: Enter Case Data (establishment without cases)

Now, for another one of your establishments you don’t have any cases to enter data for.

In Section 2, enter 0 (zero) for “Total number of cases with days away from work” and 1 for “Total number of other recordable cases”.

Under “Injury and Illness Types” enter 1 in #1 to indicate that there was one injury. Press “Save & Continue” to proceed to Section 3.

When you get to Section 3 a ski lift supervisor comes by and tells you he remembers two cases that should have been recorded, but were mistakenly left off the forms. After making the corrections, he gives you the revised forms that should be used to respond to the survey. Use this new set of forms to make any appropriate changes for this establishment.

Debriefing probes:

• (Show Screen 3)

• On this screen, what did you expect the “Back” button to do?

• What about that “Continue” button, what did you think that would do?

• The screen originally said “In Section 2 you reported: 0 cases with days away from work” (show screen shot), what did that mean to you?

• Was it clear that you did not need to submit a detailed case form for the Other Recordable Case?

• Was there anything confusing about this page?

Comparison Probes

Again, we have an alternate version of this page, take a look and let me know what features you like better or worse in this version

• (Show Screen A3)

• On this screen, what would you expect the “Enter Case” button to do?

• What about the “Continue” button, what did you think that would do?

• What do you think the “Remove” button would do?

• What if you had an additional case to add, what would you do?

Attachment V: Tasks and Debriefing Protocol – Control Group

Task 1: Add Two Establishments

For the purposes of this task and those that follow, you will be reporting for the calendar year 2012.

You have received three mailings indicating the surveys that you need to report for. You have just logged in to the IDCF website using the account for establishment ID 02-88811107-0.  In this task, you need to make sure all three surveys are entered in the system.  Please use the survey forms to do that, and don’t forget to think out loud as you go through the task!

Now you want to report for establishment number 02-888111007-0, please show me how you would do that.

Post-task debriefing probes:

• (Show Screen A1)

• What did you think when you first saw this screen?

• How difficult was it for you to add establishments?

• Was there any language on the website that confused you?

• Were there any features of the website layout that you found confusing or frustrating, or which you think could be improved?

• How difficult was it for you to locate your establishment ID on your survey mailing?

Task 2: Enter Case Data (with cases)

Now, for one of your establishments you need to enter additional data. You have been provided copies of the forms that employers are required to use to keep track of workplace injuries and illnesses. Please use the information that has been entered in these forms to complete Sections 2 and 3 of the survey. If you have any questions as you work please let me know, and remember to talk out loud as you work.

Debriefing probes:

• (Show Screen A3 )

• How easy or difficult was it to indicate that you needed to enter information about a case?

• On this screen, what did you expect the “Enter Case” button to do?

• What about that “Continue” button, what did you think that would do?

• What if you had an additional case to add, what would you do?

• Was there anything confusing about this page?

Task 3: Enter Case Data (without cases)

Now, for another one of your establishments you don’t have any cases to enter data for.

In Section 2, enter 0 (zero) for “Total number of cases with days away from work” and 1 for “Total number of other recordable cases”.

Under “Injury and Illness Types” enter 1 in #1 to indicate that there was one injury. Press “Save & Continue” to proceed to Section 3.

When you get to Section 3 a ski lift supervisor comes by and tells you he remembers two cases that should have been recorded, but were mistakenly left off the forms. After making the corrections, he gives you the revised forms that should be used to respond to the survey. Use this new set of forms to make any appropriate changes for this establishment.

Debriefing probes:

• (Show Screen A4)

• The screen said “In Section 2 you reported: 0 cases with days away from work” (show screen shot), what did that mean to you?

• Was it clear that you did not need to submit a detailed case form for the Other Recordable Case?

• Was there anything confusing about this page?

Comparison Probes

Again, we have an alternate version of these pages, take a look and let me know what features you like better or worse in this version

• (Show Screens 3 and 4)

• On this screen, what would you expect the “Enter Case 1” button to do?

• What about that “Continue” button, what did you think that would do?

• What do you think the “Remove” button would do?

• Does the number of rows in the table mean anything to you?

  • How do you think the computer decided how many rows to show?

• What if you had an additional case to add, what would you do?

• Which set of pages do you think would be easier to use?

Attachment VI: OSHA Log for Task 2 (Separate File)

Attachment VII: OSHA Log for Task 3 (Separate File)

Attachment VIII: Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you approximately 60 minutes to participate in this research study.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141, and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 02-28-2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

21