Attachment C Lab testing protocol

Attachment C: OSMR Lab Testing

Protocol

<Provide respondent consent form and have them read and sign it.>

Thank you for participating in our research study today. Today we’re going to look at materials to help make improvements to one of our surveys, the Occupational Employment Statistics (OES) report. This survey collects data from businesses on employees’ occupations and wages.

Today you will be interacting with some websites designed to collect that information. As part of the process, we are going to ask you to do a few tasks so you can experience how easy or difficult it is to use. Our goal is to make the process as easy to use as possible.

As you're working, please think out loud and describe what you're looking for, and mention anything that is confusing or surprising. I will take notes while you're working. If you are reading anything (either on screen or on paper), please read it aloud.

I'd like to stress that we're testing the product, and not your abilities. If you find parts of the product difficult to use or understand, so will other people, and it will be our job to make sure we make the appropriate changes to improve it.

I will give you the tasks one at a time. Go as far as you would if you were alone. When you're ready to stop, please say "I've finished" or "I would stop here." Please remember to think aloud.

<Participants will start at a mock-up of the OES IDCF respondent website. A screenshot is provided below>

Task 1:

Imagine you receive the following request from the Bureau of Labor Statistics (BLS). Please skim over the first page.

Please skim over this page, you don’t have to read it in detail.

What are your impressions of this page?

What do you think it’s asking you to do?

Task 2:

Next, imagine you moved company locations recently, and you want to give BLS the new address of your company. Where would you click to do that?

Probes:

• Was the link “Update Respondent Information” easy or difficult to find?

• What did you think it meant?

• Is there an alternative placement or instruction that would make it easier? What would that be?

Task 3:

Now, imagine you click the link and you are brought to a page to update your business’s information. This is what that looks like:

(Note: mock-ups will be built and partially functional in Excel)

Probes:

What are your impressions of this form?

Is this what you would expect to see on the form to update your contact information?

Do you have any questions/concerns about this form?

What would you expect to see after clicking ‘Continue’?

Task 4:

Now, imagine you are submitting data to BLS for the Occupational Employment Statistics (OES) report. You know that you will need to provide the job title, description of job duties, wage rate, and wage amount for each of your employees.

You figure the best way to get that information would be to take a look at your payroll records, seen here:

Employee name	Department	Position	Hours worked for week	Wage
Joe Smith	Warehouse	Forklift operator	60	$11.00
Richard King	Warehouse	Forklift operator	50	$11.00
Sybil Bedford	Warehouse	Forklift operator	50	$12.35
Bernard Oberland	Warehouse	Truck loader/unloader	45	$9.00
Helen Ledwidge	Warehouse	Truck loader/unloader	47	$9.00
Mary Amberly	IT	Programmer	40	$13.24
David Langford	IT	Programmer	40	$13.24
James Garcia	IT	Programmer	45	$15.00
Michelle Yang	Administration	General manager	40	$65,000
Jean West	Administration	Logistics manager	42	$60,000
Sam Brown	Office	Administrative assistant	40	$9.00

Job descriptions:

• Forklift operators:  Operate powered industrial trucks to load and unload materials

• Truck loader/unloaders:  helps to load freight onto trucks and unload freight shipped to the warehouse facility.

• Programmer: Writes computer programs, tests, and fixes computer software.

• General Manager: Directs and coordinates activities across company departments, administers organization policies, oversees employees.

• Logistics manager: Oversees warehouse, inventory control, material handling, customer service, and planning workers.

• Administrative assistant: Provides clerical support to managers and employees, assists with daily office needs and activities.

Now, you need to take this payroll information and enter the data needed to complete the OES report. How would you go about doing this? Please talk aloud as you work through the task.

(Note: This is a screenshot of an interactive Excel document that will be used to mimic the process of completing OES data in IDCF as closely as possible. The researchers will observe the participants taking the fake payroll data above and entering it into the Excel spreadsheet. Any problems, confusion, or issues will be assessed in the process).

Probe as needed:

• What are you thinking now?

• How are you going about completing the spreadsheet?

• Was anything confusing, difficult to understand about the task?

• What could be improved in the process?

Those are all of the tasks we had for you today. Do you have any other questions or feedback for us?

Thank you!