May 15, 2018
NOTE TO THE REVIEWER OF:
|
OMB CLEARANCE #1220-0141 “Cognitive and Psychological Research” |
FROM: |
Bill Mockovak Survey Methodologist Office of Survey Methods Research
|
SUBJECT: |
Submission of Materials for Usability Testing of the BLS Taxonomy
|
Please accept the enclosed materials for approval under the OMB clearance package 1220-0141, “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research.
The maximum number of burden hours is estimated to be 39.
If there are any questions regarding this project, please contact Bill Mockovak at (202) 691-7414.
Attachments
I. Introduction and Purpose
For the past several years, the BLS Taxonomy Team has been developing a structure (taxonomy) for organizing BLS content on its public website.
This structure, or information scaffolding, has also been used to develop a hierarchical menu, which allows data seekers to navigate through the information structure to find desired information.
BLS has already used the taxonomy to develop a beta tool called the Data Finder, which can be found on the BLS public data website.1 However, testing with the Data Finder and internal reviews have revealed that further development and refinement of the taxonomy would benefit from usability testing with a non-BLS population.
The primary objectives of this research are to determine how well users can navigate the taxonomy using the existing menu system and if a subset of the existing categories and category names are the most appropriate.
II. Respondents
Participants will be recruited through Amazon’s Mechanical Turk (MTurk) and TryMyUI for both the navigation testing and card-sorting tasks. The number of participants from each source is listed below:
Navigation Testing (10 tasks)
60 MTurk participants (30 per persona/group: students & business owners)
6 TryMyUI participants (3 students, 3 business owners)
Card Sorting (1 task)
60 MTurk participants (30 per persona/group: students & business owners)
6 TryMyUI participants (3 students, 3 business owners)
Total = 132; 120 MTurk participants, 12 TryMyUI participants
Although most participants will be recruited through MTurk, an additional 12 participants will be tested using TryMyUI2 because it provides a video recording and a richer description of the thought processes that participants engage in as they use the menu system and conduct card-sorting tasks.
III. Research Design
Navigation Testing. Using a specially designed online tool for testing navigation called TreeJack,3 participants will complete 10 randomly presented tasks that ask them to answer a variety of questions (see Attachment 1 for the list of tasks). For example,
“You want to find out what percentage of employees are in unions”
“You want to find out how many people in the United States want to work, but have
given up looking for work”
“Your daughter, who is a freshman in high school, is interested in becoming a personal
trainer. Where would you look to find advice on steps she would need to take?” etc.
To determine the effectiveness of the menu system with different types of data seekers, participants will be recruited from two groups (personas) that are known users of BLS data:
College students – 18 or older and in college or graduate school
Business owners (any size business)
Analysis for the navigation task.
The following measures will be used to evaluate results from the navigation testing:
Direct success. This is defined as users read the task, navigate the menu without hesitation, and find the right link without getting distracted by incorrect links. Counts are given for:
how many people got it right
how many people got it wrong
the paths people took before they selected an answer
how long it took people to complete the task
Time required for direct success. This is defined as time elapsed from start to end of the task.
Indirect success. This is defined as users who try different branches within the tree before finding the correct answer.
Time required for indirect success. This is defined as time elapsed from start to end of the task.
Failure. This is defined as users who choose the wrong link as the final answer.
Skip. This is defined as a user who gives up on the task entirely.
In addition, video recordings from TryMyUI will be analyzed and summarized.
Card Sorting Task. Using the software provided in TreeJack, items from two of the categories currently existing in the taxonomy will be tested using online, open-ended card-sorting. Attachment 2 shows the 27 items that will be sorted. The team selected these items for testing because they had difficulty coming up with categories that meaningfully distinguished between them (the team came up with two categories). The team wants to find out how a non-BLS audience would categorize and describe the items.
Using the open-ended card-sorting approach, participants are given a list of items and asked to place them into one or more groups of their own creation. Participants are also asked to give each group a name that reflects its content. The purpose of this testing is to compare the names that non-BLS participants generate with the category names currently assigned in the taxonomy, as well as to examine the content of the categories. The estimated time for completing this card sorting task is 15 minutes.
Analysis for the card-sorting task. Output measures include the following:
Number of cards sorted
Number of categories created and content
Names assigned to categories (which will be compared to existing names)
Time taken to create the categories and assign items
IV. Burden Hours
Recruitment. Since recruitment will be handled through MTurk and TryMyUI, there are no recruitment costs.
Data Collection.
The estimated completion times are based on dry runs of the protocol conducted using BLS volunteers.
Navigation Task
66 participants x 20 min = 1,320 min divided by 60 min = 22 hours
Card-Sorting Task
MTurk: 60 participants x 15 min = 900 min divided by 60 min = 15 hours
TryMyUI: 6 participants x 20 min = 120 min divided by 60 min = 2 hours
Total burden hours are therefore 22 + 15 + 2 = 39 hours
V. Payment to Respondents
MTurk participants will be reimbursed 15 cents per minute: $3 for the navigation task and $2.25 for the card-sorting task. The average pay rate for MTurk workers is 10 cents per minute.4 A slightly higher rate is desired because of the challenging nature of the tasks and to encourage participants to view the task seriously, work conscientiously, and spend the targeted amount of time on the tasks. TryMyUI sessions cost $35, $10 of which goes to the participant.
V. Data Confidentiality
BLS will not provide a guarantee of confidentiality for the MTurk or TryMyUI participants.
VI. Attachments
Attachment 1 – Navigation Tasks
Attachment 2 – Items in the Card-Sorting Task
Attachment 3 – Instructions for TryMyUI and Amazon’s Mechanical Turk
Attachment 1 - Navigation Tasks
Your daughter, who is a freshman in high school, is interested in becoming a mechanical engineer. Where would you look to find advice on steps she would need to take?
You want to find out how many people in the United States want to work but have given up looking for work.
You notice that everything seems more expensive at the grocery store. Where would you look to find out how much the cost of groceries has gone up?
You want to learn more about people who work for themselves in the United States.
You are thinking of sending your children to a local private school, and want to know what others are spending on private schools.
You heard a story on the news about how dangerous some jobs can be, and you want to find out how many people die at work in a typical year.
Since you work in an amusement park, you wonder how many amusement parks there are around the country.
You are looking to see what information BLS has about Boston.
You’re interested in finding out what types of perks employers offer to attract and keep employees.
You live in Dallas and want to find out how many people broke bones while working there.
Attachment 2 – Items in the Card-Sorting Task
Age
Disability status
Educational attainment
Eldercare
Families and marital status
Foreign born status
Home ownership
Income
Veteran status
Race/Hispanic
School enrollment
Sex
How People Spend Their Time
Earnings status
Hours and full/part-time status
Job Level / Complexity / Pay Grade
Self-employed
Unpaid family workers
Wage and salary workers
Multiple jobholding status
Union membership / bargaining status
Length of employment
Alternative worker arrangement status
Absence from work
Strikes and Work Stoppages
Worker activity at time of fatality
Worker location at time of fatality
Attachment 3 – Instructions for TryMyUI and Amazon’s Mechanical Turk
Instructions for TryMyUI
Recruitment criteria: We are looking for students in [college or graduate school/owners of businesses (any size)] to participate in this study.
Scenario for the Navigation Task:
This is not a typical usability test. Instead, we would like you to use a menu system to answer some questions. We want to find out how easy the menu system is to use. Each task will start with the following links:
Home
Measure
Characteristic
You don’t actually have to answer each question. Instead, click on the link that you think would lead to the best answer to the question. Once you reach the correct level in the menu, the message “I’d find it here” will appear in a green box. When that message appears, you can click one of the links shown if you think it’s the right link. If you think you went down the wrong path, you can back up and try a different path. However, do not back up using the back arrow on your browser. Click on Home or Measure or Characteristic instead. These instructions will be available when you start the first task, which should appear to the right of these instructions on your monitor. Please be sure to think out loud as you complete the tasks. Click the Start Task button when you are ready to begin.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 20 minutes to complete. This study is being administered by OptimalWorkshop.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of study responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.
Scenario for the Card-Sorting Task:
This is not a typical usability test. In this exercise you are going to use online software to sort a list of 27 items into categories that you create. Your participation in this task will help us better organize the information on our website.
You begin the sorting task by using your mouse to drag an item to the right side of the screen. As you do that, the computer will automatically create a category for you. You can name the category right away or wait until you have a few items in it. If you want to create a new category, drag an item to a blank spot on the screen. You can create as many categories as you like. Items that you put together in a single category should have something in common, but there are no right or wrong answers, and you can use whatever logic you choose. Put the items in categories that make sense to you. When you are done, be sure to name each category that you created and then conduct a final check to make sure the items are in the categories you want them in. Please be sure to think out loud as you complete the sorting.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 15 minutes to complete. This study is being administered by OptimalWorkshop.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of study responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.
Instructions for Amazon’s Mechanical Turk (MTurk)
Navigation Task
Recruitment criteria: We are looking for students in [college or graduate school/owners of businesses (any size)] to participate in this study. This task is estimated to take no longer than 20 minutes, and you will be paid $3.00 for completing it.
Title: Help test a menu system
Description: Answer 10 questions that require you to look for information on a website using a menu.
Keywords: Menu, navigation, information-seeking
Instructions for the Navigation Task:
We are conducting a study to improve a menu system for a website with statistical information. You will be presented with 10 questions and asked to find the menu link that would best lead to the answer. Each task will start with the following links:
Home
Measure
Characteristic
You don’t actually have to answer each question. Instead, click on the link that you think would lead to the best answer to the question. Once you reach the correct level in the menu, the message “I’d find it here” will appear in a green box. When that message appears, you can click one of the links shown if you think it’s the right link. If you think you went down the wrong path, you can back up and try a different path. However, do not back up using the back arrow on your browser. Click on Home or Measure or Characteristic instead.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141.. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 20 minutes to complete. This study is being administered by OptimalWorkshop.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of study responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.
Card-Sorting Task
Recruitment criteria: We are looking for students in [college or graduate school/owners of businesses (any size)] to participate in this study. This task is estimated to take no longer than 15 minutes, and you will be paid $2.25 for completing it.
Title: Use online card sorting to help us better organize information on a website
Description: Sort 27 items into categories that make sense to you. When done, name the categories.
Keywords: Card sorting, sorting information, organization of information, website design
Instructions for the Card-Sorting Task:
Begin the sorting task by using your mouse to drag an item to the right side of the screen. As you do that, the computer will automatically create a category for you. You can name the category right away or wait until you have a few items in it. You can either add items to the first category you created or pull an item to a blank part of the screen to create a new category.
Create as many categories as you like. Items that you put together in a single category should have something in common, but there are no right or wrong answers. Put the items in categories that make sense to you.
When you are done, be sure to name each category that you created and then conduct a final check to make sure the items are in the categories you want them in.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141.. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 15 minutes to complete. This study is being administered by OptimalWorkshop.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of study responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.
2 TryMyUI is an online usability testing service that provides participants and a 20 minute video for $35 per session (see www.TryMyUI.com).
3 optimalworkshop.com
4 Mathematica Policy Research, “Use of Online Crowdsourcing and Online Survey Sample Providers for Survey Operations,” January 13, 2017
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mockovak, William - BLS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |