Note to Reviewer - BLS.gov Usability Testing

March 21, 2010

NOTE TO THE REVIEWER OF:	OMB No. 1220-0141 Cognitive and Psychological Research
FROM:	William Mockovak, Director Behavioral Sciences Research Center, Office of Survey Methods Research (OSMR)
SUBJECT:	Usability Testing of Drop-Down Menu Prototypes for the BLS Public Data Dissemination Website

Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.”

In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research at least two weeks prior to the beginning of the study.

This usability testing will compare four versions of a drop-down menu for the BLS public data dissemination website: www.BLS.gov.

We anticipate this research will require 54 burden hours.

If there are any questions regarding this project, please contact Bill Mockovak (202) 691-7414.

1. Introduction and Purpose

Previous usability testing has indicated that the drop-down menus currently in use on the BLS.gov website can lead to errors in navigation, user frustration, inefficiency, and low satisfaction, especially among less skilled computer and Internet users.

To address this issue, BLS website designers have reviewed industry standards for menus and have developed three prototypes that appear to be good candidates for replacing the existing menus. Current industry standards include the following desirable features:

• A drop-down with more than two levels should be avoided (especially true of fly-out menus).

• A hover menu or clickable menu should be provided.

• A hover menu is most user-friendly when used with a delay (or click-off area).

• A clickable menu should have an obvious open/close function.

The purpose of this research is to determine which of the prototypes (see Attachment 1) performs best on standard usability measures such as the successful completion of tasks, time to complete tasks, overall proportion of tasks completed, and user satisfaction. In addition, other supplemental measures will be evaluated as well.

II. Data Collection Methodology

Usability Testing: Objectives

As noted, the primary objective of the usability testing is to compare the performance of different user groups on four prototypes of drop-down menus. To compare the prototypes, participants will be given a series of tasks to complete. Both quantitative and qualitative measures of performance will be collected. For example, quantitative measures will include time to complete a task, success on tasks, proportion of tasks completed, number of mouse clicks, and a measure of errant navigation (i.e., where the participant indicated he or she wanted to do something but an erroneous click led to an unexpected outcome). Qualitative measures will include ratings of ease of use, preference ratings, and open-ended feedback about preferences for different versions, aesthetics, layout, formatting, etc.

The prototypes to be compared are shown in Attachment 1. They differ on the key features shown in the following table.

Prototype	Type of Menu Control	Display Width	List of Subject Areas
A	Click to show/close	Fixed, 3 columns	Partial list of content, with ‘view all’ option
B	Hover, no delay	Fixed, 1 column	Compact
C	Hover, with delay	Fixed, 3 columns	Current content
D*	Hover, no delay	Fixed, 3 columns	Current content

* Prototype D is the current menu approach on BLS.gov

Comparing Prototypes B and D will show the effects of changes in menu formatting, with a compact 1-column menu being compared to a full 3-column menu.

Comparing Prototypes A and D will show the combined effects of changing menu control and menu content.

Comparing Prototypes C and D will show the effects of adding a slight delay when users drag their mouse over the menu.

Testing and Recruiting Strategy for Participants

There will be two groups of participants: one group will be tested in the BLS lab, and the second will be tested through an online usability testing service (Trymyui). The group tested in the BLS lab will include up to 13 high school students from a local D.C. charter school who are taking statistics. The students will participate in the lab study so that they can experience firsthand how data are collected. In addition, the students will work with the raw data provided by the Trymyui participants, so that they can see how statistical analyses lead to decisions about the effectiveness of different Web design features.

As shown in the table, based on past testing we anticipate that an average participant in the in-house test will be able to complete 10 tasks, whereas 8 tasks will be the limit for online testing. In addition to the tasks, participants in the in-house test will be asked to complete demographic questions, satisfaction questions, and participate in a short debriefing.

	BLS Lab Testing		Trymyui (online testing)
Prototype	Number of Participants	Number of tasks	Number of Participants	Number of tasks
A	10	10	10	8
B	10	10	10	8
C	10	10	10	8
D	10	10	10	8

Although a within-subjects testing design has the advantage of having each participant work with each prototype, it did not seem to be the best approach in this case for the following reasons.

First, since each online test is limited to 20 minutes, there is only time to have a participant work with one prototype.

Second, although the prototypes differ significantly in some respects (e.g., layout of menus), one of the key difference is how the menu is activated using a mouse. Because there are four prototypes and an in-house testing session is limited to 60 minutes, it would not be possible to give each participant adequate experience with each prototype. More importantly, if we attempted to switch among the four prototypes, this would likely change participants’ behavior because we would have to inform them that a new approach was going to be tested.

Finally, another drawback to having participants work with all the prototypes is that it will be difficult to completely counterbalance the order of the prototypes and the tasks that are completed for each one. For example, if the counterbalancing is not perfect, it’s possible that simpler tasks might be paired more frequently with a given prototype, which would lead to the erroneous conclusion that a prototype’s functionality is better, when it may be largely a function of the task difficulty.

Therefore, the planned approach will be to have 10 participants complete 10 tasks for each prototype in the in-house test. Participants will be recruited based on computer experience (beginning to intermediate vs. expert user) and age (under 35 vs. 35 or older). The BLS recruiter and researchers will work to see that participants are distributed as evenly as possible among the prototypes on these three characteristics. In-house participants will receive $40 per session.

The usability testing will be conducted in-house using a standard protocol, as well as online, using a testing service (Trymyui), which has been used in past usability tests and for unmoderated cognitive interviewing.¹

The cost for each online session varies depending on the number of sessions ordered, but is usually in the $27-30 range. The online tester receives $10 of this amount, with the rest going to Trymyui. Trymyui provides an online, interactive development site for preparing and testing protocols, and also stores videos of each testing session. The online usability testing service is viewed as a supplemental source of usability data and has the following advantages:

• Ease of recruiting: a diverse user population can be recruited with specific characteristics (for example, age and computer skill)

• Links to high quality video recordings are provided that can be shared within BLS. Moreover, portions of the videos can be edited and built into presentations.

• Pricing is competitive versus other sources.

• Results are extremely timely (often start coming within 2-3 hours of a test being ordered)

• Participants tend to provide excellent “think aloud” feedback, for example, they provide well articulated ‘first impressions’ and extensive think-aloud comments about usability and design features.

• Written feedback is obtained from participants to supplement verbal feedback

As with any research approach, there are also disadvantages. These include the following:

• Each online session is limited to 20 minutes.

• Instructions must be clear and precise, and “task restart” instructions are critical when participants make navigation errors.

• Since there is no moderator to intervene if the participant goes off path (for example, due to a navigation error), participants must carefully follow the instructions to restart at the proper place for the next task.

• No automated measures of time-on-task or success are produced. These must be manually derived.

Despite the drawbacks, past experience has convinced us that online testing adds a valuable new tool to our usability testing arsenal.

Examples of Tasks

Participants will be asked to search for information starting from the home page of BLS.gov. The following tasks will be used in the in-house testing. A sub-set of these tasks will be used in the online testing.

The general instructions to a participant will be to start on the home page, and without using the search box, complete each of the following tasks (Note: Tasks were selected so that they could be completed most easily by using the drop-down menus on the home page. However, we don’t want to tell participants directly to use the menus because this will distort normal behavior. Instead, we want them to discover this on their own.)

1. Find a publication called the Occupational Outlook Quarterly. This is an online magazine about jobs and careers.

2. Find the section of this website that provides useful information for students and teachers.

3. What is the average yearly wage for all people working in New York state?

4. BLS counts the number of mass layoff events in the United States. Find out how many mass layoff events occurred during the month of January 2012

5. Find the section on this website that provides information about how Americans spend their time.

6. Find the section on this website that shows how the productivity of American workers has changed over time for different manufacturing industries.

7. Find information about the number of unemployed workers in different countries.

8. Find the section that provides information about efforts to measure the number of green jobs in the U.S.

9. Find the section that presents information about the number of injuries and illnesses in the workplace.

10. Find the section that presents information about the number of strikes by workers or lockouts of workers by employers.

Protocol for In-House Testing

When usability testing is conducted in the lab in a face-to-face setting, participants will:

1. Be briefed about the study and its objectives and complete a consent form. Consent forms will be obtained from the parents or guardians of students, ages 14-17 (see Attachment 2). In addition, students will be asked to sign a standard consent form before the study is started (see Attachment 3).

2. Complete a series of usability tasks.

3. Participate in a general debriefing, including completing a brief questionnaire to obtain some basic demographic and background information about each participant and ratings of the website using the SUS scale (Attachment 5).

The in-person testing protocol is shown in Attachment 4. The procedure will be modified for online testing.

Online Testing

The online testing protocol is also shown in Attachment 4.

Analysis

Analysis will focus on time taken to complete tasks, task success, comments made during the tasks, and the follow-up debriefing session.

A checklist used to summarize data collected in the online testing is shown in Attachment 6.

III. Burden Hours

A maximum of 54 burden hours is expected. We estimate an in-house session will last an average of 45 minutes, and not exceed an hour. Online sessions are limited to 20 minutes.

IV. Data Confidentiality

Participants in the in-house testing will be informed as to the voluntary nature of the study. These participants will be informed that the study will be used for internal purposes to improve the design of the BLS.gov website. In house participants, will be notified that in accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Participants obtained through Trymyui sign separate agreements with Trymyui, and although BLS follows steps to ensure that their privacy is maintained, no pledge of confidentiality is provided to these respondents.

V. Attachments

Attachment 1: Illustrations of Drop-Down Menus

Attachment 2: Parent/Guardian Consent Form

Attachment 3: Participant Consent Form

Attachment 4: Protocol for In-House and Online Usability Testing

Attachment 5: Debriefing and Rating Questions for In-House Participants

Attachment 6: Checklist for Evaluating Online Testing

Attachment 1 - Illustrations of Drop-Down Menus

Prototype A – Click on/off Option

• Need to click on tab to open, and click on X to close.

Prototype B – Compact Menus

• Dragging mouse over menu will open it

• Menu closes when you move mouse off it (no clicking necessary)

• Higher-level categories are displayed (not as many sub-categories displayed)

Prototype C – Same as current menu, but with hover delay

• Menu drops, with slight delay, when you move mouse over it

• Moving mouse off menu makes it disappear (no clicking necessary)

• Have to scroll to see entire menu

Prototype D – Current menu (baseline)

• Current menu on BLS.gov

• Menu drops immediately (no delay) with mouseover

• Moving off menu makes it disappear quickly

• Have to scroll to see entire menu

ATTACHMENT 2: Parent/Guardian Consent Form

Parental Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its web-based materials. This specific study is designed to determine the effectiveness of menus provided for navigation throughout the BLS.gov website.

During this research your child will be observed by a BLS researcher, comments will be audiotaped, and a computer program will be used to record entries and timing data for later analysis. Please note that this is a not a test of your child’s abilities. Instead, we are interested in how well the website is designed and how easy it is to use. Any difficulty your child might encounter with certain tasks will provide us with information we can use to redesign the website to make it easier to use.

Participation in this research project is voluntary, and your child has the right to stop at any time. We estimate the session will last an average of 45-60 minutes. As a thank you for making the effort to come into our lab and participate, we will give your child a check for $40. If you would like to talk with a researcher before giving your permission, please see the contact information below. If you agree to allow your child to participate, please sign below.

The BLS, its employees, agents, and partner statistical agencies, will use the information your child provides for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your child’s responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 28, 2015.

-------------------------------------------------------------------------------------------------------------------------------

I have read and understand the statements above, and I give my consent for my child to participate in this study.

___________________________________ ___________________________

Parent’s or Guardian’s signature Date

___________________________________ Contact Information for BLS Researchers:

Parent’s or Guardian’s printed name Dr. Bill Mockovak, 202-691-7414

OMB Control Number: 1220-0141

Expiration Date: 2/28/2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

ATTACHMENT 3: Participant Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its web-based materials. This specific study is designed to determine the effectiveness of menus provided for navigation throughout the BLS.gov website.

During this research you will be observed by a BLS researcher, comments will be audiotaped, and a program will be used on the computer to record entries and timing data for later analysis.

We estimate it will take you an average of 45-60 minutes to participate in this research. Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Please note that this is a not a test of your abilities. Instead, we are interested in how well the website is designed and how easy it is to use. Any difficulty you might encounter with certain tasks will provide us with information we can use to redesign the website to make it easier to use.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 28, 2015.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 2/28/2015

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Attachment 4 - Protocol for In-House and Online Usability Testing

In-House Introduction

• Thank you for coming in today. Before we begin, have you participated in one of our studies before? (a yes answer reduces the amount of general background information about the purpose of lab studies and BLS that is provided).

• Background. Let me give you some background for what you will be doing today. The Bureau of Labor Statistics, or BLS, would like to make it as easy as possible for people to find information on its website. Therefore, I am going to ask you to search for some information on the BLS website. You will be presented with a series of tasks and asked to search for a specific section on the website or find information to answer a question.

• Now this next point is important. It’s easy to think that we’re evaluating you because we’re going to ask you to complete tasks, and I’ll be watching, but we’re really not evaluating you. We’re evaluating the website. We might have done a great job designing the website, or we might have done a poor job. The only way we can find out is to have a wide variety of people use it. So, please keep that in mind. We are not evaluating you. So relax and try to have fun today.

• Also, when some people come into our lab, they view whatever we ask them to do as a personal challenge, and they really want to do their best. That’s a good attitude up to a point, but what we’re trying to do is simulate what you would do if you were at home, or somewhere else, and you came to the BLS website to find some information. We want you to try as hard as you would if you were at home, but if you reach a point where if you were at home you would just skip the task and go to another website, please do that. This is important information for us. If you skip a task, it tells us that we haven’t made the website easy enough to use.

• Also, we use software on the computer to record what you are doing. For example, where you move the mouse, what you click on, and so forth. This just helps us summarize what people did. When we produce reports, we do not identify individuals.

• Talk aloud. One more point. As you search for information on the website, it would really be helpful to me if you would talk out loud to explain what you are doing and why. Some people feel very self conscious doing this, but it really helps us determine what you are thinking, and it gives us clues for why a task may be difficult. We have found it helps to give people a little practice doing this to warm up, so before we begin, let’s practice a bit. What I want you to do is sit back now and describe what you see when you unlock the door to your apartment or home. Just describe the first room you see after you open the door, including tables, chairs, pictures on the wall, and so on. (Allow description)

• Do you have any questions before we begin?

Consent Form

• Before we begin, I have a consent form that I would like you to read and sign. This acknowledges that you are participating freely in this study.

Tasks

Now I’m going to walk you through a couple of practice tasks on the computer, so you can see how they will be presented.

Complete 2 practice tasks (mention not to use search box)

Okay, from this point on, you are on your own, so work at your own pace, but please remember to talk out loud about. Tell me what you are thinking and why you are doing something. If you get stuck, feel free to ask me a question, and if you get frustrated and would skip a task, go ahead and skip it.

One more thing before we begin, remember not to use the search box. We’re not interested in that today.

Any questions before we begin?

Online Protocol

Note: Trymyui testers are thoroughly trained in “talking aloud.” Also, no training is required on using their system.

Introduction for Tasks

The Bureau of Labor Statistics (BLS) would like to make it as easy as possible for people to find information on its website. This voluntary study is being collected under OMB No. 1220-0141 and it is estimated it will take you no longer than 20 minutes to complete. You will be presented with a series of tasks and asked to answer a question or search for a specific page on its website. However, this next point is important. When looking for something, please do not use the search box. Make believe it’s not even there. Also, if you find that you cannot complete a task, stop and move to the next task. Please return to the starting page (home page) for each new task. Have fun!

Tasks

1. Find a publication called the Occupational Outlook Quarterly. This is an online magazine about jobs and careers.

2. Find the section of this website that provides useful information for students and teachers.

3. What is the average yearly wage for all people working in New York state?

4. BLS counts the number of mass layoff events in the United States. Find out how many mass layoff events occurred during the month of January 2012

5. Find information comparing the number of people unemployed in different countries.

6. Find the section that provides information about efforts to measure the number of green jobs in the U.S.

7. Find the section that presents information about the number of injuries and illnesses in the workplace.

8. Find the section that presents information about the number of strikes by workers or lockouts of workers by employers.

Written Feedback

You can ask for written feedback on four questions at the conclusion of the tasks.

1. Overall, how easy or difficult were these tasks to complete? Were they very easy, easy, neither easy nor difficult, difficult, or very difficult? How easy do you think these tasks would be for an average American citizen using this website?

2. Please tell us what you think about the drop-down menus that appear at the top of each page. For example, did you like the way they worked? Did you like the formatting and the choice of colors?

3. Please give us more reactions to the drop-down menus at the top of each page. Did the lists that were shown make sense to you? How do these menus compare with other websites you have used?

4. Overall, how easy or difficult were the menus to use? Would you say they were very easy, easy, neither easy nor difficult, difficult, or very difficult?

Attachment 5 – Debriefing and Rating Questions for In-House Participants

1. Overall, how easy or difficult were these tasks to complete?

 Very easy

 Very

 Neither easy nor difficult

 Difficult

 Very difficult

2. How easy or difficult was it to navigate, or move around, on this site?

 Very easy

 Very

 Neither easy nor difficult

 Difficult

 Very difficult

3. Overall, how easy or difficult were the menus to use?

 Very easy

 Very

 Neither easy nor difficult

 Difficult

 Very difficult

Debriefing Questions

1. Please tell us what you think about the drop-down menus that appear at the top of each page. For example, did you like the way they worked? Did you like the formatting and the choice of colors?

2. Please give us more reactions to the drop-down menus at the top of each page. Did the lists that were shown make sense to you? How do these menus compare with other websites you have used?

System Usability Scale

Strongly Strongly

disagree agree

1. I think that I would like to use this

website frequently

2. I found the website unnecessarily

complex

3. I thought the website was easy

to use

4. I think that I would need the

support of a technical person to

be able to use this website

5. I found the various functions in

this website were well integrated

6. I thought there was too much

inconsistency in this website

7. I would imagine that most people

would learn to use this website

very quickly

8. I found the website very

cumbersome to use

9. I felt very confident using the

website

10. I needed to learn a lot of

things before I could get going

with this website

Using SUS

The SUS scale is generally used after the respondent has had an opportunity to use the website being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time.

All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the center point of the scale.

Attachment 6 – Checklist for Evaluating Online Testing

Prototype - __________

Task	Location of First Click	Total Number of Clicks	Success 1=Yes 2=No	Time to Complete (success only)	# of Errant Clicks
1
2
3
4
5
6
7
8
9
10

Location of First Click:

SA –Subjects Areas

DT – Database & Tools

P- Publications

ER – Economic Release

B – Beta

E - Elsewhere

1 Trymyui: http://www.trymyui.com/