Usability Testing for the OOH website

March 1, 2010

NOTE TO THE REVIEWER OF:	OMB No. 1220-0141 Cognitive and Psychological Research
FROM:	William Mockovak, Director Behavioral Sciences Research Center, Office of Survey Methods Research (OSMR)
SUBJECT:	Usability testing of prototypes for the redesigned Occupational Outlook Handbook website

Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.”

In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research at least two weeks prior to the beginning of the study.

As a brief overview, this request covers two phases of data collection requiring a total of 90 burden hours:

Phase 1: April 2010

Phase 2: June- July 2010

If there are any questions regarding this project, please contact Bill Mockovak (202) 691-7414.

1. Introduction and Purpose

The BLS Employment Projections program produces a variety of outputs, ranging from technical materials to career information for kids. Its most widely used product is the Occupational Outlook Handbook, which accounts for about one-third of BLS.gov website's traffic (the Web version receives 6 to 9 million visits monthly).

Previous research conducted with the online Occupational Outlook Handbook (OOH) first focused on identifying major groups that use the OOH. Once these groups were identified, focus groups and usability testing were used to produce both qualitative and quantitative assessments of the current website (http://www.bls.gov/oco/) and to determine user requirements. These requirements were then used to develop prototypes for a new OOH website.

The purpose of this next round of research is to conduct a series of usability tests with user populations to determine the impact of the changes that have been implemented. These user tests will be conducted using both in-house and remote-usability testing in the OSMR lab, as well as in-person usability testing conducted at professional conferences.

II. Data Collection Methodology

Usability Testing: Objectives

The primary objective of the usability testing is to compare the performance of user groups on different prototypes of the OOH website.¹ Both quantitative and qualitative measures will be collected. For example, quantitative measures will include time to complete a task and success rate. Qualitative measures will include ratings of ease of use, preference ratings for the prototypes, and open-ended feedback about aesthetics, layout, formatting, etc.

OOH User Populations to be Tested

The following groups will be tested in the April through July 2010 timeframe:

Group	Estimated Number Phase 1	Estimated Number Phase 2	$25 incentive offered for remote testing
High school students	6	6	Yes
College students	6	6	Yes
High school career counselors	6	6	Yes
College career counselors	6	6	Yes
Adults (post college)	6	6	No
Attendees at professional conferences	0	30	No
Phase 1 Total	30	-
Phase 2 Total	-	60
Overall Total	90

Although we are unlikely to exceed the total number of participants, the exact breakdown by user group is only our best guess at this time. Actual numbers will depend on the success of recruiting efforts. Column 4 identifies those groups in which we would like to use a $25 incentive.

Importance of Incentives

Given ongoing budget constraints, our intention is to solicit voluntary (non-incentive) participation whenever possible. For example, we are fairly confident that we will be able to recruit an adequate number of volunteers when we attend professional conferences such as the American School Counselors Association and the National Career Development Association. We are hoping that members of these groups are interested enough in the product to volunteer their time.

However, we know from past attempts to organize focus groups that recruiting representatives from other groups is more difficult, especially if we desire a good show rate. For example, we conducted focus groups with Fairfax County high school career counselors and would like to follow up with a subsample of those participants. Because of the counselors’ schedules, it is likely they would only be able to participate in a remote usability testing situation during a lunch break or after school on their own time. We will begin by soliciting non-incentive participants, but if unsuccessful, we would like the option to offer a $25 incentive to cover the person’s time and to encourage their active involvement.

As far as students are concerned, our past experience has clearly revealed that we need to offer an incentive to obtain participants from this group.

In the case of college career counselors, as with the County high school career counselors, we would like to follow up with a subsample of those participants. However, the college career counselors were given an incentive to participate in our early focus groups.

We know from past studies that although we can struggle through and complete a study without incentives, offering incentives has several immediate benefits:

1. There is less likelihood people will miss the session.

2. Participants generally give more effort during a session when they know they are receiving an incentive.

3. We can usually recruit a more representative group of participants.

The first point is especially important because it takes time to prepare for a remote testing session. We also encourage members of the development team to observe testing sessions, so it is extremely frustrating for everyone involved when a participant does not show. It is also extremely costly since the staff time wasted when someone does not show easily exceeds the cost of the incentive.

Test Phases

We anticipate two separate testing phases. Phase 1 will begin approximately April 1, and last through the month. After analysis of the data and redesign of the prototypes, Phase 2 will begin on June 1, and last through July. Our objective is to arrive at a final design by September 1.

Tasks

Participants will be asked to complete a series of information-seeking tasks starting from the home page of the Occupational Outlook Handbook or from a Google search list (results of a Google search). Sample tasks might be to determine the educational requirements for a registered nurse, the starting salary for hair dressers, and the number of different specializations.

In-Person Protocol

When usability testing is conducted in a face-to-face setting, participants will:

1. Be briefed about the study, read a description of the study (or both), and complete a consent form. Consent forms will be obtained from the parents or guardians of students, ages 14-17 (see Attachment 2). In addition, students will be asked to sign a standard consent form before the study is started (see Attachment 3).

2. Complete a brief questionnaire to obtain some basic demographic and background information about each participant.

3. Complete a series of usability tasks with the prototypes (the order of presentation of the prototypes will be counterbalanced).

4. After each task, the participant will rate the difficulty of the task.

5. After all tasks are completed, the participant will complete the SUS scale (see Attachment 1).

6. The participant will be shown the other prototypes, given an opportunity to review them, and asked for general reactions. A general debriefing will follow.

The procedure will be modified as follows for remote testing.

Remote Testing

1. Remote testing is coordinated using an application, called UserVue, produced by TechSmith. TechSmith’s privacy policy is shown in Attachment 5. We believe current consent procedures cover the main points of TechSmith’s policy.

2. Participants will be contacted in advance and mailed a consent form and, if an incentive is involved, a BLS administrative form (SF-1034), which is required to confirm receipt of convenience checks. A handout will also be enclosed or sent via e-mail that explains the purpose of the study and the participant’s role (see Attachment 4).

3. To avoid asking a participant to mail back the consent form and SF-1034 in separate envelopes, the BLS researcher will obtain either a taped verbal consent or ask the participant to respond to an e-mail stating that he or she is participating voluntarily in the scheduled session.

4. Participants will be able to see and navigate through the BLS web pages using an approach approved by BLS security (UserVue). Once this connection is made, the steps in the in-person procedure will be followed.

5. After the session is completed, participants will return the signed consent form and Form SF-1034 in a pre-addressed envelope.

6. Once OSMR receives the SF-1034 and signed consent form, we will mail a $25 check to the person.

Analysis

Analysis will focus on time taken to complete tasks, task success, and comments made during the tasks and in the follow-up debriefing session.

III. Burden Hours

A maximum of 90 burden hours is expected (this allows for some possible time overruns). We estimate it will take an average of 45 minutes and not to exceed 1 hour per respondent.

IV. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will be informed that the study will be used for internal purposes to improve the design of the Occupational Outlook Handbook (OOH). No information related to this study will be released to the public in any way that would allow identification of individuals except as permitted by the Privacy Act.

V. Attachments

Attachment 1: Pre- and Post-Evaluation Questions and SUS scale

Attachment 2: Parent/Guardian Consent Form

Attachment 3: Participant Consent form

Attachment 4: Information Sheet

Attachment 5: TechSmith’s Privacy Policy

ATTACHMENT 1 – Pre and Post Evaluation Questions and SUS scale

Questions Completed Prior to the Usability Test

Please check the box below that best describes you.

 Career counselor, for students

 Career counselor, for adults (non-students)

 Teacher/educator

 Staff member in a career center

 Student, age 13-17

 Student, college

 Adult, 18 or older, not currently in school

 Adult, in graduate or technical school

 Other – Please describe

How would you describe your general level of experience working with the Occupational Outlook Handbook website?

 Very experienced

 Experienced

 Neither experienced nor inexperienced

 Inexperienced

 Very inexperienced

 Have never visited the website

What is your gender?

 Male

 Female

Rating Completed After Each Task

Please rate how easy or difficult it was to complete this task.

 Very easy

 Easy

 Neither easy nor difficult

 Difficult

 Very difficult

Ratings Completed at Conclusion of Session

Considering the limited amount of experience you have had with the different versions of the web pages, which would you prefer if you had to make a selection?

 Version A

 Version B

 Version C

Questions about features that distinguish the prototypes

Once the prototypes have been developed, and we know what the key differences are, we will include some questions about these features. However we will begin with an open-ended question, as follows: Tell me about the design of the web pages that you saw today. What were your reactions to them? What did you like? What did you dislike?

SUS Scale

Starts on next page.

System Usability Scale

Strongly Strongly

disagree agree

1. I think that I would like to

use this website frequently

2. I found the website unnecessarily

complex

3. I thought the website was easy

to use

4. I think that I would need the

support of a technical person to

be able to use this website

5. I found the various functions in

this website were well integrated

6. I thought there was too much

inconsistency in this website

7. I would imagine that most people

would learn to use this website

very quickly

8. I found the website very

cumbersome to use

9. I felt very confident using the

website

10. I needed to learn a lot of

things before I could get going

with this website

Using SUS

The SUS scale is generally used after the respondent has had an opportunity to use the website being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time.

All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the center point of the scale.

ATTACHMENT 2: Parent/Guardian Consent Form

Parental Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its web-based materials. This specific study is intended to compare the effectiveness of different designs of web pages for the online version of the Occupational Outlook Handbook, which provides a variety of career information.

During this research your child will be observed by a BLS researcher, comments will be audiotaped, and a computer program will be used to record entries and timing data for later analysis. Please note that this is a not a test of your child’s abilities. Instead, we are interested in how well the website is designed and how easy it is to use. Any difficulty your child encounters with certain tasks will provide us with information we can use to redesign the website to make it easier to use.

Participation in this research project is voluntary, and your child has the right to stop at any time. We estimate the session will last an average of 45 minutes. As a thank you for participating, we will give your child a check for $25. If you would like to talk with a researcher before giving your permission, please see the contact information below. If you agree to allow your child to participate, please sign below.

The BLS, its employees, agents, and partner statistical agencies, will use the information your child provides for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your child’s responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 29^th, 2012.

-------------------------------------------------------------------------------------------------------------------------------

I have read and understand the statements above, and I give my consent for my child to participate in this study.

___________________________________ ___________________________

Parent’s or Guardian’s signature Date

___________________________________ Contact Information for BLS Researchers:

Parent’s or Guardian’s printed name Dr. Jean Fox, 202-691-7370

Dr. Bill Mockovak, 202-691-7414

OMB Control Number: 1220-0141

Expiration Date: 2/29/2012

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

ATTACHMENT 3: Participant Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its web-based materials. This specific study is intended to compare the effectiveness of different designs of web pages for the online version of the Occupational Outlook Handbook, which provides a variety of career information.

During this research you will be observed by a BLS researcher, comments will be audiotaped, and a program will be used on the computer to record entries and timing data for later analysis.

We estimate it will take you an average of 45 minutes to participate in this research. Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Please note that this is a not a test of your abilities. Instead, we are interested in how well the website is designed and how easy it is to use. Any difficulty you encounter with certain tasks will provide us with information we can use to redesign the website to make it easier to use.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 29^th, 2012.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 2/29/2012

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Attachment 4 – Information Sheet

Purpose of this study	You will be asked to work with different web page designs being considered for the online version of the Occupational Outlook Handbook. This study will determine which design is most effective.
Your participation is: Voluntary and Confidential	You will be asked to sign a consent form before participating in this study. That consent form emphasizes the following points: Your participation is completely voluntary. You can stop at any time. You will be observed and your key strokes will be recorded for later analysis. Your responses will be kept confidential. Your name will not appear in any reports.
What will you be doing?	You will be completing tasks that visitors to the online Occupational Outlook Handbook are likely to attempt. For example, you might be asked to find an occupation or to determine what type of education is required for an occupation.
Is this a test of my ability?	We are not testing you! We are testing the website. Observing problems you encounter will help us improve the website.
How long will the session last?	Times will vary, but the session should last about an average of 45 minutes and not exceed 60 minutes.
Why is my participation important?	We are trying to obtain feedback from a variety of people who might use the online version of the Occupational Outlook Handbook.
What if I have questions about this study?	Call or e-mail one of the following persons: Bill Mockovak; 202-691-7414, [email protected] Jean Fox; 202-691-7370, [email protected]

Attachment 5 – TechSmith’s Privacy Policy

TechSmith's Use of Cookies and E-mail

TechSmith collects cookies on our Web site to capture information about page visits. This information is anonymous and TechSmith uses this information only internally - to deliver the most effective content to our visitors. Information from the cookies is used to gauge page popularity, analyze traffic patterns on our site and guide development of other improvements to our site. TechSmith does not require that you accept cookies, however, some functionality on our Web site, our product or service check-out process, and products and services may be disabled if you decline to accept cookies. You can set your browser to notify you when you receive a cookie, giving you the chance to decide whether or not to accept it. You may also change your cookie settings through preferences options in our products and/or services, where applicable. TechSmith never gives away information about our users. If you choose to e-mail us and provide personally identifiable information about yourself, TechSmith will use this information only to respond to your inquiry. TechSmith will not sell, rent or otherwise disclose that information to third parties unless such disclosure is necessary for the purposes set forth in this Policy, by law or a policy or notice contained or associated with a specific TechSmith product or service. If you have any questions about our privacy practices, please contact our webmaster.

1 We are currently planning to test 3 prototypes, but since the team is still evaluating alternatives, this number may change.

0