Usability Test for the OOH Website - Unemployed

June 2, 2009

NOTE TO THE REVIEWER OF:	OMB CLEARANCE #1220-0141 “Cognitive and Psychological Research”
FROM:	William Mockovak Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of materials for usability test of the Occupational Outlook Handbook website with unemployed data users

Please accept the enclosed materials for approval under the OMB clearance package #1220-0141 “Cognitive and Psychological Research.”

This research is an extension of previous research approved on May 19, 2009 (NOA 5-19-09). That previously approved research covers a focus group with middle school students to be held on June 12^th. The current research request was originally planned as a focus group, but considering the nature of the information required, it was decided that a usability test, accompanied by structured and unstructured questions, would provide more useful information.

Therefore, in accordance with our agreement with OMB, I am submitting a brief description of this study. We estimate 20 burden hours will be required for this research.

If there are any questions regarding this project, please contact Bill Mockovak at (202) 691-7414 or [email protected].

Attachments

1. Introduction and Purpose

This research is an extension of previous research approved on May 19, 2009 (NOA 5-19-09). That previously approved research described a focus group with middle school students to be held in mid June. The current research request was also originally planned as a focus group, but after further deliberation about the nature of the information required, it was decided that a usability test, accompanied by structured and unstructured questions, would provide more useful (for redesign purposes) information about the OOH website.

2. Research Design

A combination survey, usability test, and individual interview will be used to determine what information unemployed workers are interested in when considering potential jobs, and how they handle and react to information-seeking tasks on the Occupational Outlook Handbook (OOH) website.

3. Participants

The participants will be 20 unemployed workers, age 21 or older, recruited from local D.C. unemployment offices. An attempt will be made to obtain as diverse a group as possible.

4. Payment

Participants will receive $40 for an hour-long session.

5. Burden Hours

We anticipate that 20 adults will each complete a self-administered survey, a usability test, and a brief debriefing session. This research will be conducted in the BLS cognitive laboratory.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used to help staff of the Occupational Outlook Handbook better decide what information is of interest to adult job seekers and how it might be presented effectively. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

7. Attachments

A – Usability testing protocol

B – Announcement asking for volunteers to participate in usability testing

C – Consent form

Attachment A – Usability Testing Protocol

Introduction

• Thank participant for coming.

• Inform participant of videotaping.

• Get signed consent form.

• Introduce BLS – We collect data about employment in the U.S., including the monthly unemployment estimate that you may hear about in the news each month. We also publish the Consumer Price Index, which is a measure of inflation.

• However, in addition to business surveys, we also produce products that are designed to help people plan for a career. You will be looking at some of these products today.

• Goal for today: We are talking with a variety of people who might use one of our websites to see if we can improve it. Today, I’d like to talk about how you might use this website to find information about jobs or careers. You’ll have an opportunity to work with our website, then we'll talk about your impressions. I want to emphasize that we are testing the website itself, not your ability to use a computer or to find information.

• Any questions?

General Questions

Before we begin, we would like to collect some background information about you. The information you provide will be kept confidential and not linked to your name in any way.

1. Is there a job or type of work that is of most interest to you?

 No (skip to 3)

 Yes (Please describe the job/type of work)

2. What specific things about this job or type of work appeals to you?

3. Is there a job or type of work that you want to avoid?

 No (skip to 5)

 Yes (Please describe the job/type of work)

4. What specific things make you want to avoid certain types of work?

4. Please think about a job that you would like to do. If you could talk to someone currently doing that job, what questions would you like to ask that person about the job?

4. What comes to mind when you hear the term “work environment” for a job.

4. Using the 10-point scale below, where 1 is “not important” and 10 is “important,” please indicate for each item how important that type of information is to you when you are considering a job or type of work. (If you are not sure what something describes, skip the item)

Not important Important

1 5 10

How important is this type of information about a job to you?
What the workplace setting is (for example, outside, in an office)
How common work-related injuries are
How much education is required
How workers get trained or learn job skills
Benefits provided (for example, life insurance, health insurance, retirement plans, etc.)
What jobs are similar or related
What a typical career path might be
Whether the number of jobs is growing or declining in the short-term (next 1-2 years)
Whether the number of jobs is growing or declining in the long-term (next 10 years)
Factors that will cause changes in the number of jobs in this field
How stable employment is (job security)
How easy or difficult it is to find a job
A written summary about a ‘day in the life’ of someone doing this job
A video about a ‘day in the life’ of someone doing this job
An interview with someone currently working in the job
Specific skills required (for example, typing, math, language, physical strength)
How many workers there currently are in this field
Advice about finding a job in this field
What the starting pay or salary is
What the job pays after different amounts of time (for example, 5, 10, 15 years)
What the history of an occupation is
What types of personalities tend to do best in this type of work
How many hours per week a person usually works

Background Questions

Please answer the following background questions.

6. What is your age? ________

6. What is your gender?

 Male

 Female

6. What is your race? (check one)

 American Indian or Alaska Native

 Asian

 Black or African American

 Native Hawaiian or Other Pacific Islander

 White

 Other (please explain)

6. Are you Hispanic or Latino?

 No

 Yes

6. What is the highest level of education you have completed?

 Some high school

 Some high school, but with technical training

 High school graduate or GED

 Some college, but no degree

 Technical training past high school

 Associate college degree (2-year program, for example, A.A. or A.S.)

 College degree (for example, B.A. or B.S.)

 Some post-graduate education, but no degree

 Post-graduate degree (for example, M.A., M.S., Ph.D., or professional

degree)

Additional Questions

6. Have you looked at any materials, either printed or on the Internet that provide information about possible jobs or choosing a career? The products can be from the Bureau of Labor Statistics or anyone else.

 No (go to question 13)

 Yes (Go to next question)

6. (If yes) Do you remember what materials or website you looked at?

 No, I don’t remember (go to question 13)

 Yes (Please describe)

6. Had you ever heard of the Occupational Outlook Handbook before volunteering for this session?

 No (go to end)

 Yes

 Don’t know, not sure (go to end)

6. Have you ever looked at the website for the Occupational Outlook Handbook?

 No (go to end)

 Yes

 Don’t remember, not sure (go to end)

6. What kinds of information were you looking for in the Occupational Outlook Handbook?

Brief Introduction to the Occupational Outlook Handbook Website

This is the website that you are going to be working with today.

In a little while, I will be asking you to search for information from this page (BLS.gov/oco). Each task that you will be asked to do will start on this page.

Usability Tasks (start UTE)

As I mentioned, I’m going to ask you to complete a few tasks on this website, so that you can become more familiar with it.

The tasks will appear at the top of the screen. Read each one, and do your best to complete it. When you finish a task, the computer should pop up a message saying you are finished. For example, let’s do a practice task that asks you to find a web page.

Practice Task 1

Go to the web page that provides general information about lawyers.

Practice Task 2

Go to the web page that provides general information about lawyers and using your mouse highlight the text that describes what education is required to become a lawyer.

Some tasks, like the one you just completed, will just ask you to go to a page and highlight some information using your mouse. For other tasks a multiple-choice question will appear that you will be asked to answer. Let’s try one of those. When you are ready to answer this question, click the "Answers" tab at the top of the screen.  If you don't know the answer to a question and want to skip a task, click the "Skip" tab at the top of the screen. 

Practice Task 3

Once again go to the web pages for a lawyer, and answer the following question.    Question:  How many years of formal education are required to become a lawyer?

Task 1

Go to the page that provides general information about construction jobs for laborers.

Task 2

Go to the page that provides general information about jobs for accountants.

Task 3

Go to the page that provides general information about nursing jobs.

Task 4

Go to the page that provides general information about cooks. Using your mouse, highlight the paragraph that describes what short-order cooks do.

Task 5

Find out what the job prospects are for elementary school teachers over the next few years. How fast will the number of these jobs grow in the next few years in comparison to all jobs? Answer the following question:

 Much slower than average

 Slower than average

 About as fast as average

 Faster than average

 Much faster than average

Task 6

Find out how most electricians learn their trade (acquire their skills). Use your mouse to highlight the right information.  Question: Which of the following is true?  Most electricians learn their trade ...

 By graduating from a 4-year technical or trade high school

 By serving in the military for 4 years

 By completing an apprenticeship program lasting 4 to 5 years

 By attending a 2-year technical college

Task 7

What type of education is recommended for someone who would like to be a chef? When you get to the relevant page, highlight the information.

Task 8

Find out if a license is required to sell real estate, like homes. Highlight the text that provides the information. Question: Which of the following answers is correct?

 A license is not required to sell real estate

 Only a few States require a license to sell real estate

 Most States require a license to sell real estate

 All States require a license to sell real estate

Task 9

Find and highlight the text that describes the alternative educational paths for becoming a registered nurse.  Question: What educational paths exist for becoming a registered nurse?

 Bachelor’s degree

 Associate degree

 Diploma from an approved nursing program

 All of the above

 None of the above

Task 10

Do you have a question about a job that you would like answered? If so, please tell the question to the session moderator and then see if you can find the answer.

Post-Usability Tasks

Following completion of the usability tasks, participants will be asked to complete the SUS scale (follows), then they will be asked some open-ended debriefing questions.

System Usability Scale

Strongly Strongly

disagree agree

1. I think that I would like to

use this system frequently

2. I found the system unnecessarily

complex

3. I thought the system was easy

to use

4. I think that I would need the

support of a technical person to

be able to use this system

5. I found the various functions in

this system were well integrated

6. I thought there was too much

inconsistency in this system

7. I would imagine that most people

would learn to use this system

very quickly

8. I found the system very

cumbersome to use

9. I felt very confident using the

system

10. I needed to learn a lot of

things before I could get going

with this system

Using SUS

The SUS scale is generally used after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time.

All items should be checked. If participants feel that they cannot respond to a particular item, they should mark the center point of the scale.

Follow-up Questions

1. What is your opinion about the amount of information provided about jobs on this website?

 Not enough information (Probe: What information was missing?)

 About the right amount of information

 Too much information

2. Would you rather see a lot of detailed information about jobs or just short summaries (a few paragraphs or sentences)?

 A lot of detailed information (the content as presented is fine)

 Short paragraphs

 Just short summaries (a few sentences)

General Open-Ended Questions (asked by facilitator)

a. So, tell me what you think about this website. What do you like about it?

b. What do you dislike about it? If you have visited similar sites, is there information

missing from our site that they provide?

c. What could we do to better make it meet your needs?

A

Volunteers Needed!

Earn $40 by helping us evaluate our website

The Bureau of Labor Statistics (BLS), which is part of the U.S. Department of Labor, needs adults (21 or older) to help us evaluate a website that provides information about jobs and careers.

Participate in a short survey, complete a few tasks about job-searching strategies, and then tell us what you like or dislike about the website. Help us design a better site and improve our career resources!

Important Facts:

• You must be currently unemployed and 21 or older.

• You will be reimbursed $40 (paid by check), for an hour session.

• You must have prior experience using the Internet.

• Sessions are held at the Bureau of Labor Statistics, which is near Union Station

(2 Massachusetts Ave, NE).

• Use the 1^st Street Visitors’ entrance to enter the building. A picture ID is required.

For more information, or to schedule a session:

Call Stella Godbolt (202) 691-7524

or E-mail [email protected]

ttachment b – Announcement for volunteers

Attachment C – Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of materials presented on one of its websites: the Occupational Outlook Handbook.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you will be audio and video taped, and you may be observed by researchers from BLS.

We estimate this session will last about an hour. Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 29, 2012.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: February 29, 2012

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

14