Usability Test of the Occupational Handbook - college students

August 6, 2009

NOTE TO THE REVIEWER OF:	OMB CLEARANCE #1220-0141 “Cognitive and Psychological Research”
FROM:	William Mockovak Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of materials for usability test of the Occupational Outlook Handbook (OOH) website with college students

Please accept the enclosed materials for approval under the OMB clearance package #1220-0141 “Cognitive and Psychological Research.”

This research is a further extension of research initially approved on May 19, 2009 (NOA 5-19-09). That initial research request was approved for a focus group with middle school students that was held on June 12^th.

Since then, a follow-up usability study of the OOH website with unemployed adults (21 or older) was conducted. The current research request is for an extension of this usability test, but with college students as participants.

Therefore, in accordance with our agreement with OMB, I am submitting a brief description of this study. We estimate 10 burden hours will be required for this research.

If there are any questions regarding this project, please contact Bill Mockovak at (202) 691-7414 or [email protected].

Attachments

1. Introduction and Purpose

This research is an extension of usability research approved on June 2, 2009, and an initial research project (focus group with middle school children) first approved in May 2009 (see NOA 5-19-09).

The usability test approved on June 2^nd has been completed, and the program office has requested some additional data from another user group of interest: college students.

The protocols for both tests are very similar, with some minor differences. Changes have been made to the contextual interview that precedes the actual usability tasks to make the questions more relevant to college students, plus participants will be asked to rate the difficulty of tasks immediately after completion of a task.

2. Research Design

A combination contextual interview, usability test, and individual interview will be used to determine what information college students are interested in when considering potential jobs and careers, and how they handle and react to information-seeking tasks on the Occupational Outlook Handbook (OOH) website.

3. Participants

The participants will be 10 undergraduate college students recruited from the Washington, D.C. area.

4. Payment

Participants will receive $40 for an hour-long session.

5. Burden Hours

We anticipate that 10 students will each complete an interview, a usability test, and a brief debriefing session. This research will be conducted in the BLS cognitive laboratory.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will also be informed that the study will be used to help staff of the Occupational Outlook Handbook better decide what information is of interest to adult job seekers and how it might be presented effectively. Information related to this study will not be released to the public in any way that would allow identification of individuals except as prescribed under the conditions of the Privacy Act Notice.

7. Attachments

A – Usability testing protocol

B – Consent form

Attachment A – Usability Testing Protocol

Introduction

• Thank participant for coming.

• Inform participant of videotaping.

• Get signed consent form.

• Introduce BLS – We collect data about employment in the U.S., including the monthly unemployment estimate that you may hear about in the news each month. We also publish the Consumer Price Index, which is a measure of inflation.

• However, in addition to business surveys, we also produce products that are designed to help people plan for a career. You will be looking at some of these products today.

• Goal for today: We are talking with a variety of people who might use one of our websites to see if we can improve it. Today, I’d like to talk about how you might use this website to find information about jobs or careers. You’ll have an opportunity to work with our website, then we'll talk about your impressions. I want to emphasize that we are testing the website itself, not your ability to use a computer or to find information.

• Any questions?

General Questions

Before we begin, we would like to collect some background information about you. The information you provide will be kept confidential and not linked to your name in any way.

1. What is your current year in college?

 1^st year

 2^nd year

 3^rd year

 4^th year

 Graduate in 2009

2. What is your college major? (Any minors?)

3. When you think about working in a future job or career, what specific things or characteristics about the job or career appeal to you? What things draw you to a particular job or career? (Probe: Anything else?)

4. Are there jobs or careers that you definitely want to avoid if you have the choice?

 No (skip to 6)

 Yes (What is the name of the job/type of work that you want to avoid?)

5. What specific things about these jobs or careers do you want to avoid or are unappealing to you?

6. Please think about a career that you would like to pursue. If you could talk to someone currently in that career, what questions would you like to ask that person?

5. Using the 10-point scale below, where 1 is “not important” and 10 is “important,” please indicate for each item how important that type of information is to you when you are considering a job or type of work. (If you are not sure what something describes, skip the item)

Not important Important

1 5 10

How important is this type of information about a job to you?
What the workplace setting is (for example, outside, in an office)
How common work-related injuries are
How much education is required
How workers get trained or learn job skills
Benefits provided (for example, life insurance, health insurance, retirement plans, etc.)
What jobs are similar or related
What a typical career path might be
Whether the number of jobs is growing or declining in the short-term (next 1-2 years)
Whether the number of jobs is growing or declining in the long-term (next 10 years)
Factors that will cause changes in the number of jobs in this field
How stable employment is (job security)
How easy or difficult it is to find a job
A written summary about a ‘day in the life’ of someone doing this job
A video about a ‘day in the life’ of someone doing this job
An interview with someone currently working in the job
Specific skills required (for example, typing, math, language, physical strength)
How many workers there currently are in this field
Advice about finding a job in this field
What the starting pay or salary is
What the job pays after different amounts of time (for example, 5, 10, 15 years)
What the history of an occupation is
What types of personalities tend to do best in this type of work
How many hours per week a person usually works

Background Questions

Please answer the following background questions.

6. What is your age? ________

6. What is your gender?

 Male

 Female

6. Are you Hispanic or Latino?

 No

 Yes

6. What is your race? (please select one or more)

 American Indian or Alaska Native

 Asian

 Black or African American

 Native Hawaiian or Other Pacific Islander

 White

Additional Questions

6. Have you looked at any materials within the past 6 months, either printed or on the Internet that provide information about possible jobs or choosing a career? The products can be from the Bureau of Labor Statistics or anyone else.

 No (go to question 12)

 Yes (Go to next question)

6. (If yes) Do you remember what materials or website you looked at?

 No, I don’t remember (go to question 13)

 Yes (Please describe)

6. Had you ever heard of the Occupational Outlook Handbook before volunteering for this session?

 No (go to end)

 Yes

 Don’t know, not sure (go to end)

6. Have you ever looked at the website for the Occupational Outlook Handbook?

 No (go to end)

 Yes

 Don’t remember, not sure (go to end)

6. What kinds of information were you looking for in the Occupational Outlook Handbook?

Brief Introduction to the Occupational Outlook Handbook Website

This is the website that you are going to be working with today.

In a little while, I will be asking you to search for information from this page (BLS.gov/oco). Each task that you will be asked to do will start on this page.

Usability Tasks (start UTE)

As I mentioned, I’m going to ask you to complete a few tasks on this website, so that you can become more familiar with it.

The tasks will appear at the top of the screen. Read each one, and do your best to complete it. When you finish a task, the computer should pop up a message saying you are finished. For example, let’s do a practice task that asks you to find a web page.

Note: To obtain their immediate reactions, participants will be asked to rate each task on three criteria using a 7-point scale. The ratings include the following items:

• Overall, this task was … (where 1=very easy and 7=very difficult)

• I was satisfied with the ease of completing this task (where 1=strongly disagree and

7= strongly agree)

• I was satisfied with the amount of time it took to complete the task (where 1=strongly disagree and 7= strongly agree)

Practice Task 1

Go to the web page that provides general information about lawyers.

Practice Task 2

Go to the web page that provides general information about lawyers and using your mouse highlight the text that describes what education is required to become a lawyer.

Some tasks, like the one you just completed, will just ask you to go to a page and highlight some information using your mouse. For other tasks a multiple-choice question will appear that you will be asked to answer. Let’s try one of those. When you are ready to answer this question, click the "Answers" tab at the top of the screen.  If you don't know the answer to a question and want to skip a task, click the "Skip" tab at the top of the screen. 

Practice Task 3

Once again go to the web pages for a lawyer, and answer the following question.    Question:  How many years of formal education are required to become a lawyer?

Task 1

Go to the page that provides general information about construction jobs for laborers.

Task 2

Go to the page that provides general information about jobs for accountants.

Task 3

Go to the page that provides general information about nursing jobs.

Task 4

Go to the page that provides general information about cooks. Using your mouse, highlight the paragraph that describes what short-order cooks do.

Task 5

Find out what the job prospects are for elementary school teachers over the next few years. How fast will the number of these jobs grow in the next few years in comparison to all jobs? Answer the following question:

 Much slower than average

 Slower than average

 About as fast as average

 Faster than average

 Much faster than average

Task 6

Find out how most electricians learn their trade (acquire their skills). Use your mouse to highlight the right information.  Question: Which of the following is true?  Most electricians learn their trade ...

 By graduating from a 4-year technical or trade high school

 By serving in the military for 4 years

 By completing an apprenticeship program lasting 4 to 5 years

 By attending a 2-year technical college

Task 7

What type of education is recommended for someone who would like to be a chef? When you get to the relevant page, highlight the information.

Task 8

Find out if a license is required to sell real estate, like homes. Highlight the text that provides the information. Question: Which of the following answers is correct?

 A license is not required to sell real estate

 Only a few States require a license to sell real estate

 Most States require a license to sell real estate

 All States require a license to sell real estate

Task 9

Find and highlight the text that describes the alternative educational paths for becoming a registered nurse.  Question: What educational paths exist for becoming a registered nurse?

 Bachelor’s degree

 Associate degree

 Diploma from an approved nursing program

 All of the above

 None of the above

Task 10

Do you have a question about a job that you would like answered? If so, please tell the question to the session moderator and then see if you can find the answer.

Post-Usability Tasks

Following completion of the usability tasks, participants will be asked to complete the SUS scale (follows), then they will be asked some open-ended debriefing questions.

System Usability Scale

Strongly Strongly

disagree agree

1. I think that I would like to

use this system frequently

2. I found the system unnecessarily

complex

3. I thought the system was easy

to use

4. I think that I would need the

support of a technical person to

be able to use this system

5. I found the various functions in

this system were well integrated

6. I thought there was too much

inconsistency in this system

7. I would imagine that most people

would learn to use this system

very quickly

8. I found the system very

cumbersome to use

9. I felt very confident using the

system

10. I needed to learn a lot of

things before I could get going

with this system

Using SUS

The SUS scale is generally used after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time.

All items should be checked. If participants feel that they cannot respond to a particular item, they should mark the center point of the scale.

Follow-up Questions – These questions will be asked after the participant has completed the SUS ratings

1. What is your opinion of the usefulness of information presented about jobs and careers on this website?

 Very useful

 Somewhat useful

 A little useful

 Not at all useful

 Don’t know/Not sure

2. How would you rate the formatting and display of the information on this website?

 Excellent

 Good

 Neither good nor fair

 Fair

 Poor

3. When reading information about jobs or careers would you rather see a lot of detailed information as presented on the website, shorter summaries (a few paragraphs or sentences), or bullets?

 A lot of detailed information (the content as presented is fine)

 Short paragraphs that break up the content

 Bullets of information (just a few sentences about each key point)

General Open-Ended Questions (asked by facilitator)

a. So, tell me what you think about this website. What do you like about it?

b. What do you dislike about it? If you have visited similar sites, is there information

missing from our site that they provide?

c. What could we do to better make it meet your needs?

Attachment B – Consent Form

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of materials presented on one of its websites: the Occupational Outlook Handbook.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you will be audio and video taped, and you may be observed by researchers from BLS.

We estimate this session will last about an hour. Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires February 29, 2012.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: February 29, 2012

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

13