Research into use of the Occupational Outlook Handbook

August 21, 2008

NOTE TO THE REVIEWER OF:	OMB No. 1220-0141 Cognitive and Psychological Research
FROM:	William Mockovak, Director Behavioral Sciences Research Center, Office of Survey Methods Research (OSMR)
SUBJECT:	Submission of materials for research into use of the Occupational Outlook Handbook (updated proposal)

Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research at least two weeks prior to the beginning of the study.

Please note that this submission is a revision of a package originally submitted on October 3, 2007. That package included both usability testing and the use of focus groups. This package only describes a follow-up usability test.

We are requesting 20 additional participants for further usability testing of alternative prototypical screens used on the OOH website. The usability testing protocol remains the same. The 20 additional participants will result in a total of 40 burden hours for this research (20 hours from the previous research). We hope to recruit the following participants:

• 5 job seekers (any age)

• 5 older workers (30+), some of whom might be considering a job switch

• 5 high school students (ages 13-18)

• 5 college students (18-22)

If there are any questions regarding this project, please contact Bill Mockovak (202) 691-7414.

1. Introduction and Purpose

The BLS Employment Projections program produces a variety of outputs, ranging from technical materials to career information for kids. Its most widely used product is the Occupational Outlook Handbook, which accounts for about one-third of the BLS.gov website's traffic. Organizationally, the Employment Projections program includes two divisions, the Division of Industry Employment Projections and the Division of Occupational Outlook.

The EPP program is at the very beginning stages of strategic planning. In preparation for a strategic planning session, tentatively scheduled for January 2008, the EPP program wants to gather input from its major customers, which includes students (middle, high school, and college), educational and career counselors, teachers, state labor marker information staff, academic and government researchers, career information developers, and the media. The purpose of this research is to begin to obtain feedback from two of those groups: high school students and high school counselors.

Since about a third of the visitors to the BLS.gov website visit the Occupational Outlook Handbook, the purpose of this study will be to conduct a usability analysis of the website followed by a debriefing.

II. Data Collection Methodology

Plan for Usability Testing: Objectives

The objectives of the usability test are to gather information on the ease, comfort level, and performance of participants accessing the Occupational Outlook Handbook (OOH) website. In addition, feedback about strengths and weaknesses of the website and alternative screen designs will be obtained.

Scenarios

Usability tests will be structured around tasks that visitors to the Occupational Outlook Handbook typically desire to accomplish. For example, tasks such as the following will be used:

1. From the BLS.gov home page, find the Occupational Outlook Handbook.

2. From the Occupational Outlook Handbook website, answer the following questions about (for example) electrical engineers: What type of degree is required? What is the job outlook over the next 10 years for this profession? What subjects will a person need to be good in to succeed in this field? What are some related occupations?

In order to get a better idea of what respondents are thinking as they attempt each task, and to help delineate between errors of omission and commission, the usability protocol will employ the “think aloud” technique, which asks respondents to provide a running monologue of their thoughts as they attempt to complete tasks. The researcher will brief the respondent on how to provide this type of protocol. They will be encouraged to say whatever comes to mind, but told that the researchers are especially interested in hearing about any aspects of the system that the student finds difficult, why they cause problems, and how they might try to resolve these difficulties. At the completion of the usability task, students will be asked to complete an evaluation form (see Attachment 2) and will be debriefed by the researcher using open-ended questions about what the student liked or did not like about the website and alternative screen designs.

Each usability session will last about one hour. In a previous study, we recruited 20 students. We will attempt to recruit 20 participants in this study grouped as follows:

• 5 job seekers (any age)

• 5 older workers (30+), some of whom might be considering a job switch

• 5 high school students (ages 13-18)

• 5 college students (18-22)

Participants

Participants will be recruited from a database of volunteers maintained at BLS. In addition, we may use other sources, such as Craig’s List, to try to identify job seekers who might be willing to participate in the test. Participants will be reimbursed $40 for a session that will last between 45 and 60 minutes depending on the skill level of the participant.

Consent forms will be obtained from the parents or guardians of all students, age 13-18. In addition, the students will be asked to sign a consent form before the study is started (see attachments).

Analysis

Analysis of the usability test will focus on time to complete tasks, task failures, comments made during the tasks, and an in-depth debriefing.

III. Burden Hours

A maximum of 20 burden hours is expected (this allows for some possible time overruns) for this second study, for a total of 40 burden hours for both studies.

IV. Data Confidentiality

Participants will be informed as to the voluntary nature of the study. Participants will be informed that the study will be used for internal purposes to improve the design of the Occupational Outlook Handbook (OOH). No information related to this study will be released to the public in any way that would allow identification of individuals except as permitted by the Privacy Act (see Attachment 3).

V. Attachments

Attachment 1: Post-test evaluation questions (SUS scale)

Attachment 2: Parental/Guardian Consent form

Attachment 3: Student Consent Form

ATTACHMENT 1 –– Post-Test Questions

System Usability Scale

Strongly Strongly

disagree agree

1. I think that I would like to

use this system frequently

2. I found the system unnecessarily

complex

3. I thought the system was easy

to use

4. I think that I would need the

support of a technical person to

be able to use this system

5. I found the various functions in

this system were well integrated

6. I thought there was too much

inconsistency in this system

7. I would imagine that most people

would learn to use this system

very quickly

8. I found the system very

cumbersome to use

9. I felt very confident using the

system

10. I needed to learn a lot of

things before I could get going

with this system

Using SUS

The SUS scale is generally used after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time.

All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the center point of the scale.

ATTACHMENT 2: Parental/Guardian Consent Form

Parental Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its printed and web-based materials. This study is intended to identify improvements to the Occupational Outlook Handbook, and related publications (for example, the Career Guide to Industries). These publications are designed to provide career information to a wide variety of people, including students and adults.

In this study, we are especially interested in how students, ages 13-18, react to and use versions of these materials that were developed for the World Wide Web. We are asking your permission to have your child participate in a usability study at the Bureau of Labor Statistics, which is located by Union Station. In a usability study, the student will be asked to complete a series of tasks, such as finding the website, locating certain occupations, and answering questions about those occupations. This is a not a test of the student’s abilities. Instead, we are interested in how well the website is designed and how easy it is to use. If students have difficulty with certain tasks, that will provide us with information we can use to redesign the website to make it easier to use.

The BLS, its employees, agents, and partner statistical agencies will use the information your child provides for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your child’s responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the next page of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research your child will be observed by a BLS researcher, and a program will be used on the computer to record entries and timing data for later analysis. Other observers may also be present, but if so, they will always be introduced to your child.

This research session will last no longer than an hour, and as a thank you for participating, we will give your child a check for $40. Your child’s participation in this research project is voluntary, and he or she has the right to stop at any time. If you agree to allow your child to participate, please sign below. If you would like to talk with a researcher before giving your permission, please see the contact information below.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above, and I give my consent for my child to participate in this study.

___________________________________ ___________________________

Parent’s or Guardian’s signature Date

___________________________________ Contact Information for BLS Researchers:

Parent’s or Guardian’s printed name Dr. Jean Fox, 202-691-7370

Dr. Bill Mockovak, 202-691-7414

Note:

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141, and it expires February 28, 2009.

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The BLS may release individually identifiable information to individuals designated as agents of the BLS in accordance with Public Law 107-347 to perform exclusively statistical activities. Individuals designated as agents of the BLS may be imprisoned for not more than 5 years or fined not more that $250,000 or both for any knowing and willful disclosure of respondent information to unauthorized persons. Such designated agents may include individuals from other sponsoring agencies; to contractors, grantees, and their employees or volunteers who are working on this study for the BLS and who need access to the information; or to the National Archives and Records Administration or the General Services Administration for records management purposes. Under written agreements to protect the confidentiality and security of individually identifiable information, the BLS may provide individually identifiable information to other researchers designated as agents of the BLS to conduct statistical research projects that further the mission and functions of the BLS.

ATTACHMENT 3: Student Consent Form

Student Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its printed and web-based materials. This specific study is intended to identify improvements to the Occupational Outlook Handbook, and associated career information (for example, the Career Guide to Industries).

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you will be audiotaped and may be videotaped. You may also be observed. Audiotaping and videotaping are done only to assist with the accuracy of our note taking, and your comments will not be associated with your name in any written reports. If you do not wish to be taped or observed, please inform the researcher.

We estimate it will take you an average of 60 minutes to participate in this research. Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Please note that this is a not a test of your abilities. Instead, we are interested in how well the website is designed and how easy it is to use. If you have difficulty with certain tasks, that will provide us with information we can use to redesign the website to make it easier to use.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2009.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

OMB Control Number: 1220-0141

Expiration Date: 2/28/09

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The BLS may release individually identifiable information to individuals designated as agents of the BLS in accordance with Public Law 107-347 to perform exclusively statistical activities. Individuals designated as agents of the BLS may be imprisoned for not more than 5 years or fined not more that $250,000 or both for any knowing and willful disclosure of respondent information to unauthorized persons. Such designated agents may include individuals from other sponsoring agencies; to contractors, grantees, and their employees or volunteers who are working on this study for the BLS and who need access to the information; or to the National Archives and Records Administration or the General Services Administration for records management purposes. Under written agreements to protect the confidentiality and security of individually identifiable information, the BLS may provide individually identifiable information to other researchers designated as agents of the BLS to conduct statistical research projects that further the mission and functions of the BLS.

9