Research into use of the Occupational Outlook Handbook
Cognitive and Psychological Research
OMB Usab and Counselors 1019
Research into use of the Occupational Outlook Handbook
OMB: 1220-0141
October 3, 2007
NOTE TO THE REVIEWER OF: |
OMB No. 1220-0141 Cognitive and Psychological Research
|
FROM: |
William Mockovak, Director Behavioral Sciences Research Center, Office of Survey Methods Research (OSMR)
|
SUBJECT: |
Submission of Materials for research into use of the Occupational Outlook Handbook (OOH)
|
Please accept the enclosed materials for approval under the OMB clearance package “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research at least two weeks prior to the beginning of the study.
Participants will be recruited from Gonzaga High School in Washington, DC. The maximum number of burden hours is estimated to be 30.
If there are any questions regarding this project, please contact Bill Mockovak (202) 691-7414.
Introduction and Purpose
The BLS Employment Projections program produces a variety of outputs, ranging from technical materials to career information for kids. Its most widely used product is the Occupational Outlook Handbook, which accounts for about one-third of the BLS.gov website's traffic. Organizationally, the Employment Projections program includes two divisions, the Division of Industry Employment Projections and the Division of Occupational Outlook.
The EPP program is at the very beginning stages of strategic planning. In preparation for a strategic planning session, tentatively scheduled for January 2008, the EPP program wants to gather input from its major customers, which includes students (middle, high school, and college), educational and career counselors, teachers, state labor marker information staff, academic and government researchers, career information developers, and the media. The purpose of this research is to begin to obtain feedback from two of those groups: high school students and high school counselors.
Since about a third of the visitors to the BLS.gov website visit the Occupational Outlook Handbook, the purpose of the first study will be to conduct a usability analysis of the website followed by a debriefing. The second study will rely on in-depth interviews with high school counselors who are familiar with the Occupational Outlook Handbook, either the printed or Web version.
II. Data Collection Methodology
Plan for Usability Testing: Objectives
The objectives of the usability test are to gather information on the ease, comfort level, and performance of students accessing the Occupational Outlook Handbook (OOH) website. In addition, feedback about strengths and weaknesses of the website will be obtained.
Scenarios
Usability tests will be structured around tasks that visitors to the Occupational Outlook Handbook typically desire to accomplish. For example, tasks such as the following will be used:
From the BLS.gov home page, find the Occupational Outlook Handbook.
From the Occupational Outlook Handbook website, answer the following questions about (for example) electrical engineers: What type of degree is required? What is the job outlook over the next 10 years for this profession? What subjects will a person need to be good in to succeed in this field? What are some related occupations?
In order to get a better idea of what respondents are thinking as they attempt each task, and to help delineate between errors of omission and commission, the usability protocol will employ the “think aloud” technique, which asks respondents to provide a running monologue of their thoughts as they attempt to complete tasks. The researcher will brief the respondent on how to provide this type of protocol. They will be encouraged to say whatever comes to mind, but told that the researchers are especially interested in hearing about any aspects of the system that the student finds difficult, why they cause problems, and how they might try to resolve these difficulties. At the completion of the usability task, students will be asked to complete an evaluation form (see Attachment 2) and will be debriefed by the researcher using open-ended questions about what the student liked or did not like about the website.
We will attempt to recruit 20 students. Each usability session will last about one hour.
Plan for Individual Interviews: Counselors
Because they are knowledgeable about the Occupational Outlook Handbook and its design, counselors will be asked a detailed series of questions (see Attachment 1).
We will attempt to recruit four counselors. Each interview will last about one hour.
Participants
Participants will be recruited primarily from Gonzaga High School. We will also attempt to recruit students from local high schools in the District of Columbia to achieve a more diverse group of participants. Contact has already been made with the Headmaster of Gonzaga who has approved the voluntary participation of Gonzaga’s students and counselors in this effort. Participants will be reimbursed $40. Contact with some local D.C. schools has also been made, but without any success as of October 19, 2007.
Consent forms will be obtained from the parents or guardians of all students, age 13-18. In addition, the students will be asked to sign a consent form before the study is started (see attachments).
Analysis
Analysis of the usability test will focus on time to complete tasks, task failures, and comments made during the tasks. Analysis of the in-depth questionnaires will focus on the identification of major issues related to use of the Occupational Outlook Handbook.
III. Burden Hours
A maximum of 30 burden hours is expected (this allows for some possible time overruns).
IV. Data Confidentiality
Participants will be informed as to the voluntary nature of the study. Participants will be informed that the study will be used for internal purposes to improve the design of the Occupational Outlook Handbook (OOH). No information related to this study will be released to the public in any way that would allow identification of individuals except as permitted by the Privacy Act (see Attachment 3).
V. Attachments
Attachment 1: Draft Protocol for Counselors, Teachers, and Career Center Staff
Attachment 2: Post-test evaluation questions (SUS scale)
Attachment 3: Parental/Guardian Consent form
Attachment 4: Student Consent Form
ATTACHMENT 1 - Draft Protocol for Counselors, Teachers, and Career Center Staff
Purpose: The primary purpose of this interview would be to obtain feedback on the Occupational Outlook Handbook, either the Web version or the printed version. In addition, we could find out about the possible use of other resources.
Approach: We would try to get respondents who are experienced with either the website or with the printed Occupational Outlook Handbook. However, if we are unsuccessful finding either type of user, we could ask potential participants to first complete a series of information-seeking tasks on the website before we interview them, just so they have some experience with the website. If someone has used both versions, we have to decide which version to ask about. For now, the default is to ask about the Web version, since that would seem to be the version most likely to be used in the future, and then ask some follow-up questions about which version they prefer, and why (see Part C). Our goal is to generate a time commitment with a respondent that lasts 30-60 minutes (approximately a 30 minute interview for those with prior experience).
Respondent is a:
Counselor (for students)
Counselor (for adults)
Teacher
Career Center Staff
Screening Question
Before we contacted you, had you ever used the Occupational Outlook Handbook, either the paper version or the Web version?
Used paper version only (skip to Part B)
Used Web version only
Used both paper and Web versions
Used neither version before being contacted for this project (skip to 5)
Part A - Questions about the Web Version
On average, how frequently have you used the Occupational Outlook Handbook website within the past year, either for your own purposes or while working with someone else?
Once a month
Once a week
More than once a week
Every day
Other (explain)
How would you describe your general level of experience working with the Occupational Outlook Handbook website?
Very experienced
Experienced
Neither experienced nor inexperienced
Inexperienced
Very inexperienced
a. Can you tell me a little bit about yourself and how you use the Occupational
Outlook Handbook website?
b. What type of information do you typically look for, either for yourself or when
working with a (student/adult that you’re counseling)?
What do you personally like about the website? What are its good features?
Which links or features on the website do you tend to use a lot?
Career Guide to Industries
Career articles from the Occupational Outlook Quarterly
Employment projections
Information about the job market in each State
Job search tips
Links about general occupations (management, professional, services, etc.)
Tomorrow’s jobs
Other (explain)
Do (students/adults you counsel) use the website on their own?
Yes
No (skip next question)
Don’t know (skip next question)
When (students/adults you counsel) use the website, how do they generally react to it?
In this age of sophisticated media messages, what steps could we take to make the website more appealing to (students/adults you counsel)?
How easy or hard is it to find information about different occupations on the website?
Very easy
Easy
Neither easy nor hard
Hard
Very hard
In general, approximately what grade level do you think we should aim for when preparing written materials for the website?
Grade Level: _______
Or
6-8
9-10
11-12
Don’t know/No opinion
Other (please specify)
Considering the reading skills of (high school students/adults that you work with), how easy or difficult is it for them to understand the material on the website?
Very easy
Easy
Neither easy nor difficult
Difficult
Very difficult
What about the career information, is it comprehensive enough or is some critical piece of information missing?
Comprehensive
Inadequate, information is missing
(if inadequate) What information is missing?
In general, how useful would you say the career information on the website is for (your students/adults you counsel)?
Very useful
Useful
Not useful
(if not useful) Why?
How does the quality of the information on the website compare to other sources of career information that you use, would you say it is … (read the categories)
Much better
Better
About the same
Worse
Much worse
Don’t use other sources
(if worse) Why are the other sources better?
What other sources of career information do you recommend to your (students/adults you counsel)?
Have you ever used the “Career Guide to Industries,” which also appears on the BLS website?
Yes
No
Not sure
(if yes) How useful have you found this information to be for (students/adults you
work with)?
Very useful
Useful
Not useful
(if not useful) Why?
Have you ever used the “Teachers Guide to the Occupational Outlook Handbook,” which can also be found on the website?
Yes
No
Not sure
(if yes) How useful have you found this information to be for (students/adults you
work with)?
Very useful
Useful
Not useful
(if not useful) Why?
(if used “Teacher’s guide”) Have you ever used the “Kid’s Page link,” which also appears on the teacher’s guide website?
Yes
No
Not sure
(if yes) How useful have you found this information to be for students?
Very useful
Useful
Not useful
(if not useful) Why?
(if useful) What about the Kid’s Link page has made it useful?
a. Have you ever used the “Employment Projections” link on the website?
Yes
No
Don’t know
(if yes) How useful have you found this information to be for (students/adults you
work with)?
Very useful
Useful
Not useful
(if not useful) Why?
(if useful) What have you found to be useful about the employment projections link?
b. BLS projects the demand for jobs in an occupation 10 years into the future. Is
this an appropriate time interval or should it be shorter or longer?
Should be shorter → by how much?
Appropriate time interval
Should be longer → by how much?
a. Have you ever used the “Occupational Outlook Quarterly” link on the website?
Yes
No
Don’t know
(if yes) How useful have you found this information to be for (students/adults you
work with)?
Very useful
Useful
Not useful
(if not useful) Why?
(if useful) What have you found to be useful about the Occupational Outlook
Quarterly link?
What do you dislike about the website, if anything? If you could change one thing, what would it be?
Do you know how (students/adults you work with) use the website? For example, do they search for a specific piece of information, such as pay and educational requirements, or do they read the more detailed descriptions that are given?
Is there a particular type of (student/adult you work with) that the website is most useful for?
Yes
No
Not sure
(if yes) What type?
What steps could we take to increase (students’/adults you work with) awareness of the website?
Similarly, are there steps we could take to increase counselors’ awareness of the website?
If you were free to design the website, what information would you include and how would you present that information for (high school students/adults you work with)?
Would this website you design also work for middle school students? If not, what should be done differently for this age group?
Part B – Questions about the Printed (paper) Occupational Outlook Handbook
On average, how frequently have you used the Occupational Outlook Handbook within the past year, either for your own purposes or while working with someone else?
Once a month
Once a week
More than once a week
Every day
Other (explain)
How would you describe your general level of experience working with the Occupational Outlook Handbook website?
Very experienced
Experienced
Neither experienced nor inexperienced
Inexperienced
Very inexperienced
a. Can you tell me a little bit about yourself and how you use the Occupational
Outlook Handbook website?
b. What type of information do you typically look for, either for yourself or when
working with a (student/adult that you’re counseling)?
What do you personally like about the Handbook? What are its good features?
Do (students/adults you counsel) use the Handbook on their own?
Yes
No (skip next question)
Don’t know (skip next question)
When (students/adults you counsel) use the Handbook, how do they generally react to it?
In this age of sophisticated media messages, what steps could we take to make the Handbook more appealing to (students/adults you counsel)?
How easy or hard is it to find information about different occupations in the Handbook?
Very easy
Easy
Neither easy nor hard
Hard
Very hard
In general, approximately what grade level do you think we should aim for when preparing written materials for the website?
Grade Level: _______
Or
6-8
9-10
11-12
Don’t know/No opinion
Other (please specify)
Considering the reading skills of (high school students/adults that you work with), how easy or difficult is it for them to understand the material on the website?
Very easy
Easy
Neither easy nor difficult
Difficult
Very difficult
What about the career information, is it comprehensive enough or is some critical piece of information missing?
Comprehensive
Inadequate, information is missing
(if inadequate) What information is missing?
In general, how useful would you say the career information in the Handbook is for (your students/adults you counsel)?
Very useful
Useful
Not useful
(if not useful) Why?
How does the quality of the information on the website compare to other sources of career information that you use, would you say it is … (read the categories)
Much better
Better
About the same
Worse
Much worse
Don’t use other sources
(if worse) Why are the other sources better?
Other than the Handbook, what other sources of career information do you recommend to your (students/adults you counsel)?
Have you ever used the “Career Guide to Industries?”
Yes
No
Not sure
(if yes) How useful have you found this information to be for (students/adults you work with)?
Very useful
Useful
Not useful
(if not useful) Why?
a. Have you ever used the “Occupational Outlook Quarterly” publication?
Yes
No
Don’t know
(if yes) How useful have you found this information to be for (students/adults you
work with)?
Very useful
Useful
Not useful
(if not useful) Why?
(if useful) What have you found to be useful about the Occupational Outlook
Quarterly?
Although you said you don’t use our website, by any chance have you ever heard of the “Teachers Guide to the Occupational Outlook Handbook?”
Yes
No
Not sure
(if yes) Where did you hear about it?
a. Have you ever used a publication called the “Occupational Projections and
Training data” supplement?
Yes
No
Don’t know
(if yes) How useful have you found this information to be for (students/adults you
work with)?
Very useful
Useful
Not useful
(if not useful) Why?
b. We project the demand for jobs in an occupation 10 years into the future. Is
this an appropriate time interval or should it be shorter or longer?
Should be shorter → by how much?
Appropriate time interval
Should be longer → by how much?
What do you dislike about the Handbook, if anything? If you could change one thing, what would it be?
Do you know how (students/adults you work with) use the Handbook? For example, do they search for a specific piece of information, such as pay and educational requirements, or do they read the more detailed descriptions that are given?
Is there a particular type of (student/adult you work with) that the Handbook is most useful for?
Yes
No
Not sure
(if yes) What type?
What steps could we take to increase (students’/adults you work with) awareness of the Handbook or our associated website?
Similarly, are there steps we could take to increase counselors’ awareness of the website?
If you were free to design the Handbook, what information would you include and how would you present that information for (high school students/adults you work with)?
Would the Handbook you design also work for middle school students? If not, what should be done differently for this age group?
Part C - Question for respondents who have use the printed, Web, or both versions
(if have used printed handbook only) Can you tell me why you have only used the printed version of the handbook?
(if have used Web version only) Can you tell me why you have only used the Web version of the handbook?
(if have used both printed and Web version) Since you have used both the paper and Web versions of the Handbook, which do you prefer and why?
Paper
Web
Both the same
Why?
ATTACHMENT 2 –– Post-Test Questions
System Usability Scale
Strongly Strongly
disagree agree
1. I think that I would like to
use this system frequently
2. I found the system unnecessarily
complex
3. I thought the system was easy
to use
4. I think that I would need the
support of a technical person to
be able to use this system
5. I found the various functions in
this system were well integrated
6. I thought there was too much
inconsistency in this system
7. I would imagine that most people
would learn to use this system
very quickly
8. I found the system very
cumbersome to use
9. I felt very confident using the
system
10. I needed to learn a lot of
things before I could get going
with this system
Using SUS
The SUS scale is generally used after the respondent has had an opportunity to use the system being evaluated, but before any debriefing or discussion takes place. Respondents should be asked to record their immediate response to each item, rather than thinking about items for a long time.
All items should be checked. If a respondent feels that they cannot respond to a particular item, they should mark the center point of the scale.
ATTACHMENT 3: Parental/Guardian Consent Form
Parental Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its printed and web-based materials. This study is intended to identify improvements to the Occupational Outlook Handbook, and related publications (for example, the Career Guide to Industries). These publications are designed to provide career information to a wide variety of people, including students and adults.
In this study, we are especially interested in how students, ages 13-18, react to and use versions of these materials that were developed for the World Wide Web. We are asking your permission to have your child participate in a usability study at the Bureau of Labor Statistics, which is located by Union Station. In a usability study, the student will be asked to complete a series of tasks, such as finding the website, locating certain occupations, and answering questions about those occupations. This is a not a test of the student’s abilities. Instead, we are interested in how well the website is designed and how easy it is to use. If students have difficulty with certain tasks, that will provide us with information we can use to redesign the website to make it easier to use.
The BLS, its employees, agents, and partner statistical agencies will use the information your child provides for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your child’s responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the next page of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research your child will be observed by a BLS researcher, and a program will be used on the computer to record entries and timing data for later analysis. Other observers may also be present, but if so, they will always be introduced to your child.
This research session will last no longer than an hour, and as a thank you for participating, we will give your child a check for $40. Your child’s participation in this research project is voluntary, and he or she has the right to stop at any time. If you agree to allow your child to participate, please sign below. If you would like to talk with a researcher before giving your permission, please see the contact information below.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above, and I give my consent for my child to participate in this study.
___________________________________ ___________________________
Parent’s or Guardian’s signature Date
___________________________________ Contact Information for BLS Researchers:
Parent’s or Guardian’s printed name Dr. Jean Fox, 202-691-7370
Dr. Bill Mockovak, 202-691-7414
Note:
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141, and it expires February 28, 2009.
PRIVACY ACT STATEMENT
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The BLS may release individually identifiable information to individuals designated as agents of the BLS in accordance with Public Law 107-347 to perform exclusively statistical activities. Individuals designated as agents of the BLS may be imprisoned for not more than 5 years or fined not more that $250,000 or both for any knowing and willful disclosure of respondent information to unauthorized persons. Such designated agents may include individuals from other sponsoring agencies; to contractors, grantees, and their employees or volunteers who are working on this study for the BLS and who need access to the information; or to the National Archives and Records Administration or the General Services Administration for records management purposes. Under written agreements to protect the confidentiality and security of individually identifiable information, the BLS may provide individually identifiable information to other researchers designated as agents of the BLS to conduct statistical research projects that further the mission and functions of the BLS.
ATTACHMENT 4: Student Consent Form
Student Consent Form
The Bureau of Labor Statistics (BLS) is conducting research to improve the quality of its printed and web-based materials. This specific study is intended to identify improvements to the Occupational Outlook Handbook, and associated career information (for example, the Career Guide to Industries).
The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.
During this research you will be audiotaped and may be videotaped. You may also be observed. Audiotaping and videotaping are done only to assist with the accuracy of our note taking, and your comments will not be associated with your name in any written reports. If you do not wish to be taped or observed, please inform the researcher.
We estimate it will take you an average of 60 minutes to participate in this research. Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.
Please note that this is a not a test of your abilities. Instead, we are interested in how well the website is designed and how easy it is to use. If you have difficulty with certain tasks, that will provide us with information we can use to redesign the website to make it easier to use.
Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141, and expires February 28, 2009.
------------------------------------------------------------------------------------------------------------
I have read and understand the statements above. I consent to participate in this study.
___________________________________ ___________________________
Participant's signature Date
___________________________________
Participant's printed name
___________________________________
Researcher's signature
OMB Control Number: 1220-0141
Expiration Date: 2/28/09
PRIVACY ACT STATEMENT
In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The BLS may release individually identifiable information to individuals designated as agents of the BLS in accordance with Public Law 107-347 to perform exclusively statistical activities. Individuals designated as agents of the BLS may be imprisoned for not more than 5 years or fined not more that $250,000 or both for any knowing and willful disclosure of respondent information to unauthorized persons. Such designated agents may include individuals from other sponsoring agencies; to contractors, grantees, and their employees or volunteers who are working on this study for the BLS and who need access to the information; or to the National Archives and Records Administration or the General Services Administration for records management purposes. Under written agreements to protect the confidentiality and security of individually identifiable information, the BLS may provide individually identifiable information to other researchers designated as agents of the BLS to conduct statistical research projects that further the mission and functions of the BLS.
| File Type | application/msword |
| File Title | October 3, 2007 |
| Author | BLS User |
| Last Modified By | BLS User |
| File Modified | 2007-10-19 |
| File Created | 2007-10-03 |
© 2024 OMB.report | Privacy Policy