ATTACHMENT 1A - Protocol for In-House Usability Testing
Thank participant for coming.
Ask if person has participated in previous studies in BLS.
Introduce any observers.
Goal for today. We are in the process of redesigning our website that provides detailed career information about hundreds of occupations. It’s called the Occupational Outlook Handbook. Have you heard about it by any chance? Have you ever visited the website for the Handbook?
As I mentioned, we’re in the process of redesigning this website but we’re not sure what the best design is for the homepage, so today I’m going to ask you to search for some information about different occupations using different homepage designs we are considering. After you have finished the tasks, we’ll discuss the different versions, and you can tell me which design you prefer, and why.
Reminder. An important point to keep in mind is that we are not evaluating you today. We are evaluating different designs for the homepage, so if you have problems with a task or one of the designs, that’s actually a good thing because then we know we have to make changes. Since this isn’t a test, try to relax. We don’t care how much of a computer whiz you are. Try to make believe you are working on your home computer with no one looking at you.
Procedures & Consent Form. As part of what you’ll be doing today, we have software on the computer that captures your keystrokes and tells us where you clicked on the screen. This software does a lot of work for us, so we don’t have to worry about missing something. Although we collect this information, we will not share it with anyone. We use it only for statistical purposes, and we don’t link your name with it. Now, before I ask you to actually do anything, I’d like you to read and sign this form which states that you are participating voluntarily in this research (Obtain signed consent form).
Start. Start test.
Note. Order of prototypes will vary.
General Instructions – I am going to ask you to look for information today about different occupations. In each case, your search will start on one of two versions of the homepage that we have developed.
For some of the tasks, all you have to do is go to a web page, and you’ll be done, but other tasks will ask you to find some information on the page and then answer a question. Please remember that this is not a test. We’re interested in seeing how easy it is for you to find the answer to a question, so it’s not important if you get the right answer or not. In fact, if a lot of people come up with the wrong answer, that’s very useful information for us because it tells us that our current approach is not the best one, and that the information is too difficult to find.
I also want to point out that you will not be working with the actual Occupational Outlook Handbook today. Instead, we have set up a simplified version of the web site, and some of the actual pages you will be seeing are from our old Handbook before it was redesigned for the Web. However, you will also be looking at some pages that have a new design. I don’t think you will have any problem figuring out which ones these are.
Talk aloud. As you complete the form, I would like you to talk out loud to explain what you are doing and why you are doing it. Some people feel self conscious doing this, but it really helps us determine what you are doing and why. We have found it helps to give people a little practice doing this to warm up, so before we begin, let’s practice a bit. What I want you to do is sit back now and describe what you see when you unlock the door to your apartment or home. Just describe the first room you see after you open the door, including tables, chairs, pictures on the wall, and so on. (Allow time for description) Now, instead of describing a room, I want to describe your thoughts and reactions as you complete our online form in a couple of minutes.
Explain how UTE will present tasks, and work through practice tasks.
Instructions: You are going to work through two practice tasks to give you the chance to get comfortable with what we are doing, and then you will complete 6 tasks that ask you to look for information. After the sixth task, you will see a homepage that is formatted differently. I’ll alert you when this happens so the change doesn’t surprise you. You will then complete six more tasks with the second version of the homepage.
Tasks
The following tasks will be used in the usability test. The order of the homepages (1 or 2) will be changed for each participant. For example, Participant 1 will work with Version A first, Participant 2 will work with Version B first, but each participant will work with both versions. P1 and P2 are practice tasks.
Order 1
|
Homepage Version |
Question |
Answer |
P1 |
A |
Assume you are a math major in college. Since you like math, you are interested in finding out which occupations require math skills, even though you don’t want to be a pure mathematician. Where would you start looking? |
Click Math under Groups of Occupations tab, and then Mathematicians on landing page |
P2 |
A |
Assume you are good with your hands. You also like machines and tools, and are interested in being some type of mechanic. Where would you look to find out what occupations mechanics could work in? |
|
1 |
A |
Your cousin has told you he is thinking about becoming a physical therapist, and was wondering what he could expect to earn, on average (median) annually. What is the answer to his question? |
$72,790 annually |
2 |
A |
A friend asked you if you knew what occupation was growing the fastest. What would you tell her? |
Biomedical engineers |
3 |
A
|
A family friend thinks she would like to work in healthcare, and asks you if you could help her find out what general types of occupations are in that field. Go to the pages that provide this information. |
Under “Groups of Occupations,” click Healthcare Practitioners and Healthcare support (use Detailed Landing Page) |
4 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, find out what the annual pay of audiologists is (figure is for 2008). |
$44,342 (use Photo Grid Landing Page) |
5 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the most growth in the number of jobs between 2008 and 2018? |
Speech-language pathologists (use Growth Landing Page) |
1 |
B |
A member of your family is changing jobs and said he is thinking about becoming an athletic trainer, and wonders what he could expect to earn each year. What is the answer to his question? |
$39,640 |
2 |
B |
A friend asked you if you knew which occupation had the most new job openings. What would you tell her? |
Registered nurses |
3 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, what is the projected growth rate in jobs for audiologists between 2008 and 2018? |
Minus 28% (use Photo Grid Landing Page) |
4 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the biggest decline in the number of jobs between 2008 and 2018? |
Dentists (-56%) (use Growth Landing Page) |
5 |
B |
A family member is leaving the military and wonders what jobs might be open to him that require only a high school education, but still pay at least $50,000 a year. What would you tell him? |
Railroad conductors & yardmasters, rotary drill operators, subway/streetcar operators |
Order 2
|
Homepage Version |
Question |
Answer |
P1 |
B |
Assume you are a math major in college. Since you like math, you are interested in finding out which occupations require math skills, even though you don’t want to be a pure mathematician. Where would you start looking? |
Click Math under Groups of Occupations tab. |
P2 |
B |
Assume you are good with your hands. You also like machines and tools, and are interested in being some type of mechanic. Where would you look to find out what occupations mechanics could work in? |
|
1 |
B |
A member of your family is changing jobs and said he is thinking about becoming an athletic trainer, and wonders what he could expect to earn each year. What is the answer to his question? |
$39,640 |
2 |
B |
A friend asked you if you knew which occupation had the most new job openings. What would you tell her? |
Registered nurses |
4 |
B
|
A family friend thinks she would like to work in healthcare, and asks you if you could help her find out what general types of occupations are in that field. Go to the pages that provide this information. |
Under “Groups of Occupations,” click Healthcare Practitioners and Healthcare support (use Detailed Landing Page) |
5 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, what is the projected growth rate in jobs for audiologists between 2008 and 2018? |
Minus 28% (use Photo Grid Landing Page) |
6 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the biggest decline in the number of jobs between 2008 and 2018? |
Dentists (-56%) (use Growth Landing Page) |
1 |
A |
Your cousin has told you he is thinking about becoming a physical therapist, and was wondering what he could expect to earn, on average (median) annually. What is the answer to his question? |
$72,790 annually |
2 |
A |
A friend asked you if you knew what occupation was growing the fastest. What would you tell her? |
Biomedical engineers |
5 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, find out what the annual pay of audiologists is (figure is for 2008). |
$44,342 (use Photo Grid Landing Page) |
6 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the most growth in the number of jobs between 2008 and 2018? |
Speech-language pathologists (use Growth Landing Page) |
3 |
A |
A family member wants to change jobs and wonders what jobs might be open to him that only require only a high school education, pay at least $25,000 a year, and have a growth rate between 8 and 15 percent over the next 10 years. What would you tell him? |
Bus drivers |
Debriefing Questions
The debriefing questions are described next. For easier administration, these have been combined into two attachments: one for the general population, and one for counselors.
Homepage comparison (paper copies of the homepages will be handed to participant)
We used two different versions of the homepage today. Please look them over again, and tell me which version you prefer.
Version A
Version B
No preference
Why do you prefer that version?
Occupation Groups
As you can see, we varied the format of Occupation Groups a bit. On Version A, they are listed in a column down the left hand side of the page. And, on Version B, they are shown in two columns. Do you have a preference for either version?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
Location of Filter & Number of Options
Another thing we varied between the two versions is this feature that allows you to sort occupations by different criteria like pay, required education, and the number of projected new jobs. That feature appears here on Version A, and here on Version B. Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
Looking at the sorting feature again, Version A offers five ways to sort occupations, and Version B offers three ways. Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
When we were thinking about this sorting feature, we came up with five characteristics of occupations that we thought would be of most interest. These are: education level, the projected number of new jobs, pay in 2010, the training required, and the anticipated growth rate.
Can you rank order these from 1 to 5, where 1 is the most important characteristic to you.
Rank 1 ___
Rank 2 ___
Rank 3 ___
Rank 4 ___
Rank 5 ___
Search Boxes
We also varied the number of search boxes between the two versions. Version A has one search box up located up here, and Version B has two search boxes: one in the same location, and one farther down the page (point them out). Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
Fastest Growing Jobs (projected) & Most New Jobs (projected)
We also added some links that we thought would be of general interest to people visiting this website. For example, you can click on the highest paying jobs, the fastest growing jobs, the jobs that most visitors to this site are looking at (most viewed), and jobs with the most new job openings.
I wanted to talk to you about this link: Fastest Growing Jobs (projected). Can you tell me what that means in your own words?
We also have the link Most New Jobs (projected). Can you tell me what the difference is between these two links?
Finally, I wanted to get your opinion about the Most Viewed link (point it out). How likely do you think it is that visitors to this website will find that link useful. Would you say it was (read categories):
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
Filter Occupations - “Click go” instruction
Version A has this Click go when done instruction following the heading “Filter Occupations By” (point this out). Version B does not have this instruction.
Did you notice this instruction?
Yes
No
Not sure
Did you think it’s helpful?
Yes
No
Not sure
OOH Occupation Finder Page
Do you remember this page?
Yes
No
Not sure
Did you read these instructions (Filter & Sort)?
Yes
No
Not sure
How clear would you say these instructions are?
Very clear
Clear
Neither clear nor unclear
Unclear
Very unclear
Landing Page comparisons (paper copies of the landing pages will be handed to participant)
In some career fields, there are related types of occupations. For example, today you saw occupations that fell under Healthcare Practitioners and Healthcare Support. We used three different formats for presenting this information. I want you to look over these versions and then order them from 1 to 3, where 1 is the version you like most, 2 is the next preferred, and 3 is the one you like the least.
Rank 1 _____
Rank 2 _____
Rank 3 _____
What do you like about the version you ranked first?
What do you dislike about the version you ranked last?
OOH Occupation Finder (a paper copy of the advanced search occupation finder will be handed to participant)
We developed this table to help you search quickly for occupations that met certain criteria, for example, jobs that pay more than a certain amount of money, jobs that are expected to experience a lot of growth, and jobs that require a certain level of education.
How easy or difficult was this table to use?
Very easy
Easy
Neither easy nor hard
Difficult
Very difficult
Did not use
If you had encountered this table on the website, how likely or unlikely is it that you would have used it?
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
Do you have any comments about the table that would help us improve it or make it easier to use?
Background Questions (These will be collected on the computer after the session)
Before we begin, there are a few questions about yourself that I would like you answer.
How would you rate your level of experience using the Internet?
I am fairly new to it
I have some experience, but still much to learn
I have moderate (about average) experience
I have much (considerable) experience
I use the Internet a lot and understand its intricacies
What is your age?
18-21
22-25
26-30
31-40
41-50
51-60
61 or older
What is the highest level of education you have completed?
Some high school
Some high school, but with technical training
High school graduate or GED
Some college, but no degree
Technical training past high school
Associate college degree (2-year program, for example, A.A. or A.S.)
College degree (for example, B.A. or B.S.)
Some post-graduate education, but no degree
Post-graduate degree (for example, M.A., M.S., Ph.D., or professional
degree)
A. Homepage comparison (hand paper copies of the homepages to participant)
We used two different versions of the homepage today. Please look them over again, and tell me which version you prefer.
Version A
Version B
No preference
Why do you prefer that version?
B. Occupation Groups
As you can see, we varied the format of Occupation Groups a bit. On Version A, they are listed in a column down the left hand side of the page. And, on Version B, they are shown in two columns. Do you have a preference for either version?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
C. Location of Filter & Number of Options
Another thing we varied between the two versions is this feature that allows you to sort occupations by different criteria like pay, required education, and the number of projected new jobs. That feature appears here on Version A, and here on Version B. Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
Looking at the sorting feature again, Version A offers five ways to sort occupations, and Version B offers three ways. Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
When we were thinking about this sorting feature, we came up with five characteristics of occupations that we thought would be of most interest. These are: education level, the projected number of new jobs, pay in 2010, the training required, and the anticipated growth rate.
Please rank order these from 1 to 5, where 1 is the most important characteristic to you.
Rank 1 ___
Rank 2 ___
Rank 3 ___
Rank 4 ___
Rank 5 ___
D. Search Boxes
We also varied the number of search boxes between the two versions. Version A has one search box up located up here, and Version B has two search boxes: one in the same location, and one farther down the page (point them out). Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
E. Fastest Growing Jobs (projected) & Most New Jobs (projected)
We also added some links that we thought would be of general interest to people visiting this website. For example, you can click on the highest paying jobs, the fastest growing jobs, the jobs that most visitors to this site are looking at (most viewed), and jobs with the most new job openings.
I wanted to talk to you about this link: Fastest Growing Jobs (projected). Can you tell me what that means in your own words?
We also have the link Most New Jobs (projected). Can you tell me what the difference is between these two links?
Finally, I wanted to get your opinion about the Most Viewed link (point it out). How likely do you think it is that visitors to this website will find that link useful. Would you say it was (read categories):
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
F. Filter Occupations - “Click go” instruction
Version A has this Click go when done instruction following the heading “Filter Occupations By” (point this out). Version B does not have this instruction.
a. Did you notice this instruction?
Yes
No
Not sure
b. Did you think it’s helpful?
Yes
No
Not sure
G. OOH Occupation Finder Page
Do you remember this page?
Yes
No
Not sure
Did you read these instructions (Filter & Sort)?
Yes
No
Not sure
How clear would you say these instructions are?
Very clear
Clear
Neither clear nor unclear
Unclear
Very unclear
H. Landing Page comparisons (hand paper copies of the landing pages to participant)
In some career fields, there are related types of occupations. For example, today you saw occupations that fell under Healthcare Practitioners and Healthcare Support. We used three different formats for presenting this information. I want you to look over these versions and then order them from 1 to 3, where 1 is the version you like most, 2 is the next preferred, and 3 is the one you like the least.
Rank 1 _____
Rank 2 _____
Rank 3 _____
a. What do you like about the version you ranked first?
b. What do you dislike about the version you ranked last?
I. OOH Occupation Finder (hand a paper copy to participant)
We developed this table to help you search quickly for occupations that met certain criteria, for example, jobs that pay more than a certain amount of money, jobs that are expected to experience a lot of growth, and jobs that require a certain level of education.
How easy or difficult was this table to use?
Very easy
Easy
Neither easy nor hard
Difficult
Very difficult
Did not use
If you had encountered this table on the website, how likely or unlikely is it that you would have used it?
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
a. Do you have any comments about the table that would help us improve it or make it easier to use?
A. Homepage comparison (had paper copies of the homepages to participant)
We used two different versions of the homepage today. Please look them over again, and tell me which version you prefer.
Version A
Version B
No preference
Why do you prefer that version?
B. Occupation Groups
As you can see, we varied the format of Occupation Groups a bit. On Version A, they are listed in a column down the left hand side of the page. And, on Version B, they are shown in two columns. Do you have a preference for either version?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
C. Location of Filter & Number of Options
Another thing we varied between the two versions is this feature that allows you to sort occupations by different criteria like pay, required education, and the number of projected new jobs. That feature appears here on Version A, and here on Version B. Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
Looking at the sorting feature again, Version A offers five ways to sort occupations, and Version B offers three ways. Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
When we were thinking about this sorting feature, we came up with five characteristics of occupations that we thought would be of most interest. These are: education level, the projected number of new jobs, pay in 2010, the training required, and the anticipated growth rate.
Please rank order these from 1 to 5, where 1 is the most important characteristic to you.
Rank 1 ___
Rank 2 ___
Rank 3 ___
Rank 4 ___
Rank 5 ___
D. Search Boxes
We also varied the number of search boxes between the two versions. Version A has one search box up located up here, and Version B has two search boxes: one in the same location, and one farther down the page (point them out). Which version do you prefer?
Version A – Is there a reason you prefer this version?
Version B – Is there a reason you prefer this version?
No preference
E. Fastest Growing Jobs (projected) & Most New Jobs (projected)
We also added some links that we thought would be of general interest to people visiting this website. For example, you can click on the highest paying jobs, the fastest growing jobs, the jobs that most visitors to this site are looking at (most viewed), and most new jobs.
a. Do you think your students (clients) will know the difference between fastest growing jobs (projected) and most new jobs (projected)?
b. Do you have a preference for the order in which these links are shown?
Yes – if yes, what is your preferred order?
No
We debated between using percentage change versus adjective descriptors for describing the growth rates for occupations. Which do you prefer and why?
Percentage changes
Adjectives (average, faster than average, etc.)
Other - please explain
Finally, as a counselor, how useful do you think this link will be to your students (clients)?
Very useful
Somewhat useful
A little useful
Not at all useful
F. Filter Occupations - “Click go” instruction
Version A has this Click go when done instruction following the heading “Filter Occupations By” (point this out). Version B does not have this instruction.
a. Did you notice this instruction?
Yes
No
Not sure
b. Did you think it’s helpful?
Yes
No
Not sure
c. Do you have a preference for the order in which the characteristics that you can sort by are
shown?
Yes – if yes, what is your preferred order?
No
H. Landing Page comparisons (hand paper copies to participant)
In some career fields, there are related types of occupations. For example, today you saw occupations that fell under Healthcare Practitioners and Healthcare Support. We used three different formats for presenting this information. I want you to look over these versions and then order them from 1 to 3, where 1 is the version you like most, 2 is the next preferred, and 3 is the one you like the least.
Rank 1 _____
Rank 2 _____
Rank 3 _____
a. What do you like about the version you ranked first?
b. What do you dislike about the version you ranked last?
This version uses the largest photos. One drawback to this approach is that the occupations cannot be sorted, so that occupations that start with A, B, C, etc., will always be at the top of the page. Do you see any problems with this?
Yes – if yes, please explain
No
This version presents some general job summary information about each occupation. How useful is this information compared to other items that could be shown here?
Very useful
Somewhat useful
A little useful
Not at all useful
a. What other items do you think could be shown here?
I. OOH Occupation Finder (hand a paper copy to participant)
We developed this table to help you search quickly for occupations that met certain criteria, for example, jobs that pay more than a certain amount of money, jobs that are expected to experience a lot of growth, and jobs that require a certain level of education.
Do you remember this page?
Yes
No
Not sure
Did you read these instructions (Filter & Sort)?
Yes
No
Not sure
How clear would you say these instructions are?
Very clear
Clear
Neither clear nor unclear
Unclear
Very unclear
Do you have a preference for the order in which the characteristics that you can search by are shown?
Yes – if yes, what is your preferred order?
No
How easy or difficult was this table to use?
Very easy
Easy
Neither easy nor hard
Difficult
Very difficult
Did not use
If your students encounter this table on the website, how likely or unlikely is it that you think they will use it?
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
a. Do you have any comments about the table that would help us improve it or make it easier to use?
ATTACHMENT 1B - Protocol for Remote Usability Testing
Protocol for Remote Usability Testing
Introduction for when WebEx is used (see Attachment for separate Loop11 protocol)
Thank you for agreeing to participate today.
Ask if person has participated in previous studies in BLS.
Introduce any observers.
Note: The facilitator and participant will share the computer screen, but not see each other. Communication will be via telephone.
Background. Before we begin, let me give you some background for what you will be doing today. We are in the process of redesigning our website that provides detailed career information about hundreds of occupations. It’s called the Occupational Outlook Handbook. Have you heard about it by any chance? Have you ever visited the website for the Handbook?
As I mentioned, we’re in the process of redesigning this website but we’re not sure what the best design is for the homepage, so today I’m going to ask you to search for some information about different occupations using different homepage designs we are considering. After you have finished the tasks, I’ll ask you to rate how easy or difficult it was to find information, and then at the end of this session we’ll discuss the different versions, and you can tell me which design you prefer, and why.
Reminder. An important point to keep in mind is that we are not evaluating you today. We are evaluating different designs for the homepage, so if you have problems with a task or one of the designs, that’s actually a good thing because then we know we have to make changes. Another thing I want you to do is relax. This isn’t a test. We’re don’t care how much of a computer whiz you are. Try to make believe you are working on your home computer with no one observing you.
Procedures & Consent Form. As part of what you’ll be doing today, we have software on the computer that captures your keystrokes and tells us where you clicked on the screen. This software does a lot of work for us, so we don’t have to worry about missing something. Although we collect this information, we will not share it with anyone. We use it only for statistical purposes, and we don’t link your name with it.
Reminder. E-mail or phone call should have been received acknowledging voluntary participation.
General Instructions – I am going to ask you to look for information today about different occupations. In each case, your search will start on one of two versions of the homepage that we have developed.
For some of the tasks, all you have to do is go to a web page, and you’ll be done, but other tasks will ask you to find some information on the page and then answer a question. Since this isn’t a test, we’re interested in seeing how easy it is for you to find the answer to a question, so it’s not important if you get the right answer or not. In fact, if a lot of people come up with the wrong answer, that’s very useful information for us because it tells us that our current approach is not the best one, and that the information is too difficult to find.
I also want to point out that you will not be working with the actual Occupational Outlook Handbook today. Instead, we have set up a simplified version of the web site, and some of the actual pages you will be seeing are from our old Handbook before it was redesigned for the Web. You will be looking at some pages that have a new design. I don’t think you will have any problem figuring out which ones these are.
Talk aloud. As you complete the form, I would like you to talk out loud to explain what you are doing and why you are doing it. Some people feel self conscious doing this, but it really helps us determine what you are doing and why. We have found it helps to give people a little practice doing this to warm up, so before we begin, let’s practice a bit. What I want you to do is sit back now and describe what you see when you unlock the door to your apartment or home. Just describe the first room you see after you open the door, including tables, chairs, pictures on the wall, and so on. (Allow description) Now, instead of describing a room, I want to describe your thoughts and reactions as you complete our online form.
Start. Start test.
Note. Order of prototypes will vary.
Explain how UTE will present tasks, and work through practice tasks.
Tasks
The following tasks will be used in the usability test. The order of the homepages (1 or 2) will be changed for each participant. For example, Participant 1 will work with Version A first, Participant 2 will work with Version B first, but each participant will work with both versions. P1 and P2 are practice tasks.
Order 1
|
Homepage Version |
Question |
Answer |
P1 |
A |
Assume you are a math major in college. Since you like math, you are interested in finding out which occupations require math skills, even though you don’t want to be a pure mathematician. Where would you start looking? |
Click Math under Groups of Occupations tab, and then Mathematicians on landing page |
P2 |
A |
Assume you are good with your hands. You also like machines and tools, and are interested in being some type of mechanic. Where would you look to find out what occupations mechanics could work in? |
|
1 |
A |
Your cousin has told you he is thinking about becoming a physical therapist, and was wondering what he could expect to earn, on average (median) annually. What is the answer to his question? |
$72,790 annually |
2 |
A |
A friend asked you if you knew what occupation was growing the fastest. What would you tell her? |
Biomedical engineers |
3 |
A
|
A family friend thinks she would like to work in healthcare, and asks you if you could help her find out what general types of occupations are in that field. Go to the pages that provide this information. |
Under “Groups of Occupations,” click Healthcare Practitioners and Healthcare support (use Detailed Landing Page) |
4 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, find out what the annual pay of audiologists is (figure is for 2008). |
$44,342 (use Photo Grid Landing Page) |
5 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the most growth in the number of jobs between 2008 and 2018? |
Speech-language pathologists (use Growth Landing Page) |
1 |
B |
A member of your family is changing jobs and said he is thinking about becoming an athletic trainer, and wonders what he could expect to earn each year. What is the answer to his question? |
$39,640 |
2 |
B |
A friend asked you if you knew which occupation had the most new job openings. What would you tell her? |
Registered nurses |
3 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, what is the projected growth rate in jobs for audiologists between 2008 and 2018? |
Minus 28% (use Photo Grid Landing Page) |
4 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the biggest decline in the number of jobs between 2008 and 2018? |
Dentists (-56%) (use Growth Landing Page) |
5 |
B |
A family member is leaving the military and wonders what jobs might be open to him that require only a high school education, but still pay at least $50,000 a year. What would you tell him? |
Railroad conductors & yardmasters, rotary drill operators, subway/streetcar operators |
Order 2
|
Homepage Version |
Question |
Answer |
P1 |
B |
Assume you are a math major in college. Since you like math, you are interested in finding out which occupations require math skills, even though you don’t want to be a pure mathematician. Where would you start looking? |
Click Math under Groups of Occupations tab. |
P2 |
B |
Assume you are good with your hands. You also like machines and tools, and are interested in being some type of mechanic. Where would you look to find out what occupations mechanics could work in? |
|
1 |
B |
A member of your family is changing jobs and said he is thinking about becoming an athletic trainer, and wonders what he could expect to earn each year. What is the answer to his question? |
$39,640 |
2 |
B |
A friend asked you if you knew which occupation had the most new job openings. What would you tell her? |
Registered nurses |
4 |
B
|
A family friend thinks she would like to work in healthcare, and asks you if you could help her find out what general types of occupations are in that field. Go to the pages that provide this information. |
Under “Groups of Occupations,” click Healthcare Practitioners and Healthcare support (use Detailed Landing Page) |
5 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, what is the projected growth rate in jobs for audiologists between 2008 and 2018? |
Minus 28% (use Photo Grid Landing Page) |
6 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the biggest decline in the number of jobs between 2008 and 2018? |
Dentists (-56%) (use Growth Landing Page) |
1 |
A |
Your cousin has told you he is thinking about becoming a physical therapist, and was wondering what he could expect to earn, on average (median) annually. What is the answer to his question? |
$72,790 annually |
2 |
A |
A friend asked you if you knew what occupation was growing the fastest. What would you tell her? |
Biomedical engineers |
5 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, find out what the annual pay of audiologists is (figure is for 2008). |
$44,342 (use Photo Grid Landing Page) |
6 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the most growth in the number of jobs between 2008 and 2018? |
Speech-language pathologists (use Growth Landing Page) |
3 |
A |
A family member wants to change jobs and wonders what jobs might be open to him that only require only a high school education, pay at least $25,000 a year, and have a growth rate between 8 and 15 percent over the next 10 years. What would you tell him? |
Bus drivers |
Homepage comparison
We used two different versions of the homepage today. Let me show them to you again (Versions will be shown on computer). Which version do you prefer?
Version A
Version B
No preference
Why do you prefer that version?
Landing Page comparisons
In some career fields, there are related types of occupations. For example, today you saw occupations that fell under Healthcare Practitioners and Healthcare Support. We used three different formats for presenting this information. Let me show them to you again (Versions will be shown on computer). As I do this, please determine which you liked best, next best, and least of all.
Rank 1 _____
Rank 2 _____
Rank 3 _____
What do you like about the version you ranked first?
What do you dislike about the version you ranked last?
OOH Occupation Finder
We developed this special table to help you search quickly for occupations that met certain criteria (table will be shown on computer), for example, jobs that pay more than a certain amount of money, jobs that are expected to experience a lot of growth, and jobs that require a certain level of education.
How easy or difficult was this table to use?
Very easy
Easy
Neither easy nor hard
Difficult
Very difficult
Did not use
If you had encountered this table on the website, how likely or unlikely is it that you would have used it?
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
Do you have any comments about the table that would help us improve it or make it easier to use?
Background Questions (These will be collected on the computer after the test)
Before we begin, there are a few questions about yourself that I would like you answer.
How would you rate your level of experience using the Internet?
I am fairly new to it
I have some experience, but still much to learn
I have moderate (about average) experience
I have much (considerable) experience
I use the Internet a lot and understand its intricacies
What is your age?
18-21
22-25
26-30
31-40
41-50
51-60
61 or older
What is the highest level of education you have completed?
Some high school
Some high school, but with technical training
High school graduate or GED
Some college, but no degree
Technical training past high school
Associate college degree (2-year program, for example, A.A. or A.S.)
College degree (for example, B.A. or B.S.)
Some post-graduate education, but no degree
Post-graduate degree (for example, M.A., M.S., Ph.D., or professional
degree)
Attachment – Loop11 Protocol (Loop11 allows a usability test to be conducted without any direct human facilitation)
Introduction. Thank you for participating in this evaluation. Your feedback will help improve the functionality of the Occupational Outlook Handbook website. We estimate that this exercise will last about 20 minutes. Please read all instructions carefully.
Before you begin, please note that you will not be working with the actual Occupational Outlook Handbook today. Instead, a simplified version of the web site has been set up, and some of the pages you will be seeing are from our old Handbook before it was redesigned for the Web. You will also be looking at some pages that have a new design. I don’t think you will have any problem figuring out which ones these are.
NOTE: This survey is being administered by Loop11.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.
Loop11 Instructions.
Start. Start test.
Note. Order of prototypes will vary.
Explain how Loop11 will present tasks
We would now like you to complete some tasks and then answer several questions relating to your experience with each task. In each task you will be asked a question, such as "What is the median salary for a plumber?" In this example, your task would be to use the tools on the website to navigate to a webpage that has this information on it. We are interested in whether you can find the information, how you get there, and how long it takes. Once you believe you have found the answer click the Complete Task button.
Approach using our site as if you were at home, school, etc. and if, after expending a certain amount of effort, you believe you would “bail out” and go elsewhere to find the information, then that is what you should do. This isn’t a test of your ability, it’s a test of the website, so if you do abandon the task, that’s useful information for us because we know we have to make improvements.
*** NOTE: Please be patient and wait for each webpage to load. Because some of the pages are still in development, they may take a few seconds to load properly. Do not click the Task Complete or Abandon Task button until the webpage has loaded and you have completed or attempted the task.
When you are ready to begin, click OK and then click Next.
Tasks. The following tasks will be used in the usability test. The order of the homepages (A or B) will be changed for each participant. For example, Participant 1 will work with Version A first, Participant 2 will work with Version B first, but each participant will work with both versions.
Note: Since Loop11 runs without a facilitator, to avoid having the session run too long (20 minutes is probably the maximum), participants may be asked to complete the 8 tasks (2 practice, 6 regular tasks) for only one version. However, a couple of tasks will also be presented with the alternative version.
Order 1
|
Homepage Version |
Question |
Answer |
P1 |
A |
Assume you are a math major in college. Since you like math, you are interested in finding out which occupations require math skills, even though you don’t want to be a pure mathematician. Where would you start looking? |
Click Math under Groups of Occupations tab, and then Mathematicians on landing page |
P2 |
A |
Assume you are good with your hands. You also like machines and tools, and are interested in being some type of mechanic. Where would you look to find out what occupations mechanics could work in? |
|
1 |
A |
Your cousin has told you he is thinking about becoming a physical therapist, and was wondering what he could expect to earn, on average (median) annually. What is the answer to his question? |
$72,790 annually |
2 |
A |
A friend asked you if you knew what occupation was growing the fastest. What would you tell her? |
Biomedical engineers |
3 |
A
|
A family friend thinks she would like to work in healthcare, and asks you if you could help her find out what general types of occupations are in that field. Go to the pages that provide this information. |
Under “Groups of Occupations,” click Healthcare Practitioners and Healthcare support (use Detailed Landing Page) |
4 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, find out what the annual pay of audiologists is (figure is for 2008). |
$44,342 (use Photo Grid Landing Page) |
5 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the most growth in the number of jobs between 2008 and 2018? |
Speech-language pathologists (use Growth Landing Page) |
1 |
B |
A member of your family is changing jobs and said he is thinking about becoming an athletic trainer, and wonders what he could expect to earn each year. What is the answer to his question? |
$39,640 |
2 |
B |
A friend asked you if you knew which occupation had the most new job openings. What would you tell her? |
Registered nurses |
3 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, what is the projected growth rate in jobs for audiologists between 2008 and 2018? |
Minus 28% (use Photo Grid Landing Page) |
4 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the biggest decline in the number of jobs between 2008 and 2018? |
Dentists (-56%) (use Growth Landing Page) |
5 |
B |
A family member is leaving the military and wonders what jobs might be open to him that require only a high school education, but still pay at least $50,000 a year. What would you tell him? |
Railroad conductors & yardmasters, rotary drill operators, subway/streetcar operators |
Order 2
|
Homepage Version |
Question |
Answer |
P1 |
B |
Assume you are a math major in college. Since you like math, you are interested in finding out which occupations require math skills, even though you don’t want to be a pure mathematician. Where would you start looking? |
Click Math under Groups of Occupations tab. |
P2 |
B |
Assume you are good with your hands. You also like machines and tools, and are interested in being some type of mechanic. Where would you look to find out what occupations mechanics could work in? |
|
1 |
B |
A member of your family is changing jobs and said he is thinking about becoming an athletic trainer, and wonders what he could expect to earn each year. What is the answer to his question? |
$39,640 |
2 |
B |
A friend asked you if you knew which occupation had the most new job openings. What would you tell her? |
Registered nurses |
4 |
B
|
A family friend thinks she would like to work in healthcare, and asks you if you could help her find out what general types of occupations are in that field. Go to the pages that provide this information. |
Under “Groups of Occupations,” click Healthcare Practitioners and Healthcare support (use Detailed Landing Page) |
5 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, what is the projected growth rate in jobs for audiologists between 2008 and 2018? |
Minus 28% (use Photo Grid Landing Page) |
6 |
B |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the biggest decline in the number of jobs between 2008 and 2018? |
Dentists (-56%) (use Growth Landing Page) |
1 |
A |
Your cousin has told you he is thinking about becoming a physical therapist, and was wondering what he could expect to earn, on average (median) annually. What is the answer to his question? |
$72,790 annually |
2 |
A |
A friend asked you if you knew what occupation was growing the fastest. What would you tell her? |
Biomedical engineers |
5 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, find out what the annual pay of audiologists is (figure is for 2008). |
$44,342 (use Photo Grid Landing Page) |
6 |
A |
Starting at the link for Healthcare Practitioners and Healthcare Support, which occupation is expected to see the most growth in the number of jobs between 2008 and 2018? |
Speech-language pathologists (use Growth Landing Page) |
3 |
A |
A family member wants to change jobs and wonders what jobs might be open to him that only require only a high school education, pay at least $25,000 a year, and have a growth rate between 8 and 15 percent over the next 10 years. What would you tell him? |
Bus drivers |
Homepage Preference. We would like you to take a minute or two to review each of the homepages we showed you earlier. Browse long enough to develop an impression of each page. When you are ready to begin, click OK and then click Next. (Note: We will include links to the two homepages).
Review Homepage Version A
Review Homepage Version B
Which version do you prefer?
Version A
Version B
No preference
Why do you prefer that version?
Landing Page comparisons
In some career fields, there are related types of occupations. For example, today you saw occupations that fell under Healthcare Practitioners and Healthcare Support. We used three different formats for presenting this information. Please look at them again and determine which you liked best, which you liked the next best, and then which you liked least of all.
Review Version A
Review Version B
Review Version C
Which version do you most prefer?
Version A, with list of occupations, columns with bar chart/histogram and median wages
Version B, with small pictures in first (left) column
Version C, with larger pictures, 4 across each row
No preference
What do you like about the version you ranked first?
What do you dislike about the version you ranked last?
OOH Occupation Finder
We developed a special table to help you search quickly for occupations that met certain criteria, for example, jobs that pay more than a certain amount of money, jobs that are expected to experience a lot of growth, and jobs that require a certain level of education. Please look at this table again now.
Review Advanced Search Table
How easy or difficult was this table to use?
Very easy
Easy
Neither easy nor hard
Difficult
Very difficult
Did not use
If you had encountered this table on the website, how likely or unlikely is it that you would have used it?
Very likely
Likely
Neither likely nor unlikely
Unlikely
Very unlikely
Do you have any comments about the table that would help us improve it or make it easier to use?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | KINCAID_N |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |