Note to Reviewer - K-12 Website

3/29/2016

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141
FROM:	Robin Kaplan Research Statistician Office of Survey Methods Research
SUBJECT:	Submission of Materials for the BLS K-12 Website Feedback Study for Students and Teachers

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are estimated to be 9.33.

If there are any questions regarding this project, please contact Robin Kaplan at

202-691-7383 or Jean Fox at 202-691-7370.

1. Introduction and Purpose

The Bureau of Labor Statistics (BLS) published its K-12 website on September 30^th, 2013 (link: http://www.bls.gov/k12/). The website contains resources to teach students about BLS products through activities and games and provide information about careers. The website is divided into various sections. The first section, Games and Quizzes, contains activities to appeal to younger students and teach them about BLS facts, terminology, and different occupations. The second section, Student Resources, draws from the popular Occupational Outlook Handbook, and provides K-12 friendly content regarding career information, employment projections, and a glossary of BLS terms. Another section provides a history of BLS that contains a timeline of major events throughout history and biographies of BLS commissioners.

In addition to these sections tailored to students, there is also a section titled Teacher’s Desk, which contains BLS-themed classroom activities designed for teachers to use in the classroom, along with ready-to-print informational posters, and other resources that may be of interest to teachers who incorporate economics, career planning, and social studies into their curriculums.

Since its launch, the BLS K-12 website has been very popular, receiving approximately 100,000 visits per month. Thus, the BLS K-12 team would like to get feedback from both students and teachers on the overall appeal and usability of the website, as well as feedback on the resources and what we could do to improve the website. This type of testing is crucial to determine what portions of the website need improvement, whether students are learning from the content, if teachers find the information clear, relevant to their curriculum, and easy to implement, and recommendations for future content as the BLS K-12 website expands.

To gain insight on the issues listed above, we seek to solicit feedback from students and teachers on the BLS K-12 website from the online usability testing website TryMyUI.com. TryMyUI is an online testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes. Students and teachers recruited from the TryMyUI panel will be asked to think aloud while completing tasks and/or reviewing material from the BLS K-12 website. The sessions will be conducted iteratively so that any modifications to the protocol can be assessed and changes made to clarify question or task wording.

Research Design

Protocol

We plan to get feedback from 28 testers through the TryMyUI panel. We have developed separate protocols for students and teachers. See Appendix A for the Student Protocol and Appendix B for the Teacher Protocol. We expect that modifications may be made during the course of the study based on initial feedback from the first few TryMyUI participants. Although the goals of the testing, and overall design, will remain the same, findings from preliminary results may be used to improve the protocol.

We plan to test equal numbers (n = 14) of students and educators/teachers, but may adjust this number during the process if we find that we need more feedback from a particular testing group. As mentioned, TryMyUI is a testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes. Students and educators/teachers recruited from the TryMyUI panel will be asked to think aloud while completing tasks and/or reviewing material from the BLS K-12 website. Due to age restrictions on TryMyUI panels, testers must be 18 years old or older. We decided to get feedback from middle school teachers, as most of the classroom activities on the website are geared toward this particular age group.

Recruitment

We will use the following recruitment criteria for obtaining students and teachers from the TryMyUI panel of testers:

• Students Recruitment criteria: We are looking for high school students (ages 18 or 19) to participate in this study.

• Teachers Recruitment criteria: We are looking for testers who are middle school counselors and teachers of math, statistics, economics, or social studies, who might use Bureau of Labor Statistics data

3. Participants and Burden Hours

Our goal is to obtain feedback from up to 14 students and 14 teachers. Each session is expected to last 20 minutes, for a maximum total of 9.33 burden hours (28 participants X .33 hours).

4. Payment

A total of 28 participants from the TryMyUI panel will be recruited. Participants will be compensated $10.00 for participating in the study, the standard rate that TryMyUI offers. The payment for these 28 participants will total $280.

5. Data Confidentiality

Recruiting of participants will be handled by TryMyUI.com. Participants will be informed that the study is about providing feedback on the BLS K-12 website. The data collected as part of this study will be stored on TryMyUI.com servers. Participants in this study will not be asked to provide any personally identifiable information. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey will take no more than 20 minutes to complete. This survey is being administered by TryMyUI.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Attachments

Appendix A – Students Protocol

Appendix B – Teachers Protocol

Appendix A

High School Student Protocol

Recruitment criteria: We are looking for high school students (ages 18-19) to participate in this study.

Scenario: The Bureau of Labor Statistics would like your feedback on our K-12 site. The site provides educational resources for children, teens, and educators.  For this study, we are looking for feedback from high school students. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey will take no more than 20 minutes to complete. This survey is being administered by TryMyUI.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

We would like to make the website as easy to use as possible.  We are going to ask you to do a few tasks so you can experience how easy or difficult it is to use. Your feedback is very important to this process, so please be honest and let us know if you see anything we can improve. 

Please don't spend too much time on any one task to ensure you get through each of them, and read the instructions for each task carefully. 

Because you will be moving around different pages on the website, before you begin each task, please be sure you are on the main K12 Page (the current page you see).  

Tasks:

1.) State your first impressions of the site. For instance, you can discuss any or all of the following: 
  • What strikes you first? 
  • What is the purpose of this website?  What can you do here? 
  • Who do you think the target audience of the site is? 
  • What types of information, data, and resources are available? 

2.) Start at the BLS K-12 homepage (http://www.bls.gov/k12). 
  Find the BLS definitions for “Occupation” and “Industry.”  Explain the terms in your own words.  How are these concepts different? 

3.) Start at the BLS K-12 homepage (http://www.bls.gov/k12).  Find a BLS milestone from 2009 
  • What milestone did you pick? 
  • Was it easy or difficult to find a milestone from 2009 on the timeline? Please explain. 
  • Would you change anything on the timeline?  If so, what? 

4.) Start at the BLS K-12 homepage (http://www.bls.gov/k12). 
  Find and report the unemployment rate for Texas in October 2015. 
  Was it easy or difficult to find the unemployment rates? Please explain. 

5.) Start at the BLS K-12 homepage (http://www.bls.gov/k12). 

 In what occupation will there be the most new jobs created between 2012 and 2022? 
  • Was it easy or difficult to find this information? Please explain. 

Survey questions:

What did you like or dislike about the BLS K-12 website?

Would the site be a good resource (for example, for school work or career planning), for students your age? Why or why not?

How could the website be improved?

What types of content would you like to see on the website?

Appendix B

Protocol for Educators

Recruitment criteria: Middle school counselors and teachers of math, statistics, economics, or social studies, who might use Bureau of Labor Statistics data

Scenario: The Bureau of Labor Statistics would like your feedback on our K-12 site. The site provides educational resources for children, teens, and educators.  For this study, we are looking for feedback from middle school teachers and counselors. 

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey will take no more than 20 minutes to complete. This survey is being administered by TryMyUI.com and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

We would like to make the website as easy to use as possible.  We are going to ask you to do a few tasks so you can experience how easy or difficult it is to use. Your feedback is very important to this process, so please be honest and let us know if you see anything we can improve. 

Please don't spend too much time on any one task to ensure you get through each of them, and read the instructions for each task carefully. 

Because you will be moving around different pages on the website, before you begin each task, please be sure to follow the instructions for the starting page of each task.  

Tasks:

1.) Please let us know what you teach. 
  • What grade do you currently teach? 
  • Do you teach a specific subject?  If so, what is it? 

2.) Now start at http://www.bls.gov/k12/. State your first impressions of the overall site. For instance, you can discuss any or all of the following: 
  • What strikes you first? 
  • What is the purpose of this website?  What can you do here? 
  • Does the site contain content you would teach to students in your classroom?  Why or why not? 

3.) Now start at the BLS Teacher’s Desk page (http://www.bls.gov/k12/teachers.html).  Talk about your first impressions of the page: 
  • What do you notice about the site? 
  • Do any of the resources look relevant for your classroom? If so, which ones and why? 

4.) Start at the BLS K-12 Teacher’s Desk page (http://www.bls.gov/k12/teachers.htm).   

   Find and review the poster titled “How BLS Collects and Publishes Statistics.” 
  • What are your impressions of the poster? 
  • What grade level do you think the poster is most appropriate for? 
  • Is this something you could incorporate into your classroom?  Why or why not? 

5.) Find and review the poster titled “Learning and Earning.” 
  • What are your impressions of the poster? 
  • What grade level do you think the poster is most appropriate for? 
  • Is this something you could incorporate into your classroom?   

6.) Start at the BLS Teacher’s Desk page again (http://www.bls.gov/k12/teachers.html).   
    There are five classroom activities here. Please briefly read the description for each of them. Select and open the “Worksheet” (pdf) for one activity that looks relevant to your curriculum. For example, if you want to select the “How do you spend your time?” activity, then open the “Overview” pdf. 
     
    Once you have read the Overview and/or Worksheet for the activity that you selected, please answer the following questions: 
  • What are your impressions of this activity? 
  • What age group do you think is most appropriate for this classroom activity? 
  • Are the instructions on how to implement the classroom activity clear? 
  • Would this classroom activity fit well with your current curriculum? Why or why not? 
  • What improvements could we make to this classroom activity?

Survey Questions

What resources on the site are the most useful to you? Why?

What other types of content (e.g., classroom activities, posters, or other resources) or topics would you like to see on the BLS K-12 page in the future?

What other websites do you use to get ideas for economic lesson plans?

How could we improve the site?

Page 7 of 7