Note to Reviewer

November 9, 2018

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Jennifer Edgar Office of Survey Methods Research
SUBJECT:	Submission of Materials for Usability Testing of Interactive Charts

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 42 hours.

If there are any questions regarding this project, please contact Jennifer Edgar at 202-691-7528 or Jean Fox at 202-691-7370.

1. Introduction

BLS has been moving away from traditional, static, presentation of data to the use of interactive charts and tables which allow users to tailor the information to their needs. While some of these charts have already been incorporated into BLS.gov, no usability testing or other evaluation has been done to ensure that they are effective tool. This study aims to evaluate both charts that are currently on the website, as well as new charts that are being developed. Results from this study will be used to improve the charts.

2. Methodology

This study will use a multi-prong approach to collect the necessary information in the most time and cost-efficient way possible. Traditional, in-lab, usability testing will be supplemented with remote usability testing. The combination of these methods will allow for information to be collected from the target user groups (e.g., we anticipate it would be difficult to recruit journalists or other data users to come to the BLS during a work day).

For all sessions, a semi-structured protocol (Attachment A) will be used. The order of the charts will be randomized, to allow for each chart to be seen first by several participants.

3. Participants

In addition to testing with BLS employees (not included in the burden calculations), we will recruit from two sources:

1. Known data users identified from subscriber lists maintained by BLS program offices. These lists include reporters, academics, educators and others who have requested to be automatically sent newly released data. An email (Attachment B) will be sent to up to 100 users, in batches of 10, with the goal of recruiting up to 20 participants. Participants will be interviewed via WebEx, which will allow OSMR researchers to observe, and record, the participants interacting with the chart. These participants will not be given a consent form or an incentive, unless they come into the lab to complete the study, in which case they’ll receive the standard consent form (attachment C) and $40.

2. Office of Survey Methods Research’s general population database. This database includes participants who have responded to advertisements in the past. Participants will be recruited based on their demographic characteristics, with the goal of obtaining a sample with at least a high school education and diverse in terms of race and gender. Up to 20 participants will be contacted by email or telephone, with a target of 10 to be brought into the OSMR lab. Those who agree to participate and come into the OSMR usability lab will be given a consent form (Attachment C) and paid $40.

4. Burden Hours

Total expected burden for this study is 41.66 hours, as shown in the table below.

Respondents	# Contacted	Time Spent Recruiting	Screening Burden Hours	# Interviewed	Interview Length	Interview Burden Hour	Total Hours
Data Users	100	5 minutes	8.33	20	60 minutes	20	28.33 hours
General Population	20	10 minutes (including screening)	3.33	10	60 minutes	10	13.33 hours
Total Burden Hours							42 hours

5. Payment to Respondents

Participants who come into the OSMR lab will be paid $40.

6. Data Confidentiality

Participants interviewed via WebEx will be asked for their verbal consent, outlined in Attachment C.

Participants interviewed in person will be asked to sign a consent form (Attachment D).

7. Attachments

1. Attachment A: Protocol

2. Attachment B: Recruitment Email

3. Attachment C: Verbal Consent

4. Attachment D: Consent Form

Attachment A: Protocol

OPUBBS Interactive Chart Testing

Protocol November 9, 2018

1. Introduction

1. Looking to get feedback on several types of charts BLS uses to display information.

2. Not necessary to fully understand the statistics being presented, we’re most interested in how the charts work for you, what you like and don’t about them and how we can make them better

3. As you work with the charts, we’d like you to talk about what you’re seeing and thinking. This gives us a lot of information about how the charts are working for you. As you work, I may remind you to let me know what you are thinking.

4. Any questions?

1. First Chart, regardless of type

1. Play with chart

2. Chart Specific Tasks

3. Chart specific questions

4. General Questions

2. Second Chart, regardless of type

1. Play with chart

2. Chart Specific Tasks

3. Chart specific questions

4. General Questions

2. All Remaining Charts

1. Play with chart

2. Chart specific questions

2. Conclusion

1. Thanks, we’re about done. Now that you’ve seen the different types of charts that BLS uses, what do you think?

2. Which type of chart did you like the best? Which one did you like the least? Why?

3. Any other suggestions about how BLS can improve the charts?

4. In person – give check

5. WebEx – thank & disconnect; end recording

Line Chart Protocol

Chart Specific Tasks – go through if chart presented first or second

CPI: https://www.bls.gov/charts/consumer-price-index/consumer-price-index-by-category-line-chart.htm

1. Here is a chart, take a few minutes to explore it. Remember to think aloud as you work, sharing what you think the chart is showing, doing, what you expect to happen, that type of thing.

2. What is the chart telling you?

3. What time period is shown?

4. In the most recent data, which category had higher inflation, shelter or new vehicles? (Goal: compare two categories at one point in time)

5. For medical care services, what was largest 12-month change in the CPI in the last 20 years? (Goal: use tooltips to get actual data values)

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. Can you change the chart so it shows a different time period?

   1. Can you get the chart back to the original time period?

3. (Go through any features that the P didn’t interact with yet). Without touching your mouse, let’s talk about what you think about some features in the charts.

1. Is there a way to have the chart display only some of the categories?

2. Is there a way to have the chart show all of the categories?

3. Is there a way to download the chart?

4. Is there a way to see a table of the data the chart is showing?

4. Does BLS have any other charts about the Consumer Price Index? How would you get to them?

5. Is there anything else you think this chart should be able to do but can’t?

6. Anything you particularly don’t like about this chart?

Line Chart Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. What time period is being shown?

3. In the most recent data, which category had higher inflation, housing or new cars? (Goal: compare two categories at one point in time)

4. For medical care services, what was the highest value for the CPI in the last 20 years? (Goal: use tooltips to get actual data values)

5. Can you change the chart so it shows a different time period?

   1. Can you get the chart back to the original time period?

6. (Click original link & go through any features that the P didn’t interact with yet). Without touching your mouse, let’s talk about what you think about some features in the charts.

   1. What are all the things on this chart that you think are interactive?

   2. What do you think will happen if you put your mouse over the categories above the chart? Now go ahead and do it, did that do what you expected?

Drill Down Columns

Chart Specific Tasks – go through if chart presented first or second

https://www.bls.gov/charts/census-of-fatal-occupational-injuries/fatal-occupational-injuries-by-event-drilldown.htm

1. Here is a chart, take a few minutes to explore it. Remember to think aloud as you work, sharing what you think the chart is showing, doing, what you expect to happen, that type of thing.

2. What is the chart telling you?

3. What time period is being shown?

4. Which industry category had the lowest fatal work injuries? (Goal: Overview task with the chart)

   1. How many of those type of injuries were there? (Goal: Hover over)

5. Of the fire and explosion industry, what specific type of fire had the most injuries?

6. Now, let’s look at another type of injury. Which type of injury led to the highest fatal work injuries? (Goal: go back to the top and dig down another category)

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. How many fatal injuries are shown on the X or vertical axis?

   1. Does this number change as you look at different types of injuries?

3. (Go through any features that the P didn’t interact with yet). Now, without touching your mouse, let’s talk about what you think about some features in the charts.

1. What do you think will happen if you put your mouse over a column? Now go ahead and do it, did that do what you expected?

2. How would you get back to the major cateogries that were shown at first?

3. Is there a way to download the chart?

4. Is there a way to see a table of the data the chart is showing?

7. Does BLS have any other charts about Fatal Occupational Injuries? How would you get to them?

8. Is there anything else you think this chart should be able to do but can’t?

9. Anything you particularly don’t like about this chart?

Drill Down Chart Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. What time period is being shown?

3. Which industry category had the lowest fatal work injuries? (Goal: Overview task with the chart)

1. Of the fire and explosion industry, what specific type of fire had the most injuries?

4. Of the fire and explosion industry, what specific type of fire had the most injuries?

5. Now, let’s look at another type of injury. Which type of injury led to the highest fatal work injuries? (Goal: go back to the top and dig down another category)

6. How many fatal injuries are shown on the X or vertical axis?

1. Does this number change as you look at different types of injuries?

7. What do you think will happen if you put your mouse over a column? Now go ahead and do it, did that do what you expected?

   1. How would you get back to the major cateogries that were shown at first?

Tabbed Charts

Chart Specific Tasks – go through if chart presented first or second

https://www.bls.gov/charts/productivity-wholesale-retail/percent-change-in-productivity.htm

1. Here is a chart, take a few minutes to explore it. Remember to think aloud as you work, sharing what you think the chart is showing, doing, what you expect to happen, that type of thing.

2. What is the chart telling you?

3. What time period is being shown?

4. What sector had the largest 1-year change in productivity? (Goal: Basic overview of chart)

1. What was the percent change for that year? (Goal: Hover over)

2. Within Wholesale trade, which category had the highest 1-year change in productivity? (Goal: Drill Down)

5. Over a longer period than a year, which sector had largest change in productivity? (Goal: Tab to other chart)

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. What is the range of 1-year productivity change shown? l

   1. Does this range change as you look different categories?

3. (Go through any features that the P didn’t interact with yet). Now, without touching your mouse, let’s talk about what you think about some features in the charts.

   1. What do you think will happen if you put your mouse over a line? Now go ahead and do it, did that do what you expected?

   2. What do you think will happen if you put your mouse over a bar? Now go ahead and do it, did that do what you expected?

      1. How would you get back to the major cateogries that were shown at first?

   3. Is there a way to download the chart?

   4. Is there a way to see a table of the data the chart is showing?

10. Does BLS have any other charts about productivity? How would you get to them?

11. Is there anything else you think this chart should be able to do but can’t?

12. Anything you particularly don’t like about this chart?

Tabbed Chart Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. What sector had the largest 1-year change in productivity? (Goal: Basic overview of chart)

   1. What was the percent change for that year? (Goal: Hover over)

   2. Within Wholesale trade, which category had the highest 1-year change in productivity? (Goal: Drill Down)

   3. Within Durable goods, which category had the highest 1-year change in productivity? (Goal: Drill Down)

   4. Can you get back to the chart that showed Wholesale Trade, Retail Trade and Food Services? (Goal: Reset view)

3. Over a longer period than a year, which sector had largest change in productivity? (Goal: Tab to other chart)

Butterfly Charts

https://www.bls.gov/opub/ted/20 13/ted_20130404.htm

Chart Specific Tasks – go through if chart presented first or second

1. Here is a chart, take a few minutes to explore it. Remember to think aloud as you work, sharing what you think the chart is showing, doing, what you expect to happen, that type of thing.

2. What is the chart telling you? (make sure they say both employment and wages)

3. What time period is being shown?

4. Which occupation gets paid the most? (Goal: Overview of chart, reading axis labels)

5. Which occupation has the most number of people? (Goal: Overview of chart, reading axis labels)

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. How would you find out more information about a specific occupation?

3. What is the range of employment shown?

4. What is the range of wages shown?

5. (Go through any features that the P didn’t interact with yet). Now, without touching your mouse, let’s talk about what you think about some features in the charts.

   1. What do you think will happen if you put your mouse over a bar? Now go ahead and do it, did that do what you expected?

   2. What do you think will happen if you click on a bar? Now go ahead and do it, did that do what you expected?

   3. What do you think will happen if you click on an occupation title? Now go ahead and do it, did that do what you expected?

   4. Is there a way to see a table of the data the chart is showing?

13. Is there anything else you think this chart should be able to do but can’t?

14. Anything you particularly don’t like about this chart?

Tabbed Chart Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. Which occupation gets paid the most? (Goal: Overview of chart, reading axis labels)

3. Which occupation has the most number of people? (Goal: Overview of chart, reading axis labels)

4. How would you find out more information about a specific occupation?

5. What is the range of employment shown?

6. What is the range of wages shown?

Maps

https://www.bls.gov/charts/state-employment-and-unemployment/state-unemployment-rates-animated.htm

Chart Specific Tasks – go through if chart presented first or second

1. Here is a map, take a few minutes to explore it. Remember to think aloud as you work, sharing what you think the chart is showing, doing, what you expect to happen, that type of thing.

2. What is this chart showing you?

3. What time period is being shown?

4. Where is the highest unemployment rate currently? (Goal: Overview of current chart, use highlighting)

5. In September of 2016, what was the unemployment rate in Texas? (Goal: use time playback)

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. (Go through any features that the P didn’t interact with yet). Without touching your mouse, let’s talk about what you think about some features in the charts.

1. What do you think will happen if you put your mouse over a state? Now go ahead and do it, did that do what you expected?

2. What do you think will happen if you click the play button at the top? Now go ahead and do it, did that do what you expected?

3. Is there a way to see a table of the data the chart is showing? Now go ahead and do it, did that do what you expected?

4. What do you think will happen if you put your mouse over the labels showing what each color means? Now go ahead and do it, did that do what you expected?

3. Does BLS have any other charts about unemployment? How would you get to them?

4. Is there anything else you think this chart should be able to do but can’t?

5. Anything you particularly don’t like about this chart?

Tabbed Chart Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. Where is the highest unemployment rate currently? (Goal: Overview of current chart, use highlighting)

3. In September of 2016, what was the unemployment rate in Texas? (Goal: use time playback)

4. What do you think will happen if you click the play button at the top? Now go ahead and do it, did that do what you expected?

Linked Charts

https://blsdev.bls.gov/patricia/next_gen_project/mockups/DashboardMockup/MockupCPI-OptionA.htm Note these are currently not aviaable outside of BLS, but will be by the time of testing. Here is what the charts look like:

Chart Specific Tasks – go through if chart presented first or second

1. Here are two charts that work together. Take a few minutes to explore them. Remember to think aloud as you work, sharing what you think the charts are showing, doing, what you expect to happen, that type of thing.

2. What are these charts showing you?

3. What time period is being shown in the bar chart? What about in the line graph?

4. What was the highest monthly percent change in the current month?

5. In November 2009, what was the monthly percent change in the CPI for Food costs?

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. (Go through any features that the P didn’t interact with yet). Without touching your mouse, let’s talk about what you think about some features in the charts.

1. What do you think will happen if you put your mouse over a column? Now go ahead and do it, did that do what you expected?

2. What do you think will happen if you put your mouse over the line? Now go ahead and do it, did that do what you expected?

3. Is there a way to download the charts?

3. Is there anything else you think this chart should be able to do but can’t?

4. Anything you particularly don’t like about this chart?

Linked Chart Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. What are these charts showing you?

3. What time period is being shown in the bar chart? What about in the line graph?

4. What was the highest monthly percent change in the current month?

5. In November 2009, what was the monthly percent change in the CPI for Food costs?

Expandable Tables

https://blsdev.bls.gov/patricia/next_gen_project/mockups/DashboardMockup/MockupCPI-OptionA.htm Note these are currently not aviaable outside of BLS, but will be by the time of testing. Here is what the table looks like:

Chart Specific Tasks – go through if chart presented first or second

1. Here is a table. Take a few minutes to explore themit Remember to think aloud as you work, sharing what you think the table is showing, doing, what you expect to happen, that type of thing.

2. What is this table showing you?

3. In July 2017, what was the average unadjusted index for Fruits and vegetables?

General Questions – go through if chart is presented first or second

1. (Click the original chart link.) Great, now I have a few specific questions about the chart. Can you show me all the things on this chart that you think are interactive?

2. (Go through any features that the P didn’t interact with yet). Without touching your mouse, let’s talk about what you think about some features in the charts.

4. What do you think will happen if you click on one of the blue plus circles? Now go ahead and do it, did that do what you expected?

5. What do you think will happen if you click on one of the blue minus circles? Now go ahead and do it, did that do what you expected?

6. Is there a way to see all the rows in the table at once?

7. Is there a way to collapse all the rows in the table at once?

3. Is there anything else you think this chart should be able to do but can’t?

4. Anything you particularly don’t like about this chart?

Expandable Table Specific Questions – go through if chart is NOT presented first or second

1. What is the chart telling you?

2. What are these charts showing you?

3. In July 2017, what was the average unadjusted index for Fruits and vegetables?

4. What do you think will happen if you click on one of the blue plus circles? Now go ahead and do it, did that do what you expected?

5. What do you think will happen if you click on one of the blue minus circles? Now go ahead and do it, did that do what you expected?

Attachment B: Recruitment Email

Hello

The Bureau of Labor Statistics (BLS) has been exploring ways to present our data in a more interactive way. We are reaching out to invite you to participate in a research study to evaluate some of our charts and tables. As a subscriber to BLS data, your feedback will be valuable in our efforts to ensure that our materials are user friendly.

This research study would take about an hour of your time. Using WebEx, we’ll connect with you remotely, both showing you charts and then watching as you work with them to complete some tasks. Alternatively, you could come to our office in Washington, DC next to Union Station to participate in person.

Please reply to this email, or call me at 202-691-7528, if you would be willing to help with this important study.

Thank you,

Jennifer Edgar, PhD
Office of Survey Methods Research
Bureau of Labor Statistics
Suite 5930, 2 Massachusetts Ave., NE
Washington, DC 20212
202-691-7528

Attachment C: Verbal Consent Outline

Upon beginning a remote testing session, participants will be told the following information and asked for their verbal consent prior to beginning the study:

• During this research you will be recorded. If you do not wish to be recorded, you still may participate in this research.

• We estimate it will take you an average of 60 minutes to participate in this research.

• Your participation in this research project is voluntary, and you have the right to stop at any time.

• Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141 and expires March 31, 2021.

Attachment D: Consent Form

Consent Form

OMB Control Number: 1220-0141

Expiration Date: March 31, 2021

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 60 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141 and expires March 31, 2021.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2.  Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections.  The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection.  Information on routine uses can be found in the system of records notice,  DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418). 

23