Note to Reviewer - Next Gen News Release Usability Test_Final

December 31, 2019

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Struther Van Horn Office of Survey Methods Research
SUBJECT:	Submission of Materials for Usability Testing of the Next Generation of BLS News Releases

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden for this study is 137 hours.

If there are any questions regarding this project, please contact Struther Van Horn at 202-691-7568 or Jean Fox at 202-691-7370.

1. Introduction

BLS is moving away from the traditional static presentation of data to the use of interactive charts and tables. This allows users to tailor the information to their needs. In FY19, members of the Office of Survey Methods Research (OSMR) conducted usability testing on several interactive charts and identified recommendations for improving the charts’ usability. Where possible, the improvements have been made to the charts, and the charts have been incorporated into prototypes for the next generation BLS news releases.

This study will examine the usability of the updated interactive charts as well as the prototype news releases as a whole. Most participants will use a desktop or laptop computer for the testing, but some will use a mobile device, as some users are likely to access the news releases remotely. BLS plans to conduct the usability testing with data users. In addition to conducting the usability testing, BLS is also requesting approval through this clearance request to conduct focus groups with members of the media. Focus groups will allow for a greater in-depth discussion of satisfaction with the new prototypes than usability testing alone.

2. Methodology

This study will use a multi-prong approach to collect the necessary information in the most time and cost-efficient way possible. We will supplement traditional in-lab usability testing with remote usability testing. Since each method has its own strengths and weaknesses, using a combination of methods will allow for more thorough evaluation of the news releases. For the usability testing, we will use several methods: (1) moderated usability testing in the OSMR cognitive laboratory, (2) remote unmoderated usability testing using the service TryMyUI¹, and (3) remote moderated usability testing using the web conferencing service Zoom². The strengths and weaknesses of the methods are shown below.

Method	Strengths	Weaknesses
In-lab testing	Easy to observe participants and record sessions Works with both desktop and mobile devices	It is difficult to recruit data users, so most participants don’t understand basic BLS concepts (and so are not necessarily the target audience)
TryMyUI	Easy to set up Some ability to screen on certain characteristics Easy and fast to run participants Audio recording of participants ‘think aloud’ reactions as they complete the tasks and video recording of their screens No technical issues with cell phone users	Unpredictable success rates for screening participants (but it has been fairly good in the past) Max of 30 minutes per participant Limit of 4 post-test questions
Zoom	Able to recruit actual data users Able to view and record desktops/laptops and mobile devices.	Difficult to recruit actual data users Possible technical issues in starting the remote session

For all the moderated usability test sessions (in-lab and Zoom), we will use a semi-structured protocol (Attachment A). In this format, participants will first explore the news release on their own while thinking aloud, so we can see what they discover on their own. Then, we will ask them to complete several specific tasks to see how they use the news releases. Tasks will include finding and interpreting information in the news release as well as using interactive features of the news release (see Attachment B for the specific tasks). We will randomize the order of the news releases so half the participants see each version first. The in-lab participants will receive $40 for their participation. The remote participants will not receive an incentive.

With TryMyUI, we will only have 30 minutes per participant, so each will work with just one of the prototypes. They will complete the same tasks for the news release they use as the other participants with the prototype they see. Half of the TryMyUI participants will complete the study with each of the prototypes. The TryMyUI participants will receive a $10 incentive through the TryMyUI service.

In addition to the usability testing, OSMR will conduct focus groups with members of the media to obtain reactions on the new format and content of the news releases. We hope to recruit up to 20 participants for up to four focus groups (average of five participants per group). The focus groups will follow the protocol outlined in Attachment C.

3. Participants

We will recruit from four sources:

1. For the remote moderated testing with Zoom, we will identify known data users from subscriber email lists maintained by BLS program offices. These lists include reporters, academics, educators and others who have requested to be automatically sent newly released data. We will attempt to target business professionals and financial experts as best as we can based on their email. We will send an email (Attachment D) to up to 1,000 users, in batches of 100, with the goal of recruiting up to 14 participants. The email will explain the purpose of the test and describe what the participants will do.

2. For the TryMyUI testing, we will recruit 10 participants, screening for participants that are likely to have experience with BLS data.

3. For the in-lab testing, we will recruit from OSMR’s general population database of participants. This database includes people who have participated in previous studies or responded to advertisements for studies. We will recruit participants based on their demographic characteristics and job title, with the goal of obtaining a sample with at least a high school education and diverse in terms of race and gender, and who may be likely to be familiar with BLS data through their job. We will contact up to 25 participants by email or telephone, with a target of 6 to participate in the testing.

4. For the focus groups, we will work with the publications office at BLS to identify potential participants. We will target members of the media known to use the current news releases. We will send an email (Attachment E) to recruit these members, explaining the purpose of the focus group and ask if they’re interested in participating. We will follow up with phone calls as necessary, and estimate that it will take an average of 5 minutes to recruit each participant by phone and that we will need to contact 40 people by phone to recruit 20 participants.

4. Burden Hours

Total expected burden for this study is 137 hours, as shown in the table below.

Respondents	# Contacted	Time Spent Recruiting	Total Screening Burden (Hours)	Number of Participants	Session Length (Hours)	Total Session Burden (Hours)	Total Burden (Hours)
General Population	25	10 minutes (including screening)	4.17	6	1.0	6	10.17
Remote (Zoom)	Up to 1,000	5 minutes	83.33	14	1.0	14	97.33
Remote (TryMyUI)	10	5 minutes	0.83	10	0.5	5	5.83
Focus Group	40	5 minutes	3.33	20	1.0	20	23.33
Total Burden Hours							136.66

5. Payment to Respondents

Participants who come into the OSMR lab will be paid $40.

Participants in the TryMyUI tests will receive $10 through TryMyUI.

Remote participants using Zoom and participants in the focus groups will not be paid.

6. Data Confidentiality

Those participating in the usability and focus groups in person will sign a consent form (Attachment F), which includes the standard BLS confidentiality statement for usability testing.

Those participating in the remote usability test via Zoom will be asked for their verbal consent, outlined in Attachment G. No pledge of confidentiality will be provided.

Those participating through TryMyUI give their consent through the TryMyUI service. In addition, we will include the following text in the introduction:

Please read the following silently to yourself:

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 30 minutes to complete. This study is being administered by TryMyUI.com and resides on a server outside of the BLS Domain. BLS cannot guarantee the protection of responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

7. Attachments

1. Attachment A: Usability Testing Protocol

2. Attachment B: Tasks for Usability Testing

3. Attachment C: Focus Group Protocol

4. Attachment D: Recruitment Email for usability tests

5. Attachment E: Recruitment Email for focus groups

6. Attachment F: Consent Form for In-Lab Usability Test and Focus Groups

7. Attachment G: Verbal Consent Outline for Remote Test Sessions with Zoom

Attachment A: Usability Testing Protocol

BLS Next Generation News Release Testing

Protocol December 20, 2019

1. Introduction

1. We are looking to get feedback on two samples of a new design for our news releases.

2. We are interested in how the news releases work for you, what you like and don’t about them and how we can make them better

3. As you work with the news releases, we’d like you to talk about what you’re seeing and thinking. This gives us a lot of information about how the news releases are working for you. As you work, I may remind you to let me know what you are thinking.

4. Any questions?

5. Consent

   1. In-lab participants will sign the consent form in Attachment F.

   2. Remote participants will agree to the verbal consent provided in Attachment G.

   3. TryMyUI participants will read the OMB information below, which will be included in the TryMyUI protocol.

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 30 minutes to complete. This study is being administered by TryMyUI.com and resides on a server outside of the BLS Domain. BLS cannot guarantee the protection of responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

1. First News Release

1. Explore news release

2. News release specific tasks and follow-up probes

2. Second News Release

Note: TryMyUI participants will only work with one release

1. Explore news release

2. News release specific tasks and follow-up probes

2. Debriefing

A. Moderated (Lab and Zoom)

1. Thanks, we’re about done. Now that you’ve seen the new news releases, what do you think?

2. What did you like the best about the news releases? What did you like the least? Why?

3. What else can BLS do to improve the news releases?

4. Thank the participant

   1. In person – give check

   2. Zoom – thank & disconnect; end recording

B. Unmoderated (TryMyUI)

1. What are your overall impressions of the news releases?

2. What did you like the best about the news releases and why?

3. What did you like the least and why?

4. What else can BLS do to improve the news releases?

Attachment B: Tasks for Usability Testing

• Half the participants will see each news release first.

• For each news release, we will ask participants to explore the news release and talk about what they see for a few minutes. Then, we will ask them to complete the tasks and ask follow-up probes as needed.

Employment Situation

General Exploration

Here is a prototype for a news release for the Employment Situation. Take a few minutes to explore it. Please think aloud as you work, sharing what you think the news release is showing, what the features do, and anything else you might experience.

Goal: What do participants notice first? What do they think of what they see?

Tasks

Note: Duplicate tasks could be asked with either prototype but only need to be asked with one.

Task 1: What are the values for unemployment and payroll employment for July 2019? [3.7% and +164,000 respectively]

Follow up-probe: What does the payroll employment value represent?

Goal: Can participants find the basic statistics on the news release?

Goal: Do they know that payroll employment represents a change in employment?

Task 2: Look at the headers of the two sections with charts. What do these titles mean to you?

Goal: Do participants understand the chart titles? Do they know what “establishment survey data” and “household survey data” mean?

(duplicate) Task 3: Now I’d like to ask you to find some specific data. What is the change in employment in Government jobs in July 2019? [+44,000, Establishment data / Employment tab]

Follow-up probe: What is the full name of the Transportation industry? [Transportation and warehousing] (Note: on mobile devices, participants will have to go into the table to see the label)

Follow-up probe: What time period is covered in this chart? [1-month, shown in the legend]

Follow-up probe: What do the blue bars mean? [90% confidence interval]

Goal: Do participants see the other tabs? Can they interpret the chart correctly?

Goal: Can they find the full text for the truncated labels?

Task 4: Tell me what you think about the tab labels. What do they mean to you?

Goal: Do participants understand the tab labels?

Task 5: Can you describe what the “alternative measures” are?

Follow-up probe: Is there a way to find definitions for any of the terms used here?

Goal: Have them look around the alternative measures tab. See if they find the hover-over definitions for the underlined text.

(duplicate) Task 6: Find the chart showing unemployment rates. Where would you find important information related to the July 2019 data?

Goal: Do participants get the connection between the chart and the bullet points?

Task 7: What is the change in total nonfarm employment for June 2019? [166,000, Establishment data / total nonfarm tab]

Goal: Do participants recognize the values in the chart are in thousands?

(duplicate) Task 8: How does BLS define those who are considered to be unemployed? [additional informationabout this release]

Follow-up probe: Can you describe the information on this page? What do you think of it?

Goal: Can participants find the “about this release” information (the old tech notes)? Can they find the definitions? Do they understand the “more information” arrows?

(duplicate) Task 9: What are the options for downloading tables of data?

Goal: Do participants see the “Show Table Data” link?

Goal: Can participants view tables of data from the table link?

(duplicate) Task 10: What do you think the Interact links at the bottom of the chart do?

Goal: Do participants know what the labels mean?

(duplicate) Task 11: How would you download one of the charts?

Goal: Can participants download charts? Do they understand the button label (the label doesn’t show on mobile devices)?

(duplicate) Task 12: What do the tweet buttons do? Would you use them? Have you used a similar feature elsewhere on BLS.gov or on other sites?

Goal: Do participants find value in the Tweet buttons? They may be very salient.

Task 13: In the Establishment Surveys, describe what you see with the Earnings chart.

Follow-up probe: Look at the data for July 2019. What does that value tell you?

Goal: Do participants understand this is showing percent change as opposed to percents, which is what many of the charts show?

CPI

General Exploration

Here is a prototype for a news release for the Consumer Price Index. Take a few minutes to explore it. Please think aloud as you work, sharing what you think the news release is showing, what the features do, and anything else you might experience.

Goal: What do participants notice first? What do they think of what they see?

Tasks

Note: duplicate tasks could be asked with either prototype but only need to be asked for the first one participants are exposed to.

Task 1: What are the July 2019 values for 1-month change and 12-month change?

Goal: Can participants find the basic statistics on the news release?

Task 2: Look at the headers of the two sections with charts. What do these titles mean to you?

Goal: Do participants understand the section titles?

(duplicate) Task 3: Looking at the chart showing the 1-month change for all items, can you describe what the “CPI-U” is?

Follow-up probe: Is there a way to find a definitions for any of the terms used here?

Goal: Have them look around the CPI, 1-Month Changes Tab. See if they find the hover-over definition for the underlined text “The consumer price index for all urban consumers”.

(duplicate) Task 4: Now I’d like to ask you to find some specific data. How has the price of fruits and vegetables changed from June to July 2019? [+0.3%, 1-month change chart, Food tab]

Goal: Do participants see the additional tabs?

Task 5: Tell me what you think about the tab labels. What do they mean to you?

Goal: Do participants understand the tab labels?

(duplicate) Task 6: Find the chart showing one-month change in food prices for July 2019. Where would you find important information about the July 2019 data?

Goal: Do participants get the connection between the chart and the bullet points?

(duplicate) Task 7: Can you find information about how BLS collects data for the CPI? [additional informationabout this release]

Follow-up probe: Can you describe the information on this page? What do you think of it?

Goal: Can participants find the “about this release” information (the old tech notes)?

(duplicate) Task 8: What are the options for downloading tables of data?

Goal: Do participants see the “Show Table Data” link?

Goal: Can participants view tables of data from the table link?

(duplicate) Task 9: What do you think the [other] Interact links at the bottom of the chart do?

Goal: Do participants know what the labels mean?

(duplicate) Task 10: What do the tweet buttons do? Would you use them? Have you used a similar feature elsewhere on BLS.gov or on other sites?

Goal: Do participants find value in the Tweet buttons? They may be very salient.

(duplicate) Task 11: How would you download one of the CPI charts?

Goal: Can participants download charts? Do they understand the button label (the label doesn’t show on mobile devices)?

(duplicate) Task 12: On the chart showing “all items less food & energy” what is the full label for the last category?

Goal: Do participants know how to find the full text for the labels?

Attachment C: Focus Group Protocol

Thank you for joining us today. My name is ______ and I am a researcher here at the Bureau of Labor Statistics. Before we begin, let me give you some context for our objectives today.

We have invited you to join us today to give feedback about the news releases we currently provide for our Principal Federal Economic Indicators.

Since you report on our data, we want to hear from you about how you use the news releases and what your needs are. In particular, we have a redesigned version of the News Releases and we are especially interested in your feedback on the new format and content.

Before we continue, to allow the conversation to flow more smoothly, I’d like to go over a few ground rules so that everyone gets a chance to participate and to ensure that the session will end on time:

- First, I want to emphasize that there are no right or wrong answers to the questions I’ll be asking. The whole point of this discussion is to hear your opinions about the news releases. It doesn’t matter whether you have a positive or a negative opinion, both are important to hear.

- Next, I want to assure you that everything we discuss today will remain confidential. Whatever information we obtain from you will not be shared with anyone in a way that identifies who you are. For example, we won’t use your name or the name of your organization in the report of the findings that we’ll be writing. To facilitate this outcome, I’ll just be calling you by your first name, and when you talk about your organization, try not to mention it by name.

- We also expect that you’ll treat the information we exchange today as confidential. You should not identify anyone who participates in this group after we’re finished today.

- With your consent, we’ll be recording the audio of our discussion today. This helps reduce the burden of note taking, plus others who could not be here today will be able to hear what you have to say. The only people who will listen to the tape will be BLS research staff. Does anyone have any objection to audio-taping? Please take a minute to review and sign the consent form at your seat. It summarizes the information I just explained. (If anyone has objections to the audio recording, hand written notes will be taken)

- Finally, I want everyone to get a chance to speak when they have something to say. So I ask that you try to keep your comments brief and related to the issue at hand. And I’d appreciate it if you wait until another person is finished speaking before speaking up yourself.

Are there any questions about what I have just covered? Everyone should have received links to the redesigned news releases for the Consumer Price Index and the Employment Situation (along with a printable copy) last week. A copy is at your seat for your reference, though it’s important to know that these are intended to be distributed digitally only.

How about if we quickly introduce ourselves? Please tell us your first name, your position, how long you have used BLS data, and which news releases you use.

Issue A: Icebreaker Questions about the News Releases

Before we begin our discussion about the new version of the News Releases, I want to ask some general questions about your thoughts on the current BLS news releases.

1. What is your general opinion of current BLS news releases? If you were asked to give them a grade, A, B, C, D or F, what grade would you assign, and why?

2. In comparison, based on what you’ve seen so far, what grade would you give the new design for the news releases?

3. Before we take a closer look at the redesigned news releases, I’d like to know how you use the current BLS news releases, for example, what information in the news releases do you focus on?

(Probe, if necessary): How much of the explanatory, descriptive text do you use?

Issue B: Specific Questions about Revised News Releases

Let’s turn our attention to the revised versions of the news releases. As we go through this quick review, please feel free to mention what you like and what you don’t like about the design.

Using a laptop and projector system, access the following website: https://beta.bls.gov/nextgen/empsit.htm Walk through the basics of new layout for the Employment Situation, showing news release on the projector screen. Moderator Note: Allow comments to be made during the walkthrough.

1. Based on what you’ve seen, how well does this new format for the Employment Situation meet your needs?

(Probe, if necessary): How does it fail to meet your needs?

2. What do you think about the amount of information that is provided in this new design for the Employment Situation? Is there too much, too little, or the right amount? Are all the important facts covered?

Probe: If necessary, probe about the following:

• Unemployment rate for different demographic groups (e.g., women, older workers, etc.) not being discussed in the text

• Embedded tables

• Data tables

• Charts

• Technical notes

Using a laptop and projector system, access the following website: https://beta.bls.gov/nextgen/cpi.htm Walk through the basics of new layout for the Consumer Price Index, showing news release on the projector screen. Moderator Note: Allow comments to be made during the walkthrough.

3. How well does this new design for the Consumer Price Index meet your needs?

(Probe, if necessary): How does it fail to meet your needs?

4. What do you think about the amount of information that is provided in this new design for the Consumer Price Index? Is there too much, too little, or the right amount? Are all the important facts covered?

Probe: If necessary, probe about the following:

• Embedded tables

• Data tables

• Charts

• Technical notes

5. Did these walk-throughs change anyone’s overall rating of the new design? (discuss) (Probe: Are we headed in the right direction with our new design or do we need to rethink our approach?)

Wrap-Up

This session is almost over and before we end, I’d like to get your final thoughts.

1. If you could wave a magic wand and change one thing about the new version of the News Releases, what would it be?

2. Is there any topic that we’ve missed that you think is important to our discussion about improving the New Releases?

Thank you very much for participating today; your thoughts and feedback will be very useful. If you have any other thoughts, comments or questions following this session, please feel free to contact me at [give email address or business card].

Moderator Note: If a participant wants to talk about specific PFEI issues or has questions that are not appropriate for the focus group, say something like “I can’t give you any direct feedback on that issue/suggestions, but will I will make a note of it and make sure that it gets passed on to the right people at BLS.”

The Employment Situation Handout

The Consumer Price Index Handout

Attachment D: Recruitment Email for Usability Testing

Subject: Invitation from BLS

Hello

At the Bureau of Labor Statistics (BLS), we have been exploring more interactive ways to present our news releases. We would like to get input from data users to be sure that they are easy to use. As a subscriber to BLS data, we would like to invite you to participate in a research study to evaluate our prototype designs. Your feedback will help us ensure that our final products are user friendly.

This research study would take about an hour of your time. Using Zoom, we’ll connect with you remotely and ask you to share your screen as you work with the news releases to complete some tasks. We are looking for people who would like to participate either on a desktop/laptop computer or on a smart phone.

If you would be willing to help with this important study, please let me know. We are hoping to complete the study by late January.

Thank you,

Attachment E: Recruitment Email for Focus Groups

Subject: Invitation from BLS

Hello

At the Bureau of Labor Statistics (BLS), we have been exploring update our news releases and present the data in more interactive ways. We would like to get input from members of the media to be sure that the charts and tables in our news releases are easy for you to use. As a user of BLS data and member of the media, we would like to invite you to participate in a focus group to evaluate our prototype designs. Your feedback will help us ensure that our materials are user friendly.

This discussion would take about an hour of your time, and will aim to schedule the session at your convenience. We are conducting the focus group sessions at the Bureau of Labor Statistics office near Union Station (2 Massachusetts Ave., Washington, DC 20212).

If you would be willing to help with this important study, please reply to this email.

Thank you,

Attachment F: Consent Form for In-Lab Usability Tests and Focus Groups

Consent Form

OMB Control Number: 1220-0141

Expiration Date: March 31, 2021

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to disseminate survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be recorded, or you may be observed. If you do not wish to be recorded, you still may participate in this research.

We estimate your participation will last approximately 60 minutes.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires March 31, 2021.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2.  Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections.  The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection.  Information on routine uses can be found in the system of records notice,  DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418). 

Attachment G: Verbal Consent Outline for Remote Test Sessions with Zoom

Upon beginning a remote testing session, participants will be told the following information and asked for their verbal consent prior to beginning the study:

• During this study, we will record the screen you share and the audio from our call. If you do not wish to be recorded, you still may participate in this research.

• We estimate the session will last about 60 minutes.

• Your participation in this research project is voluntary, and you have the right to stop at any time.
  
  

• The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this research.

Are you ok with this?

1 TryMyUI is an online usability testing platform that provides 10 participants and a 30 minute video for $300 per month used (see www.TryMyUI.com).

2 Zoom allows remote participants to share their cell phone screen, which they can’t do with WebEx.

26