Form Survey Protocol Survey Protocol Attachments A-G

Attachment A: Usability Testing Protocol

BLS Next Generation News Release Testing

Protocol December 20, 2019

1. Introduction

1. We are looking to get feedback on two samples of a new design for our news releases.

2. We are interested in how the news releases work for you, what you like and don’t about them and how we can make them better

3. As you work with the news releases, we’d like you to talk about what you’re seeing and thinking. This gives us a lot of information about how the news releases are working for you. As you work, I may remind you to let me know what you are thinking.

4. Any questions?

5. Consent

   1. In-lab participants will sign the consent form in Attachment F.

   2. Remote participants will agree to the verbal consent provided in Attachment G.

   3. TryMyUI participants will read the OMB information below, which will be included in the TryMyUI protocol.

The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this survey. Your participation is voluntary, and you have the right to stop at any time. This study will take no more than 30 minutes to complete. This study is being administered by TryMyUI.com and resides on a server outside of the BLS Domain. BLS cannot guarantee the protection of responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

1. First News Release

1. Explore news release

2. News release specific tasks and follow-up probes

2. Second News Release

Note: TryMyUI participants will only work with one release

1. Explore news release

2. News release specific tasks and follow-up probes

2. Debriefing

A. Moderated (Lab and Zoom)

1. Thanks, we’re about done. Now that you’ve seen the new news releases, what do you think?

2. What did you like the best about the news releases? What did you like the least? Why?

3. What else can BLS do to improve the news releases?

4. Thank the participant

   1. In person – give check

   2. Zoom – thank & disconnect; end recording

B. Unmoderated (TryMyUI)

1. What are your overall impressions of the news releases?

2. What did you like the best about the news releases and why?

3. What did you like the least and why?

4. What else can BLS do to improve the news releases?

Attachment B: Tasks for Usability Testing

• Half the participants will see each news release first.

• For each news release, we will ask participants to explore the news release and talk about what they see for a few minutes. Then, we will ask them to complete the tasks and ask follow-up probes as needed.

Employment Situation

General Exploration

Here is a prototype for a news release for the Employment Situation. Take a few minutes to explore it. Please think aloud as you work, sharing what you think the news release is showing, what the features do, and anything else you might experience.

Goal: What do participants notice first? What do they think of what they see?

Tasks

Note: Duplicate tasks could be asked with either prototype but only need to be asked with one.

Task 1: What are the values for unemployment and payroll employment for July 2019? [3.7% and +164,000 respectively]

Follow up-probe: What does the payroll employment value represent?

Goal: Can participants find the basic statistics on the news release?

Goal: Do they know that payroll employment represents a change in employment?

Task 2: Look at the headers of the two sections with charts. What do these titles mean to you?

Goal: Do participants understand the chart titles? Do they know what “establishment survey data” and “household survey data” mean?

(duplicate) Task 3: Now I’d like to ask you to find some specific data. What is the change in employment in Government jobs in July 2019? [+44,000, Establishment data / Employment tab]

Follow-up probe: What is the full name of the Transportation industry? [Transportation and warehousing] (Note: on mobile devices, participants will have to go into the table to see the label)

Follow-up probe: What time period is covered in this chart? [1-month, shown in the legend]

Follow-up probe: What do the blue bars mean? [90% confidence interval]

Goal: Do participants see the other tabs? Can they interpret the chart correctly?

Goal: Can they find the full text for the truncated labels?

Task 4: Tell me what you think about the tab labels. What do they mean to you?

Goal: Do participants understand the tab labels?

Task 5: Can you describe what the “alternative measures” are?

Follow-up probe: Is there a way to find definitions for any of the terms used here?

Goal: Have them look around the alternative measures tab. See if they find the hover-over definitions for the underlined text.

(duplicate) Task 6: Find the chart showing unemployment rates. Where would you find important information related to the July 2019 data?

Goal: Do participants get the connection between the chart and the bullet points?

Task 7: What is the change in total nonfarm employment for June 2019? [166,000, Establishment data / total nonfarm tab]

Goal: Do participants recognize the values in the chart are in thousands?

(duplicate) Task 8: How does BLS define those who are considered to be unemployed? [additional informationabout this release]

Follow-up probe: Can you describe the information on this page? What do you think of it?

Goal: Can participants find the “about this release” information (the old tech notes)? Can they find the definitions? Do they understand the “more information” arrows?

(duplicate) Task 9: What are the options for downloading tables of data?

Goal: Do participants see the “Show Table Data” link?

Goal: Can participants view tables of data from the table link?

(duplicate) Task 10: What do you think the Interact links at the bottom of the chart do?

Goal: Do participants know what the labels mean?

(duplicate) Task 11: How would you download one of the charts?

Goal: Can participants download charts? Do they understand the button label (the label doesn’t show on mobile devices)?

(duplicate) Task 12: What do the tweet buttons do? Would you use them? Have you used a similar feature elsewhere on BLS.gov or on other sites?

Goal: Do participants find value in the Tweet buttons? They may be very salient.

Task 13: In the Establishment Surveys, describe what you see with the Earnings chart.

Follow-up probe: Look at the data for July 2019. What does that value tell you?

Goal: Do participants understand this is showing percent change as opposed to percents, which is what many of the charts show?

CPI

General Exploration

Here is a prototype for a news release for the Consumer Price Index. Take a few minutes to explore it. Please think aloud as you work, sharing what you think the news release is showing, what the features do, and anything else you might experience.

Goal: What do participants notice first? What do they think of what they see?

Tasks

Note: duplicate tasks could be asked with either prototype but only need to be asked for the first one participants are exposed to.

Task 1: What are the July 2019 values for 1-month change and 12-month change?

Goal: Can participants find the basic statistics on the news release?

Task 2: Look at the headers of the two sections with charts. What do these titles mean to you?

Goal: Do participants understand the section titles?

(duplicate) Task 3: Looking at the chart showing the 1-month change for all items, can you describe what the “CPI-U” is?

Follow-up probe: Is there a way to find a definitions for any of the terms used here?

Goal: Have them look around the CPI, 1-Month Changes Tab. See if they find the hover-over definition for the underlined text “The consumer price index for all urban consumers”.

(duplicate) Task 4: Now I’d like to ask you to find some specific data. How has the price of fruits and vegetables changed from June to July 2019? [+0.3%, 1-month change chart, Food tab]

Goal: Do participants see the additional tabs?

Task 5: Tell me what you think about the tab labels. What do they mean to you?

Goal: Do participants understand the tab labels?

(duplicate) Task 6: Find the chart showing one-month change in food prices for July 2019. Where would you find important information about the July 2019 data?

Goal: Do participants get the connection between the chart and the bullet points?

(duplicate) Task 7: Can you find information about how BLS collects data for the CPI? [additional informationabout this release]

Follow-up probe: Can you describe the information on this page? What do you think of it?

Goal: Can participants find the “about this release” information (the old tech notes)?

(duplicate) Task 8: What are the options for downloading tables of data?

Goal: Do participants see the “Show Table Data” link?

Goal: Can participants view tables of data from the table link?

(duplicate) Task 9: What do you think the [other] Interact links at the bottom of the chart do?

Goal: Do participants know what the labels mean?

(duplicate) Task 10: What do the tweet buttons do? Would you use them? Have you used a similar feature elsewhere on BLS.gov or on other sites?

Goal: Do participants find value in the Tweet buttons? They may be very salient.

(duplicate) Task 11: How would you download one of the CPI charts?

Goal: Can participants download charts? Do they understand the button label (the label doesn’t show on mobile devices)?

(duplicate) Task 12: On the chart showing “all items less food & energy” what is the full label for the last category?

Goal: Do participants know how to find the full text for the labels?

Attachment C: Focus Group Protocol

Thank you for joining us today. My name is ______ and I am a researcher here at the Bureau of Labor Statistics. Before we begin, let me give you some context for our objectives today.

We have invited you to join us today to give feedback about the news releases we currently provide for our Principal Federal Economic Indicators.

Since you report on our data, we want to hear from you about how you use the news releases and what your needs are. In particular, we have a redesigned version of the News Releases and we are especially interested in your feedback on the new format and content.

Before we continue, to allow the conversation to flow more smoothly, I’d like to go over a few ground rules so that everyone gets a chance to participate and to ensure that the session will end on time:

- First, I want to emphasize that there are no right or wrong answers to the questions I’ll be asking. The whole point of this discussion is to hear your opinions about the news releases. It doesn’t matter whether you have a positive or a negative opinion, both are important to hear.

- Next, I want to assure you that everything we discuss today will remain confidential. Whatever information we obtain from you will not be shared with anyone in a way that identifies who you are. For example, we won’t use your name or the name of your organization in the report of the findings that we’ll be writing. To facilitate this outcome, I’ll just be calling you by your first name, and when you talk about your organization, try not to mention it by name.

- We also expect that you’ll treat the information we exchange today as confidential. You should not identify anyone who participates in this group after we’re finished today.

- With your consent, we’ll be recording the audio of our discussion today. This helps reduce the burden of note taking, plus others who could not be here today will be able to hear what you have to say. The only people who will listen to the tape will be BLS research staff. Does anyone have any objection to audio-taping? Please take a minute to review and sign the consent form at your seat. It summarizes the information I just explained. (If anyone has objections to the audio recording, hand written notes will be taken)

- Finally, I want everyone to get a chance to speak when they have something to say. So I ask that you try to keep your comments brief and related to the issue at hand. And I’d appreciate it if you wait until another person is finished speaking before speaking up yourself.

Are there any questions about what I have just covered? Everyone should have received links to the redesigned news releases for the Consumer Price Index and the Employment Situation (along with a printable copy) last week. A copy is at your seat for your reference, though it’s important to know that these are intended to be distributed digitally only.

How about if we quickly introduce ourselves? Please tell us your first name, your position, how long you have used BLS data, and which news releases you use.

Issue A: Icebreaker Questions about the News Releases

Before we begin our discussion about the new version of the News Releases, I want to ask some general questions about your thoughts on the current BLS news releases.

1. What is your general opinion of current BLS news releases? If you were asked to give them a grade, A, B, C, D or F, what grade would you assign, and why?

2. In comparison, based on what you’ve seen so far, what grade would you give the new design for the news releases?

3. Before we take a closer look at the redesigned news releases, I’d like to know how you use the current BLS news releases, for example, what information in the news releases do you focus on?

(Probe, if necessary): How much of the explanatory, descriptive text do you use?

Issue B: Specific Questions about Revised News Releases

Let’s turn our attention to the revised versions of the news releases. As we go through this quick review, please feel free to mention what you like and what you don’t like about the design.

Using a laptop and projector system, access the following website: https://beta.bls.gov/nextgen/empsit.htm Walk through the basics of new layout for the Employment Situation, showing news release on the projector screen. Moderator Note: Allow comments to be made during the walkthrough.

1. Based on what you’ve seen, how well does this new format for the Employment Situation meet your needs?

(Probe, if necessary): How does it fail to meet your needs?

2. What do you think about the amount of information that is provided in this new design for the Employment Situation? Is there too much, too little, or the right amount? Are all the important facts covered?

Probe: If necessary, probe about the following:

• Unemployment rate for different demographic groups (e.g., women, older workers, etc.) not being discussed in the text

• Embedded tables

• Data tables

• Charts

• Technical notes

Using a laptop and projector system, access the following website: https://beta.bls.gov/nextgen/cpi.htm Walk through the basics of new layout for the Consumer Price Index, showing news release on the projector screen. Moderator Note: Allow comments to be made during the walkthrough.

3. How well does this new design for the Consumer Price Index meet your needs?

(Probe, if necessary): How does it fail to meet your needs?

4. What do you think about the amount of information that is provided in this new design for the Consumer Price Index? Is there too much, too little, or the right amount? Are all the important facts covered?

Probe: If necessary, probe about the following:

• Embedded tables

• Data tables

• Charts

• Technical notes

5. Did these walk-throughs change anyone’s overall rating of the new design? (discuss) (Probe: Are we headed in the right direction with our new design or do we need to rethink our approach?)

Wrap-Up

This session is almost over and before we end, I’d like to get your final thoughts.

1. If you could wave a magic wand and change one thing about the new version of the News Releases, what would it be?

2. Is there any topic that we’ve missed that you think is important to our discussion about improving the New Releases?

Thank you very much for participating today; your thoughts and feedback will be very useful. If you have any other thoughts, comments or questions following this session, please feel free to contact me at [give email address or business card].

Moderator Note: If a participant wants to talk about specific PFEI issues or has questions that are not appropriate for the focus group, say something like “I can’t give you any direct feedback on that issue/suggestions, but will I will make a note of it and make sure that it gets passed on to the right people at BLS.”

The Employment Situation Handout

The Consumer Price Index Handout

Attachment D: Recruitment Email for Usability Testing

Subject: Invitation from BLS

Hello

At the Bureau of Labor Statistics (BLS), we have been exploring more interactive ways to present our news releases. We would like to get input from data users to be sure that they are easy to use. As a subscriber to BLS data, we would like to invite you to participate in a research study to evaluate our prototype designs. Your feedback will help us ensure that our final products are user friendly.

This research study would take about an hour of your time. Using Zoom, we’ll connect with you remotely and ask you to share your screen as you work with the news releases to complete some tasks. We are looking for people who would like to participate either on a desktop/laptop computer or on a smart phone.

If you would be willing to help with this important study, please let me know. We are hoping to complete the study by late January.

Thank you,

Attachment E: Recruitment Email for Focus Groups

Subject: Invitation from BLS

Hello

At the Bureau of Labor Statistics (BLS), we have been exploring update our news releases and present the data in more interactive ways. We would like to get input from members of the media to be sure that the charts and tables in our news releases are easy for you to use. As a user of BLS data and member of the media, we would like to invite you to participate in a focus group to evaluate our prototype designs. Your feedback will help us ensure that our materials are user friendly.

This discussion would take about an hour of your time, and will aim to schedule the session at your convenience. We are conducting the focus group sessions at the Bureau of Labor Statistics office near Union Station (2 Massachusetts Ave., Washington, DC 20212).

If you would be willing to help with this important study, please reply to this email.

Thank you,

Attachment F: Consent Form for In-Lab Usability Tests and Focus Groups

Consent Form

OMB Control Number: 1220-0141

Expiration Date: March 31, 2021

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to disseminate survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be recorded, or you may be observed. If you do not wish to be recorded, you still may participate in this research.

We estimate your participation will last approximately 60 minutes.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires March 31, 2021.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2.  Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections.  The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection.  Information on routine uses can be found in the system of records notice,  DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418). 

Attachment G: Verbal Consent Outline for Remote Test Sessions with Zoom

Upon beginning a remote testing session, participants will be told the following information and asked for their verbal consent prior to beginning the study:

• During this study, we will record the screen you share and the audio from our call. If you do not wish to be recorded, you still may participate in this research.

• We estimate the session will last about 60 minutes.

• Your participation in this research project is voluntary, and you have the right to stop at any time.
  
  

• The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on March 31, 2021. Without this currently approved number, we could not conduct this research.

Are you ok with this?