Safety Alert and Outbreak Advisory Templates Testing

Appendix III

Moderator’s Guide

Safety Alert and Outbreak Advisory

Focus Groups

OMB No: 0910-0497 Expiration Date: 10/31/2020

Paperwork Reduction Act Statement: According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0910-0497. The time required to complete this information collection is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.

Send comments regarding this burden estimate or any other aspects of this collection of information, including suggestions for reducing burden to [email protected].

SUMMARY

Research Objectives

The goal of this focus group study is to understand consumer preferences toward different ways of formatting FDA safety alert and outbreak advisory messages. Specifically, to explore how to best frame the FDA safety alerts and outbreak advisories so the public can be informed and able to take actions to protect themselves during such public health emergencies.

Research Questions

• Are the messages on the templates easy to understand? What are the components that work or do not work?

• What are the actions consumers would take after reading the safety alerts and outbreak advisories?

• What key information are consumers looking for on safety alerts and outbreak advisories?

• What are the best ways to present the key information to consumers (e.g. segmentation of information, placement of texts and photos, call-out boxes, bullet points, order of information, etc.)?

• What are consumers’ reading habits when it comes to these templates? (e.g. Do consumers read the whole article from beginning to end, skim through it from top to bottom while looking for key information, or jump around?)

• Does the placement (where it shows up) and format of photos (e.g. commercial photos vs. actual photos) matter to consumers?

• Would consumers share the information on safety alerts and outbreak advisories?

  • If so, how?

  • Would consumers use any online features to share? Which features? Can anything be done to make the sharing features easier to use?

Notes to Moderator

This focus group moderator’s guide is not a script and, therefore, should not be read verbatim – although responses to all main questions are needed. The moderator should use these questions as a roadmap and probe as needed to maintain the natural flow of conversation.

Section 1. Welcome and Ground Rules (5 mins)

Thank you for taking the time to join us today. I am _______ from ________ and we are conducting a research study on behalf of the U.S. Food and Drug Administration, or FDA. The purpose of tonight’s discussion is to talk about some ways that the FDA communicates with consumers. Your feedback is very important to us. Our discussion will last about 90 minutes.

Before we start, I want to review a few important things:

• Your participation today is voluntary, and you have the right to stop being in the focus group at any time.

• None of my questions are very personal, but you can choose not to answer any of them if you are uncomfortable. Just let me know that you prefer not to answer.

• Anyone can join the conversation at any time; you don’t need to wait for me to call on you, just make sure you speak up and speak one at a time. However, please keep in mind that we have a lot to talk about and materials to review, so it’s important that I hear from everyone and that we discuss all the topics.

• Audio/video/observers:

• We are audio and video recording this discussion so that I can give you my full attention and not have to take notes. When writing up our findings, we will not include any information that could identify you. Your name, address, and phone number will not be given to anyone, and no one will contact you about this research after this group is over.

• I have colleagues listening to our discussion and taking notes behind the glass. At the end of our conversation, I may go into the back to see if there are any last-minute questions for you.

• We are also live-streaming our session so that other members of the research team who couldn’t be here today can observe our discussion remotely.

• Please mute your cell phones or turn them completely off. If you need to use the restroom, please feel free to step out and join us once you are done.

• Most importantly, there are no right or wrong answers. We want to know your honest opinions and experiences.

• I am not selling anything, I do not work for FDA, and I haven’t created any of the things we’ll look at, so please provide your honest opinions.

• Also, I am not an expert about the topics we are going to discuss today so you may have questions that I can’t answer.

• All of your comments will be kept secure to the extent provided by law, so feel free to say exactly what is on your mind. When our team drafts a report based on focus group findings, we will never attribute direct quotes to any particular person or use any names.

• Please remember to speak up so that everyone can hear you clearly.

• Are there any questions before we continue?

Section 2. Participant Introductions and Warm-Up (5 mins)

Let’s go around the room and say your first name, what is your favorite food and when is the last time you ate it?

Section 3. General Questions (10 mins)

As you might know, one of the most important responsibilities the U.S. Food and Drug Administration (FDA) has is to protect public health by ensuring the safety of our nation's food and cosmetic products. Today we are going to show you several messages posted on FDA’s website, which are intended to inform the public about safety issues of FDA regulated products. Specifically, we will be looking at messages related to food and cosmetic products.

To start, I would like to ask you all to imagine there is a new contamination problem related to a food that you eat or a cosmetic product that you use. By saying “cosmetic product”, we mean hair and body products as well as products you apply to your face, like toothpaste, shaving cream, lotion or moisturizers, facial masks, and of course make-up. It was discovered that some people have had bad reactions or have gotten sick. The contamination problem could result in a foodborne illness outbreak, a food or cosmetic product recall, or something like that.

• How do you think you would first learn about a contamination problem in foods or cosmetics?

• Is there a source of information you trust the most?

• Is there a source of information you prefer?

• What kind(s) of information is usually provided?

  • Is it what you most want to know?

• What information would you want to know?

• What’s the most important piece of information?

• After you heard about the safety problem, what would you do?

[Probe if not mentioned]

• Would you look for more information? Where would you look?

  • Have you ever visited the FDA website for more information?

• Would you check to see whether you have the product in your household?

Section 4. Usability Testing and Templates (65 mins)

INTRODUCTION

Now we are going to show you several messages from the FDA website. All the examples we will be discussing today are actual cases that happened. However, the website and messages you are about to see are for testing purpose and might have a different format compared to the actual messages that were posted on the FDA website. [Note to Moderator: please monitor participants and make sure no one takes photos or screenshots of the pages.]

[Notes to Moderator: Refer to the grid for order of template showing.]

INFANT FORMULA TEMPLATE (with Usability Testing)

The first message we are going to show you today is related to [infant formula]. In this message, FDA is informing the public about a safety problem related to an infant formula product. [For Tattoo Group participants ONLY: You may have never used infant formula but for the purpose of this study, please try your best to imagine you have a young child who is fed with infant formula daily.]

• If you heard found out about the problem on the news or from friends or family, what information would you want to know before seeing the safety alert?

• What would be the most important piece of information for you?

Now, please enter this link [provide URL] into your laptop/cellphone browser but don’t click “start” until I say so. We are going to do a quick exercise that asks you to react in real time to a safety alert similar to what you would see on FDA’s website. I want to emphasize that this is NOT a test. Your input will be extremely valuable for us to collect feedback and further improve the message. So please try your best to imagine you found this page on your own and read it in a way you usually read this kind of information.

Usability Task 1

Please click the place where you would normally look FIRST.

Usability Task 2

Please click on what you think is the most important information you want to know.

Usability Task 3

Please click on FDA’s recommendations to consumers.

[Notes to Moderator: Give participants approximately 5 minutes to read. The moderator can read the room and call the time]

OK, now let’s talk about the safety alert you just read.

• What are your first thoughts after seeing this alert?

  • How do you feel about the way the alert looks visually?

  • What did you like about it? Why?

  • What did you dislike about it? Why?

  • Would you change anything? If so, what?

• What did you think about the length of the alert?

[Prove if not mentioned]

• Was it too long, too short, or just about right? If too long, what information did you not need to know? If too short, what information was missing?

• Was anything confusing or hard to understand? If so, how could it be changed so it is easier to understand?

• What was the most important information you were looking for? Where did you find it? How easy or hard was it to find it?

• What would you change to make it easier to find the information?

• What is the main take-way?

[Probe if not mentioned]

• What is the problem with the product?

• What do you think this alert is telling you to do, if anything?

• Did you see FDA’s recommendations in the message? Could you find the recommendations? How easy or hard was it to find them?

• What actions would you take after reading this alert?

[Probe if not mentioned]

• Would you search for further information? Where would you search? Is there a source of information you trust the most?

• Is there a source of information you prefer?

• Would you be interested in getting updates on this matter? If so, in what form(s) (email, social media, text message, etc.)?

• Would you look to see whether you have the product in your household?

• Would you share this information with anyone?

• If you were to share this page, how would you do it?

• What would you change to make it easier to share this page?

• On what type of device do you usually read this kind of message (e.g., a computer, smart phone, or tablet)?

• Think about when you were reading the message earlier, can you describe how you read it?

[Probe if not mentioned]

• Where is your eye drawn first?

• What part of this alert would you read first?

• Did you skim this alert or read it line by line?

• Did you look at the images? Can you comment on the photo? How about where the photo is located in the message?

• Can you comment on the photo? How about the placement of the photo?

TATTOO INK TEMPLATE

Next, we are going to show you another message on FDA website. This one is related to a safety issue of a [tattoo ink product]. [For Caregiver Group participants ONLY: You may not have a tattoo or are not considering getting a tattoo, but for the purpose of this study, please try your best to imagine you or someone you’re close to are thinking about getting a tattoo sometime soon.]

• If you heard found out about the problem on the news or from friends or family, what information would you want to know before seeing the safety alert?

• What would be the most important piece of information for you?

Now, please enter this link [provide URL] into your laptop/cellphone browser. Please don’t read it yet. Again, this is NOT a test. Please try your best to imagine you found this page on your own and read it in a way you usually read this kind of information.

[Notes to Moderator: Give participants approximately 5 minutes to read. The moderator can read the room and call the time]

OK, now let’s talk about the safety alert you just read.

• What are your first thoughts after seeing this alert?

  • How do you feel about the way the alert looks visually?

  • What did you like about it? Why?

  • What did you dislike about it? Why?

  • Would you change anything? If so, what?

• What did you think about the length of the alert?

[Prove if not mentioned]

• Was it too long, too short, or just about right? If too long, what information did you not need to know? If too short, what information was missing?

• Was anything confusing or hard to understand? If so, how could it be changed so it is easier to understand?

• What was the most important information you were looking for? Where did you find it? How easy or hard was it to find it?

• What would you change to make it easier to find the information?

• What is the main take-way?

[Probe if not mentioned]

• What is the problem with the product?

• What do you think this alert is telling you to do, if anything?

• Did you see FDA’s recommendations in the message? Could you find the recommendations? How easy or hard was it to find them?

• What actions would you take after reading this alert?

[Probe if not mentioned]

• Would you search for further information? Where would you search? Is there a source of information you trust the most?

• Is there a source of information you prefer?

• Would you be interested in getting updates on this matter? If so, in what form(s) (email, social media, text message, etc.)?

• Would you share this information with anyone?

• If you were to share this page, how would you do it?

• What would you change to make it easier to share this page?

• On what type of device do you usually read this kind of message (e.g., a computer, smart phone, or tablet)?

• Think about when you were reading the message earlier, can you describe how you read it?

[Probe if not mentioned]

• Where is your eye drawn first?

• What part of this alert would you read first?

• Did you skim this alert or read it line by line?

• What else would be useful to include in this message?

ROMAINE LETTUCE TEMPLATE

Now, we are going to show you the last message for today. This one is related to [the E. coli outbreak happened this past November].

• If you heard found out about the problem on the news or from friends or family, what information would you want to know before seeing the safety alert?

• What would be the most important piece of information for you?

Now, please enter this link [provide URL] into your laptop/cellphone browser. Please don’t read it yet. Again, this is NOT a test. Please try your best to imagine you found this page on your own and read it in a way you usually read this kind of information.

[Notes to Moderator: Give participants approximately 5 minutes to read. The moderator can read the room and call the time]

OK, now let’s talk about the outbreak advisory you just read.

• What are your first thoughts after seeing this advisory?

  • How do you feel about the way the advisory looks visually?

  • What did you like about it? Why?

  • What did you dislike about it? Why?

  • Would you change anything? If so, what?

• What did you think about the length of the advisory?

[Prove if not mentioned]

• Was it too long, too short, or just about right? If too long, what information did you not need to know? If too short, what information was missing?

• Was anything confusing or hard to understand? If so, how could it be changed so it is easier to understand?

• What was the most important information you were looking for? Where did you find it? How easy or hard was it to find it?

• What would you change to make it easier to find the information?

• What is the main take-way?

[Probe if not mentioned]

• What is the problem?

• What do you think this advisory is telling you to do, if anything?

• Did you see FDA’s recommendations in the message? Could you find the recommendations? How easy or hard was it to find them?

• What actions would you take after reading this advisory?

[Probe if not mentioned]

• Would you search for further information? Where would you search? Is there a source of information you trust the most?

• Is there a source of information you prefer?

• Would you be interested in getting updates on this matter? If so, in what form(s) (email, social media, text message, etc.)?

• Would you look to see whether you have romaine lettuce in your household?

• Would you share this information with anyone?

• If you were to share this page, how would you do it?

• What would you change to make it easier to share this page?

• On what type of device do you usually read this kind of message (e.g., a computer, smart phone, or tablet)?

• Think about when you were reading the message earlier, can you describe how you read it?

[Probe if not mentioned]

• Where is your eye drawn first?

• What part of this alert would you read first?

• Did you skim this alert or read it line by line?

• Did you look at the images? Can you comment on the photo? How about where the photo is located in the message?

• Can you comment on the photo? How about the placement of the photo?

Section 5: Conclusions (5 mins)

False Close

Thank you very much for participating in this focus group. I appreciate you sharing your time and valuable feedback. Is there anything that you would like to share that you did not have a chance to share yet?

If you will excuse me for just a moment, I would like to check with my team to see if there are any follow-up questions for you. [Moderator goes to back room to see if there are any additional questions.]

Close

Great, I believe we are all done here. Thank you so much for your time!

[Moderator stops recording and directs participants to the assistant who will administer incentives and obtain receipts.]

2