Appendix 5. Pretest Plan and Scripts

Appendix 5. Pretest Plan and Scripts.docx

Generic Clearance for Survey Research Studies

Appendix 5. Pretest Plan and Scripts

OMB: 0536-0073

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0536-0073 can be found here:

2024-05-07 - No material or nonsubstantive change to a currently approved collection

Document [docx]

Download: docx | pdf

OMB Control Number: 0536-0073

Exp. Date: 04/30/2025

Appendix 5

Pretest Plan and

Survey Question Pretesting Scripts

I. Introduction

II. Survey Pretesting Scripts

A. Acute Version of the Survey

B. Chronic Version of the Survey

C. Focus Groups

I. Introduction

This document lays out our plan for revising and finalizing a survey developed and fielded in the UK for use in the U.S. This involves pretesting using qualitative research methods (focus groups and cognitive interviews with individual subjects) and quantitative methods (pilot testing of the revised survey instrument]. The purpose of the pretesting is to refine the language and logical flow of the survey. The purpose of the pilot testing is to assure that survey is functioning properly as programmed and to make final adjustments to price/cost levels used in choices presented in the survey.

The survey instrument has two versions, one on acute foodborne illness and the other on the long-term outcomes that sometimes follow acute foodborne illnesses.

The goals of the pretesting are:

Assure that the survey questions are comprehensible to respondents with a wide range of reading ability and is communicating with respondents as intended;
Minimize the cognitive and time burden on respondents;
Test adaptations made to adapt UK survey for US use and identify and make any other needed adaptations, for example, appropriate income brackets;
Test redesign of chronic health choice questions and the risk communication tutorial that accompanies the redesign to assure that respondents understand them as intended;
Maximize respondent’s interest in and attention to questions on the survey;
Simplify the survey, as possible, to reduce respondent burden;
Assess and minimize the time needed to complete the survey and the associated time and cost burden while collecting needed information.

Drafts of the acute and chronic versions of the survey reflecting preliminary adaptations to the UK surveys are attached as Appendices 1 and 2.

The goals of the pilot testing are:

To assure that the programed survey is functioning properly;
To evaluate and adjust price/cost variables used in the survey choices to make certain that responses from the fielded survey can be used to produce valid estimates of willingness to pay.

The time to complete the cognitive interviews will be no more than 90 minutes in length. Focus groups will be two hours in length with a break. The research team has regularly used sessions of these lengths successfully in prior survey development projects. A mix of activities including read alouds, discussing the subject’s own understanding of the language, discussing how particular parts of the survey might be improved, and breaks are used to keep the subject’s interest.

Our goal is that pilot tests will take no more than an average of 30 minutes.

We are asking for approval to engage 270 participants in pretesting this survey:

We assume we will have to contact 1020 potential respondents in order to find 270 participants for the following pretests and pilot testing:
- 24 cognitive interviews on the acute illness version of the survey,
- 30 cognitive interviews on the chronic illness version, and
- 2 focus groups of 8 participants each exploring disease outcome descriptions and price/cost levels for choice sets in the survey.
- 200 respondents for pilot testing of the two versions of the programmed survey instrument

Pretesting Acute Illness Survey Version

1. We will start pretesting using use 2-4 cognitive interviews to identify areas of the survey that need further refinement. During these interviews, the participants will “think aloud” as they are reading the entire survey. We will ask the participant to tell us what they are thinking as they read each screen of the survey, and if there is anything that they find difficult to understand or awkward. We will use probes to follow up, e.g.:

-- Can you tell us what went through your mind as you read this screen?

-- Were there any words that you didn’t understand or found strange?

-- Could you tell us in your own words what this question is asking?

The outcome of these 2 pretests will be a list of survey questions or groups of survey questions that require refinement and further testing.

2. Most of the remaining cognitive interviews will focus on particular areas of the survey. We will break the survey into different areas and focus on these with individual participant. Pretest script questions are written for each survey question so that pretesting can focus on particular questions or areas of the survey as need for refinement is identified. Following each interview, the research team will write up notes on observations made during the interview. The team will then meet to discuss what was learned about needed revisions and will make revisions. Subsequent interviews will focus on how well these revisions are functioning.

3. We will ask a few subjects at the beginning, middle, and end of the pretesting process to take the survey without interruption in order to evaluate how long it takes to complete the survey. Because this activity is not expected to take 90 minutes, these cognitive interviews may also be used for further cognitive interviewing on areas identified as needing further refinement.

4. We also will use some of our cognitive interviews to evaluate price/cost ranges for the survey choice sets. We will do this by asking the cognitive interviewees to take the survey but will substitute open-ended for closed-ended follow up questions after each choice asking the maximum or minimum the respondent would be willing to pay to avoid pain and suffering from acute foodborne illnesses or from their long-term health impacts. This will provide initial bids that will be finalized through pilot testing of the survey instruments.

Pretesting Chronic Illness Survey Version

Pretesting of the chronic illness version of the survey will follow the same process as the acute version.

Particular attention will be paid to screens on risk choice questions. The risk communication screens have been used in several prior surveys by multiple research teams, and thus are very well vetted, but they will still be pretested to assure they are functioning in the context of this survey (Hoffmann et al. 2017, OECD 2023).

Focus Groups on Disease Descriptions

The survey design randomly assigns sets of symptoms and illness durations with intervention options to respondents in a conjoint experiment design. The conjoint experiment design assures that survey responses will trace out demand for the disease/risk reduction.

Disease descriptions were developed and pretested by a team of economists and physicians for the UK survey. The UK disease descriptions have been revised to enhance consistency with those used in the USDA Cost of Foodborne Disease estimates. The USDA Cost of Foodborne Disease estimates disease descriptions were developed by an interdisciplinary team of economists and physicians, reviewed by an expert panel including physician/epidemiologists from the U.S. CDC and academia, and finally published in a peer reviewed foodborne disease journal. We will ask CDC foodborne disease specialists, some with clinical medicine experience, to again review these disease descriptions and comment on any further changes needed.

We will then use two focus groups of 8 people each (one for acute, and one for chronic outcomes) to explore how well the disease descriptions are likely to be understood and accepted by survey respondents. We will use what we learn from these focus groups to adjust disease descriptions and then will further test them in cognitive interviews along with other survey questions.

We will also use our focus groups to explore an initial range for bids through discussion about the minimum and maximum amounts people might be willing to pay.

Pilot Testing Revised Survey Instrument

Once both survey versions are finalized through pretesting, we will have the surveys programmed for online administration and will pilot test them. We will use pilot tests on 100 respondents for each version of the survey (200 respondents in total). Prior research has shown that pilot tests on 100 respondents generally provides enough statistical power to finalize a well-functioning stated preference survey. These respondents will be drawn from NORC’s AmeriSpeak panel, a probability-weighted panel that is representative of the U.S. public. More information on the AmeriSpeak panel is provided in Part A.

Prior to pilot testing, the survey will be programmed by staff at NORC at the University of Chicago (NORC), the survey firm that we plan to have field the final survey. Pilot testing will provide an opportunity to conduct quality control checks on the programmed survey. Importantly, it also provides the statistical power needed to assure that the price/cost levels used in the disease outcome choices provide respondents’ the range of options needed to capture demand for the health outcomes across the population.

References

Hoffmann, Sandra, Alan Krupnick, and Ping Qin. "Building a set of internationally comparable value of statistical life studies: estimates of Chinese willingness to pay to reduce mortality risk." Journal of Benefit-Cost Analysis 8, no. 2 (2017): 251-289.

OECD. “The Surveys on Willingness-to-Pay to Avoid Chemicals-Related Health Impacts (SWACHE) Project”. Accessed at https://www.oecd.org/chemicalsafety/costs-benefits-chemicals-regulation.htm Jan. 26, 2023.

II. Survey Pretesting Scripts

I. Acute Version of the Survey

Opening the pretest session

Welcome. My name is _________. I work with ____________.

Thank you for helping us with our survey, OMB Control Number 0536-0073, Expiration 4/30/2025.

The time to complete the pretest will take no more than 90 minutes.

We appreciate your willingness to help us improve a survey that will help the Department of Agriculture value the health impacts of foodborne illnesses. These estimates are used by the Department in developing new food safety policy and by the Department and others in public health education efforts.

____ will be taking notes as I talk with you about the survey. We will keep anything you say confidential. We’ll use our notes to help us improve the survey. They won’t be used for any other purpose. Our notes will be stored in a secure data storage facility and will destroy them once we finish the study. We understand staff from NORC have explained this to you and that you’ve signed an informed consent form. We are also required to comply with the terms explained in this informed consent form.

[Version 1: Think Alouds]

First, we are going to ask you to read through the survey out loud. While you are reading, we’d like you to tell us what you are thinking, or what is going through your mind. Please also mention things that may appear to you to be unimportant or unclear. This will help us get a better feeling for whether the survey is clearly written. It will help us identify places where we need to improve the survey and how we might be able to shorten it.

[Version 2: Other Cognitive Interviews]

We’ll be asking you to read through different questions we’ve developed. We have a number of questions we’d like to ask about the survey questions. Your input will help us make sure that anyone taking this survey can understand it and to help us make it as short and enjoyable as possible for the people who take the survey. It will be helpful to have you answer the questions as you read.

Thank you.

Full read aloud [for 2-4 respondents]

We’d like you to read through the survey out loud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant or unclear. This will help us get a better feeling for whether the survey is clearly written. It will help us identify places where we need to improve the survey and how we might be able to shorten it.

To get used to doing this, try to visualize the place where you live. Think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.¹

When they’re done: Let’s get started with the survey.

Probes:

[The pretester will listen to the participant as they read through the survey. The goal is to limit interruptions and just listen to what the participant is saying, noting questions where the participant appears to have difficulties, or expresses incredulity or discomfort. Comments by the pretester should be limited to statements like, “tell me what you’re thinking” when the subject pauses. Note the time spent on different questions.

[Once they’ve read through the survey, if there were particular questions that raised issues, use the question scripts below to follow up on those questions as time permits].

Questions on Individual Slides.

Note that no one cognitive interview participant will be asked all of the questions below. The “think aloud” and cognitive interviews will be used to identify areas of the survey that need further refinement. Individual cognitive interviews will focus on particular sections of the survey and/or particular survey questions identified in prior cognitive interviews as needing further refinement. Respondent burden is controlled by limiting each cognitive interview to 90 minutes. Our estimate of the number of cognitive interviews needed is based on extensive prior experience in developing stated preference surveys.

Slide 1. Welcome

Please read this slide aloud. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
Probes
1. What came to mind when you read “foodborne illness”?
2. Can you tell us in your own words how you talk about this kind of illness?
3. Are there any words on this page that you don’t understand or think someone else might not understand?
4. Is there anything on this page that is new to you? Is any of it surprising?
  1. [if yes, do you find the information on this page believable?]
5. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 2. Background Information

Please read this slide aloud. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
Probes
1. Is there anything on this page that is unclear to you?
2. Are there any words on this page that you don’t understand or think someone else might not understand?
3. Can you tell us in your own words what you understand the purpose of this survey is?
4. Can you tell us in your own words what you understand the section “Confidentially of collected data” to be telling you.
5. Can you tell us in your own words what the section “Voluntary Participation” is telling you?
6. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 3. Informed Consent

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Again, go ahead and answer the questions.

Probes
1. Is there anything on this page that is unclear to you?
2. Are there any words on this page that you don’t understand or think someone else might not understand?
3. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 4. Initial Background Questions

Please read this slide aloud. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
Probes
1. Is there anything on this page that is unclear to you?
2. Are there any words on this page that you don’t understand or think someone else might not understand?
3. Do you feel comfortable with the way these questions are asked?
4. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 5. Foodborne Illness Background

Please read this slide aloud. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant. Again, go ahead and answer the questions.

Probes
1. What came to mind when you read “foodborne illness”?
2. Can you tell us in your own words how you talk about these kinds of illnesses?
3. Are there any words on this page that you don’t understand or think someone else might not understand?
4. Is there anything on this page that is new to you? Is any of it surprising?
  1. [if yes, do you find the information on this page believable?]
5. Can you tell us in your own words what bacteria, viruses, and parasites are? What words do you use for these?
6. Did you know that food can be contaminated with bacteria, viruses, or parasites?
  1. [If no:] Did you find the statement “that illness can be caused by eating food that is contaminated with bacteria, viruses, or parasites” to be believable?
7. Can you tell us in your own words what you understand by each of these words:
  1. Diarrhea (runny stools)
  2. Stomach cramps
  3. Vomiting
  4. Nausea
  5. fever
8. Did you know that the symptoms of foodborne illness include: “diarrhea (runny stools), stomach cramps, vomiting, nausea, and/or fever”?
  1. If no: Did you find the sentence that “Common symptoms include: diarrhea (runny stools), stomach cramps, vomiting, nausea, and/or fever” to be believable?
9. What did you think when you read that CDC estimates that each year 1 in 6 Americans get sick from a foodborne illness?
  1. [if not believable follow up with: In fact this is true, CDC does estimate that 1 in 6 Americans get a foodborne illness every year. Is there a way we could have said this that might have been more believable to you?
10. Did it surprise you to read that people can be hospitalized from a foodborne illness?
  1. [if yes] Did you believe this statement?
  2. [if no] Why?
11. The purpose of this slide is to provide people with some basic background on foodborne illness and to let them know that pretty much everyone in the U.S. will get a foodborne illness at some point in their life. Is there anything you think we missed that was important to remind people of?
12. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 6. Your Recent Experience with Foodborne Illness Symptoms

Please read this slide aloud. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
Probes
1. What came to mind when you read “foodborne illness”?
2. Can you tell us in your own words how you talk about this kind of illness?
3. How easy would it be for you to answer this question if you were taking a survey?
4. Is there anything on this page that is new to you? Is any of it surprising?
  1. [if yes, do you find the information on this page believable?]
  2. [if no, Why?
5. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 7. You and Symptoms of Foodborne Illness

I’d like you to read through this list of symptoms out loud and as you do so, tell us what comes to mind as you read each term. Then tell us what each one means in your own words.

Are there any terms that you’re unfamiliar with?

[Follow up if there are terms the participant is unfamiliar with:

describe the symptom and ask “what do you call this?”

Can you remember back 5 years? [If no: what’s the worst stomach bug you’ve had? How long ago did it occur?]]

Slide 8. Avoiding the Symptoms of Foodborne Illness

Please read this slide aloud. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Is there anything on this slide that is unclear?
We try to keep surveys as short as possible. Is there anything on this slide that seems unimportant to you.
Were you aware that you can be infected with a bug and not have any symptoms?
Did you find it believable that there are things that you could do to prevent the symptoms of an infection you might get?
1. [If no] Did you know that colds are called by “bugs”/viruses? Are there things that you do to prevent the symptoms of a cold or make them better?
  1. [if yes] How do you think we could explain this in a way that is clearer or more believable.
  2. [If no] Did you know you can take an over-the-counter drug to clear up a stuffy or runny nose? How could we have made it clearer to you that that is the sort of thing we’re talking about?
Did you find it believable that your medical costs would be covered?

If no, why?

Did you find it believable that you would not lose wages or income?

If no, why?

Did you find it believable that someone would help handle your other responsibilities?

If no, why?

[if no to 6-8]. We want people to really focus on the benefit of not experiencing the pain and suffering of the symptoms, not its impact on medical costs, income, or being able to handle other responsibilities. Do you have any thoughts on how we could have made this clearer or helped you believe that you could ignore these other things?

Slide 9. But first, remember your budget

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Is there anything on this slide that is unclear?
Do you think after reading this slide that you will keep your budget in mind when you make the choices presented in this survey?
1. [if no] Is there anything we could have said that would have helped you keep your budget in mind?

Slide 10. Choice 1

Follow up slide after making the above choice:

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Again, go ahead and answer the questions. Doing so can help you notice how easy or hard it is to answer. That’s important for us to know.

How easy would it be for you to answer this question if you were taking a survey?
Did you remember that:
1. Your medical costs would be covered
2. That you would not lose wages due to the illness (or earnings if you have your own business)
3. That you would have help to handle any other responsibilities while you’re sick?
4. We want people to really focus on the benefit of not experiencing the pain and suffering of the symptoms, not its impact on medical costs, income, or being able to handle other responsibilities. Do you have any thoughts on how we could have made this clearer or help you believe that you could ignore these other things?
Did the list of symptoms you’d experience in Option A in the box on the slide we’re showing you seem realistic to you?
1. [if no] what about it seemed unrealistic?
Did the duration of the symptoms in Option A seem realistic to you?
1. [if no] why? What would have been more realistic to you?
Does option B seem realistic to you?
1. [if no] why? What would have been more realistic to you?
2. [if yes] why?
Do the monetary values seem realistic to you?
Would you have chosen option B
1. [if yes] Would you have chosen it if it were more expensive? [if yes] go through and ask if they would have chosen it if it were more expensive … in $5 intervals.
Would you have chosen option A
1. [if yes] Would you have chosen it if it cost $10? $5? [If they would have chosen it if it were $10, ask if they would have been willing to choose it if it were more expensive (in increments of $5 up to $30.
What did you think you were being asked to pay for?
Did you state the $ payment on behalf of yourself or on behalf of your household?
When making choices about paying to avoid the health impacts of food poisoning, did you also consider how much it would cost you to take sick leave; to pay for extra childcare and for the medical expenses?

a. if no: what do you think we could do to help respondents remember this?

What did you think when you saw the initial suggested price/cost amount?
What went into your thinking on whether to offer more (less) for the risk reduction?
What is the maximum you would pay for this risk reduction?
What is the minimum you would pay for this risk reduction?

Slide 11. An alternative way to rate illness severity

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?

Slide 12. Illness Severity Rating Tool

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?

Probes:

Are there any words or expressions that were hard to understand?
Are there any descriptions that you thought had nothing to do with foodborne illness?
How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 13. More Choices

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Is there anything on this slide that seems unclear to you?

Slide 14. Choice 2

Follow up slide after making the above choice:

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Think about, but don’t tell us which option you have chosen.
How easy would it be for you to answer this question if you were taking a survey?
Did you remember that:
1. Your medical costs would be covered?
2. That you would not lose wages due to the illness (or earnings if you have your own business)?
3. You would have help to handle any other responsibilities while you’re sick?
Did the list of symptoms you’d experience in Option A seem realistic to you?
1. [if no] what about it seemed unrealistic?
Did the duration of the symptoms in Option A seem realistic to you?
1. [if no] why? What would have been more realistic to you?
2. [if yes] why?
Does option B seem realistic to you?
1. [if no] why? What would have been more realistic to you?
2. [if yes] why?
Do the monetary values seem realistic to you?
1. [if no] Were they too high or too low? Is there a value you would have found more realistic?
Would you have chosen option B?
1. [if yes] Would you have chosen it if it were more expensive? [if yes] go through and ask if they would have chosen it if it were more expensive … in $5 intervals.
Would you have chosen option A?
1. [if yes] Would you have chosen it if it cost something?
2. [if yes] go through and ask if they would have chosen it if it were more expensive … in $5 intervals.
3. Having read this set of choices. Was the information on page 14 helpful? Do you think we could have dropped page 14?
  1. [if yes, why? Is there other information that might have been more useful on page 14?
  2. [if no, why? Is there other information that might have been more useful on page 14?
What did you think when you saw the initial suggested price/cost amount?
What went into your thinking on whether to offer more (less) for the risk reduction?
What is the maximum you would pay for this risk reduction?
What is the minimum you would pay for this risk reduction?

Slide 15. Questions about your survey experience

Please read questions 1-5 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind? Go ahead and answer the questions. Again, sometimes actually answering the question helps people notice things about the questions that they wouldn’t otherwise.

For question 1. [and do the same for question 2, 3, 4, and 5]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

Please read questions 6-11 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind?
For question 6. [and do the same for question 7-11]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

Please read questions 12-14 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind?

For question 12. [and do the same for question 13-14]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

Slide 16. General Health Questions

Please read questions 1-5 on slide 16 aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind?

For question 1. [and do the same for question 2, 3, 4, and 5]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

Slide 17. Other Background Questions (1)

Please read the questions on slide 17 aloud.

While you are reading, can you tell me what you are thinking, or what is going through your mind?

For question 1.
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

For question 2.
1. Can you tell us in your own words what the question is asking?
2. Does the place you live fit into these categories?
3. Are there any words that you don’t understand or think someone else might not understand?
4. Is there anything we could have done to make the wording easier to understand?
For question 3.
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

Slide 18. Other background question (2)

Please read the questions slide 18 aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind? Again, sometimes actually answering the question helps people notice things about the questions that they wouldn’t otherwise.
How easy was it for you to answer question 4?
1. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

How easy was it for you to answer question 5? Is there anything we could have done to the wording to make it easier to answer?
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?
How easy was it for you to answer question 6?
1. Can you tell us in your own words what the question is asking?
2. Does your marital status fit into these categories?
3. Are there any words that you don’t understand or think someone else might not understand?
4. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?
How easy was it for you to answer question 7?
1. Can you tell us in your own words what the question is asking?
2. Does your educational experience fit into these categories.
3. Are the categories used new word here for you?
4. Are there any words that you don’t understand or think someone else might not understand?
5. How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to make the wording easier to understand?

Slide 19. Other background question (3)

Please read the questions on slide 19 aloud.
While you are reading, can you tell me what you are thinking, or what is going through your mind?
How easy was it for you to answer question 8? How easy would it be for you to answer this question if you were taking a survey? Is there anything we could have done to the wording to make it easier to answer?

Additional Acute Illness Cognitive Questions

1. We want you to think about the worst case of foodborne illness, stomach bug, or flu that you have experienced recently.

It will have caused you some degree of pain and suffering.

Now imagine that it would have been possible to avoid that pain and suffering: what is the most you would have been prepared to pay to achieve that?

$_____________

3. After having conducted enough cognitive interviews to adjust the language in the survey, again ask a number of interviewees to take the survey and time how long it took.

If needed, ask the interviewees:

Are there any ways that you would suggest that we could shorten the survey?

4. After completing focus groups on disease outcome descriptions, substitute alternative disease descriptions in slides 10 and 14 to further test descriptions of alternative sets of disease symptoms.

II. Chronic Version of the Survey

Opening the pretest session

Welcome. My name is _________. I work with ____________.

Thank you for being willing to help us with our survey, OMB Control Number 0536-0073, Expiration 4/30/2025. The time to complete the pretest will take no more than 90 minutes.

We’re going to be working through questions on a draft survey that we’ve developed to help the Department of Agriculture value the health impacts of foodborne illnesses. These estimates are used by the Department in developing new food safety policy and by the Department and others in public health education efforts.

To keep your identity private, please don’t tell us your full name. We can use your first name or any other name you’d like us to use for this interview. What would you like us to call you?

[Version 1: Think Alouds]

[Version 2: Other Cognitive Interviews]

Thank you.

Instructions for Interviewers:

Think Alouds. The pretester will listen to the participant as they read through the survey. The goal is to limit interruptions and just listen to what the participant is saying, noting questions where the participant appears to have difficulties, or expresses incredulity or discomfort. Note the time spent on different questions. The session also will be recorded as well for timing.

Once they’ve read through the survey, if there were particular questions that raised issues, use the question scripts below to follow up on those questions as time permits.

Individual Slide Questions:

Note that no one cognitive interview participant will be asked all of the questions below. The “think aloud” read through will be used to identify areas of the survey that need further refinement. Individual cognitive interviews will focus on particular sections of the survey and particular survey questions. Respondent burden is controlled by limiting each cognitive interview to 90 minutes. Our estimates of the number of cognitive interviews needed is based on extensive prior experience in developing stated preference surveys.

Slide 1. Introduction [Note, this is the same as the slide on the acute survey, so should not need additional work]

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Probes

What came to mind when you read “foodborne illness”?
Can you tell us in your own words how you talk about this kind of illness?
Are there any words on this page that you don’t understand or think someone else might not understand?
Is there anything on this page that is new to you? Is any of it surprising?
1. [if yes, does you find the information on this page believable?]
How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 2. Background Information [Note, this is the same as the slide on the acute survey, so should not need additional work]

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
Probes
1. Is there anything on this page that is unclear to you?
2. Are there any words on this page that you don’t understand or think someone else might not understand?
Can you tell us in your own words what you understand the purpose of this survey is?
Do you have any thoughts on how we could have made this slide clearer?
How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 3. Informed Consent [Note, this is the same as the slide on the acute survey, so should not need additional work]

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
1. Again, go ahead and answer the questions.

Probes
1. Is there anything on this page that is unclear to you?
2. Are there any words on this page that you don’t understand or think someone else might not understand?
3. How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 4. Initial Background Questions [Note, this is the same as the slide on the acute survey, so should not need additional work]

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.
Probes
1. Is there anything on this page that is unclear to you?

Are there any words on this page that you don’t understand or think someone else might not understand?
Do you feel comfortable with the way these questions are asked?
How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 5. Foodborne Illness

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant. Again, go ahead and answer the questions.

Probes
1. What came to mind when you read “foodborne illness”?
2. Can you tell us in your own words how you talk about this kind of illness?
3. Are there any words on this page that you don’t understand or think someone else might not understand?
4. Is there anything on this page that is new to you? Is any of it surprising?
  1. [if yes, do you find the information on this page believable?]
  2. [if no, in fact it is all true. Is there anything we could have said to make it believable to you?]
5. Can you tell us in your own words what bacteria, viruses, and parasites are? What words do you use for these?
6. Did you know that food can be contaminated with bacteria, viruses, or parasites?

[If no:] Did you find the statement “that illness can be caused by eating food that is contaminated with bacteria, viruses, or parasites” to be believable?

Can you tell us in your own words what you understand by each of these words:
1. Diarrhea (runny stools)
2. Stomach cramps
3. Vomiting
4. Nausea
5. fever
Did you know that the symptoms of foodborne illness include: “diarrhea (runny stools), stomach cramps, vomiting, nausea, and/or fever”?
1. If no: Did you find the sentence that “Common symptoms include: diarrhea (runny stools), stomach cramps, vomiting, nausea, and/or fever” to be believable?
The purpose of this slide is to provide people with some basic background on foodborne illness and to let them know that pretty much everyone in the U.S. will get a foodborne illness at some point in their life. Is there anything you think we missed that was important to remind people of?

Did you know that most people in the U.S. will get a foodborne illness one or more time in their lives? [

If they answer no ask]: When you read this, did you believe that you probably have had or will have a foodborne illness at some point in your life?

[Ask]: Would it have made a difference if we’d included this statement in this slide: “Foodborne illness is fairly common, even in the United States. CDC estimates that each year 1 in 6 Americans will have a foodborne illness.”

Do you have any thoughts on how we could have made this clearer?

How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]

Slide 6. Complications and Long-term Outcomes

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Did you know that foodborne illnesses can have long term health impacts?
Did you find the material presented here believable?

a. [If not}: what in particular did you not find believable?

Is there anything on this slide that is unclear?
Are there any words used that you think people would not understand?
We try to keep surveys as short as possible. Is there anything on this slide that seems unimportant to you.
Do you have any thoughts on how we could have made this clearer?
[Show the participant Material Q1 }: Here is another way we thought we could present this material. Could you read it out loud? Again, please tell me what you’re thinking or what is going through your mind as you read it.

[once they’ve read it] Do you prefer any of this to the version we currently have in the survey?

Q1 (alternative text)

2. Long-term consequences of food poisoning

Although having a long-term health problem after a foodborne infection is very rare, they do occur and can be very serious.

Long-term outcomes following foodborne infections include:

Irritable Bowel Syndrome (IBS)

Sometimes the symptoms of foodborne illness return periodically even though you’ve gotten over the initial infection. This is because the foodborne illness damaged your gut. Sometimes people get fully better after several months, but studies show people can experience periodic diarrhea, cramping, gas, and bloating as long as 7 years after an initial foodborne illness.

Guillan-Barré Syndrome (GBS)

Sometimes in trying to kill the bug causing the foodborne illness, a person’s immune system attacks their own nervous system causing muscle weakness. This can affect them in several different ways including temporary paralysis, difficulty speaking, breathing, swallowing, or digesting food or may lead to fluctuations in heart rate or blood pressure. Any one person may experience one or many of these outcomes. Recovery from GBS can take months to years. 70% of people fully recover, but 20 % still have problems 3 years after their illness.

Reactive Arthritis

Inflammation from a foodborne infection can affect a person’s joints, urinary tract, and/or your eyes. Symptoms such as joint pain, swelling, lower back and buttock pain, swelling in the fingers and toes, joint stiffness, pain or burning sensation while urinating, frequent urination, and fluid discharge from the genitals. Most people with reactive arthritis recover within a few months with treatment, although their symptoms may last up to a year. Others may develop long-term, mild arthritis, while a few others may have chronic, severe arthritis that is difficult to treat.

Permanent Renal Failure

Some foodborne diseases can cause kidneys to permanently quit working. Kidneys are critical normal body function. They filter waste products out of your blood creating urine. They also produce some hormones your body needs to function well. Permanent kidney failure is treated by dialysis several times a week. Eventually, a person with permanent kidney failure may need a kidney transplant.

Meningitis

Meningitis is an inflammation of the membranes covering the brain and spinal cord. Meningitis from foodborne illness is more likely to affect older adults, infants, and children. Early symptoms include: sudden high fever, stiff neck, severe headache, confusion or difficulty concentrating, and seizures. Most people who are treated quickly will make a full recovery. But some people may experience long-term problems including: vision and/or hearing problems, difficulty with memory and concentration, recurrent seizures, difficulty with balance or muscle coordination.

Sepsis

Sepsis can be a serious complication of any infection, including foodborne infections. Sepsis occurs when the body’s immune system goes into overdrive in response to an infection and attacks the body’s own tissue and organs. It may require treatment in an Intensive Care Unit.

Most people fully recover from sepsis. But it can take time. People who have had sepsis, particularly serious sepsis, may have symptoms for months or even years. These include: fatigue, difficulty sleeping, difficulty breathing, muscle and joint pain, decreased mental function, poor concentration, depression and mood swings, nightmares or flashbacks.

Slide 7. Choices about long-term impacts

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Probes

Is there anything on this page that is unclear to you?
Are there any words on this page that you don’t understand or think someone else might not understand?
Do you feel comfortable with the way these questions are asked?
How hard to understand are these materials? [1= very easy, 2= somewhat easy, 3= somewhat difficult, 4=very difficult]
We want people to really focus on the benefit of not experiencing the pain and suffering of the symptoms, not its impact on medical costs, income, or being able to handle other responsibilities. Do you have any thoughts on how we could have made this clearer or help you believe that you could ignore these other things?

Slide 8. Understanding risk (1)

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Probes:

Can you tell us in your own words what this slide is saying?
Do you have any thoughts on how we could have made this clearer?
If you were taking this survey, would you want to close out of your browser at this point?

Slide 9. Understanding risk (2)

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Probes:
1. Can you tell us in your own words what this slide is saying?
2. Do you have any thoughts on how we could have made this clearer?

Slide 10. Changes out of 100

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Probes:
1. Can you tell us in your own words what this slide is saying?
2. Is there anything in this slide that you found unclear?
3. Is there any way you think we could simplify this slide?
4. Did you find the prior two slides helpful in working up to this slide?
5. If you were taking this survey, would you want to close out of your browser at this point?

Slide 11. Changes out of 1000

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Probes:
1. Can you tell us in your own words what this slide is saying?
2. Is there anything in this slide that you found unclear?
3. Is there any way you think we could simplify this slide?

Slide 12. Comparing risks using these figures

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Please also mention things that may appear to you to be unimportant.

Which answer would you have chosen?

Probes:
1. Can you tell us in your own words what this slide is saying?
2. Is there anything in this slide that you found unclear?
3. Is there any way you think we could simplify this slide?
4. Did you find the slides with coins and die helpful in understanding this slide comparing risks? We’d love to shorten this “tutorial” on risk. Is there any way you would suggest we do that?

Slide 13. Choices about Long-term Impacts

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Is there anything on this slide that is unclear?
We try to keep surveys as short as possible. Is there anything on this slide that seems unimportant to you.
Did you find it believable that there are things that you could do to reduce the risk of experiencing a long-term outcome when you get a foodborne illness?
Did you find it believable that your medical costs would be covered?
Did you find it believable that you would not lose wages or income?
Did you find it believable that someone would help handle your other responsibilities?

[if no to 6-7]. We want people to really focus on the benefit of not experiencing the pain and suffering of the symptoms, not its impact on medical costs, income, or being able to handle other responsibilities. Do you have any thoughts on how we could have made this clearer or help you believe that you could ignore these other things?

Slide 14. “Remember your budget”

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Is there anything on this slide that is unclear?
Do you think after reading this slide that you will keep your budget in mind when you make the choices presented in this survey?
1. [if no] Is there anything we could have said that would have helped you keep your budget in mind?

Slide 15. “Choice 1 (the risk)”

1. Please read this slide. Tell me in your own words what is it says to you.

2. Which option shows the larger risk of having a long-term health problem from a foodborne illness?

3. What do you understand by the risk being 63 out of 1000 over 5 years?

4. Is there anything that you find confusing on this slide?

If yes, what do you understand by _____?

Would it be clearer if we said _________?

5. Do you have any thoughts on how we could have made this clearer?

Slide 16. “Choice 1 (the choice)”

1. Did you think about what the short-term illness would be like when you answered this question? If yes, what kinds of symptoms did you think about?

2. If yes: we would like people not to focus on the nature of the short-term illness, just on the long-term outcome. Would it have helped not to have mentioned the short-term illness in the outcome’s description? Do you have other suggestions on how we could help people focus only on the long-term outcomes?

3. Is it clearer if we use the label: “Chance of having the long-term illness after a short-term foodborne illness” rather than “Chance of having the long-term illness”

4. What did you think when you read “Preventive expenditure per year for the next 5 years”?

5. What would you have thought if you had instead read “Extra expenditure per year for the next 5 years”?

6. It could be more or less expensive to prevent this outcome.

If they chose to pay, ask: What is the maximum amount you would be willing to pay?

If they chose the risk, ask: How low would the price have to be before you’d be willing to choose to reduce the risk?

7. What did you think when you saw the initial suggested price/cost amount?

What went into your thinking on whether to offer more (less) for the risk reduction?
What is the maximum you would pay for this risk reduction?
What is the minimum you would pay for this risk reduction?

8. Do you have any thoughts on how we could have made this clearer?

Slide 17. “Choice 2 (the risk)”

1. Please tell me your own words what this slide is telling you.

2. Does this slide differ from the one you saw earlier?

If yes, how?

3. We could use your help thinking about how to talk about short-term or long-term illnesses

a. Most foodborne illness lasts only a few days and is quite mild.

How would you refer to these illnesses?

Here are some terms we thought of (in addition) to those you thought of

Short-term illness

Acute illness?

Have you heard these terms before you looked at this survey?

What do you think of when you hear these terms?

b. As we said earlier, and as you may know from COVID, some infections cause long-term outcomes.

Have you heard of this happening?

How do you refer to these kinds of illnesses?

Here are some terms we thought of (in addition) to those you thought of

Long-term illness,

Long-term health problems

Long-term health impacts

Chronic illness (or health impacts)

Have you heard these terms before you looked at this survey?

What do you think of when you hear these terms?

What term do you prefer?

4. Is there anything you find unclear about this slide?

5. Do you have any thoughts on how we could have made this clearer?

Slide 18. “Choice 2 (the choice)”

1. Please tell me your own words what this slide is telling you.

2. What would you chose to do?

a. It could be more or less expensive to prevent this outcome.

If they chose to pay, ask: What is the maximum amount you would be willing to pay?

If they chose to take the risk, ask: is there any cost of prevention low enough for you to have chosen to take precaution?

3. Does this slide differ from the choice you just saw?

If yes, how?

4. Is there anything in the description of symptoms or health impacts that you find unclear?

5. Do you find it believable that a foodborne illness could cause these kinds of health problems?

a. if no: In fact, they do. Is there anything you think we could do to make it more believable?

6. Did you find it believable that there is something you could do to prevent this long- term health impact? If no: Probe on why not If yes: Probe on why yes

7. How long will the risk reduction last?

8. How much will you need to pay each year? Over the next 5 years?

9. It could be more or less expensive to prevent this outcome.

If they chose to pay, ask: What is the maximum amount you would be willing to pay?

If they chose the risk, ask: How low would the price have to be before you’d be willing to choose to reduce the risk?

10. What did you think when you saw the initial suggested price/cost amount?

What went into your thinking on whether to offer more (less) for the risk reduction?
What is the maximum you would pay for this risk reduction?
What is the minimum you would pay for this risk reduction?

11. Do you have any thoughts on how we could have made this slide clearer?

Slide 19. An Alternative Way to Rate Illness Severity

1. Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?

2. Do you have any thoughts on how we could have made this slide clearer?

Slide 20. “Rating your current health”

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
How relevant did you think these questions were in general? How relevant did you think they were in relation to the long-term health impacts of a foodborne illness?

Slide 21. Choice 3 (the risk)

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?

What would you think when you read this if we said “normal” rather than “current”?

What if we said “usual”?

3. Do you have any thoughts on how we could have made this slide clearer?

Slide 22. Choice 3 (the choice)

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
1. Probe: What do you understand by the disease description? Can you imagine what it would be like to experience this health state? Can you tell us what it would be like in your own words?

Can you make this choice?

Do you find this more or less difficult to answer than the prior choices? Why?

Do you have any thoughts on how we could have made this slide clearer?

It could be more or less expensive to prevent this outcome.
1. If they chose to pay, ask: What is the maximum amount you would be willing to pay?
2. If they chose the risk, ask: How low would the price have to be before you’d be willing to choose to reduce the risk?
What did you think when you saw the initial suggested price/cost amount?
1. What went into your thinking on whether to offer more (less) for the risk reduction?
2. What is the maximum you would pay for this risk reduction?
3. What is the minimum you would pay for this risk reduction?

Slide 23. Choice 4 (the risk)

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?
Do you feel you are still able to focus on what this slide is saying?
If you were taking this survey, would you want to close out of your browser at this point?

Slide 24. Choice 4 (the choice)

Please read this slide aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind?

Can you make this choice?
Were you able to focus on this question?
1. If yes, how many of these questions do you think you could answer before you started to find them tedious?
2. If no, is there anything we could do to make it more interesting?

It could be more or less expensive to prevent this outcome.
1. If they chose to pay, ask: What is the maximum amount you would be willing to pay?
2. If they chose the risk, ask: How low would the price have to be before you’d be willing to choose to reduce the risk?

What did you think when you saw the initial suggested price/cost amount?
1. What went into your thinking on whether to offer more (less) for the risk reduction?
2. What is the maximum you would pay for this risk reduction?
3. What is the minimum you would pay for this risk reduction?

Slide 25. Questions about your survey experience

Please read questions 1-5 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind? Go ahead and answer the questions. Again, sometimes actually answering the question helps people notice things about the questions that they wouldn’t otherwise.
2. Do you have any thoughts on how we could have made this question clearer?

For question 1. [and do the same for question 2, 3, 4, and 5]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

Please read questions 6-11 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind?
2. Do you have any thoughts on how we could have made this question clearer?

For question 6. [and do the same for question 7-11]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

Please read questions 12-14 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind?
For question 12. [and do the same for question 13-14]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

Slide 26. General health questions

Please read questions 1-5 on this slide aloud.
1. While you are reading, can you tell me what you are thinking, or what is going through your mind?
2. Do you have any thoughts on how we could have made this question clearer?

For question 1. [and do the same for question 2, 3, 4, and 5]
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

Slide 27. Other background questions (1)

Please read the questions on slide 27 aloud.
While you are reading, can you tell me what you are thinking, or what is going through your mind?
Do you have any thoughts on how we could have made this question clearer?

For question 1.
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

For question 2.
1. Can you tell us in your own words what the question is asking?
2. Does the place you live fit into these categories?
3. Are there any words that you don’t understand or think someone else might not understand?
4. Is there anything we could have done to make the wording easier to understand?

Slide 28. Other background questions (2)

Please read the questions on slide 28 aloud.While you are reading, can you tell me what you are thinking, or what is going through your mind?

For question 3.
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

How easy was it for you to answer question 4?
1. Is there anything we could have done to make the wording easier to understand?

How easy was it for you to answer question 5? Is there anything we could have done to the wording to make it easier to answer?
1. Can you tell us in your own words what the question is asking?
2. Are there any words that you don’t understand or think someone else might not understand?
3. Is there anything we could have done to make the wording easier to understand?

Slide 29. Other background questions (3)

Please read the questions on slide 29 aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Think about, but don’t tell us what you’d chose.

How easy was it for you to answer question 6?
1. Can you tell us in your own words what the question is asking?
2. Does your marital status file in these categories?
3. Are there any words that you don’t understand or think someone else might not understand?
4. Is there anything we could have done to make the wording easier to understand?

How easy was it for you to answer question 7? Can you tell us in your own words what the question is asking?
1. Does your educational experience fit in these categories.
2. Are the categories used comfortable for you?
3. Are there any words that you don’t understand or think someone else might not understand?
4. Is there anything we could have done to make the wording easier to understand?

Slide 30. Other background questions (4)

Please read the questions on slide 30 aloud. While you are reading, can you tell me what you are thinking, or what is going through your mind? Think about, but don’t tell us what you’d chose.

How easy was it for you to answer the questions? Is there anything we could have done to the wording to make it easier to answer?

Additional Questions for Chronic Disease Survey Version Cognitive Interviews

We want you to think about the worst case of foodborne illness, stomach bug, or flue that you have experienced recently.

It will have caused you some degree of pain and suffering.

Now imagine that it would have been possible to avoid that pain and suffering: what is the most you would have been prepared to pay to achieve that?

$_____________

After having conducted enough cognitive interviews to adjust the language in the survey, again ask a number of interviewees to take the survey and time how long it took.
1. If needed, ask the interviewees:
2. Other than the background and debriefing questions, was there any information in the survey that you felt you did not need to make your choices?
3. Are there any ways that you would suggest that we could shorten the survey?

After completing focus groups on long-term disease outcome descriptions, substitute alternative disease descriptions in slides 16, 18, 22, and 24 to further test descriptions of alternative sets of disease symptoms.

III. Focus Groups on Disease Descriptions

Notes to the moderator are written in italics.

Visual materials to be presented to the focus group are referenced in bold italics.

The rest of the text is to be read out to the participants.

INTRODUCTION [5 minutes]

Welcome. My name is _________. I work with ____________.

Thank you for being willing to help us with our survey, OMB Control Number 0536-0073, Expiration 4/30/2025.

The time to complete this Focus Group will last up to 2 hours. We’ll take a break halfway through. We’ll start by looking at descriptions of relative mild illnesses. We’ll then then look at hospitalized cases. After a break, we’ll look at more serious complications and possible, but rare, long-term outcomes.

Our understanding from Ameri-Speak is that you’ve all said it was OK to record the session. As was explained when you were recruited for this focus group, the recording and your comments will be kept confidential and stored in a secure file. We’ll destroy the files once we finish the survey. Please let me know if you have any questions.

We are designing a survey to help the U.S. Department of Agriculture value the health impacts of foodborne illnesses. These estimates are used by the Department in developing new food safety policy and by USDA and others in public health education efforts.

This Focus Group is to help us understand what foodborne illness means to people, and to test some of the materials we are preparing for the survey.

The purpose of this focus group Is to help us make the survey's disease descriptions as clear and easy to understand as possible. You can help us by talking about how clear/unclear these descriptions are to you.

Your comments and opinions will help us make sure that anyone taking this survey can understand it and to help us make it as short and enjoyable as possible for the people who take the survey.

There are no “right” or “wrong” answers in this session, so please be open, honest and speak directly to us.

If we show you some materials that don’t make sense then please tell us – that’s part of the purpose of the session.

WHAT IS FOODBORNE ILLNESS [10 min.]

We’ll start just by getting on the same page about what we mean when we say foodborne illness.

By foodborne illness we mean illnesses that are caused by living things -- bacteria, viruses, or parasites -- on ingested foods. We’re not talking about chemical contamination of food.

Most of these illnesses affect the gut, but a few have other effects.

Most of these illnesses are mild, but they can sometimes cause serious but short-term complications. On rare occasions short-term foodborne illnesses can lead to serious long-term (or chronic) illnesses.

What does foodborne illness involve?

What is your experience with foodborne illness? Have you, your family or close friends ever had a foodborne illness?
Describe the symptoms you associate with having a foodborne illness.

Moderator: Prompt on symptoms they don’t include, using:

Feeling sick
High Temperature
A lack of energy
Aching muscles
Headache
Diarrhea/loose stools
Vomiting / explosive uncontrolled vomiting
Stomach cramps/pains
Blood in stools

How long do you think a typical foodborne illness lasts?

Exploring drivers for perceived severity

Foodborne illness varies in severity. We’ll be looking at illness of varying severity. To start, I’d like you to think about illnesses that’s serious enough that might get you to go to the doctor. Which of these, if any, would prompt you to go to the doctor?

watery stools for 2 days (no)

Fever of 99 for 2 days (no)

Bloody stools for 1 day (yes)

Fever of 103 for 1 day (yes)

Watery stools for 4 days (yes)

Nausea and vomiting for 1 day (no)

Dizziness, especially when getting up from lying down or sitting to standing (yes)

Vomiting for an afternoon (no)

Neck ache with a fever of 100 (yes)

EXPLORING SURVEY DESCRIPTIONS

3.1. Vignettes (Mild acute illness) [15 min.]

As part of the survey, we will be showing people descriptions of a case of foodborne illness.

We’d like your views on some illness descriptions we’ve developed.

These illnesses vary in severity – more or fewer symptoms, more or less severe symptoms, lasting shorter or longer time etc.

We’d like to know how clear/unclear they are….

There are no “right” or “wrong” answers here, so please be frank with us.

If we show you some materials that don’t make sense then please tell us.

Moderator: Show sequentially vignettes - material M1 M2 M3

Asking after each one:

What did you think about as you read this description?

How clear/unclear is the description?

Do you think the description is realistic?

i. if not, what about it didn’t seem right to you?

Are there ways we could make this clearer?

i. prompt on M1. Would it be clearer if instead of saying the “illness lasts 2 day” we said, “it takes 2 days for you to get back to feeling normal”

Exploring duration (mild cases)

Ask: What is the longest you’ve ever been sick from a stomach bug? By sick, we mean from the time you came down with it to the time you felt like your normal self.

People who study foodborne illnesses have found that people who get illness that is mild enough that they get better without getting medical care typically are sick for 1-4 days, but they can be sick as long as two weeks.

Ask: Would you find it believable if we described a mild illness (no doctor visit needed) as lasting up to 4 days?

Would you find it believable if we described it as lasting 14 days? What would help you believe foodborne illness could last that long?

Vignettes (Moderate acute illness (Dr. visit) [15 min.]

The descriptions we’ve shown you are for mild foodborne illness. Now I’d like to show you some descriptions of foodborne illness that’s severe enough that you should (and probably would) see a doctor.

What would make you visit a doctor if you had the symptoms of foodborne illness?

Prompts:

-- how long would you have to have diarrhea before you thought you should see a doctor?

-- how high a fever would you have to have, and for how long before you would want to see a doctor?

-- are there any symptoms that might make you want to see a doctor after being sick for a day?

Moderator: Show sequentially vignettes - material M4 M5 M6

Asking after each one:

What did you think about as you read this description?

How clear/unclear is the description?

Do you think the description is realistic?

i. if not, what about it didn’t seem right to you?

Are there ways we could make this clearer?

Exploring Duration (moderate illness that requires doctor’s care)

People who study foodborne illnesses have found that people who get an illness that is bad enough that see a doctor typically are sick for 4-8 days, but they can be sick as long as two weeks.

Moderator ask:

Would you find it believable if we showed a foodborne illness that was bad enough that you went to see a doctor as lasting 8 days?

Would you find it believable if we described it as lasting 10 days? 14 day? 15 day?

What would help you believe it could last that long?

Vignettes (Hospitalized cases) [20 min.]

Occasionally foodborne illness can be serious enough that the doctor they see will admit them to a hospital. Now I’d like to show you some descriptions of foodborne illness that’s severe enough that you could be admitted to a hospital.

Moderator: Show sequentially vignettes - material M7- M9

Asking after each one:

What did you think about as you read this description?

How clear/unclear is the description?

Do you think the description is realistic?

i. if not, what about it didn’t seem right to you?

Are there ways we could make this clearer?

Exploring Duration (moderate illness that requires doctor’s care)

Studies of hospitalizations from foodborne illness show they range from 4 to 13 days

Ask: Would you find it believable if we described a hospitalization from foodborne illness as lasting only 4 days?

If no: What would help you believe this?

Would you find it believable if we described a hospitalization from foodborne illness as lasting 13 days?

If no: What would help you believe it could last that long?

Estimates of how long it takes people to fully recover from foodborne illnesses once they’ve been discharged from the hospital estimate it can take from 9 to 37 days to be back to normal.

Ask: Would you find it believable if we said it took 9 days after someone had been hospitalized for 4 days to feel back to normal?

If no: What would help you believe this?

Would you find it believable if we said it took 37 days after someone had been hospitalized for 12 days to feel back to normal?

If no: What would help you believe this?

Great job! Let’s take a 5 minute break.

3.4. Complications and Long-term Health Impacts [40 min]

Welcome back.

Sometimes a case of foodborne illness can lead to serious complications and long-term health problems. This is rare but it does happen.

Have you heard this information before? Does it sound believable?
If yes, what are some of the complications or long-term health problems you’ve heard of from foodborne illness
If no, have you ever heard of someone getting a chronic illness from any other infection?

If yes, what were some of those?

If no, prompt

-- have you heard of Lyme’s disease?

-- meningitis in children as a cause of cognitive problems

-- long COVID

Here are some examples:

Moderator: materials: m10 general descriptions of complications and long-term effects from foodborne illness

Probe understanding

Is it believable that a case of foodborne illness from a poorly cooked burger or chicken can lead to these sorts of serious effects?
Moderator, go through each illness listed and probe credibility. Are there particular aspects of the illness description that they find difficult to accept.
Ask if they would you find it believable if we described a hospitalization from foodborne illness as lasting 13 days?

If no: What would help them believe it could last that long?

Now we are going to present some descriptions of long-term health impacts of foodborne illnesses in the form that we’ll use in our survey. We’d like to know how clear or confusing they are.

Moderator: Show sequentially vignettes - material M11, M12

Asking after each one:

What did you think about as you read this description?

How clear/unclear is the description?

Do you think the description is realistic?

i. if not, what about it didn’t seem right to you?

Are there ways we could make this clearer?

Exploring Duration (moderate illness that requires doctor’s care)

Studies of hospitalizations from foodborne illness show they range from 4 to 13 days

Ask:

1. Reactive arthritis is generally temporary swelling or pain in the joints, but it can last as long as 6 months.

Do you believe that Reactive arthritis can last this long?

Would you find it believable that the condition can be temporary?

Is there anything that would have made this more believable?

2. The symptoms of Irritable Bowel Syndrome are periodic bowel problems (diarrhea or constipation), excessive gas or bloating. The best studies available show the number of people with IBS who seek medical care tapers off over an 8-year period. It’s not known whether these people have finally recovered or if they’ve just given up on the idea that there’s anything the medical profession can do to help.

Would you find it believable if we were to say these symptoms could go on for 8 years?

3. As we said, meningitis is an inflammation of the lining of the brain. It can lead to vision problems, hearing problems, or cognitive problems (particularly in children and the elderly).

Would you find that difficult to believe that these conditions are permanent?

If yes, is there anything that would made this more believable?

4. STEC (E coli 157) can cause permanent kidney damage and even permanent kidney failure. This is a lifetime health problem that requires treatment on dialysis or a kidney transplant.

Have you ever heard of this disease?

If so, have you heard it happening from a foodborne illness?

Do you find it believable that STEC could permanently damage a kidney?

If not, what would make it believable for you?

OPTIONAL: EXPLORING BID RANGES – material M13 [10 min.]

Many of us learned during the COVID pandemic that it is possible to get an infection and not experience any symptoms.

We’re now going to ask you about choices you could make to avoid experiencing the symptoms of foodborne illness, even if you get infected by a foodborne bug. However, there will be a cost to preventing the symptoms.

We want you to focus solely on avoiding the pain and suffering associated with foodborne illness.

For the purposes of this study, assume that if you suffer from long-term illness:

your medical costs would be covered,
you would not lose wages due to the illness (or earnings if you have your own business), and
you would have help to handle any other responsibilities while you’re sick.

Please treat the following questions as if you were in a real-life situation, so that your answers are as accurate as possible.

Do not agree to pay an amount that you cannot afford or that there are more important ways to spend your money.

When making these choices, please keep in mind:

your income and savings
that the payment would reduce your ability to buy other things you value.

FOCUS GROUP MATERIALS

Mild Acute Illness (no Drs. Visit)

You have runny stools and stomach cramps for 2 days.

You have little energy or appetite.

You don’t need to visit the doctor.

The illness lasts 4 days.

You spend 2 of those days in bed.

You have runny stools, stomach cramps, nausea and vomiting for 1 days.

You have little energy or appetite.

You don’t need to visit the doctor.

The illness lasts 3 days.

You spend 1 of those days in bed.

You have runny stools, stomach cramps, nausea and vomiting 2 days.

You have a mild fever and have little energy or appetite.

You don’t need to visit the doctor.

The illness lasts 5 days (z=3,4,5).

You spend 3 of those days in bed.

Moderate Acute Illness (Dr’s. visit needed)

You have bloody stools and stomach cramps.

You have a mild fever and have little energy or appetite.

You visit your GP, who prescribes an antibiotic, tells you to rest, drink plenty of fluids and to take over- the-counter medicine for fever and aches and to come back in 2 days if you’re not better or sooner if symptoms worsen.

You improve and fully recovered.

The illness lasts 5 days.

You spend 2 of those days in bed.

You have runny stools, stomach cramps that goes on for more than 2 days.

You also have a mild fever and have little energy or appetite.

You visit your GP, who tells you to rest, drink plenty of fluids, to take over- the-counter medicine for fever and aches, and to come back for another visit if you don’t start feeling better in 7 days.

You improve and fully recover.

The illness lasts 6 days.

You spend 3 of those days in bed.

You have runny stools, stomach cramps.

You also have a moderate fever that isn’t better at the end of 2 days.

You visit your GP, who tells you to rest, drink plenty of fluids, to take over- the-counter medicine for fever and aches, and to come back for another visit if you don’t feel better in 7 days.

You improve and fully recover.

The illness lasts 8 days.

You spend 4 of those days in bed.

Hospitalization for Acute Illness

Severe gastroenteritis

You have severe bloody stools, stomach cramps, and a mild fever.

You visit your doctor, who admits to the hospital.

In the hospital you are given an antibiotic and treated for dehydration with intravenous fluids. They run tests and rule out more serious causes.

You are in the hospital for 5 days.

When you get home, you rest for a few days and fully recover.

The illness lasts a total of 12 days.

You spend 7 of those days in bed.

You have severe diarrhea, stomach cramps that isn’t better at the end of 2 days.

You also have a mild fever and feel light-headed.

You visit a doctor who admits you to the hospital.

In the hospital you are given an antibiotic and treated for dehydration with intravenous fluids.

You are in the hospital for 4 days.

When you get home, you rest for a few days and fully recover.

The illness lasts a total of 10 days.

You spend 7 of those days in bed.

Mis-diagnosed appendicitis

You have fever, diarrhea, and abdominal pain that feels like an appendicitis. You see a doctor who admits you to the hospital.

In the hospital you have your appendix removed.

You’re in the hospital 7 days.

You need to rest when you get home from the hospital and are able to resume your normal activities after 13 days.

You see a physical therapist to regain muscle strength after your abdominal surgery.

The illness lasts a total of 22 days.

You spend 15 of those days in bed.

Serious neurological symptoms

You have fever and headache, muscle aches.

You develop a stiff neck, have more difficulty balancing than usual, and feel a bit confused.

You see a doctor who admits you to the hospital.

In the hospital you learn that you have an infection affecting the lining of your brain and your spinal cord. You’re put on intravenous antibiotics and recover. You are in the hospital for 11 days.

When you get home, you have to rest for a few weeks, but fully recover.

The illness lasts a total of 42 days.

You spend 28 of those days in bed.

M10 Complications and Long-term Health Impacts

Although having a serious complication or long-term health problem after a foodborne infection is very rare, they do occur and can be very serious.

These include:

Guillan-Barre Syndrome (GBS)

GBS occurs when the body’s immune system attacks part of the body’s nervous system. This can lead to muscle weakness, which results in symptoms that can include:

temporary paralysis of the legs, arms, or face
temporary paralysis of the respiratory system
blurred or double vision
difficulty speaking
difficulty chewing or swallowing, resulting in the need to be fed through a tube
difficulty with digestion or bladder control
fluctuations in heart rate or blood pressure

Irritable Bowel Syndrome (IBS)

IBS can be long last, but the symptoms occur sporadically. These can include:

abdominal (gut) pain and cramping
a change in your bowel habits – such as diarrhea, constipation, or sometimes both
excessive gas
occasionally having an urgent need to go to the toilet
a feeling that you have not fully emptied your bowels after going to the toilet
passing mucus with your bowels

Reactive Arthritis

Inflammation from an infection can cause inflammation in your joints, the urinary system, and/or your eyes. This can cause:

joint pain and swelling – usually in the knees, feet or ankles
pain in your lower back and rump
swelling in your fingers and toes
joint stiffness – particularly in the morning

pain or a burning sensation when you pee
urinating more often than usual
having a sudden urge to pee
a discharge of fluid from the penis or vagina

eye pain
swollen eye lids

Kidney Failure

The kidneys are essential to maintaining normal body function. They clean waste products from the blood. Some foodborne illnesses can lead to kidney failure or chronic kidney disease that can eventually cause kidney failure. When a person’s kidneys permanently fail, this person will need dialysis or a kidney transplant to survive.

Meningitis

Meningitis is an inflammation of the lining of the brain or spinal cord. If not treated quickly it can lead to vision and/or hearing impairment, cognitive impairment, difficulty with balance or muscle coordination, or recurrent seizures.

Sepsis

Sepsis occurs when the body’s immune system goes into overdrive as it fights an infection and attacks the body’s own tissues and organs. It requires quick treatment, sometimes in ICU. Most people fully recover from sepsis. But it can take time. People who have had sepsis, particularly serious sepsis, may have symptoms for months or even years. These include: fatigue, difficulty sleeping, difficulty breathing, muscle and joint pain, decreased mental function, poor concentration, and emotional issues similar to PTSD.

M11

M12

M 13 Making choices about avoiding symptoms of foodborne illness [on next page]

Imagine that you face a choice between two options: Option A. You get a foodborne illness and experience the symptoms shown below. After the illness you will return to your current state of health. Option B. You get a foodborne illness but avoid the symptoms. You stay at your current state of health for the whole period. But there is a cost to avoiding the symptoms as shown below. If these were the only two options available to you, which would you choose? Remember your medical costs would be covered, you would not lose wages due to the illness (or earnings if you have your own business), and you would have help to handle any other responsibilities while you’re sick. Remember your budget! If you spend money to avoid the symptoms of foodborne illness, you won’t have it for other things you may want or need.

	Option A. Get a foodborne illness with symptoms	Option B. Get a foodborne illness but avoid the symptoms.
Symptoms you experience	You develop a high temperature, aching muscles, and chills. You have little energy and no appetite. You develop diarrhea. You don’t visit a doctor. The illness lasts 3 days, 1 spent in bed, after which you return to your pre-illness state of health.	You stay in your current state of health
How long symptoms last	3 days	0 days
Extra cost	$0	$30

Which one would you choose?	☐	☐

It’s possible that these preventive expenditures could be more (less) expensive.

[If chosen]

Would you change your choice if Option B cost:

$10?

Yes/No

[If chosen]

Would you change your choice if Option B cost:

$40?

Yes/No

1 Willis, Gordon. 2005. Cognitive Interviewing: A “How to” Guide. Short course presented at the 1999 Meeting of the American Statistical Association.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Sandra Hoffmann
File Modified	0000-00-00
File Created	2023-11-20