Intent to Vaccinate Pretest Memo

Memorandum on Household Pulse Intent to Vaccinate Question Pretesting

Center for Behavioral Science Methods

December 28, 2020

Staff from the Center for Behavioral Science Methods tested a series of intent to vaccinate questions using a Qualtrics web probing instrument. We sampled 5000 email addresses from the Census Bureau’s Affinity Panel and received approximately 200 responses. This memo details our findings and recommendations for the series of intent to vaccinate questions.

GETVACC

Once a vaccine to prevent COVID-19 is available to you, would you… 

1. Definitely get a vaccine – skip to PROBE1

2. Probably get a vaccine  - ask WHYNOT

3. Probably NOT get a vaccine – ask WHYNOT

4. Definitely NOT get a vaccine – ask WHYNOT

To evaluate this question we used a retrospective probe. We showed respondents the text of the question and response options and then asked how difficult they found the question. Most respondents (approximately 80%) reported that the question was “not at all difficult.” Those that did report some level of difficulty were asked to elaborate. Those respondents tended to report that the question itself was easy to understand but that deciding on their answer made the question difficult. We do not recommend changes to this question. However, we do note that as the vaccine roll out continues, the embedded assumption in this question that most have not received or been offered a vaccine will become stale. We recommend dropping this question when COVID-19 vaccines have been made available to most potential respondents.

WHYNOT

4. Which of the following, if any, are reasons that you [only probably will /probably won’t/definitely won’t] [get a COVID-19 vaccine/won’t receive all required doses of a COVID-19 vaccine]? (Select all that apply.)

Scripter: randomize

1. I am concerned about possible side effects of a COVID-19 vaccine

2. I don’t know if a COVID-19 vaccine will work

3. I don’t believe I need a COVID-19 vaccine – go to WHYNOT2

4. I don’t like needles

5. I don’t like vaccines

6. My doctor has not recommended I get a COVID-19 vaccine

7. I plan to wait and see if it is safe and may get it later

8. I don’t know how to get it

9. I am concerned about the cost of a COVID-19 vaccine

10. Other (please specify: _____) [ANCHOR]

One of our main concerns was the list of response options for the WHYNOT item. We used a semi-retrospective web probing approach to evaluate this question. The first open-ended probe was administered to all participants and was designed to evaluate the list of response options in the WHYNOT item. Two analysts coded the open-ended responses. Most of our sample responded that they would “definitely” get a vaccine once it is made available (approximately 60%) and were not asked the WHYNOT follow-up item. The remaining 40% of respondents answered both the WHYNOT item and the open-ended probe. We found that most of the responses to the probe aligned with the response options offered in the WHYNOT item. In addition, we found that most of the responses in the text box provided for those who chose “other, specify” also mentioned reasons that were accounted for in the closed response options.

Drop two response options: I don’t know how to get it and I don’t like needles

We recommend dropping two response options based on low response: 1) “I don’t know how to get it, and 2) “I don’t like needles.’ The response option for “I don’t know how to get it” was only chosen by very few respondents. In addition to the low frequency, this option is vague and is not relevant for those who do not yet have access to a vaccine. This response option could be added back in when the vaccine is made available to most potential respondents. We also recommend dropping the response option “I don’t like needles” since very few respondents chose this option.

Add response options: I don’t trust COVID-19 vaccines, I don’t trust the government, and I think other people need it more than I do right now.

However, the theme of trust emerged from the data that was not covered by our original response options. Both analysts independently identified the theme of trust in the data. Respondents mentioned not trusting drug manufacturers and the speed of vaccine development. There was also mention of distrust of political parties and the current administration. There was mention of the Tuskegee Experiment, which we interpreted as related to distrust in doctors and medicine in general and in the African American community more specifically. Respondents also questioned the speed of vaccine development, citing that vaccines for other diseases, like AIDS have never been invented. We recommend adding two response options to capture this sentiment. The new response options would read: “I don’t trust COVID-19 vaccines” and “I don’t trust the government.”

Another common sentiment in the open ended text was that other people need the vaccine more than the person responding and that others should receive it first, in a benevolent way. For this reason, we recommend also adding a response option that “I think other people need it more than I do right now.”

Response option randomization

We recommend randomizing the response options for this question as well as for the WHYNOT2 follow-up item. This reduces the chances of any primacy effects of the sequence of response options.

WHYNOT2

Why do you believe that you don’t need a COVID-19 vaccine? (mark all that apply)

Scripter: randomize

1. I already had COVID-19

2. I am not a member of a high-risk group

3. I plan to use masks or other precautions instead

4. I don’t believe COVID-19 is a serious illness

5. Other (please specify: _____) [ANCHOR]

Add “I don’t think vaccines are beneficial” to I don’t need follow-up

This item is a follow-up question for those who choose the response option “I do not believe I need a COVID-19 vaccine.” While we did not have many responses to this question or the probe designed to evaluate it, we recommend adding an additional response option. The additional response option would read: “I don’t think vaccines are beneficial.” We recommend this option to give those who generally tend not to vaccinate an appropriate response option if they follow this path.

CBDRB-FY21-CBSM001-002