Attachment NSC-2 - Phase 2 Cognitive Testing

Census Household Pulse Survey Phase 2 Cognitive Testing

Robin Kaplan & Brandon Kopp

Office of Survey Methods Research

3/26/2021

Executive Summary

Background

This research was conducted to assess respondents’ understanding of new response options added to two questions designed to measure shopping and spending changes, and reasons for those changes, that were not previously tested in the web-probing study for Phase 2 of the Census Household Pulse Survey. New response options were added to measure changes in spending behavior that may have occurred due to businesses starting to re-open or expand hours.

Questions tested

The first spending question included new response options that aimed to measure spending changes in the last 7 days related to shopping in-stores, eating at restaurants, attending medical appointments, or resuming/starting housekeeping or caregiving services. The second spending question aimed to measure reasons for these spending changes in the last 7 days. The new response options included items about shops/businesses reopening, resuming onsite work, and not being concerned about going out in public, being laid off, or no longer having concerns about the economy. Full question wording can be found on page 4 of the full report that follows and the Appendix.

Participants and procedure

A convenience sample of 9 participants was recruited for the cognitive interviews, each lasting between 30-45 minutes. Interviews took place remotely on Microsoft Teams using video chat with one interviewer and one observer to take notes. The interviewers could speak to and see the participants, and also display the questions being tested via screen-sharing. Participants were asked a few questions from the Pulse survey that precede the spending questions in the production survey, and then the two spending questions were administered. Afterward, the interviewer debriefed the participants on the two spending questions, including general comprehension of the question and response options, and then specific probes aimed to get at participants’ understanding of the new response options.

Results

Results showed that participants understood both of the questions and all of the new response options as intended. Participants were able to answer the questions and recall spending behaviors for the past week accurately for themselves and other household members. Several participants interpreted the questions as being related to COVID-19, although the questions did not explicitly state that. Some minor confusion occurred about the 7-day reference period (e.g., questioning why a broader reference period wasn’t used, or why the question did not refer back to mid-March when COVID restrictions first began). However, when debriefing participants at the end, and explaining this would be a weekly survey designed to capture change over time, they understood the purpose of the 7-day reference period better.

Several participants expressed that there were many response options and being confused about the mutually exclusive nature of some of the response options. However, this did not prevent participants from correctly understanding the meaning of the question and response options. This issue may have been exacerbated by the experience of having an interviewer present and the pressure of responding verbally in real-time, and would likely be less of an issue when answering in a self-administered mode (as in the Pulse survey).

For the second spending question, the majority of participants questioned response options that implied people are no longer concerned about going to public or crowded places or being exposed to high-risk people, about being laid off, and no longer have concerns about the economy. Participants felt that these response options were a bit unrealistic and people are likely to still have some lingering concerns about those issues, even as businesses start to re-open.

Recommendations

No changes were recommended for the first spending question. For the second spending question, we suggest that several response options be changed from “no longer concerned” to “less concerned” as such:

Original response option	Revised response option
No longer concerned about going to public or crowded places or having contact with high-risk people	Less concerned about going to public or crowded places or having contact with high-risk people
No longer concerned about being laid off or having hours reduced	Less concerned about being laid off or having hours reduced
No longer concerned about the economy	Less concerned about the economy

Overall, the new response options were understood correctly, and participants could answer and recall the information needed accurately. We recommend the questions be included in the Pulse survey as-written, aside from the change outlined in the above table to the response options for the second spending question. The full report below describes the research, main findings, and recommendations in more detail.

Overview

The goal of this research was to assess participants’ understanding of new response options added to two spending questions that were not previously tested in the web-probing study for Phase 2 of the Census Household Pulse Survey. The new response options were added to measure changes in spending and shopping behavior that may have occurred due to businesses starting to re-open or expand hours recently due to the COVID-19 pandemic. The other response options were previously tested and performed well.

The two questions tested are displayed below, and the new response options are highlighted in yellow:

1. Q21d. In the last 7 days, which of the following changes have you or your household made to your spending or shopping? Select all that apply. 

1. Made more purchases online (as opposed to in store) 

2. Made more purchases by curbside pick-up (as opposed to in store) 

3. Made more purchases in-store (as opposed to purchases online or curbside pickup)

4. Increased use of credit cards or smartphone apps for purchases, instead of using cash 

5. Avoided eating at restaurants 

6. Resumed eating at restaurants 

7. Canceled or postponed in-person medical or dental appointments 

8. Attended in-person medical or dental appointments

9. Canceled or postponed housekeeping or caregiving services 

10. Resumed or started new housekeeping or caregiving services

11. Did not make any changes to spending or shopping behavior (SKIP 21F)

2. Q21f. (if 21d = 1 through 10) In the last 7 days, for which of the following reasons have you or your household changed spending? Select all that apply. 

1. Usual shopping places were closed or had limited hours (e.g., restaurant, doctor/dentist office, health club, hair salon, child care center, etc.)

2. Usual shopping places re-opened or increased hours

3. Concerned about going to public or crowded places or having contact with high-risk people 

4. No longer concerned about going to public or crowded places or having contact with high-risk people 

5. Loss of income 

6. Increased income

7. Concerns about being laid off or having hours reduced 

8. No longer concerned about being laid off or having hours reduced

9. Working from home/teleworking 

10. Resumed working onsite at workplace

11. Concerns about the economy  

12. No longer concerned about the economy

13. Other, specify:

Protocol

Interviews took place remotely on Microsoft Teams using video chat with one interviewer and one observer to take notes. The interviewers could speak to and see the participants, and also display the questions being tested via screen-sharing. This was done because the questions would be difficult to administer verbally and the use of show cards displaying the full question and response options was necessary. This approach also more closely mimics the experience of viewing the questions in the Pulse production survey (which is online and self-administered).

The protocol started by explaining the purpose of the research and that we were seeking feedback on questions that ask about their household’s spending. Participants were instructed to read each question out loud and then verbally state their responses. They were encouraged to think aloud about their response process and explain how they arrived at their answers in real-time.

A number of questions from the Pulse survey were also asked that precede the spending questions in the production survey, including household size, whether respondents experienced a loss of employment income or expect to in the next 4 weeks, whether respondents worked in the last 7 days, and whether it has been difficult to pay for usual expenses in the last 7 days. These questions were included to provide helpful context for the spending questions and were required for some of the later skip patterns based on household size. Afterward, the two spending questions were administered.

After participants answered the questions, they were debriefed on the two spending questions. General probes were asked about each question stem first, including probes to get at comprehension of the question and the response options. Afterward, specific probes were included to get at comprehension of the individual response options, with an emphasis on response options that the participant had selected and the new response options. At the very end of the interviews, participants were fully debriefed about the purpose of the research (i.e., testing questions for the Census Household Pulse Survey, which is administered weekly and designed to capture spending changes over time with the same respondents) and thanked for their participation. The full protocol is available in the Appendix.

Participants

A convenience sample of 9 participants were recruited for the cognitive interviews, each lasting between 30-45 minutes. The table below includes some basic demographic information about the participants:

Mean age	M = 46.7 years old (range = 32 to 56)
Gender	4 Male 5 Female
Household size	Median: 3, ranging from 1 to 6 5 participants had children under 18 years old
Employment status	8 employed 1 not employed
State of residence	7 Virginia 1 Maryland 1 New Hampshire

Due to the limited timeline to complete this research, broad recruitment criteria were used. Although attempts were made to recruit participants who had changed spending behaviors in the last 7 days, this was not always possible. Participants were recruited via OSMR staff contacts based on meeting any of the following criteria:

• Regularly shop at big box stores and/or grocery stores, or used to but stopped due to COVID

• Regularly eat at restaurants, or used to but stopped due to COVID

• Have, or used to have, caregivers or housekeepers working in their home

We were able to recruit participants who met the following broad criteria:

• 1 participant who started eating in restaurants again in the last 7 days.

• 1 participant who recently started going back to on-site work and shopping in-person at grocery stores (but not in last 7 days).

• 1 participant who recently had 2 people move into their house and had a change in spending and shopping as a result of that.

• 1 participant who recently sent a child back to offsite daycare.

• 1 participant whose family recently moved to the U.S. from overseas, and had to postpone sending children back to offsite daycare.

Results

The results focus on general comprehension of the question stems and response options. Specific findings focus primarily on the new response options since they had not been previously tested.

Question 21d

“In the last 7 days, which of the following changes have you or your household made to your spending or shopping? Select all that apply.” 

Question stem

In general, participants understood this question as-intended and understood that the question was asking about any changes in their spending that occurred in the last 7 days. Most participants mentioned that they assumed the question was COVID-related, although the question did not explicitly mention it. Many noted they thought about their spending pre and post-COVID, but then recalled the 7-day reference period and answered accordingly. Some participants seemed to have slight confusion about the reference period, wondering if it was supposed to be “compared” to some other time frame, such as the 7-days beforehand, or since the coronavirus pandemic first began:

“I wasn’t really sure. It’s asking about the last week, but I’m not sure what time period it’s comparing to. It’s comparing two time frames, last week compared to the previous week…I’m not sure what it’s comparing to. Just comparing to recent times, it’s all the same.”

“Had to think about changes I’ve made since COVID started and then how many of those actually applied to the last 7 days. Mentally tallied that up.”

“I don’t know what it’s comparing to. Are you comparing it to the year before or the week before?”

Another point of minor confusion for some participants was questioning the 7-day reference period in general. Some thought this was such a short reference period that it would be hard to capture meaningful spending habits and changes:

“What’s important about the last 7 days? Why not one month? Seven days seems like a narrow slice. I’m trying to imagine something within the last 7 days that should prompt examination.”

“In the past 7 days it’s hard to judge whether I made more purchases by different modes. Hard to say in past 7 days. Not enough days to notice a change in pattern. Need more time to determine if my behavior change and determine why.”

“It would be easier if asking about last 1-2 months.”

Similarly, many participants also mentioned that some of the changes would have applied to them if the reference period had been broader, such as 14 days or one to two months. Many also noted that they had been engaging in behaviors for a long time (often since mid-March), such as making online purchases, avoiding restaurants, or canceling appointments, but there was no change in the last 7 days:

“We haven’t eaten at a restaurant since March, I picked it because 7 days falls within that period. It didn’t change in the previous 7 days versus the 7 days before that.”

Ultimately, most respondents were able to answer the question in terms of the last 7 days without difficulty. Although some felt it might be easier to answer with a broader reference period, participants seemed to be able to recall their behaviors accurately in the past 7 days. When debriefing respondents at the end, and explaining this would be a weekly survey, participants understood the purpose of the reference period better. Overall, the question stem was understood by respondents, and they could answer the question without major problems for themselves and their household. Thus, no change in reference period is recommended as we expect the context of the rest of the Pulse survey will reduce the likelihood of this confusion occurring.

Tally of the response options

The table below shows the frequencies in which participants selected each of the response options for Q21d:

In the last 7 days, which of the following changes have you or your household made to your spending or shopping? Select all that apply. 

Response options	Frequency selected
1) Made more purchases online (as opposed to in store)	5
2) Made more purchases by curbside pick-up (as opposed to in store)	1
3) Made more purchases in-store (as opposed to purchases online or curbside pickup)	1
4) Increased use of credit cards or smartphone apps for purchases, instead of using cash	3
5) Avoided eating at restaurants	3
6) Resumed eating at restaurants	1
7) Canceled or postponed in-person medical or dental appointments	0
8) Attended in-person medical or dental appointments	2
9) Canceled or postponed housekeeping or caregiving services	1
10) Resumed or started new housekeeping or caregiving services	0
11) Did not make any changes to spending or shopping behavior	2

General feedback on the response options

When asked about general reactions to the response options, most participants indicated the items were not surprising and the list included items they would expect to see. Some participants felt that other categories might include spending more money on furniture or home offices now that people are staying and working at home more, purchasing more entertainment services such as streaming cable, or spending on school supplies for remote learning.

A common issue for participants was forgetting the question stem while (although it was still displayed on the screen) making their way through the response options. Some participants would start to answer yes or no to the response options, forgetting that the question was asking about whether their spending or shopping had changed. Participants would often have to go back and re-read the question stem as a reminder of what the question was asking before arriving at their responses:

“Since this is real-time I feel stressed about responding quickly. It is an overwhelming list of response options.”

“I forgot what the question was asking by the time I got to number 5 or 6.”

“I lost the root of the spending or shopping. Just interpreted as whether I did or not.”

Although the list of response options seemed long to participants, it is important to acknowledge that the experience of cognitive interviews is different from taking the survey online. Some participants might have felt pressured to answer quickly with an interviewer present, however, participants may find this easier in a self-administered mode, as one participant pointed out:

“It will be helpful if it is online and people can take their time.”

Several participants also mentioned noticing that some of response options were the “flip side” of one another, and wondering if they were mutually exclusive or not:

“If you select ‘Avoided eating at restaurants’ then you wouldn’t select ‘Resumed eating at restaurants, right?”

“Thought it was strange to have 7 & 6 on the same list because they are mutually exclusive. I just knew that we haven’t eaten at a restaurant so I dismissed that as a selection.”

“The first four are relative statements, five and six are ‘avoided’ and ‘resumed’ and do not need to be relative to anything. Attended didn’t have to be a change.”

Conversely, one participant expressed that the mutually exclusive response options were not confusing but “complementary” and made responding a bit easier because the options were more black and white that way (e.g., either you avoided restaurants or ate at restaurants). Similarly, several participants mentioned immediately discounting the other complementary response option once selecting the one that applied to them.

Despite some initial confusion, participants seemed to understand that some response options were mutually exclusive and took this into account when arriving at their answer. Most participants answered the questions thinking about their own experiences as well as those of other household members. A couple of participants mentioned initially answering the question for just themselves, then considered whether other household members would have the same answers – in all instances that was the case. Thus, participants seemed to take their perspective as well as the households’ into account when answering the questions and understood the meaning of the response options.

Feedback on the new response options

Probes on the previously tested response options revealed that they continued to work well, consistent with the results from the web probing. Thus, this section focuses primarily on the new response options. The table below displays each of the new response options and notable participant reactions to each. Note that participant reactions were varied and often spoke to their unique situations and thought processes, and not necessarily their comprehension of response option itself.

New Response Option	Participant Feedback
3) Made more purchases in-store (as opposed to purchases online or curbside pickup)	All participants understood this response option as referring to shopping in stores. A couple participants had varying reactions to the response option: “This question seemed political. Refers to people who shop in the store and are less respectful of the virus.” “I don’t know why people would choose to go shop in-store when they can buy most things online. Maybe they have to go in-person to places like a pharmacy to get medications.” Most participants thought of grocery, big box, and home improvement stores for this item.
6) Resumed eating at restaurants	All participants correctly understood this response option to mean physically dining indoors at a restaurant or dining in the outdoor seating or patio provided by the restaurant (as opposed to getting take out, drive-thrus, or delivery). One participant mentioned eating indoors at a restaurant in the past 7 days – this participant previously had COVID and felt more “immune” to the virus and thus safe in doing so. One participant, from a smaller state, noted that indoor dining was more common there than in a neighboring, larger state that still has more restrictions. The larger state mostly has outdoor dining only.
8) Attended in-person medical or dental appointments	Generally, participants understood this response option as referring to seeing medical professionals in-person at an office or hospital (as opposed to using telemedicine or telehealth services). Two participants mentioned attending an in-person medical appointment in the last 7 days, but when probed later, noted this appointment this hadn’t actually affected their spending: “I lost the root of the spending or shopping. Just interpreted as whether I did or not.” One participant noted that this “encompasses things that would be prescheduled. It would be difficult to anticipate a change over the last 7 days unless I had something to suddenly attend to.” One participant thought this response option was a little strange because: “I consider medical visits to be vital and wouldn’t cancel them for any reason.”
10) Resumed or started new housekeeping or caregiving services	Generally, participants understood this response option as-intended. In the context of the previous response option (#9), participants understood this as asking if they used to have a regular housekeeping or caregiving service, and whether or not they restarted that service in the past 7 days. Several participants found it strange to lump caregiving (a necessity) and housekeeping (not essential) in the same response option: “Housekeeping services seem like a luxury whereas caregiving is a necessity and so it’s strange to group them together.” A couple participants wondered what counts as caregiving services: “I wasn’t sure if caregiving would include school closures and/or spending on private school.” “I assume caregiving would include dog-walking.”

Summary of Q21d

Overall, Q21 was understood correctly by participants. Participants understood the question stem as referring to changes made in spending in the last 7 days. Although participants found the number of response options long, they were able to understand the response options and arrive at accurate answers for themselves and other household members. This is likely to be attenuated in a self-administered mode. Overall, the findings do not support making any changes to Q21d.

Question 21f

(if 21d = 1 through 10) In the last 7 days, for which of the following reasons have you or your household changed spending? Select all that apply.

Most participants correctly understood that this question was getting at the reasons for spending changes in the last 7 days. Like Q21d, similar issues regarding the reference period and mutually exclusive nature of the response options were also mentioned for this question. However, since participants had encountered this previously in Q21d, this issue wasn’t as pronounced and participants were able to answer the question. Again, several participants interpreted the question as asking about whether something happened to their finances due to COVID, although the question does not mention that explicitly:

“Although the question only asked about past 7 days, in my mind I compared back to pre-COVID times.”

“Anything that doesn’t link directly to COVID probably falls into ‘Other.’”

Most participants did not indicate that any of the response options seemed surprising or that something was missing from the list of response options. A couple participants mentioned they might have expected to see PPE on the list, again because “everyone has had to buy that lately.” One participant also mentioned “family reasons” could also be included (separate from health reasons), and one mentioned increased medical bills or unexpected housing bills or home repairs. Some participants brought up other reasons for changing spending, including having children at home (e.g., children returning from college or grade school age children not going to in-person classes) and spending more on distance learning materials or home office items. A few participants noted that the ‘Other, specify’ option was a nice 13 “catch-all,” supporting its inclusion. Consistent with the previous question, most participants correctly included both themselves and other household members when answering the question.

Tally of the response options

The table below shows the frequencies in which participants selected each of the response options for Q21f:

In the last 7 days, for which of the following reasons have you or your household changed spending? Select all that apply.

Response options	Frequency selected
1) Usual shopping places were closed or had limited hours (e.g., restaurant, doctor/dentist office, health club, hair salon, child care center, etc.)	0
2) Usual shopping places re-opened or increased hours	1
3) Concerned about going to public or crowded places or having contact with high-risk people	5
4) No longer concerned about going to public or crowded places or having contact with high-risk people	0
5) Loss of income	0
6) Increased income	1
7) Concerns about being laid off or having hours reduced	0
8) No longer concerned about being laid off or having hours reduced	0
9) Working from home/teleworking	2
10) Resumed working onsite at workplace	0
11) Concerns about the economy	3
12) No longer concerned about the economy	0
13) Other, specify:	21

Feedback on the new response options for Q21f

Probes on the previously tested response options revealed that they continued to work well, consistent with the results from the previous testing. Thus, this section focuses primarily on the new response options. The table below displays each of the new response options and notable participant feedback on each item. Note that participant reactions were varied and often spoke to their unique situations and thought processes, and not necessarily their comprehension of response option itself.

New Response Option	Participant Feedback
2) Usual shopping places re-opened or increased hours	This response option was generally understood by participants. Some participants thought of the pandemic causing stores to close or cut their hours, and now more places are now re-opening. Places like gyms, shopping malls, home improvement stores, and grocery stores came to mind for most participants. One participant mentioned businesses more generally than just shopping places, such as children’s play gyms.
4) No longer concerned about going to public or crowded places or having contact with high-risk people	This response option was generally understood by participants. Participants thought crowded places referred to busy bars, restaurants, stores during their peak hours, or taking public transportation. One participant thought of being willing to shop for “non-essentials” in-person, such as clothing or accessory stores. A majority of participants felt it was unrealistic that people would “no longer be concerned” about going to crowded places: “A more realistic wording of the question would be to say less concerned about going to crowded places.” “This is about ‘yolo’ people – younger people who are less concerned about going out.” “This reason was strange. I took it to mean you are not worried anymore about getting this virus. I was surprised that this was ‘no longer concerned’ instead of ‘less concerned’.” “This might be my own bias, but do people do that?” A couple participants noted this may refer to people who feel the health and safety precautions businesses have taken are sufficient, and so people feel safe going out to public places without getting infected.
8) No longer concerned about being laid off or having hours reduced	This response option was generally understood by participants. Most participants interpreted the question as directly referring to pandemic-related concerns due to employers being closed or reducing hours, but now reopening again. One participant felt this response option came off “arrogant – people who have economic and financial soundness of their employer making money again and surviving during the pandemic.” One participant noted: “You have a small percentage of the population whose job is 100% secure. Doesn’t seem that would apply to many people.”
10) Resumed working onsite at workplace	Participants correctly understood this response option as asking about returning to work at a physical office or location. Several participants mentioned taking public transportation and having to work in an office with other people.
12) No longer concerned about the economy	Most participants understood this response option in terms of the pandemic, and having more optimism about things like GDP and the unemployment rate improving. A few participants felt it would be unrealistic for people to have no concerns about the economy, similar to the feedback on response options 4 and 8: “Don’t know how anyone could not be concerned about the economy right now.” “I wouldn’t expect anyone to be no longer concerned about the economy at this point.” This participant had strong concerns about the uncertainty of the economy in the future, including the virus worsening, uncertainty about whether more economic support programs would pass, and student loan payments. This participant tried to reduce spending as much as possible, focusing only on essentials, and keeping an eye on policy and the economy. One participant noted he thought a clearer way to ask about concerns with the economy might be getting at spending habits (e.g., “where people are more or less likely to spend in this economic environment”), or that a term like “consumer confidence” would be better. This participant also wondered whether most people would think about the economy in general, or just how the pandemic affected them personally. Another participant was unsure what this response option meant: “I thought of the stock market. It didn’t make much sense to me that that would change my spending. I thought those were curious. Wasn’t sure which part of the economy it was about.”

Summary of Q21f

Overall, participants correctly understood Q21f as asking about reasons for changes in spending for themselves and their households in the last 7 days. The new response options were also generally understood, although response options 4, 8, and 12 had mixed reactions, with most participants questioning if people would really “no longer be concerned” with going out in public, being laid off, or the economy. Because of this, we recommend rewording those response options to start with the phrase “less concerned” rather than “no longer concerned.”

Conclusions and recommendations

Overall, results from 9 cognitive interviews showed that participants generally understood the two spending questions and the new response options added to get at businesses reopening in the wake of the COVID-19 pandemic. Although there was some minor confusion about the reference period and mutually exclusive nature of some of the response options, this did not seem to affect comprehension and participants’ ability to arrive at accurate answers for themselves and their household. The question is likely to perform better in a self-administered mode.

No changes are recommended to Q21d. We identified some changes to Q21f response options 4, 8, and 12 to better reflect the idea that people may be “less concerned” rather than having “no concerns” about going out in public, being laid off, or the economy to reflect participant feedback that it seemed unlikely people would have absolutely no concerns about these issues. These changes are reflected in the table below.

Recommended changes

Question tested	Recommended change
Q21d. In the last 7 days, which of the following changes have you or your household made to your spending or shopping? Select all that apply. 1) Made more purchases online (as opposed to in store) 2) Made more purchases by curbside pick-up (as opposed to in store) 3) Made more purchases in-store (as opposed to purchases online or curbside pickup) 4) Increased use of credit cards or smartphone apps for purchases, instead of using cash 5) Avoided eating at restaurants 6) Resumed eating at restaurants 7) Canceled or postponed in-person medical or dental appointments 8) Attended in-person medical or dental appointments 9) Canceled or postponed housekeeping or caregiving services 10) Resumed or started new housekeeping or caregiving services 11) Did not make any changes to spending or shopping behavior	No changes needed, keep question and response options as-written.
Q21f. In the last 7 days, for which of the following reasons have you or your household changed spending? Select all that apply. 1) Usual shopping places were closed or had limited hours (e.g., restaurant, doctor/dentist office, health club, hair salon, child care center, etc.) 2) Usual shopping places re-opened or increased hours 3) Concerned about going to public or crowded places or having contact with high-risk people 4) No longer concerned about going to public or crowded places or having contact with high-risk people 5) Loss of income 6) Increased income 7) Concerns about being laid off or having hours reduced 8) No longer concerned about being laid off or having hours reduced 9) Working from home/teleworking 10) Resumed working onsite at workplace 11) Concerns about the economy 12) No longer concerned about the economy 13) Other, specify:	In the last 7 days, for which of the following reasons have you or your household changed spending? Select all that apply. 1) Usual shopping places were closed or had limited hours (e.g., restaurant, doctor/dentist office, health club, hair salon, child care center, etc.) 2) Usual shopping places re-opened or increased hours 3) Concerned about going to public or crowded places or having contact with high-risk people 4) Less concerned about going to public or crowded places or having contact with high-risk people 5) Loss of income 6) Increased income 7) Concerns about being laid off or having hours reduced 8) Less concerned about being laid off or having hours reduced 9) Working from home/teleworking 10) Resumed working onsite at workplace 11) Concerns about the economy 12) Less concerned about the economy 13) Other, specify:

Appendix: Protocol

Introduction

Thank you for participating in this research today for the Bureau of Labor Statistics. We are interested in getting your feedback on questions that ask about you and your household’s spending. Before we add questions to surveys, we like to run them by people to make sure that they make sense and that people understand the questions in the same way that we do. So today we will go through a set of questions and then I will ask you some questions about your experience answering those questions.

Do you have any questions before we get started?

Survey

I am going to share my screen with you now. On the page, you will see a series of questions. Please read each question out loud, and let me know which answers you would select. As you are answering the questions, please think aloud. Your thoughts as you are answering the question are helpful to us.

Question	Response/Notes
How many total people – adults and children – currently live in your household, including yourself?
(if Q1 > 1) How many people under 18 years-old currently live in your household?
Have you, or has anyone in your household experienced a loss of employment income since March 13, 2020? 1) Yes 2) No
Do you expect that you or anyone in your household will experience a loss of employment income in the next 4 weeks because of the coronavirus pandemic? Select only one answer. 1) Yes 2) No
In the last 7 days, did you do ANY work for either pay or profit? Select only one answer. 1) Yes 2) No
In the last 7 days, how difficult has it been for your household to pay for usual household expenses, including but not limited to food, rent or mortgage, car payments, medical expenses, student loans, and so on? Select only one answer. 1) Not at all difficult 2) A little difficult 3) Somewhat difficult 4) Very difficult
Q21d. In the last 7 days, which of the following changes have you or your household made to your spending or shopping? Select all that apply.  Made more purchases online (as opposed to in store)  Made more purchases by curbside pick-up (as opposed to in store)  Made more purchases in-store (as opposed to purchases online or curbside pickup) Increased use of credit cards or smartphone apps for purchases, instead of using cash  Avoided eating at restaurants  Resumed eating at restaurants  Canceled or postponed in-person medical or dental appointments  Attended in-person medical or dental appointments Canceled or postponed housekeeping or caregiving services  Resumed or started new housekeeping or caregiving services Did not make any changes to spending or shopping behavior (SKIP 21F)
Q21f. (if 21d = 1 through 10) In the last 7 days, for which of the following reasons have you or your household changed spending? Select all that apply.  Usual shopping places were closed or had limited hours (e.g., restaurant, doctor/dentist office, health club, hair salon, child care center, etc.) Usual shopping places re-opened or increased hours Concerned about going to public or crowded places or having contact with high-risk people  No longer concerned about going to public or crowded places or having contact with high-risk people  Loss of income  Increased income Concerns about being laid off or having hours reduced  No longer concerned about being laid off or having hours reduced Working from home/teleworking  Resumed working onsite at workplace Concerns about the economy   No longer concerned about the economy Other, specify:

Debriefing

I would now like to circle back and ask you some follow-up questions that we just went through.

[SHOWCARD: GO TO Q21d]

[ALL]

Question	Response
In your own words, what was this question asking about?
Did any of the changes listed in the question seem surprising to you? Which ones?
Were there any other changes you expected to see on this list, but didn’t? What are those?
(If HH size >1) Did you answer this question thinking about your own experiences or did you also include the experiences of others in your household?

[IF SELECTED Q21d = 1-10]

• Let’s talk about some of the specific changes.

Question	Response
1) Made more purchases online (as opposed to in store) (If selected) Tell me about how your spending changed related to buying things online
3) Made more purchases in-store (as opposed to purchases online or curbside pickup) (If selected) Tell me about how your spending changed related to buying in-store (If NOT selected) In your own words, what does [repeat response option] mean to you? Optional: What types of stores came to mind when you read this question? Optional: What types of stores do you think people would use curbside pickup for?
5) Avoided eating at restaurants  (If selected) Tell me about how your spending changed related to eating at restaurants.
6) Resumed eating at restaurants  (If selected) Tell me about how your spending changed related to eating at restaurants. Did your experience eating at restaurants in the last 7 days include dining in the restaurant? (if “No”) Can you tell me more about that? (If NOT selected) In your own words, what does [repeat response option] mean to you? Optional: What types of restaurants came to mind when you read this question?
7) Canceled or postponed in-person medical or dental appointments  (If selected) Tell me about how your spending changed related to medical or dental appointments.
8) Attended in-person medical or dental appointments (If selected) Tell me about how your spending changed related to medical or dental appointments. (If NOT selected) In your own words, what does [repeat response option] mean to you?
9) Canceled or postponed housekeeping or caregiving services  (If selected) Tell me about how your spending changed related to housekeeping or caregiving services (if NOT selected) In your own words, what does [repeat response option] mean to you?
10) Resumed or started new housekeeping or caregiving services (If selected) Tell me about how your spending changed related to housekeeping or caregiving services (If NOT selected) In your own words, what does [repeat response option] mean to you?

[IF SELECTED NO CHANGES]

• Although you indicated that you did not make any changes to your spending in the last 7 days, we would like to get your general feedback on the types of changes listed in this question.

• Let’s look at the following types of changes listed in the question:

Question	Response
3) Made more purchases in-store (as opposed to purchases online or curbside pickup) In your own words, what does [repeat response option] mean to you? Optional: What types of stores came to mind when you read this question? Optional: What types of stores do you think people would use curbside pickup for?
6) Resumed eating at restaurants  In your own words, what does [repeat response option] mean to you? Optional: What types of restaurants came to mind when you read this question?
8) Attended in-person medical or dental appointments In your own words, what does [repeat response option] mean to you?
10) Resumed or started new housekeeping or caregiving services In your own words, what does [repeat response option] mean to you?

[SHOWCARD: GO TO Q21f]

[ALL]

Question	Response
In your own words, what was this question asking about?
Did any of the reasons listed in the question seem surprising to you? Which ones?
Were there any other reasons that you expected to see on this list, but didn’t? What are those?
(If HH size >1) Did you answer this question thinking about your own experiences or did you also include the experiences of others in your household?

[IF SELECTED Q21d = 1-10]

• Let’s talk about some of the specific reasons the question asked about.

Question	Response
2) Usual shopping places re-opened or increased hours (if selected): In your own words, what does [repeat response option] mean to you? What types of shopping places were you thinking of when you selected this reason? (if NOT selected) In your own words, what does [repeat response option] mean to you? What types of shopping places came to mind when reading this reason?
4) No longer concerned about going to public or crowded places or having contact with high-risk people (if selected) In your own words, what does [repeat response option] mean to you? Optional: what made you select this response? (if NOT selected) In your own words, what does [repeat response option] mean to you?
8) No longer concerned about being laid off or having hours reduced (ALL) In your own words, what does [repeat response option] mean to you?
10) Resumed working onsite at workplace (ALL) In your own words, what does [repeat response option] mean to you? (If HH size > 1) Did you consider other household members when answering this question?
12) No longer concerned about the economy (ALL) In your own words, what does [repeat response option] mean to you?

[IF SELECTED OTHER-Specify]

• Let’s talk about some of the specific reasons the question asked about.

Question	Response
You selected ‘Other reason.’ Please explain how the reason you gave differs from the reasons listed in the question.

[SHOWCARD: GO TO Q21f]

[IF Q21d = 11]

• Although you didn’t make any changes to your spending in the last 7 days, we would like to get your feedback on the follow up question. Please read it out loud, and I will have some follow-up questions for you later.

Question	Response
In your own words, what was that question asking about?
What time frame were you thinking of when reading this question?
Did any of the reasons listed in the question seem surprising to you? Which ones?
Were there any other reasons that you expected to see on this list, but didn’t? What are those?
(If HH size > 1) Did you think about your other household members when reading the question?

• Let’s talk about some of the specific reasons the question asked about.

Question	Response
2) Usual shopping places re-opened or increased hours In your own words, what does [repeat response option] mean to you? When thinking about “usual shopping places” in terms of your spending, what types of places came to mind?
4) No longer concerned about going to public or crowded places or having contact with high-risk people In your own words, what does [repeat response option] mean to you? When thinking about “public or crowded places” in terms of your spending, what types of places came to mind?
8) No longer concerned about being laid off or having hours reduced In your own words, what does [repeat response option] mean to you?
10) Resumed working onsite at workplace In your own words, what does [repeat response option] mean to you? (If HH size > 1) Did you consider other household members?
12) No longer concerned about the economy In your own words, what does [repeat response option] mean to you?

Final Questions

I have a few final questions that I would like to ask to finish up

Question	Response
How old are you as of today?
What city and state do you live in?
In the past month, how frequently have you participated in any public activities, such as attending in-person events, shopping in stores, eating in restaurants, etc.? Never Occasionally Sometimes Often
What other feedback do you have about the questions?

Thank you for your participation.

1 “Other” responses included: “Retirement” and “Changed spending because we are feeding more people.” Also note that one participant did not select any of the response options.

15