Attachment J-CPS COVID Questions Cognitive Testing Results

Cognitive Testing Report

Current Population Survey coronavirus (COVID-19) pandemic-related questions

In May 2020, the Current Population Survey (CPS) fielded five new questions with the intent of facilitating the understanding of changes to the labor market during the COVID-19 pandemic. Due to the rapid onset of the pandemic and the importance of the data the questions are meant to collect, the Office of Management and Budget (OMB) allowed the questions to be fielded with the expectation that they be cognitively tested in the near future with any appropriate adjustments to soon follow. On May 1^st, OSMR was asked to conduct cognitive testing and return a report with the results.

Methods

Telephone Interviews

Volunteers for cognitive testing were found through email solicitations asking BLS staff to reach out to friends and family members that fit the criteria below. Specifically we were looking for nine people who fit the following criteria:

• From a household with 2 or more people over age 16, and:

• Not in the Labor Force – That is, currently without a job and not currently looking for one (due to COVID or other reasons). We hoped to avoid retired people or those who aren’t at all interested in ever returning to the labor force.

• Employed– Currently have a job, either full- or part-time.

  • Those who are currently working

  • Those who are not currently working (with or without pay)

These criteria were selected based on the skip patterns of the questions to ensure we had adequate coverage of all of the questions.

All interviews were conducted over the phone. One researcher conducted the interview and, in almost all cases, a second researcher listened and took notes. In each interview, the researcher started by introducing the purpose of the study and familiarizing participants with the CPS. He then conducted an abbreviated version of the CPS including a shortened household roster and labor force questionnaire and the applicable COVID-19 questions. The instrument used for testing allowed participants to report this information for themselves and up to 2 household members.

After going through the abbreviated survey, participants were asked a series of debriefing questions about the COVID-19 questions they had received during the interview. Participants were then thanked for their time and the call was ended. The whole process - introduction, abbreviated survey, and debriefing - took between 15 and 25 minutes per participant.

Census Pulse Web Probe Online Assessment

Following the emergence of COVID-19, the U.S. Census Bureau launched the Household Pulse Survey, which was designed to quickly and efficiently collect data on how people’s lives are impacted by the COVID-19 pandemic. The Household Pulse Survey is composed of questions submitted by various federal agencies. To facilitate questionnaire evaluation, the Census Bureau also launched an online questionnaire that allows researchers evaluating COVID-19 related questions to pose their questions to a relatively large number of web participants along with closed- and open-ended probing questions and receive feedback within weeks. Two rounds of this Web Probe Assessment were conducted with independent samples. On the first, there were two questions similar to the CPS COVID questions¹. On the second, BLS researchers were able to add several of the COVID-19 questions to this web probe survey. The full set of questions relevant to this study are included in Appendix A.

Participants in the Census Pulse Web Probe Assessment were drawn from a non-probability panel maintained by the Census Bureau of individuals who expressed interest in taking part in online questionnaire testing.

Round 1 included 578 participants who were broadly dispersed geographically, coming from all states except Maine, Missouri, South Dakota, Vermont, and Wyoming. Round 2 included 866 participants from all states except Delaware and Missouri. Participants in both rounds were mostly female (62 percent in Round 1, 57 percent in Round 2) and highly educated with high percentages in both samples having either a bachelor’s or graduate degree (72 percent in Round 1, 65 percent in Round 2). Fifty-nine percent of Round 1 and 55 percent of Round 2 respondents reported engaging in some form of work for pay in the 7 days preceding their completion of the survey.

Caveat

Due to the limited number of participants, the results outlined in this memorandum are qualitative in nature and serve as a guide to spot potential question issues. The Web Probe Assessment also used a non-probability, convenience sample known not to be nationally representative. The results can be used to provide general indications of question effectiveness, but cannot be generalized to indicate how widespread these issues might be in the population or the regular CPS sample. The recommendations outlined in this report have not been cognitively tested and thus present some risk in their acceptance without further testing.

Results

The summary table below specifies the number of participants who were asked about each question during both the telephone interviews and in the Web Probing Assessment², and the summary recommendation for each.

	Topic	Cognitive Interview	Web: Round 1	Web: Round 2	Recommendation
INST	Section Instructions	9	n/a	n/a	Revision
Q1	Telework from home	9	n/a	464	No Change
Q2	Unable to work	9	n/a	810	No Change
Q3	Receive pay for lost work	7	n/a	n/a	No Change
Q4	Prevented from looking for work	5	n/a	98	No Change
Q5	Need medical care for non-virus, did not receive	9	577	720	Revision

INST. Section Instructions

Tested Wording	Recommended Wording
I now have a few questions related to work activities affected by the Coronavirus-COVID-19 pandemic. Efforts to contain the Coronavirus have included business and school closures, social distancing, and other disruptions. The following questions refer to the last 4 weeks. By the last 4 weeks I mean the 4 week period ending last Saturday.	I now have a few questions related to work activities affected by the Coronavirus-COVID-19 pandemic. Efforts to contain the Coronavirus have included business and school closures, social distancing, and other disruptions. The following questions refer to the last 4 weeks. By the last 4 weeks, I mean the 4 week period beginning on Sunday, [DATE] and ending last Saturday, [DATE].

Cognitive Interviewing Results

All participants were read the set of instructions above and no debriefing questions specifically covered the instructions. No participants expressed any confusion as the instructions were read. As will be discussed later, two participants did ask, after Question 3 was read, for the start-point of the four week period. Future updates could include wording making the time period explicit (e.g., “By the last 4 weeks, I mean the 4 week period beginning on Sunday, April 19^th and ending last Saturday, May 16^th”). This aligns the approach with how the reference period is presented to respondents before starting the labor force series.

Q1. Telework from home

Tested Wording	Recommended Wording
At any time in the LAST 4 WEEKS, did (you/name) telework or work at home for pay BECAUSE OF THE CORONAVIRUS PANDEMIC? (Enter No if person worked entirely from home before the Coronavirus pandemic) Yes No	No Change

Cognitive Interviewing Results

	Self	Proxy
Yes	2	7
No	1	1

The question related to telework was covered in all nine interviews. It was asked about three self-participants and eight proxy participants³ who were employed and currently working. In all cases, participants were able to answer the question as intended. One self-employed participant expressed some confusion about the question during the debriefing (though answered correctly and confidently during the interview). In her job, she often works from home, and occasionally goes out to meet clients. She has had to change those meetings to telephone and video conferences due to the fact that her clients’ workplaces have closed due to the coronavirus pandemic.

Only two of the eleven participants who were eligible for this question had an answer of ‘No’. Both situations were correctly represented by their answers. One was a grocery store manager who needs to perform his job in-person and one worked entirely from home prior to the coronavirus pandemic so correctly recognized that her telework situation was not affected by the pandemic. Those for whom the answer was ‘Yes’ work at offices that were closed due to the pandemic and had been teleworking during the last 4 weeks. Most of those who teleworked were working their usual schedule at home.

Round 2 Web Probing Assessment Results

	Frequency	Percent
Yes	370	79.7
No	93	20.0
Skipped	1	0.2
Total	464	100.0

The second round of the Web Probing Assessment included this question but for self-participants only. The question was asked only of those who worked for pay in the last seven days. Nearly 80 percent of participants responded ‘Yes’ to this question. Only one participant skipped the question entirely, suggesting limited sensitivity or other issues that might lead one to choose not to answer the question.

No specific follow-up probes were asked about this question. We did, however, look at the relationship between answers to this question and one open-ended question that asked participants to “List the ways your employment has changed since March 13, 2020.” Of the 270 participants who provided an answer to that question, 67 participants (25 percent) gave some indication that they were teleworking or working at home. Of these, almost all (99 percent) had reported ‘Yes’ to the telework question; this alignment is another indication that the telework question is working as intended.

The wording related to telework that these participants provided is listed below. The majority use wording similar to the wording in the question. The single participant who reported ‘No’ to the telework question and indicated in the open-end something related to telework wrote “work by phone instead of in-person.” There is no context available from the other questions to interpret this response, it may or may not indicate an incorrect answer to the telework question.

	Frequency
working from home	22
work from home	16
telework	12
working remotely	5
work at home	4
remote working	3
working at home	2
work by phone	1
working entirely from home	1
working exclusively from home	1
working remotely from home	1

Bottom Line

Overall, this question performed as intended. We have no change recommendations for this question. Participants in both the cognitive interviews and the Web Probing Assessment appeared to be able to provide adequate answers to this question. In the Web Probing Assessment, participants’ open-ended statements generally aligned with the wording in the question; using “telework”, “work from home”, and “working from home” most often.

Q2. Unable to work

Tested Wording	Recommended Wording
At any time in the LAST 4 WEEKS, (were you/was name) unable to work because (your/his/her) EMPLOYER CLOSED OR LOST BUSINESS due to the Coronavirus pandemic? (Enter Yes for the self-employed who lost work or customers because of the Coronavirus pandemic) Yes (Go to Q3) No (Go to Q4)	No Change

Cognitive Interviewing Results

	Self	Proxy
Yes	7	2
No	2	9

The question about being unable to work due to the coronavirus pandemic was asked about all self and proxy participants. The question worked as intended. All nine participants for whom ‘Yes’ was the correct answer were in situations where their employer (or their own business in four cases) shut down completely or their hours were cut back drastically due to the coronavirus pandemic. Those for whom the answer was ‘No’ were employed and were able to work their full schedule.

Round 2 Web Probing Assessment Results

	Frequency	Percent
Yes	158	19.5
No	644	79.5
Skipped	8	1.0
Total	810	100.0

This question was asked in Round 2 of the Web Probing Assessment of all participants. Eight participants (1 percent) skipped this question⁴. This high item response rate (99 percent) suggests that the question was not viewed as overly sensitive, intrusive, or irrelevant.

The 158 participants who responded ‘Yes’ to the target question also received a probe that asked “Why were you unable to work?” Participants were allowed to select all that apply from a group of responses. They could also enter an open-ended response by selecting ‘Other, specify’.

	Frequency	Percent
Employer was closed	71	35.7
Laid off temporarily or furloughed	49	24.6
Other, specify	39	19.6
Employer reduced some or all hours	38	19.1
Skipped	2	1.0
Total	199	100.0

Most of the participants (127 participants, 80 percent) selected at least one of the close-ended response options which are within-scope reasons for answering ‘Yes’ to the target question. Thirty-nine participants (20 percent) selected ‘Other’ and of those, 37 wrote an accompanying open-ended response. Twenty-nine participants (18 percent) entered only an ‘Other’ response (i.e., they did not select a close-ended option). These open-ended responses were coded and aggregated into the categories below.

	Frequency	Percent
In-Scope Responses	16
Business Closed or Reduced Hours	2	5.1
Lack of Business	12	30.8
Laid Off	2	5.1
Out-of-Scope Responses	10
Disability	5	12.8
Childcare	3	7.7
Illness	2	5.1
Not enough information to judge	11
Coronavirus	4	10.3
Other5	7	18.0
Skipped	2	5.1
Total	39	100.0

The most common category was ‘Lack of Business’ which consists of mostly self-employed individuals who reported being unable to find clients; suggesting a correct answer to the target question. There were also a number of entries that would be considered out-of-scope; responses having to do with being sick, taking care of children who are out of school, or having a disability. We recalculated the answers to this question to show how many participants gave at least one in-scope reason for being unable to work (one of the close-ended responses or a valid open-ended response) and those who gave only an out-of-scope reason. Nearly 88 percent of participants gave an in-scope reason. This suggests some misunderstanding of the question. It appears these individuals focused on the terms “unable to work” and “due to the coronavirus pandemic” and did not focus on an employer.

	Frequency	Percent
In-Scope Reason	139	88.0
Out-of-Scope Reason	19	12.0
Total	158	100.0

Bottom Line

This question worked as intended. While some participants in the Web Probing Assessment gave open-ended responses to the “Why were you unable to work?” question that might be considered out-of-scope (e.g., sick, disabled, or childcare), a majority either did so in addition to making an in-scope response, or provided a short, open-ended response without sufficient information to determine whether the responses to the target question were truly incorrect. Therefore there is no compelling evidence from the Web Probing Assessment to conclude the question was not working. The question worked well in the cognitive interviews.

Q3. Receive pay for lost work

Tested Wording	Recommended Wording
Did (you/name) receive any pay from (your/his/her) EMPLOYER for the hours (you/he/she) DID NOT work in last 4 weeks? (Enter No if self-employed and received no pay from customers) Yes No	No Change

Cognitive Interviewing Results

	Self	Proxy
Yes	3
No	4	2

This question was covered in seven of the nine interviews. It was asked about the nine self and proxy participants who reported they were unable to work because their employer closed or lost business due to the coronavirus in Q2. The question worked as intended. Three participants, responding for themselves, reported receiving pay from their employer for time not worked during the last four weeks. Two of the three reported receiving full pay despite not working and one reported receiving partial pay (pay equivalent to one day per week). All reported that this pay was not the result of taking leave. One of these participants reported still being employed and working greatly reduced hours, but still receiving full pay. Those for whom ‘No’ was the correct answer were not being paid and did not expect to receive back pay in the future.

One issue that arose for this question came during the survey portion of the interview when two participants asked for the start date of the reference period (e.g., one asked “4 weeks ago? What was the date?”). One of these participants had received payments early on during their layoff, but was not anymore. The other was self-employed and had to think about when business had fully tapered off. In both of these cases, the interviewer provided them the date and they determined they were out of scope for the question and responded ‘No’.

Bottom Line

The question, as tested in the cognitive interviews, worked as intended. Participants who responded ‘Yes’ were able to provide descriptions of the payment that fit within the intended scope of the question. Participants in the cognitive interviews were asked whether any of the questions they had been asked were sensitive or personal and none of them thought that any of the questions, including this one, were overly sensitive.

Q4. Prevented from looking for work

Tested Wording	Recommended Wording
Did the Coronavirus pandemic prevent (you/name) from looking for work in the LAST 4 WEEKS? Yes No	No Change

Cognitive Interviewing Results

	Self	Proxy
Yes	2	1
No	1	2

This question was asked in five of the nine cognitive interviews. It was asked about all self and proxy participants who were labeled as Not In The Labor Force following the labor force questions. That is, these are people for whom ‘No’ was reported for the question that asked if they had done anything to find work during the last four weeks. The question worked as intended.

A ‘Yes’ response was reported for two self and one proxy participant. In one case, the participant was self-employed and gets a lot of his business through customer referrals and stopping by open businesses and talking to managers about their computer needs. He has been unable to do this because most of the businesses he services (and similar businesses) are closed due to the coronavirus pandemic. The two other participants were siblings from the same household and had similar situations in that they were on indefinite layoff from their job and were interested in looking for work, but had not done anything beyond looking at ads because most of the businesses in the sectors they would prefer to work (entertainment and the arts) were closed due to the pandemic.

For those for whom a ‘No’ was reported, one was retired and the other two were expecting to be brought back to their former place of employment when conditions improve. When asked about other reasons for not looking for work, one participant reported needing to be home for childcare because that had been interrupted due to the pandemic. This reason was only brought up through additional probing and was not his primary concern.

Round 2 Web Probing Assessment Results

	Frequency	Percent
Yes	46	47.0
No	52	53.0
Skipped	0	0.0
Total	98	100.0

This question was asked of those who responded ‘No’ to both “In the last 7 days, did you do ANY work for either pay or profit?” and “Have you been doing anything to find work during the last 4 weeks?” In all, 98 participants received this question and 46 (47 percent) responded ‘Yes’ that the coronavirus pandemic had prevented them from looking for work. No participants skipped this question, suggesting that it is not a sensitive topic and does not present any other issue that may cause a high number of refusals in the production CPS survey.

Those who said that the coronavirus pandemic prevented them from looking for work were asked to explain why that was the case; “Please explain how the coronavirus pandemic prevented you from looking for work.” Their open-ended responses to this question were coded and aggregated into the following categories. Several participants listed more than one response. The most common response was that people are concerned about getting ill; some because they are in a high risk group, others because they have children or high-risk individuals in their home. All of these responses suggest some way, either directly or indirectly, that the coronavirus pandemic may have affected participants’ willingness or ability to search for work.

	Frequency	Percent
Do Not Want to Get Sick	17	34.7
Government Mandate	11	22.5
Lack of Job Postings	9	18.4
Waiting To Return To Job	7	14.3
Is Sick	2	4.1
Childcare	2	4.1
Eldercare	1	2.0
Total	49	100.0

Responses for ‘Waiting to Return to Job’ are loosely related to the scope of this question. In the cognitive interviews, several participants who said the coronavirus pandemic had not prevented them from looking for work cited the same reason; they expected to be going back to their job when conditions improved. The coronavirus is playing a role in why they are currently out of work and it is a factor in when they will be brought back.

Bottom Line

This question worked as intended. We do not recommend any changes to this question. In both the cognitive interviews and the Web Probing Assessment, participants who said that their job search had been affected by the coronavirus pandemic largely cited reasons that fit within the scope of the question. That is, the coronavirus pandemic is either directly or indirectly what prevented them from looking for work.

Q5 & Q5a. Need medical care for non-virus, did not receive

Tested Wording	Recommended Wording
At any time in the last 4 weeks, did you or anyone in your household need medical care for something other than Coronavirus, but not get it because of the Coronavirus pandemic? Please include all adults and children in the household. (Enter NO if all persons in the household voluntarily cancelled or delayed medical care for themselves or another household member.) Yes No	Next, I am going to ask about access to medical care over the last 4 weeks. Please include all adults and children in the household and DO NOT include dental or vision care. At any time in the last 4 weeks, did you or anyone in your household need medical care for something other than Coronavirus, but not get it because of the Coronavirus pandemic?

Cognitive Interviewing Results

	Self
Yes	3
No	6

This question was asked once for each household so was covered in all nine interviews. This question mostly worked as intended, with one exception. In this case, a participant answered ‘No’ to the question but later recalled during the debriefing that his wife was supposed to be undergoing pulmonary rehabilitation but several month’s worth of appointments (including during the four week reference period) had been cancelled. The participant said it had not come to mind at the time the question was asked.

In all other cases, participants accurately answered the question. In two of the correctly reported instances of ‘Yes’ to the question, the participants reported they or a household member had a procedure scheduled (a consult related to an injured back and an elective surgery) but those procedures were cancelled by the hospital due to the coronavirus pandemic. In another case, a participant responded ‘Yes’ when his elderly mother-in-law, who lives in the household, voluntarily decided not to seek treatment for an allergic reaction due to fear of contracting the coronavirus. She did not attempt to schedule an appointment. For the five instances where participants correctly responded ‘No’ to the question, the participants did not have medical needs during the four week period.

During the debriefing, participants were asked what the term ‘medical care’ includes as it applies to this question. Several participants included inapplicable types of care like dental and vision care. Some of these participants even recalled, at this point in the debriefing, that they had had a dental or vision appointment cancelled but they had not thought about it at the time of the question. So it is not clear if they would have actually reported it if they had thought of it at the time. Most participants defined medical care in this question as necessary medical care because of the word ‘needed’. They thought that the question was not asking about check-ups or physicals.

Round 1 Web Probing Assessment Results

	Frequency	Percent
Yes	219	38.0
No	357	61.8
Skipped	1	0.2
Total	577	100.0

All participants⁶ in the Web Probing Assessment were asked “At any time in the last 4 weeks, did you need medical care for something other than coronavirus, but DID NOT GET IT because of the coronavirus pandemic?” There were 219 participants (38 percent of the sample) who reported they were unable to get medical care due to the coronavirus. There was a low rate of non-response to the item suggesting low sensitivity.

Another question asked participants “At any time in the last 4 weeks, did you DELAY getting medical care because of the coronavirus pandemic?” The skip patterns in the survey were set so that if a participant answered ‘Yes’ to either question they would receive a follow-up probe about what type of care they were unable to get. This is problematic for interpretation because we do not know which procedures were voluntarily delayed and which were cancelled by the medical provider. The probe answers below reflect only those participants who responded ‘Yes’ to the target question⁷.

After answering the target question, the 219 participants who responded ‘Yes’ were asked a follow-up probe “What type of care were you unable to get because of the coronavirus pandemic?” Some were randomly assigned to select all that apply from a close-ended list of response options, while others were assigned to provide their response in an open-ended format. Participants asked to select all that apply were also given the option to select other and provide an open-ended response.

The ‘Closed-Ended’ columns cover the 101 participants assigned to the select all that apply format and the 215 selections they made. The ‘Other, specify’ columns cover the 16 statements provided when participants selected ‘Other’ in the select all that apply question. The ‘Open-Ended’ columns cover the 118 participants who received the open-ended version of the question. Note that several participants in the open-ended format entered multiple responses. When given the open-ended format, many participants wrote in medical specialties and not what the visit was about. These are recorded as ‘Specialized’ in the table below.

		Close-Ended			Other, specify			Open-Ended
		Freq	Perc		Freq	Perc		Freq	Perc
Treatment for ongoing condition		48	22.3
Dental care		47	21.8					41	21.5
Diagnostic or medical screening test		41	19.1		1	6.3		20	10.5
A regular check-up		32	14.9		1	6.3		25	13.1
Prescription drugs or medications		16	7.4					4	2.1
Other		16	7.4					98	4.7
A surgical procedure		8	3.7					8	4.2
Urgent care for an accident or illness		7	3.2
Immunization		--	--		1	6.3		4	2.1
Vision		--	--		1	6.3		11	5.6
Specialized (e.g., pediatrician apt.)		--	--		12	75.0		69	36.1
Total		215	100.0		16	100.0		191	100.0

Across all versions of the follow-up question, 88 participants (40 percent) reported dental care, 12 reported vision care (5 percent), and 58 reported a missed check-up (25 percent); all responses that might be considered out-of-scope for the target question. As mentioned earlier, it is not possible to tell whether these responses pertain to the question about voluntary delay or the one related to inability to get care.

Round 2 Web Probing Assessment Results

	Frequency	Percent
Yes	288	40.0
No	427	59.3
Skipped	5	0.7
Total	720	100.0

The same question was asked of all participants in the second round of the Web Probing Assessment⁹. Once again, 40 percent of the participants reported being unable to receive medical care due to the coronavirus. There was also a small number of non-respondents to this question indicating that it was not considered sensitive.

We did not discover the issue with the skip patterns leading to those who respond ‘Yes’ to both the voluntary delay and unable to get care questions receiving this probe until after the Round 2 data were collected, so the same problem exists here. Once again, a high percentage, 256 of the 288 responses (89 percent), responded ‘Yes’ to both questions. The table below covers only those who responded ‘Yes’ to the target question and not the one related to voluntary cancellations. In this version of the Web Probing Assessment, there was no open-ended version of the question. Participants only received the select all that apply probe asking them to select what types of procedures were delayed or cancelled. They were also given the option of entering an open-ended response if they selected ‘Other’. Both the close-ended and the coded and aggregated open-ended statements are tallied below.

		Close-Ended			Other, specify
		Freq	Perc		Freq	Perc
Dental care		165	23.4		1	2.4
Diagnostic or medical screening test		132	18.7		8	19.0
Treatment for ongoing condition		129	18.3		3	7.1
A regular check-up		123	17.4		1	2.4
Prescription drugs or medications		50	7.1		1	2.4
A surgical procedure		43	6.1
Other		40	5.7		110	2.4
Urgent care for an accident or illness		23	3.3		1	2.4
Immunization		--	--
Vision		--	--		12	28.6
Specialized (e.g., pediatrician appt)		--	--		14	33.3
Total		705	100.0		42	100.0

As with Round 1, large percentages of participants listed out-of-scope medical care activities when asked what care they did not receive; 165 participants (58 percent) listed dental care, 12 participants (4 percent) listed vision care, and 123 participants (43 percent) listed a regular check-up. This demonstrates that this question is likely overestimating the number of people who were unable to receive necessary medical care, the intended focus of the question.

Bottom Line

This question led to the greatest number of issues of those that we tested. Several of the cognitive interview participants and many of the Web Probe Assessment participants include dental and vision care as part of their interpretation of medical care.

The question is already quite complex and we are hesitant to suggest wording changes that will only add complexity. NCHS stakeholders should review the results and prioritize their measurement objectives given the limited space that can be dedicated to these questions. We recommend adding instructions to clarify that dental and vision care should not be considered if that distinction is of high priority.

Additionally, the measurement objectives of the question are not entirely clear. The following instructions were approved by NCHS for inclusion in the self-study packet for CPS interviewers (underlining added for clarity).

Enter No if all persons in the household voluntarily cancelled or delayed medical care for themselves or another household member. Cancellations or delays are voluntary only if the person decided that the care was not needed or could be easily postponed.

Cancellations or delays of medical care should not be considered voluntary if the person believed the care was needed, but they chose not to get it due to fear, loss of health insurance, lack of transportation, unavailability of providers, or problems getting an appointment. If any of these reasons were due to the coronavirus pandemic, enter Yes.

If a respondent decides that a procedure is not necessary after weighing it against the risk of contracting the coronavirus, as was the case with the proxy participant who had an allergic reaction, it is unclear whether a ‘Yes’ response is valid for this question. For the purposes of testing, we counted it as a correct response because the participant referenced her fear of contracting coronavirus (during the debriefing), but it really is unclear given the above instructions. A clearer distinction could be made by separating out all voluntary choices made by the respondent, regardless of the reason for that choice, from involuntary reasons for not receiving treatment (e.g., loss of insurance, unavailability of providers, etc.). The reference to “chose not to get it due to fear” should be reevaluated and possibly removed from future interviewer training pending that reevaluation.

Appendix A: Web Probe Assessment Questions

Web Probing Assessment – First Round

Blue indicates question is a follow-up probe added to evaluate the prior question

Skip Instructions	Question	Response Options
All	Q11. Now we are going to ask about your employment. In the last 7 days, did you do ANY work for either pay or profit? Select only one answer.	Yes No
If Q11 = 1	Q14. Are you receiving pay for the time you are not working? Select only one answer.	1. Yes 2. No
All	Q37. At any time in the last 4 weeks, did you DELAY getting medical care because of the coronavirus pandemic? Select only one answer.	1. Yes 2. No
All	Q38. At any time in the last 4 weeks, did you need medical care for something other than coronavirus, but DID NOT GET IT because of the coronavirus pandemic? Select only one answer.	1. Yes 2. No
If Q37 = 1 OR Q38 = 1 (randomly assigned)	Q38PA. What type of care were you unable to get because of the coronavirus pandemic? Select all that apply.	1. Urgent care for accident or illness 2. A surgical procedure 3. Diagnostic or medical screening test 4. Treatment for ongoing condition 5. A regular check-up 6. Prescription drugs or medications 7. Dental care 8. Other
If Q37 = 1 OR Q38 = 1 (randomly assigned)	Q38PB. What type of care were you unable to get because of the coronavirus pandemic?	Open-Ended

Web Probing Assessment – Second Round

Blue indicates question is a follow-up probe added to evaluate the prior question

Skip Instructions	Question	Response Options
If Q11= no	C1. In the last 4 weeks, did you do ANY work for either pay or profit? Select only one answer.	1. Yes 2. No
If C1 = No	C2. Have you been doing anything to find work during the last 4 weeks?	1. Yes 2. No
if C2 = No	C3. Did the coronavirus pandemic prevent you from looking for work in the last 4 weeks?	1.       Yes 2.       No
If C3 = Yes	C3P. Please explain how the coronavirus pandemic prevented you from looking for work.	Open-ended
If C1 = Yes	C4. At any time in the LAST 4 WEEKS, did you telework or work at home for pay because of the coronavirus pandemic?	1. Yes 2. No
	C5. At any time in the LAST 4 WEEKS, were you unable to work because your employer closed or lost business due to the coronavirus pandemic?	1. Yes 2. No
If C5 = Yes	C5P1. Why were you unable to work?	Employer was closed Employer reduced some or all hours Laid off temporarily or furloughed Other (Specify)
If C5 = Yes	C5P2. If your employer asked you to work a usual schedule, would you have been willing or able to?	1. Yes 2. No
If C5 = No	C5P3. During the LAST 4 WEEKS, were you able to work your usual schedule despite the pandemic?	1. Yes 2. No

1 One question related to pay for time not worked was sufficiently different from the proposed CPS question (Q3), that we decided not to use responses for that question in this report.

2 Skip patterns and survey drop-offs led to varying sample sizes for each question. In an effort to maximize the use of the available data, we used incomplete cases if they answered the question of interest. Item nonresponse rates are shown with the individual question results.

3 Self-participants refers to when a participant provides information about themselves, proxy participants refers to when the study participant provides information about others household members.

4 Fifty-six participants dropped out before receiving this question.

5 Responses included “my wife ceased her main operation,” “I am the business owner. I have sacrificed my paycheck,” “Unable to procure supplies,” “can’t travel due to coronavirus,” “Not hiring,” “Funding froze,” and “Unable to access customer site due to Covid restrictions.”

6 One participant broke off from the survey before reaching this question.

7 There was substantial overlap between these questions. Of the 219 who reported being unable to receive care, 187 (85%) also reported delaying care. It appears many participants in the Web Probe Assessment either had multiple medical situations with different outcomes or had overlapping interpretations of what the two questions were meant to capture.

8 Included items like “minor leg injury,” “Routine elective appointments,” and “non-urgent medical issues”

9 146 participants broke off from the survey before reaching this question

10 Answer was “personal”

Page 15 of 15