TO: Rachel Krantz-Kent
Office of Employment and Unemployment Statistics
FROM: Robin Kaplan and Douglas Williams
Office of Survey Methods Research
DATE:
SUBJECT: Cognitive Test Results for COVID-19 and Telework Related Questions in the CPS
Introduction
In May 2020, the Current Population Survey (CPS) survey fielded four1 new questions with the intent of facilitating the understanding of changes to the labor market during the COVID-19 pandemic. Data has been collected using these questions for nearly two years while the pandemic has continued. Now many states and businesses have begun to end mandates or workforce restrictions to maintain the health and safety of the public. As a result, CPS is seeking to revise these questions to measure the extent that workforce changes (such as telework) have continued, and the scope of that change by comparing telework frequency to before the COVID-19 pandemic started.
To address these changes, CPS has made questionnaire revisions, consisting of four questions to replace the existing questions. The original questions collected information about employment changes, such as telework and work shutdowns, and information about how the pandemic affected people’s job searches. The revised questions will replace these and focus on current telework frequency and whether the individual teleworked before the COVID-19 pandemic. Before testing occurred, the Office of Survey Methods Research (OSMR) conducted an expert review of these questions and provided wording recommendations. This memo includes results from one-on-one cognitive interviews and online testing conducted during June of 2022.
Methods
One-on-one cognitive interviews
Cognitive interviews provide an in-depth understanding of the respondents’ thought processes and reactions to questions to uncover possible sources of response error, bias, and to ensure respondents understand the questions as intended. For this research, one-on-one cognitive interviews were conducted remotely via telephone or Microsoft Teams by staff from OSMR who are experienced in conducting these types of interviews. Participants were asked an abbreviated version of the CPS, including a shortened household roster and labor force questionnaire, and the applicable revised COVID-19 questions. After completion, participants were asked debriefing questions to better understand the thought processes used to answer the COVID-19 questions, in particular comprehension and recall as several questions asked about the period when the pandemic first started, over two years ago. The protocol for the cognitive interviews is included in Appendix A.
Recruitment for the cognitive interviews
Participants for cognitive testing were found through placing a Craigslist ad in the city of Raleigh, NC. This city was selected because it is a major city with diverse business establishments and professional jobs, where it is expected some of the workforce would telework or work from home. Specifically, we recruited ten participants who fit the following criteria:
From a household with 1 or more people over age 18, and;
Employed– Currently have a job, either full- or part-time, and currently telework or work from home.
In each interview, the researcher started by introducing the purpose of the study and familiarizing participants with the CPS. Then an abbreviated version of the CPS was administered, including a shortened household roster, labor force questionnaire, and the applicable COVID-19 questions. The instrument used for testing allowed participants to report this information for themselves and up to 2 other household members.
After going through the abbreviated survey, participants were asked a series of debriefing questions about the COVID-19 questions they had received during the interview. Participants were then thanked for their time and the call was ended. The whole process - introduction, abbreviated survey, and debriefing - took approximately 30 minutes per participant.
Online Testing
Online testing was also conducted to complement the one-on-one cognitive interviews and to collect feedback from a larger, more geographically diverse group of participants. The questionnaire and debriefing questions were similar to those used in the one-on-one cognitive interviews, but written in a self-administered format with mostly closed-ended probes. Participants were recruited from Amazon Mechanical Turk (mTurk), an online marketplace where individuals can sign up to participate in short online research tasks for nominal compensation. Like the cognitive interviews, participants were screened (N=400) to identify those that currently have a job, either full- or part-time, and currently telework or work from home. Participants were instructed to exclude any work they do through mTurk or other online survey platforms in both the screener and full questionnaire. Only those meeting the screening requirements (n=207) were invited to complete the CPS questionnaire content, and 200 participated. However, after assessing responses to the screener versus the full questionnaire, we found that 35 participants indicated they had teleworked last week in the screener, but then went on to report they had NOT teleworked in the full questionnaire2. These 35 participants who reported inconsistent responses were removed from the sample for lack of eligibility for the study and data quality purposes, leaving a total of 165 participants. The screener took an average of 55 seconds to complete, and the full questionnaire took an average of about 10 minutes to complete. All participants resided in the United States, and ranged in age from 21 to 71 years old, with an average age of 40. A total of 61% of the sample was male and 72% reported having a Bachelor’s degree or higher. The protocol for the online study is included in Appendix B.
Results
The summary table below specifies the number of participants (both self and proxy3) who were asked about each of the COVID-19 questions during both the cognitive interviews and online study, as well as the final recommendation for each question.
|
Question Topic |
Cognitive Interview |
Online Study |
Recommendation |
Q1 |
Telework from home for pay |
10 (self) 4 (proxy) |
165 |
No change |
Q2 |
Number of hours teleworked or worked from home |
94 (self) 3 (proxy) |
165 |
No change |
Q3 |
Telework in Feb. 2020 |
10 (self) 4 (proxy) |
165 |
No change |
Q4 |
Did more, less, or same amount of telework as in Feb. 2020 |
3 (self) 1 (proxy) |
85 |
No change |
General Feedback
Participants in the cognitive interview study felt that the COVID questions were straightforward and easy to answer. It was easy for most participants to recall their jobs around the time the pandemic started and afterward. A couple participants had trouble remembering back to the reference period (pre-COVID and during February 2020), but all participants were able to arrive at an answer. Three of the four participants answering as a proxy for other household members displayed some difficulty answering. This was due to difficulty recalling when other household members started teleworking or not having the knowledge to answer the question (either because they were not in the same household during February 2020, or they did not know a lot about others household members’ jobs in general). One participant felt it was invasive to answer on behalf of other household members. However, all participants were able to understand the questions and arrive at their best estimates.
Results from the online study largely echoed the findings from the cognitive interviews. The data indicated most participants understood the questions as intended and provided consistent responses across all items. Participants did not have any major difficulties recalling the information needed to answer the questions based on their responses to closed-ended probes and open-ended feedback. Taken together, the COVID questions performed well as written and no critical issues were found. Thus, no changes to the existing questions are recommended. A summary of the findings for each of the questions is outlined below.
Q1. Telework from home
Tested Wording |
Recommended Wording |
Cognitive
interview wording: At any time
LAST WEEK, did you telework or work at home for pay?
Online study wording: At any time LAST WEEK, did you telework or work at home for pay?
|
No Change |
Cognitive Interviewing Results
The question about whether participants teleworked last week was covered in all ten interviews. It was asked about ten self-participants and four proxy participants who were employed and currently working, as seen in the table below.
|
Self |
Proxy |
Yes |
9 |
3 |
No |
1 |
1 |
All but one participant indicated they had teleworked last week, except one teacher who had recently resumed in-person teaching. In all cases, participants were able to answer the question and understood the phrase “telework or work at home for pay” as intended.
Most participants were teleworking their usual schedule at home. Several participants noted that while they primarily telework, they occasionally work on-site depending on different circumstances. For example, one participant noted they very rarely are required to go into the office when someone they act as a backup for is not available. Another participant worked as a freelance software developer and noted depending on the current gig, he might be able to work from home or may have to come into an office. Another proxy participant said her spouse is a manager and is required to come into the office once or twice per week, though it has been less often recently due to rising gas prices.
Most participants could easily recall when they started teleworking as it coincided with the start of pandemic, usually in March 2020, or whenever they were hired for the job, both of which were landmark dates in their mind. A few participants mentioned some of the hours they teleworked may have been unpaid – for instance, a manager who caught up on emails in the evenings after work hours (but was in a salaried position and not compensated for this time), a freelance software developer who worked a few hours learning new skills for a gig but was not paid for this time, and a participant who worked some unpaid internship hours remotely. Taken together, participants in the cognitive interview study understood this question well, and no major issues were found.
Online Study
The online study asked the question about teleworking last week for self-participants only. Participants reported working an average of 40 hours per week across all jobs. As shown in the table below, there were two follow-up probes asking whether participants generally telework each week and whether any of this time was unpaid work. Like the cognitive interview participants, most (almost 92%) reported they generally telework every week, and almost 92% also reported that this time was for paid work.
|
Yes |
No |
||
Percent |
Frequency |
Percent |
Frequency |
|
Do you generally telework or work from home every week? (N=165) |
91.5% |
151 |
8.5% |
14 |
Was any of this time unpaid work? (N=165) |
8.5% |
14 |
91.5% |
151 |
Conversely, around 9% of participants reported that they do not generally telework every week, and around 9% also reported that this time included unpaid work. Interestingly, 13 of the 14 participants who reported that they do NOT generally telework each week also reported that some of their telework time was unpaid. This suggests that those who do not telework each week may also do more unpaid work (e.g., they may catch up on work at home after regular business hours), consistent with some responses in the cognitive interviews.
Another follow-up probe asked where participants teleworked from the most last week, as shown in the table below. Consistent with the cognitive interviews, most participants reported working from home. Note that 6% reported teleworking from a location other than their home, suggesting that the term “telework” was broader than just working from home for some participants. No participants indicated any other locations in the ‘Other, specify’ field.
LAST WEEK, where did you telework from the most? (N=165) |
Percent |
Frequency |
From your home |
91.5% |
155 |
Another business or office location |
3.0% |
5 |
Various worksites |
3.0% |
5 |
Recommendation
Overall, this question performed as intended. We have no recommended changes for this question. Participants in both the cognitive interviews and the online study were able to provide adequate answers to this question and seemed to understand the phrase “telework or work at home for pay” as intended.
Q2. Number of hours teleworked or worked from home
Tested Wording |
Recommended Wording |
Cognitive interview wording: How many hours did you telework or work at home for pay LAST WEEK?
(Interviewer instruction: Multiple jobholders should include telework hours on all jobs combined.)
Online study wording: LAST WEEK, you worked [fill]5 hours total. How many of these hours did you telework or work at home for pay? If you have more than one job include telework hours on all jobs combined. |
No Change
|
Cognitive Interviewing Results
The question about the number of hours participants teleworked last week was asked of nine self-participants and three proxy participants who reported teleworking in the last week. This question seemed relatively easy for most participants to answer, as they reported telework hours as part of their regular work schedule.
For some participants who did not have a set work schedule (e.g., they work gigs, freelance, or have varying hours depending on business need), the question was slightly more difficult and involved trying to add the number of hours up in real-time or providing estimates. For example, one participant worked for a company whose hours differed based on business needs. She indicated “depending on the workload, it can vary. It’s hard to say. Sometimes Mondays are busier, and Fridays are less busy. It can range from 25 to 45 hours.” This same participant noted she also has a side business selling beauty products online that she initially forgot to include but remembered to once the interviewer instruction to include telework hours on all jobs combined was read. She stated her best estimate is that she works about 20 hours per week on her side business, but struggled to arrive at an answer, noting “I sometimes do work at midnight, so it’s a very rough, ballpark estimate.”
Another participant answering as a proxy for her husband noted she had to “guestimate” because he often logs back in the evenings to catch up on work. She thought he did so about two hours per weeknight on top of his 40-hour per week job, arriving at an answer of 50 hours. She mentioned these hours are technically unpaid since he does not earn overtime pay or time off for them. Two other proxy-participants were not certain about the number of hours their housemates worked, assuming it was likely 40 hours since they had salaried full-time positions.
Overall, this question worked well in the cognitive interviews. We note there might be some issues with workers who have irregular hours, change gig jobs or freelance jobs frequently, or hold more than one job where they telework. As expected, some of proxy responses were rough estimates. Interviewer instructions to remind multiple job holders to report telework hours from all jobs combined is also critical as some participants noted having multiple gigs, varying freelance hours, or “side hustles” in which they telework. In addition, some participants included telework that occurred after working hours or was unpaid. However, because the CPS includes this type of work (e.g., teachers grading papers at home or self-employed respondents working on training or professional development), and the total telework hours are anchored to the total number of hours respondents reported working earlier in the interview, this is not expected to be an issue. When collecting total telework hours, we recommend that interviewers apply the same concept used for collection of total work hours.
Online Study
In the online study, the question on number of hours teleworked last week was asked of all participants. Participants reported teleworking an average of 27.3 hours during the previous week. All participants reported teleworking an amount that was equal to or less than the total hours worked during the past week, suggesting that they interpreted each question correctly. A total of 43.6% of participants reported teleworking all of the hours they worked the previous week. Participants reported teleworking an average of 70% of their working hours during the previous week.
When asked whether the number of hours they telework is consistent each week or if it varies, the majority (almost 60%) reported working a consistent number of hours each week.
Is that number of hours consistent each week, or does it vary? (N=165) |
Percent |
Frequency |
Consistent each week |
58.8% |
97 |
Varies week to week |
41.2% |
68 |
Participants who reported teleworking a consistent number of hours each week teleworked 78.1% of their total hours during the previous week, whereas those reporting that their telework hours vary week to week reported teleworking 56.9% of their total hours. This suggests that those who telework a consistent schedule may also telework more frequently.
A total of 34% of participants reported having more than one job. Those with multiple jobs reported working more on average (42.4 hours) than those with one job (38.7 hours). Similarly, as seen in the table below, participants who had only one job were more likely to indicate that their telework hours are consistent each week compared to those working multiple jobs, where participants were evenly split on working consistent versus varying hours. This is similar to the cognitive interviewing findings, where participants seemed to report more variable hours when they held multiple jobs and had slightly more difficulty arriving at answer for the total number of hours teleworked.
|
One job |
Multiple jobs |
Telework hours consistent each week |
62.5% |
50.9% |
Telework hours vary week to week
|
37.5% |
49.1% |
Recommendation
This question worked well, and no changes are recommended. Participants in both the cognitive interviews and the online study were able to provide adequate answers to this question and seemed to understand it as intended. Some minor issues occurred with workers who have variable schedules, hold more than one job where they telework, and with proxy responses. Again, we recommend that interviewer training emphasize the need to remind respondents that multiple jobholders should include telework hours on all jobs combined to help increase data quality.
Q3. Telework in Feb. 2020
Tested Wording |
Recommended Wording |
Cognitive interview wording: Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started?
(Interviewer instruction: Enter “No” if the person was not employed in February 2020.)
Online study wording: Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started?
Remember to exclude any work through mTurk, or for other survey platforms.
|
No Change |
Cognitive Interviewing Results
In the cognitive interviews, the question on whether participants teleworked in February 2020 before the pandemic started was covered in all participants. The responses for self and proxy can be seen in the table below.
|
Self |
Proxy |
Yes |
3 |
1 |
No |
7 |
4 |
The question was generally easy for participants to understand and to remember back to when the pandemic started. Some participants started working at home in March of 2020 and were not able to telework before that time because their work required being on-site (e.g., they catered events, taught in a classroom, or worked at an airport), so that time was very distinct in their minds. Participants who had been teleworking long before the pandemic began also found this question easy to answer because they reported when they started working for their employer, which was usually a memorable date.
The question was somewhat more difficult for a couple participants (who were doing irregular gig or freelance work during the pandemic) to recall whether they were teleworking or not in February 2020 because of frequent changes in jobs, employers, or gigs. One participant had to estimate around when she and her husband began teleworking because it was not at the beginning of the pandemic. Instead, she estimated that they both began teleworking in about January 2021, when she recalled the COVID-19 variants began showing up. The question was also more challenging for some participants answering as a proxy because they did not have the knowledge of when their household member started teleworking, or their household member lived elsewhere in February 2020.
Online Study
About half of the participants in the online study reported that they had teleworked in February 2020 before the pandemic started, as seen in the table below. About 2% of participants reported that they were not employed at that time. Only 1.2% indicated they were not sure, suggesting that recall (for self-response) is not an issue for this question.
Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started? (N=165) |
Percent |
Frequency |
Yes |
51.5% |
85 |
No |
45.5% |
75 |
You were not employed in February, 2020 |
1.8% |
3 |
Not sure |
1.2% |
2 |
Participants who reported that they teleworked before the pandemic were asked a follow-up question about whether this was for the same employer they currently have. As seen in the table below, most participants reported that they currently have the same job or employer. Over 10% of participants reported having changed jobs since February 2020.
Was this for the same job or employer that you have now? (n=85) |
Percent |
Frequency |
Yes |
89.4% |
76 |
No |
10.6% |
9 |
Participants who reported they teleworked in February 2020 were asked a probe about how easy or difficult it was to recall whether they teleworked before the pandemic. As seen in the table below, most participants found this information very easy or somewhat easy to recall, consistent with the cognitive interview study. Of the nine participants who thought the question was somewhat difficult, only one had changed jobs since February 2020.
How easy or difficult was it for you to recall whether or not you teleworked in February 2020? (n=85) |
Percent |
Frequency |
Very easy |
67.1% |
57 |
Somewhat easy |
22.4% |
19 |
Somewhat difficult |
10.6% |
9 |
Very difficult |
0.0% |
0 |
Participants who reported they did NOT telework (or were unsure if they teleworked) in February 2020 were asked whether telework was available to them in February 2020, before the COVID-19 pandemic started. As seen in the table below, most participants reported that telework was also not available to them before the pandemic.
Was telework available to you in February 2020 (before the pandemic started?) (n=79) |
Percent |
Frequency |
Yes |
12.7% |
10 |
No |
87.3% |
69 |
Recommendation
This question worked well based on the cognitive testing and online study, and most participants had no problems. The cognitive interview study found that there may be some estimation involved with those who had irregular schedules during the pandemic or proxy participants who may not have the knowledge to answer on behalf of other household members. However, these memory issues were minor and participants were able to arrive at answers or estimates. The online study showed a similar pattern of results, and most participants found it easy to recall this information. No question changes are recommended.
Q4. Did more, less, or the same amount of telework as in Feb. 2020
Tested Wording |
Recommended Wording |
Cognitive interview wording: LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)?
1. More 2. Less 3. About the same 4. Don't Know 5. Refused
Online study wording: LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)?
1. More 2. Less 3. About the same 4. Not sure |
No Change |
Cognitive Interviewing Results
In the cognitive interviews, the question about whether participants worked more, less, or the same amount of telework as in February 2020, was asked of the four participants who reported that they had teleworked before the pandemic. This included three self-responses and one proxy-response, as seen in the table below.
|
Self |
Proxy |
Yes |
2 |
1 |
No |
1 |
0 |
This question was generally easy for all participants to understand. For two participants, it was very straightforward as their employer expanded telework for all employees after the pandemic. Two participants also reported this was easy to recall as their employer increased the number of regular hours that could be done via telework since February 2020. One participant switched employers since February and knew she used to work fewer telework hours then compared to now. These participants felt it was an easy comparison to make and did not report any difficulties arriving at answer. However, as noted above, this question was only administered four times.
Online Study
Participants who indicated that they had teleworked in February 2020 (n=85) were asked the question about whether they teleworked more, less, or the same amount as in February 2020. As seen in the table below, most participants indicated that they teleworked about the same amount or more. Only one participant indicated they were not sure (and later explained in an open-ended probe that this was because their work hours constantly change), suggesting that recall was not a major issue with this question.
LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)? (n=85) |
Percent |
Frequency |
More |
27.1% |
23 |
Less |
15.3% |
13 |
About the same |
56.5% |
48 |
Not sure |
1.2% |
1 |
Participants who teleworked in February 2020 were asked an open-ended probe to explain how they came up with their response to this question, and 83 respondents provided an explanation. The responses tended to fall into the following five categories:
|
Percent |
Frequency |
|
42.2% |
35 |
|
20.5% |
17 |
|
3.6% |
3 |
|
20.5% |
17 |
|
9.6% |
8 |
|
3.6% |
3 |
Representative examples of these responses are outlined below for each of the five categories:
|
Examples |
|
|
|
|
|
|
|
|
|
|
Recommendation
This question worked as intended. We do not recommend any changes to this question. Participants in both the cognitive interviews and the online study were able to answer the question with minimal recall issues and provided substantive answers for how they knew the amount of telework they did now versus before the pandemic started.
Conclusion
In summary, the cognitive test and online study results showed that participants understood the questions well and were able to provide answers to them. Participants were mostly able to recall the information needed to arrive at their answers (e.g., their work schedule and hours just prior to the pandemic), as this was a memorable and especially vivid time for them. However, the cognitive interviews, which allow for richer qualitative data collection, indicated that some participants had minor recall issues. These were mainly for participants who worked multiple jobs or changed jobs frequently (e.g., freelance or gig work), but these participants were still able to think about their responses and arrive at answers they felt were accurate. We recommend that interviewer training emphasize the need to remind multiple jobholders to include telework hours on all jobs combined to help increase data quality. When collecting total telework hours, we recommend that interviewers apply the same concept used for collection of total work hours.
As expected in any survey with proxy response, minor issues also occurred where some respondents in the cognitive interview study had issues with recall or knowledge when answering on behalf of other household members, so these responses were estimates. This is a potential data quality issue to be aware of given the questions ask about a period dating back to February 2020. Taken together, the cognitive interview and online studies showed similar results and found no critical issues, and we recommend the questions proceed as written.
Appendix A: Cognitive Interview Protocol
Introduction
Hello, my name is ______________ and I work at the Bureau of Labor Statistics. Thank you for taking the time to talk with me today. (IF NEEDED) I’m joined by my colleague __________ today.
We are going to be working with some questions that might be asked as part of the Current Population Survey, which is one of the longest running survey programs measuring the economic and social well-being of the United States. We will first ask about the people living in your household, then some questions about how you worked at different time periods.
Before new survey questions are asked, it’s important to test them out with people like you, to make sure that they make sense and are easy to follow. Your answers will help us improve the questions. There are no right or wrong answers; we’re looking for your reactions and honest feedback. Your name will not be included in the report, and nothing that you say will be attributed directly to you.
We are not here to evaluate you, we are looking to improve the questions.
Informed Consent:
Upon beginning a remote testing session, participants will read this information or be shown the following information and asked for their verbal consent prior to beginning the study:
We estimate the session will last 30 minutes. If you have any comments regarding this estimate or any other aspect of this study, send them to [email protected].
Your participation in this research project is voluntary, and you have the right to stop at any time.
The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on July 31, 2024. Without this currently approved number, we could not conduct this research. Your responses are also protected by law:
The Bureau of Labor Statistics, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.
In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection. Information on routine uses can be found in the system of records notice, DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418).
Do you have any questions before we proceed?
Do you agree to participate?
Questionnaire completion:
If reading the survey/administering over the phone:
I will read the survey aloud to you, please tell me your answers.
If a question isn’t clear or you don’t understand some wording that is used, please let me know. We want these questions to be clear and easy to understand.
If there are any questions you would rather not answer, let me know.
VAR |
Question |
Response Options |
Universe |
Roster1 |
Please tell me the names and ages of everyone who lives in your household |
Open ended |
Everyone |
Labor Force Questions, ask as a block for up to 3 HH employed members (age 16+), identified during roster |
|||
LABOR |
Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___ |
|
Everyone |
WK |
LAST WEEK, did (you/NAME) do ANY work for pay? |
|
Everyone |
IO1INT |
Now I have a few questions about the job at which (you/NAME) worked LAST WEEK.)
(Was.../Were you) employed by government, by a private company, a non-profit organization, or (was.../were you) self-employed (or working in a family business)? |
(D) Don’t Know (R) Refused |
All employed |
IO1IND |
What kind of business or industry is this? READ IF NECESSARY: What do they make or do where you work? |
Open-ended |
All employed |
IO1OCC |
What kind of work (do/did) (you/name) do, that is, what (is/was) (your/name’s) occupation? (For example: plumber, typist, farmer) |
Open-ended |
All employed |
IO1DT |
What (are/were) (your/name’s) usual activities or duties at this job? (For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick) |
Open-ended |
All employed |
Telework questions (COVID-19), ask as a block for up to 3 HH employed members, identified during roster |
|||
NEW_INTRO |
I now have some questions related to how the COVID-19 pandemic affected how people work. |
|
All employed |
NEW1 |
At any time LAST WEEK did (you/NAME) telework or work at home for pay? |
|
All employed |
NEW2 |
LAST WEEK, (you/Name) worked [FILL: value for variable HRACTT (which is the actual hours worked last week at all jobs asked in main CPS)] hours [FILL FOR MULTIPLE JOBHOLDERS: total, at all jobs]. How many of these hours did [you/NAME] telework or work at home for pay? (Interviewer instruction: Multiple jobholders should include telework hours on all jobs combined.) |
|__|__| hours (D) Don’t Know (R) Refused |
Teleworked |
NEW3 |
Did (you/NAME) telework or work at home for pay in February 2020 before the COVID-19 pandemic started? (Interviewer instruction: Enter “No” if the person was not employed in February 2020.) |
|
Teleworked |
IF NEW1 = 1 (YES) and NEW3 = 1 (YES) GO TO NEW4; ELSE GO TO END |
|||
NEW4 |
LAST WEEK, did (you/NAME) do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)? |
|
Teleworked in Feb 2020 |
Next, I would like to talk about the questions you answered and your general reactions answering these.
Start by collecting additional detail about jobs from the participant’s own perspective:
What was it like for you to participate in this survey?
Were there any questions you found difficult or confusing?
Were there any questions you found sensitive or personal?
Now I’d like to talk about some of the specific questions you answered. The first one asked about telework.
If on video call, share screen with questionnaire showing so that participant can see the questions.
If on telephone call, read questions aloud.
Show NEW_INTRO and NEW1
What does ‘telework or work at home for pay’ mean to you?
IF YES TO TELEWORK
You said that (you/NAME) teleworked last week. Can you tell me more about that?
IF NEEDED: Where did (you/NAME) telework from?
How many days did (you/NAME) telework last week?
(Do you/Does name) generally telework or work from home every week?
When did (you/NAME) start teleworking or working from home?
Was any of this time unpaid work or after working hours?
IF NO TO TELEWORK
Is telework available to (you/NAME)?
(Have you/Has Name) ever teleworked or worked from home for pay?
IF YES: Why (do you/does name) no longer telework or work from home for pay?
Show NEW2
How did you come up with (NUMBER) hours for teleworking or working from home?
Is that number of hours consistent each week, or does it vary?
IF VARIES: How does it vary?
Were any of these hours unpaid or after working hours?
Do you have more than one job or employer where some of your hours are spent teleworking?
IF YES: Did the hours you reported include telework from all jobs?
Show NEW3
IF YES TELEWORK IN FEB 2020
Was this question easy or difficult for you to answer? (Why?)
You said that (you/NAME) teleworked or worked from home for pay in Feb 2020. Can you tell me more about that?
Can you tell me what timeframe you were thinking about when answering this question?
Was this for the same job or employer that you have now?
When you teleworked, or worked from home, was it the same amount of hours every week, or did it vary?
IF VARIED: How did it vary?
IF NO TELEWORK IN FEB 2020
Can you tell me about the type of work (you/NAME) did in February 2020?
Did (you/NAME) do any work remotely, from home, or a place other than the worksite in February 2020?
IF YES: Tell me about that.
Was telework available to (you/NAME) in February 2020?
SHOW NEW4
You said that the amount of telework or work at home for pay (you/NAME) did is (AMOUNT) now than it was in February 2020. Can you tell me about that?
When you were comparing what you did last week to February 2020, how did you come up with your answer?
Was there anything about this question that was difficult to answer? (What was that?)
CLOSING
Thank you very much for the information you have shared today. I have a few final follow-up questions for you:
Is there anything else about telework, working from home, or how the COVID-19 pandemic affected how you work that we have not already talked about?
Thank you for your time. We appreciate all of your feedback.
Appendix B: Online Testing Protocol
Welcome! Thank you for your interest in our research to improve the Current Population Survey, a survey about people in the US who are working or out of work.
Throughout the survey, you will be asked to write brief but detailed explanations for a few questions to help us understand your situation.
The study should take about 10 minutes. Please only start the study when you will be able to complete the whole study without interruption. Please do your best to respond to the questions accurately.
Please do not use your browser's back button.
This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: July 31, 2024). Without this currently approved number, we could not conduct this survey. This survey will take approximately 10 minutes to complete. If you have any comments regarding this estimate or any other aspect of this study, send them to [email protected]. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. Your participation is voluntary, and you have the right to stop at any time.
---page break---
Before we begin….
What is your mTurk worker ID?
[open entry]
What is your sex?
Male
Female
What year were you born?
[open numeric entry]
What is the highest level of school you have completed or the highest degree you have received?
Less than a high school diploma
High school graduate, no college
Some college or associate degree
Bachelor's degree or higher
---page break---
Screener for identifying eligible participants.
VAR |
Question |
Response Options |
Universe |
SCR1 |
For ALL questions in this survey, exclude any work you do through mTurk or for any other survey platforms. |
|
Everyone |
SCR2 |
Excluding any work you do through mTurk or for any other survey platforms, LAST WEEK, did you do any work for pay? |
|
Everyone |
SCR3 |
Excluding any work you do through mTurk or for any other survey platforms, last week did you telework for any of your paid work hours? For example, remote work or working from home during paid working hours. |
|
Employed |
SCR4 |
Excluding any work you do through mTurk or for any other survey platforms, were you teleworking at all in February 2020? |
|
Employed and Teleworked last week. |
Page Break |
|||
S_END |
Thank you. |
GO TO ELIGIBILITY |
|
Eligibility:
If SCR2 = 1 and SCR3 = 1 then eligible. SCR4 will be monitored and is not required (e.g., SCR4 = YES) of all participants.
Participants identified as eligible will automatically be provided main survey link.
Questions for working are repeated for context and to verify screening.
VAR |
Question |
Response Options |
Universe |
LABOR |
Next, we are going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___ |
|
Everyone |
EXCL |
For all questions, remember to exclude any work you do through mTurk or for any other survey platforms. |
|
Everyone |
WK |
Excluding any work you do through mTurk or for any other survey platforms, LAST WEEK, did you do ANY work for pay? |
|
Everyone |
Page Break |
|||
S_END |
Thank you for participating in this survey. |
|
|
Page Break |
|||
MJ |
LAST WEEK, did you have more than one job, including part-time, evening or weekend work? Exclude any work you do through mTurk or for any other survey platforms. |
|
WK = 1 |
Page Break |
|||
HRACT_M |
LAST WEEK, how many hours did you work? Include total hours across all jobs. Exclude any work you do through mTurk or for any other survey platforms. |
|__|__| hours |
MJ = 1 or 2 |
Page Break |
|||
IO1INT |
Now I have a few questions about the (main) job at which you worked LAST WEEK.
Were you employed by government, by a private company, a non-profit organization, or were you self-employed or working in a family business? |
|
All employed |
IO1IND |
What kind of business or industry do you work at for your job? |
Open-ended |
All employed |
IO1OCC |
What kind of work did you do, that is, what is your occupation? |
Open-ended |
All employed |
IO1DT |
What are your usual activities or duties at this job? |
Open-ended |
All employed |
Page Break |
|||
NEW_INTRO |
I now have some questions related to how the COVID-19 pandemic affected how people work. Exclude any work you do through mTurk or for any other survey platforms. |
|
All employed |
NEW1 |
At any time LAST WEEK did you telework or work at home for pay? |
|
All employed |
PR1 |
Do you generally telework or work from home every week? |
|
All teleworking |
PR2 |
Was any of this time unpaid work? |
|
All teleworking |
PR3 |
LAST WEEK where did you telework from the most? |
|
All teleworking |
Page Break |
|||
NEW2 |
LAST WEEK, you worked [FILL: value for variable HRACT_M (total actual hours worked last week at all jobs)] hours total. How many of these hours did you telework or work at home for pay?
If you have more than one job include telework hours on all jobs combined. |
|__|__| hours |
All teleworking |
PR4 |
Is that number of hours consistent each week, or does it vary? |
|
All teleworking |
Page Break |
|||
NEW3 |
Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started?
Remember to exclude any work through mTurk, or for other survey platforms. |
|
All teleworking |
PR5 |
Was telework available to you in February 2020 (before the COVID-19 pandemic started)? |
|
Did not telework or not sure in Feb 2020 |
PR6 |
Was this for the same job or employer that you have now? |
|
Teleworked in Feb 2020 |
PR7 |
How easy or difficult was it for you to recall whether or not you teleworked in February 2020? |
|
Teleworked in Feb 2020 |
IF NEW1 = 1 (YES) and NEW3 = 1 (YES) GO TO NEW4; ELSE GO TO END |
|||
Page Break |
|||
NEW4 |
LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)? |
|
Teleworked in Feb 2020 |
PR8 |
Please briefly explain how you came up with your response. |
Open ended |
Teleworked in Feb 2020 |
END |
Thank you for participating in this
survey. |
|
|
1 Initially, five questions were fielded, but one item on medical care was dropped in 2020. The four remaining questions were: 1.) "At any time in the last 4 weeks, did you telework or work at home for pay because of the coronavirus pandemic?" 2.) "At any time in the last 4 weeks, were you unable to work because your employer closed or lost business due to the coronavirus pandemic?" 3.) "Did you receive any pay from your employer for the hours you did not work in the last 4 weeks?" 4.)"Did the coronavirus pandemic prevent you from looking for work in the last 4 weeks?"
2 It is unclear why these 35 participants changed their responses between the screener and the full questionnaire. They may have intentionally misreported their answer to teleworking last week, trying to “screen into” the study, or they may have interpreted the question differently in the screener vs. the full questionnaire.
3 Self-participant refers to when a participant provides information about themselves, proxy participant refers to when the study participant provides information about others household members. Only the cognitive interview study collected reports from proxy participants.
4 One participant noted he was currently teleworking as a teacher in the screener, but during the interview indicated he had just begun in-person teaching.
5 The online survey instrument filled in participants’ response to the previous question on the number of hours worked last week.
6 Responses falling into the Other category were mostly nonsubstantive (e.g., “good” or “very well valuable”).
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Kaplan, Robin - BLS |
| File Modified | 0000-00-00 |
| File Created | 2022-08-05 |