Nonsubstantive Attachment L - CPS COVID Questions Cognitive Testing Results_7.25.2022_Final

Nonsubstantive Attachment L - CPS COVID Questions Cognitive Testing Results_7.25.2022_Final.docx

Current Population Survey—Basic Labor Force

Nonsubstantive Attachment L - CPS COVID Questions Cognitive Testing Results_7.25.2022_Final

OMB: 1220-0100

Document [docx]
Download: docx | pdf

TO: Rachel Krantz-Kent

Office of Employment and Unemployment Statistics

FROM: Robin Kaplan and Douglas Williams

Office of Survey Methods Research

DATE: 8/5/2022

SUBJECT: Cognitive Test Results for COVID-19 and Telework Related Questions in the CPS



Introduction

In May 2020, the Current Population Survey (CPS) survey fielded four1 new questions with the intent of facilitating the understanding of changes to the labor market during the COVID-19 pandemic. Data has been collected using these questions for nearly two years while the pandemic has continued. Now many states and businesses have begun to end mandates or workforce restrictions to maintain the health and safety of the public. As a result, CPS is seeking to revise these questions to measure the extent that workforce changes (such as telework) have continued, and the scope of that change by comparing telework frequency to before the COVID-19 pandemic started.

To address these changes, CPS has made questionnaire revisions, consisting of four questions to replace the existing questions. The original questions collected information about employment changes, such as telework and work shutdowns, and information about how the pandemic affected people’s job searches. The revised questions will replace these and focus on current telework frequency and whether the individual teleworked before the COVID-19 pandemic. Before testing occurred, the Office of Survey Methods Research (OSMR) conducted an expert review of these questions and provided wording recommendations. This memo includes results from one-on-one cognitive interviews and online testing conducted during June of 2022.



Methods

One-on-one cognitive interviews

Cognitive interviews provide an in-depth understanding of the respondents’ thought processes and reactions to questions to uncover possible sources of response error, bias, and to ensure respondents understand the questions as intended. For this research, one-on-one cognitive interviews were conducted remotely via telephone or Microsoft Teams by staff from OSMR who are experienced in conducting these types of interviews. Participants were asked an abbreviated version of the CPS, including a shortened household roster and labor force questionnaire, and the applicable revised COVID-19 questions. After completion, participants were asked debriefing questions to better understand the thought processes used to answer the COVID-19 questions, in particular comprehension and recall as several questions asked about the period when the pandemic first started, over two years ago. The protocol for the cognitive interviews is included in Appendix A.

Recruitment for the cognitive interviews

Participants for cognitive testing were found through placing a Craigslist ad in the city of Raleigh, NC. This city was selected because it is a major city with diverse business establishments and professional jobs, where it is expected some of the workforce would telework or work from home. Specifically, we recruited ten participants who fit the following criteria:

  • From a household with 1 or more people over age 18, and;

  • Employed– Currently have a job, either full- or part-time, and currently telework or work from home.

In each interview, the researcher started by introducing the purpose of the study and familiarizing participants with the CPS. Then an abbreviated version of the CPS was administered, including a shortened household roster, labor force questionnaire, and the applicable COVID-19 questions. The instrument used for testing allowed participants to report this information for themselves and up to 2 other household members.

After going through the abbreviated survey, participants were asked a series of debriefing questions about the COVID-19 questions they had received during the interview. Participants were then thanked for their time and the call was ended. The whole process - introduction, abbreviated survey, and debriefing - took approximately 30 minutes per participant.

Online Testing

Online testing was also conducted to complement the one-on-one cognitive interviews and to collect feedback from a larger, more geographically diverse group of participants. The questionnaire and debriefing questions were similar to those used in the one-on-one cognitive interviews, but written in a self-administered format with mostly closed-ended probes. Participants were recruited from Amazon Mechanical Turk (mTurk), an online marketplace where individuals can sign up to participate in short online research tasks for nominal compensation. Like the cognitive interviews, participants were screened (N=400) to identify those that currently have a job, either full- or part-time, and currently telework or work from home. Participants were instructed to exclude any work they do through mTurk or other online survey platforms in both the screener and full questionnaire. Only those meeting the screening requirements (n=207) were invited to complete the CPS questionnaire content, and 200 participated. However, after assessing responses to the screener versus the full questionnaire, we found that 35 participants indicated they had teleworked last week in the screener, but then went on to report they had NOT teleworked in the full questionnaire2. These 35 participants who reported inconsistent responses were removed from the sample for lack of eligibility for the study and data quality purposes, leaving a total of 165 participants. The screener took an average of 55 seconds to complete, and the full questionnaire took an average of about 10 minutes to complete. All participants resided in the United States, and ranged in age from 21 to 71 years old, with an average age of 40. A total of 61% of the sample was male and 72% reported having a Bachelor’s degree or higher. The protocol for the online study is included in Appendix B.

Results

The summary table below specifies the number of participants (both self and proxy3) who were asked about each of the COVID-19 questions during both the cognitive interviews and online study, as well as the final recommendation for each question.


Question Topic

Cognitive Interview

Online Study

Recommendation

Q1

Telework from home for pay

10 (self)

4 (proxy)

165

No change

Q2

Number of hours teleworked or worked from home

94 (self)

3 (proxy)

165

No change

Q3

Telework in Feb. 2020

10 (self)

4 (proxy)

165

No change

Q4

Did more, less, or same amount of telework as in Feb. 2020

3 (self)

1 (proxy)

85

No change



General Feedback

Participants in the cognitive interview study felt that the COVID questions were straightforward and easy to answer. It was easy for most participants to recall their jobs around the time the pandemic started and afterward. A couple participants had trouble remembering back to the reference period (pre-COVID and during February 2020), but all participants were able to arrive at an answer. Three of the four participants answering as a proxy for other household members displayed some difficulty answering. This was due to difficulty recalling when other household members started teleworking or not having the knowledge to answer the question (either because they were not in the same household during February 2020, or they did not know a lot about others household members’ jobs in general). One participant felt it was invasive to answer on behalf of other household members. However, all participants were able to understand the questions and arrive at their best estimates.

Results from the online study largely echoed the findings from the cognitive interviews. The data indicated most participants understood the questions as intended and provided consistent responses across all items. Participants did not have any major difficulties recalling the information needed to answer the questions based on their responses to closed-ended probes and open-ended feedback. Taken together, the COVID questions performed well as written and no critical issues were found. Thus, no changes to the existing questions are recommended. A summary of the findings for each of the questions is outlined below.

Q1. Telework from home

Tested Wording

Recommended Wording

Cognitive interview wording: At any time LAST WEEK, did you telework or work at home for pay?

  1. Yes

  2. No

  3. Don’t Know

  4. Refused


Online study wording: At any time LAST WEEK, did you telework or work at home for pay?

  1. Yes

  2. No

No Change



Cognitive Interviewing Results

The question about whether participants teleworked last week was covered in all ten interviews. It was asked about ten self-participants and four proxy participants who were employed and currently working, as seen in the table below.


Self

Proxy

Yes

9

3

No

1

1



All but one participant indicated they had teleworked last week, except one teacher who had recently resumed in-person teaching. In all cases, participants were able to answer the question and understood the phrase “telework or work at home for pay” as intended.

Most participants were teleworking their usual schedule at home. Several participants noted that while they primarily telework, they occasionally work on-site depending on different circumstances. For example, one participant noted they very rarely are required to go into the office when someone they act as a backup for is not available. Another participant worked as a freelance software developer and noted depending on the current gig, he might be able to work from home or may have to come into an office. Another proxy participant said her spouse is a manager and is required to come into the office once or twice per week, though it has been less often recently due to rising gas prices.

Most participants could easily recall when they started teleworking as it coincided with the start of pandemic, usually in March 2020, or whenever they were hired for the job, both of which were landmark dates in their mind. A few participants mentioned some of the hours they teleworked may have been unpaid – for instance, a manager who caught up on emails in the evenings after work hours (but was in a salaried position and not compensated for this time), a freelance software developer who worked a few hours learning new skills for a gig but was not paid for this time, and a participant who worked some unpaid internship hours remotely. Taken together, participants in the cognitive interview study understood this question well, and no major issues were found.

Online Study

The online study asked the question about teleworking last week for self-participants only. Participants reported working an average of 40 hours per week across all jobs. As shown in the table below, there were two follow-up probes asking whether participants generally telework each week and whether any of this time was unpaid work. Like the cognitive interview participants, most (almost 92%) reported they generally telework every week, and almost 92% also reported that this time was for paid work.


Yes

No

Percent

Frequency

Percent

Frequency

Do you generally telework or work from home every week? (N=165)

91.5%

151

8.5%

14

Was any of this time unpaid work? (N=165)

8.5%

14

91.5%

151



Conversely, around 9% of participants reported that they do not generally telework every week, and around 9% also reported that this time included unpaid work. Interestingly, 13 of the 14 participants who reported that they do NOT generally telework each week also reported that some of their telework time was unpaid. This suggests that those who do not telework each week may also do more unpaid work (e.g., they may catch up on work at home after regular business hours), consistent with some responses in the cognitive interviews.

Another follow-up probe asked where participants teleworked from the most last week, as shown in the table below. Consistent with the cognitive interviews, most participants reported working from home. Note that 6% reported teleworking from a location other than their home, suggesting that the term “telework” was broader than just working from home for some participants. No participants indicated any other locations in the ‘Other, specify’ field.

LAST WEEK, where did you telework from the most? (N=165)

Percent

Frequency

From your home

91.5%

155

Another business or office location

3.0%

5

Various worksites

3.0%

5



Recommendation

Overall, this question performed as intended. We have no recommended changes for this question. Participants in both the cognitive interviews and the online study were able to provide adequate answers to this question and seemed to understand the phrase “telework or work at home for pay” as intended.

Q2. Number of hours teleworked or worked from home

Tested Wording

Recommended Wording

Cognitive interview wording: How many hours did you telework or work at home for pay LAST WEEK?


(Interviewer instruction: Multiple jobholders should include telework hours on all jobs combined.)


Online study wording: LAST WEEK, you worked [fill]5 hours total. How many of these hours did you telework or work at home for pay? 

If you have more than one job include telework hours on all jobs combined.

No Change





Cognitive Interviewing Results

The question about the number of hours participants teleworked last week was asked of nine self-participants and three proxy participants who reported teleworking in the last week. This question seemed relatively easy for most participants to answer, as they reported telework hours as part of their regular work schedule.

For some participants who did not have a set work schedule (e.g., they work gigs, freelance, or have varying hours depending on business need), the question was slightly more difficult and involved trying to add the number of hours up in real-time or providing estimates. For example, one participant worked for a company whose hours differed based on business needs. She indicated “depending on the workload, it can vary. It’s hard to say. Sometimes Mondays are busier, and Fridays are less busy. It can range from 25 to 45 hours.” This same participant noted she also has a side business selling beauty products online that she initially forgot to include but remembered to once the interviewer instruction to include telework hours on all jobs combined was read. She stated her best estimate is that she works about 20 hours per week on her side business, but struggled to arrive at an answer, noting “I sometimes do work at midnight, so it’s a very rough, ballpark estimate.”

Another participant answering as a proxy for her husband noted she had to “guestimate” because he often logs back in the evenings to catch up on work. She thought he did so about two hours per weeknight on top of his 40-hour per week job, arriving at an answer of 50 hours. She mentioned these hours are technically unpaid since he does not earn overtime pay or time off for them. Two other proxy-participants were not certain about the number of hours their housemates worked, assuming it was likely 40 hours since they had salaried full-time positions.

Overall, this question worked well in the cognitive interviews. We note there might be some issues with workers who have irregular hours, change gig jobs or freelance jobs frequently, or hold more than one job where they telework. As expected, some of proxy responses were rough estimates. Interviewer instructions to remind multiple job holders to report telework hours from all jobs combined is also critical as some participants noted having multiple gigs, varying freelance hours, or “side hustles” in which they telework. In addition, some participants included telework that occurred after working hours or was unpaid. However, because the CPS includes this type of work (e.g., teachers grading papers at home or self-employed respondents working on training or professional development), and the total telework hours are anchored to the total number of hours respondents reported working earlier in the interview, this is not expected to be an issue. When collecting total telework hours, we recommend that interviewers apply the same concept used for collection of total work hours.

Online Study

In the online study, the question on number of hours teleworked last week was asked of all participants. Participants reported teleworking an average of 27.3 hours during the previous week. All participants reported teleworking an amount that was equal to or less than the total hours worked during the past week, suggesting that they interpreted each question correctly. A total of 43.6% of participants reported teleworking all of the hours they worked the previous week. Participants reported teleworking an average of 70% of their working hours during the previous week.

When asked whether the number of hours they telework is consistent each week or if it varies, the majority (almost 60%) reported working a consistent number of hours each week.

Is that number of hours consistent each week, or does it vary? (N=165)

Percent

Frequency

Consistent each week

58.8%

97

Varies week to week

41.2%

68



Participants who reported teleworking a consistent number of hours each week teleworked 78.1% of their total hours during the previous week, whereas those reporting that their telework hours vary week to week reported teleworking 56.9% of their total hours. This suggests that those who telework a consistent schedule may also telework more frequently.

A total of 34% of participants reported having more than one job. Those with multiple jobs reported working more on average (42.4 hours) than those with one job (38.7 hours). Similarly, as seen in the table below, participants who had only one job were more likely to indicate that their telework hours are consistent each week compared to those working multiple jobs, where participants were evenly split on working consistent versus varying hours. This is similar to the cognitive interviewing findings, where participants seemed to report more variable hours when they held multiple jobs and had slightly more difficulty arriving at answer for the total number of hours teleworked.


One job

Multiple jobs

Telework hours consistent each week

62.5%

50.9%

Telework hours vary week to week


37.5%

49.1%



Recommendation

This question worked well, and no changes are recommended. Participants in both the cognitive interviews and the online study were able to provide adequate answers to this question and seemed to understand it as intended. Some minor issues occurred with workers who have variable schedules, hold more than one job where they telework, and with proxy responses. Again, we recommend that interviewer training emphasize the need to remind respondents that multiple jobholders should include telework hours on all jobs combined to help increase data quality.

Q3. Telework in Feb. 2020

Tested Wording

Recommended Wording

Cognitive interview wording: Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started?


(Interviewer instruction: Enter “No” if the person was not employed in February 2020.)


  1. Yes

  2. No

  3. Don’t Know

  4. Refused


Online study wording: Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started?


Remember to exclude any work through mTurk, or for other survey platforms.


  1. Yes

  2. No

  3. You were not employed in February 2020

  4. Not sure


No Change



Cognitive Interviewing Results

In the cognitive interviews, the question on whether participants teleworked in February 2020 before the pandemic started was covered in all participants. The responses for self and proxy can be seen in the table below.


Self

Proxy

Yes

3

1

No

7

4



The question was generally easy for participants to understand and to remember back to when the pandemic started. Some participants started working at home in March of 2020 and were not able to telework before that time because their work required being on-site (e.g., they catered events, taught in a classroom, or worked at an airport), so that time was very distinct in their minds. Participants who had been teleworking long before the pandemic began also found this question easy to answer because they reported when they started working for their employer, which was usually a memorable date.

The question was somewhat more difficult for a couple participants (who were doing irregular gig or freelance work during the pandemic) to recall whether they were teleworking or not in February 2020 because of frequent changes in jobs, employers, or gigs. One participant had to estimate around when she and her husband began teleworking because it was not at the beginning of the pandemic. Instead, she estimated that they both began teleworking in about January 2021, when she recalled the COVID-19 variants began showing up. The question was also more challenging for some participants answering as a proxy because they did not have the knowledge of when their household member started teleworking, or their household member lived elsewhere in February 2020.

Online Study

About half of the participants in the online study reported that they had teleworked in February 2020 before the pandemic started, as seen in the table below. About 2% of participants reported that they were not employed at that time. Only 1.2% indicated they were not sure, suggesting that recall (for self-response) is not an issue for this question.

Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started? (N=165)

Percent

Frequency

Yes

51.5%

85

No

45.5%

75

You were not employed in February, 2020

1.8%

3

Not sure

1.2%

2




Participants who reported that they teleworked before the pandemic were asked a follow-up question about whether this was for the same employer they currently have. As seen in the table below, most participants reported that they currently have the same job or employer. Over 10% of participants reported having changed jobs since February 2020.

Was this for the same job or employer that you have now? (n=85)

Percent

Frequency

Yes

89.4%

76

No

10.6%

9







Participants who reported they teleworked in February 2020 were asked a probe about how easy or difficult it was to recall whether they teleworked before the pandemic. As seen in the table below, most participants found this information very easy or somewhat easy to recall, consistent with the cognitive interview study. Of the nine participants who thought the question was somewhat difficult, only one had changed jobs since February 2020.

How easy or difficult was it for you to recall whether or not you teleworked in February 2020? (n=85)

Percent

Frequency

Very easy

67.1%

57

Somewhat easy

22.4%

19

Somewhat difficult

10.6%

9

Very difficult

0.0%

0









Participants who reported they did NOT telework (or were unsure if they teleworked) in February 2020 were asked whether telework was available to them in February 2020, before the COVID-19 pandemic started. As seen in the table below, most participants reported that telework was also not available to them before the pandemic.

Was telework available to you in February 2020 (before the pandemic started?) (n=79)

Percent

Frequency

Yes

12.7%

10

No

87.3%

69







Recommendation

This question worked well based on the cognitive testing and online study, and most participants had no problems. The cognitive interview study found that there may be some estimation involved with those who had irregular schedules during the pandemic or proxy participants who may not have the knowledge to answer on behalf of other household members. However, these memory issues were minor and participants were able to arrive at answers or estimates. The online study showed a similar pattern of results, and most participants found it easy to recall this information. No question changes are recommended.

Q4. Did more, less, or the same amount of telework as in Feb. 2020

Tested Wording

Recommended Wording

Cognitive interview wording: LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)?


1. More

2. Less

3. About the same

4. Don't Know

5. Refused


Online study wording: LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)?


1. More

2. Less

3. About the same

4. Not sure

No Change



Cognitive Interviewing Results

In the cognitive interviews, the question about whether participants worked more, less, or the same amount of telework as in February 2020, was asked of the four participants who reported that they had teleworked before the pandemic. This included three self-responses and one proxy-response, as seen in the table below.


Self

Proxy

Yes

2

1

No

1

0



This question was generally easy for all participants to understand. For two participants, it was very straightforward as their employer expanded telework for all employees after the pandemic. Two participants also reported this was easy to recall as their employer increased the number of regular hours that could be done via telework since February 2020. One participant switched employers since February and knew she used to work fewer telework hours then compared to now. These participants felt it was an easy comparison to make and did not report any difficulties arriving at answer. However, as noted above, this question was only administered four times.

Online Study

Participants who indicated that they had teleworked in February 2020 (n=85) were asked the question about whether they teleworked more, less, or the same amount as in February 2020. As seen in the table below, most participants indicated that they teleworked about the same amount or more. Only one participant indicated they were not sure (and later explained in an open-ended probe that this was because their work hours constantly change), suggesting that recall was not a major issue with this question.

LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)? (n=85)

Percent

Frequency

More

27.1%

23

Less

15.3%

13

About the same

56.5%

48

Not sure

1.2%

1











Participants who teleworked in February 2020 were asked an open-ended probe to explain how they came up with their response to this question, and 83 respondents provided an explanation. The responses tended to fall into the following five categories:


Percent

Frequency

  1. The participant teleworked the same amount before the pandemic started

42.2%

35

  1. Telework was increased after the pandemic

20.5%

17

  1. Telework was increased after the pandemic, but they are being phased back on-site now

3.6%

3

  1. The participant explained differences in workload back in February 2020 versus now (e.g., seasonality or changing jobs)

20.5%

17

  1. The participant explained they just knew this information from memory

9.6%

8

  1. Other6

3.6%

3



Representative examples of these responses are outlined below for each of the five categories:


Examples

  1. The participant teleworked the same amount before the pandemic started

  • My job has always been somewhat hybrid, so my telework schedule is, although pretty scattered, usually around the same amount of work.

  • I was working from home before the pandemic. The work has been similar.

  • Covid did not change the amount of telework I was completing as my company had closed our office location long before covid.

  • I have been working from home since 2017.

  1. Telework was increased after the pandemic

  • Prior to Covid, I would do telework occasionally, maybe one day a week, but now I can do most of my work remotely.

  • In February I was in the office more than I am now.

  • In Feb 2020 I had the option to telework when I wanted, but it was not a set hybrid schedule like I have now.

  • Before the pandemic started, I would work from home a couple of days per week as desired. Now, I go into the office one day per week for staff meetings in an effort to minimize exposure.

  1. Telework was increased after the pandemic, but they are being phased back on-site now

  • I did less telework because the office was opening back up and accepting workers to come on site instead of working in a hybrid function.

  • Before February 2020 I would telework occasionally. I teleworked more after Covid hit. Now I do it less than at the height of Covid.

  • My work brought us back to the office full time in the spring. From time to time, we are allowed to work from home one or two days a week, but we're primarily required to work in person now.

  1. The participant explained any differences in workload back in February 2020 versus now (e.g., seasonality or changing jobs)

  • January - March is usually a slow period in which I don't have to go into the actual workplace as often, so there were more things that could be done from home in February 2020 than there currently is in the summer time.

  • It’s been a lot of extra work the past 2 years.

  • I had a different job with different demands at that time, so I worked more hours.

  1. The participant explained they just knew this information from memory

  • I just thought about it.

  • I answered facts from memory.

  • Because I remember vividly.



Recommendation

This question worked as intended. We do not recommend any changes to this question. Participants in both the cognitive interviews and the online study were able to answer the question with minimal recall issues and provided substantive answers for how they knew the amount of telework they did now versus before the pandemic started.

Conclusion

In summary, the cognitive test and online study results showed that participants understood the questions well and were able to provide answers to them. Participants were mostly able to recall the information needed to arrive at their answers (e.g., their work schedule and hours just prior to the pandemic), as this was a memorable and especially vivid time for them. However, the cognitive interviews, which allow for richer qualitative data collection, indicated that some participants had minor recall issues. These were mainly for participants who worked multiple jobs or changed jobs frequently (e.g., freelance or gig work), but these participants were still able to think about their responses and arrive at answers they felt were accurate. We recommend that interviewer training emphasize the need to remind multiple jobholders to include telework hours on all jobs combined to help increase data quality. When collecting total telework hours, we recommend that interviewers apply the same concept used for collection of total work hours.

As expected in any survey with proxy response, minor issues also occurred where some respondents in the cognitive interview study had issues with recall or knowledge when answering on behalf of other household members, so these responses were estimates. This is a potential data quality issue to be aware of given the questions ask about a period dating back to February 2020. Taken together, the cognitive interview and online studies showed similar results and found no critical issues, and we recommend the questions proceed as written.



Appendix A: Cognitive Interview Protocol

Introduction

Hello, my name is ______________ and I work at the Bureau of Labor Statistics. Thank you for taking the time to talk with me today. (IF NEEDED) I’m joined by my colleague __________ today.

We are going to be working with some questions that might be asked as part of the Current Population Survey, which is one of the longest running survey programs measuring the economic and social well-being of the United States. We will first ask about the people living in your household, then some questions about how you worked at different time periods.

Before new survey questions are asked, it’s important to test them out with people like you, to make sure that they make sense and are easy to follow. Your answers will help us improve the questions. There are no right or wrong answers; we’re looking for your reactions and honest feedback. Your name will not be included in the report, and nothing that you say will be attributed directly to you.

We are not here to evaluate you, we are looking to improve the questions.

Informed Consent:

Upon beginning a remote testing session, participants will read this information or be shown the following information and asked for their verbal consent prior to beginning the study:

  • We estimate the session will last 30 minutes. If you have any comments regarding this estimate or any other aspect of this study, send them to [email protected].

  • Your participation in this research project is voluntary, and you have the right to stop at any time.

  • The Bureau of Labor Statistics is conducting this voluntary study under OMB No. 1220-0141, which expires on July 31, 2024. Without this currently approved number, we could not conduct this research. Your responses are also protected by law:



The Bureau of Labor Statistics, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

In accordance with the Privacy Act of 1974 as amended (5 U.S.C. 552a), this study is being conducted by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under the authority of 29 U.S.C. 2.  Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections.  The information will only be used by and disclosed to BLS personnel and contractors who need the information for activities related to improving BLS information collection.  Information on routine uses can be found in the system of records notice, DOL/BLS – 14, BLS Behavioral Science Research Laboratory Project Files (81 FR 47418). 



Do you have any questions before we proceed?

Do you agree to participate?

Questionnaire completion:

If reading the survey/administering over the phone:

  • I will read the survey aloud to you, please tell me your answers.

  • If a question isn’t clear or you don’t understand some wording that is used, please let me know. We want these questions to be clear and easy to understand.

  • If there are any questions you would rather not answer, let me know.





VAR

Question

Response Options

Universe

Roster1

Please tell me the names and ages of everyone who lives in your household

Open ended

Everyone

Labor Force Questions, ask as a block for up to 3 HH employed members (age 16+), identified during roster

LABOR

Next, I am going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___

 

Everyone

WK

LAST WEEK, did (you/NAME) do ANY work for pay?

  1. Yes, go to IO1INT

  2. No, go to END

  1. Don't Know

  1. Refused

Everyone

IO1INT

Now I have a few questions about the job at which (you/NAME) worked LAST WEEK.)


(Was.../Were you) employed by government, by a private company, a non-profit organization, or (was.../were you) self-employed (or working in a family business)?

  1. Government

  2. Private company

  3. Non-profit organization incl. tax exempt and charitable organizations

  4. Self-employed

  5. Working in family business

(D) Don’t Know

(R) Refused

All employed

IO1IND

What kind of business or industry is this?

READ IF NECESSARY: What do they make or do where you work?

Open-ended

All employed

IO1OCC

What kind of work (do/did) (you/name) do, that is, what (is/was) (your/name’s) occupation?

(For example: plumber, typist, farmer)

Open-ended

All employed

IO1DT

What (are/were) (your/name’s) usual activities or duties at this job?

(For example: typing, keeping account books, filing, selling cars, operating printing press, laying brick)

Open-ended

All employed

Telework questions (COVID-19), ask as a block for up to 3 HH employed members, identified during roster

NEW_INTRO

I now have some questions related to how the COVID-19 pandemic affected how people work.


All employed

NEW1

At any time LAST WEEK did (you/NAME) telework or work at home for pay?

  1. Yes, go to NEW2

  2. No, go to NEW3

  1. Don't Know

  1. Refused

All employed

NEW2

LAST WEEK, (you/Name) worked [FILL: value for variable HRACTT (which is the actual hours worked last week at all jobs asked in main CPS)] hours [FILL FOR MULTIPLE JOBHOLDERS: total, at all jobs].

How many of these hours did [you/NAME] telework or work at home for pay?

(Interviewer instruction: Multiple jobholders should include telework hours on all jobs combined.)

|__|__| hours

(D) Don’t Know

(R) Refused

Teleworked

NEW3

Did (you/NAME) telework or work at home for pay in February 2020 before the COVID-19 pandemic started?

(Interviewer instruction: Enter “No” if the person was not employed in February 2020.)

  1. Yes

  2. No

  1. Don't Know

  1. Refused

Teleworked

IF NEW1 = 1 (YES) and NEW3 = 1 (YES) GO TO NEW4; ELSE GO TO END

NEW4

LAST WEEK, did (you/NAME) do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)?

  1. More

  2. Less

  3. About the same

  1. Don't Know

  1. Refused

Teleworked in Feb 2020






Next, I would like to talk about the questions you answered and your general reactions answering these.

Start by collecting additional detail about jobs from the participant’s own perspective:

  • What was it like for you to participate in this survey?

  • Were there any questions you found difficult or confusing?

  • Were there any questions you found sensitive or personal?



Now I’d like to talk about some of the specific questions you answered. The first one asked about telework.

If on video call, share screen with questionnaire showing so that participant can see the questions.

If on telephone call, read questions aloud.

Show NEW_INTRO and NEW1

  • What does ‘telework or work at home for pay’ mean to you?



  • IF YES TO TELEWORK

  • You said that (you/NAME) teleworked last week. Can you tell me more about that?

    • IF NEEDED: Where did (you/NAME) telework from?

    • How many days did (you/NAME) telework last week?

    • (Do you/Does name) generally telework or work from home every week?

    • When did (you/NAME) start teleworking or working from home?

    • Was any of this time unpaid work or after working hours?



  • IF NO TO TELEWORK

  • Is telework available to (you/NAME)?

  • (Have you/Has Name) ever teleworked or worked from home for pay?

    • IF YES: Why (do you/does name) no longer telework or work from home for pay?



Show NEW2

  • How did you come up with (NUMBER) hours for teleworking or working from home?

  • Is that number of hours consistent each week, or does it vary?

    • IF VARIES: How does it vary?

  • Were any of these hours unpaid or after working hours?

  • Do you have more than one job or employer where some of your hours are spent teleworking?

    • IF YES: Did the hours you reported include telework from all jobs?



Show NEW3

  • IF YES TELEWORK IN FEB 2020

  • Was this question easy or difficult for you to answer? (Why?)

  • You said that (you/NAME) teleworked or worked from home for pay in Feb 2020. Can you tell me more about that?


    • Can you tell me what timeframe you were thinking about when answering this question?

    • Was this for the same job or employer that you have now?

    • When you teleworked, or worked from home, was it the same amount of hours every week, or did it vary?

      • IF VARIED: How did it vary?



  • IF NO TELEWORK IN FEB 2020

  • Can you tell me about the type of work (you/NAME) did in February 2020?

  • Did (you/NAME) do any work remotely, from home, or a place other than the worksite in February 2020?

    • IF YES: Tell me about that.

  • Was telework available to (you/NAME) in February 2020?



SHOW NEW4

  • You said that the amount of telework or work at home for pay (you/NAME) did is (AMOUNT) now than it was in February 2020. Can you tell me about that?

  • When you were comparing what you did last week to February 2020, how did you come up with your answer?

  • Was there anything about this question that was difficult to answer? (What was that?)



CLOSING

Thank you very much for the information you have shared today. I have a few final follow-up questions for you:

  • Is there anything else about telework, working from home, or how the COVID-19 pandemic affected how you work that we have not already talked about?



Thank you for your time. We appreciate all of your feedback.















Appendix B: Online Testing Protocol

Welcome! Thank you for your interest in our research to improve the Current Population Survey, a survey about people in the US who are working or out of work.

Throughout the survey, you will be asked to write brief but detailed explanations for a few questions to help us understand your situation.

The study should take about 10 minutes. Please only start the study when you will be able to complete the whole study without interruption. Please do your best to respond to the questions accurately.

Please do not use your browser's back button.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: July 31, 2024). Without this currently approved number, we could not conduct this survey. This survey will take approximately 10 minutes to complete. If you have any comments regarding this estimate or any other aspect of this study, send them to [email protected]. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. Your participation is voluntary, and you have the right to stop at any time.


---page break---


Before we begin….


What is your mTurk worker ID?

[open entry]


What is your sex?

Male

Female


What year were you born?

[open numeric entry]


What is the highest level of school you have completed or the highest degree you have received?

Less than a high school diploma

High school graduate, no college

Some college or associate degree

Bachelor's degree or higher


---page break---





Screener for identifying eligible participants.

VAR

Question

Response Options

Universe

SCR1

For ALL questions in this survey, exclude any work you do through mTurk or for any other survey platforms.


Everyone

SCR2

Excluding any work you do through mTurk or for any other survey platforms, LAST WEEK, did you do any work for pay?

  1. Yes, go to SCR3

  2. No, go to S_END

Everyone

SCR3

Excluding any work you do through mTurk or for any other survey platforms, last week did you telework for any of your paid work hours?

For example, remote work or working from home during paid working hours.

  1. Yes, go to SCR4

  2. No, go to S_END

Employed

SCR4

Excluding any work you do through mTurk or for any other survey platforms, were you teleworking at all in February 2020?

  1. Yes

  2. No

Employed and Teleworked last week.

Page Break

S_END

Thank you.

GO TO ELIGIBILITY




Eligibility:

  • If SCR2 = 1 and SCR3 = 1 then eligible. SCR4 will be monitored and is not required (e.g., SCR4 = YES) of all participants.

  • Participants identified as eligible will automatically be provided main survey link.



Questions for working are repeated for context and to verify screening.

VAR

Question

Response Options

Universe

LABOR

Next, we are going to ask a few questions about work related activities LAST WEEK. By last week, I mean the week beginning on Sunday, ____ and ending on Saturday, ___

 

Everyone

EXCL

For all questions, remember to exclude any work you do through mTurk or for any other survey platforms.


Everyone

WK

Excluding any work you do through mTurk or for any other survey platforms, LAST WEEK, did you do ANY work for pay?

  1. Yes, go to MJ

  2. No, go to S_END

Everyone

Page Break

S_END

Thank you for participating in this survey.



Page Break

MJ

LAST WEEK, did you have more than one job, including part-time, evening or weekend work?

Exclude any work you do through mTurk or for any other survey platforms.

  1. Yes

  2. No

WK = 1

Page Break

HRACT_M

LAST WEEK, how many hours did you work? Include total hours across all jobs.

Exclude any work you do through mTurk or for any other survey platforms.

|__|__| hours

MJ = 1 or 2

Page Break

IO1INT

Now I have a few questions about the (main) job at which you worked LAST WEEK.


Were you employed by government, by a private company, a non-profit organization, or were you self-employed or working in a family business?

  1. Government

  2. Private company

  3. Non-profit organization incl. tax exempt and charitable organizations

  4. Self-employed

  5. Working in family business

All employed

IO1IND

What kind of business or industry do you work at for your job?

Open-ended

All employed

IO1OCC

What kind of work did you do, that is, what is your occupation?

Open-ended

All employed

IO1DT

What are your usual activities or duties at this job?

Open-ended

All employed

Page Break

NEW_INTRO

I now have some questions related to how the COVID-19 pandemic affected how people work. Exclude any work you do through mTurk or for any other survey platforms.


All employed

NEW1

At any time LAST WEEK did you telework or work at home for pay?

  1. Yes, go to PR1

  2. No, go to NEW3

All employed

PR1

Do you generally telework or work from home every week?

  1. Yes

  2. No

All teleworking

PR2

Was any of this time unpaid work?

  1. Yes

  2. No

All teleworking

PR3

LAST WEEK where did you telework from the most?

  1. From your home

  2. Another business or office location

  3. Various worksites

  4. Some other place (Specify)

All teleworking

Page Break

NEW2

LAST WEEK, you worked [FILL: value for variable HRACT_M (total actual hours worked last week at all jobs)] hours total. How many of these hours did you telework or work at home for pay?


If you have more than one job include telework hours on all jobs combined.

|__|__| hours

All teleworking

PR4

Is that number of hours consistent each week, or does it vary?

  1. Consistent each week

  2. Varies week to week

All teleworking

Page Break

NEW3

Did you telework or work at home for pay in February 2020 before the COVID-19 pandemic started?


Remember to exclude any work through mTurk, or for other survey platforms.

  1. Yes, go to PR6

  2. No, go to PR5

  3. You were not employed in February, 2020, go to END

  1. Not sure, go to PR5

All teleworking

PR5

Was telework available to you in February 2020 (before the COVID-19 pandemic started)?

  1. Yes, go to END

  2. No, go to END

Did not telework or not sure in Feb 2020

PR6

Was this for the same job or employer that you have now?

  1. Yes

  2. No

Teleworked in Feb 2020

PR7

How easy or difficult was it for you to recall whether or not you teleworked in February 2020?

  1. Very difficult

  2. Somewhat difficult

  3. Somewhat easy

  1. Very easy

Teleworked in Feb 2020

IF NEW1 = 1 (YES) and NEW3 = 1 (YES) GO TO NEW4; ELSE GO TO END

Page Break

NEW4

LAST WEEK, did you do more, less, or the same amount of telework or work at home for pay as in February 2020 (before the COVID-19 pandemic)?

  1. More

  2. Less

  3. About the same

  1. Not sure

Teleworked in Feb 2020

PR8

Please briefly explain how you came up with your response.

Open ended

Teleworked in Feb 2020

END

Thank you for participating in this survey.

To receive payment for participating in this study, please copy the validation code shown below and paste it in the form on this page for this HIT and submit the task. We will check that code you paste in matches our own database and then approve your payment. 

Your code is: 418637 







1 Initially, five questions were fielded, but one item on medical care was dropped in 2020. The four remaining questions were: 1.) "At any time in the last 4 weeks, did you telework or work at home for pay because of the coronavirus pandemic?" 2.) "At any time in the last 4 weeks, were you unable to work because your employer closed or lost business due to the coronavirus pandemic?" 3.) "Did you receive any pay from your employer for the hours you did not work in the last 4 weeks?" 4.)"Did the coronavirus pandemic prevent you from looking for work in the last 4 weeks?"

2 It is unclear why these 35 participants changed their responses between the screener and the full questionnaire. They may have intentionally misreported their answer to teleworking last week, trying to “screen into” the study, or they may have interpreted the question differently in the screener vs. the full questionnaire.

3 Self-participant refers to when a participant provides information about themselves, proxy participant refers to when the study participant provides information about others household members. Only the cognitive interview study collected reports from proxy participants.

4 One participant noted he was currently teleworking as a teacher in the screener, but during the interview indicated he had just begun in-person teaching.

5 The online survey instrument filled in participants’ response to the previous question on the number of hours worked last week.

6 Responses falling into the Other category were mostly nonsubstantive (e.g., “good” or “very well valuable”).

Page 26 of 26


File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorKaplan, Robin - BLS
File Modified0000-00-00
File Created2022-08-05

© 2024 OMB.report | Privacy Policy