Cognitive Testing Results Memo (2021)

FROM: Douglas Williams, Struther Van Horn, and David Biagas

Office of Survey Methods Research

SUBJECT: Cognitive Testing of Business Response Survey 2.0

DATE: June 11, 2021

Introduction

The Bureau of Labor Statistics (BLS) has developed the first official production survey for the QCEW Business Supplement (QBS). This production version follows the Business Response Survey (BRS) which was conducted to understand the impact of the coronavirus pandemic on establishments. The production version of the QBS, will follow the same sample design (supplemental to the Annual Refiling Survey) and data collection procedures. The production version of the QBS will collect how establishments are transitioning during the economic recovery following the peak effects of the coronavirus pandemic.

Method

The production version of the QBS was developed through a team of BLS content experts, research statisticians, and survey methodologists. This ensured the question topics were relevant while considering the resulting data and potential cognitive issues. The initial questionnaire that resulted went through an expert review to address missed issues and ensure consistency in terminology use and question references (e.g., reference period). Following the expert review, cognitive testing was conducted to explore potential issues relating to the ability to provide the requested information and understanding of the questions.

A cognitive protocol was developed to outline potential issues and serve as a guide for interviewers. Interviewers could deviate from the protocol and probe as needed to address issues that may not be included in the protocol.

Participants were recruited from known industry contacts. Each interview was conducted via Microsoft Teams, where interviewers explained the purpose of the study and that participation was voluntary and confidential. Interviewers collected some background information about the establishment, then presented each question. The participant was instructed to read each question aloud and answer. Following each question, the interviewer probed following the protocol or non-standard probes as needed.

Following the interviews, a debriefing meeting was held to discuss each interviewer’s findings to share issues (or lack thereof) identified.

Participants

Participants were recruited from personal contacts of BLS staff, with one participant formerly included in the Business Response Survey (BRS) testing conducted in 2020. In recruiting participants, we attempted to reach a variety in terms of industry, establishment size, and location. A total of six participants were interviewed. The establishment characteristics these six participants represent are shown in Table 1.

Table 1. Establishment characteristics of recruited participants.

Industry	# of Employees	Location	Single or Multi-Unit
Hospital	2,200	Illinois	MU
Consulting - Defense	1,750	Virginia	MU
Healthcare – Management Services	1,300	California	MU
Retail – Grocery	400	New York	MU
Engineering - Software	20	California	SU
Technology - Accounting	18	Texas	SU

Results

Overall, participants generally had a consistent understanding of the survey questions and were able to answer all the questions. There were some issues where additional clarifying text is needed due to a few areas of confusion. This occurred for the questions listed below. We provided details on observed difficulty with these questions and offer a recommended change later in this memo.

• Q1 – add clarifying wording

• Q4 – add a response option and clarifying text

• Q5 – add clarifying text

• Q6 – add clarifying text

• Q8 – add clarifying text

• Q19 – add definition, add abbreviation to tool tip

• Q10 – add response option (Don’t know)

• Q20 – add clarifying text

There was an issue observed across the questionnaire that is important to note, as it is relevant toward how the results are interpreted. Throughout the questionnaire, there are two general references to the coronavirus pandemic that varies between questions.

• Since the start of the pandemic

• Because of the pandemic

The first reference is to a period of time, while the second is a cause of an action or behavior. Several participants noted that many business decisions asked about in this questionnaire occurred after the start of the pandemic, but that was not the impetus for the business decision – it just happened to correlate with that time. Participants shared that many decisions are made well in advance and were already planned before the coronavirus pandemic. It would be an incorrect assumption to infer that these changes occurred as a result of the pandemic. Several participants made a point to make that clarification.

Question by Question Findings

The Bureau of Labor Statistics (BLS) is looking to understand the impact that the coronavirus pandemic (also known as COVID-19) has had on your business and the U.S. economy. We appreciate your response to this important and brief survey.

This survey is concerned with how the Coronavirus pandemic has affected this business location. Please answer the following questions about this business location’s experience since the start of the Coronavirus pandemic of about March 1, 2020.

Instructions

Finding: Participants generally found the introduction to be clear and understood this to reference a specific business location. A couple participants did note that they would need the business location to be specified since they have multiple locations. One participant noted that they have many staff that work off-site – that is staff report to and work at a contractor’s location. This did not cause confusion with what business location was referenced, but whether to include those employees as a part of the selected business location.

Recommendation: No change is recommended for the introduction. For the one participant who noted ambiguity over whether to include off-site employees working at a contract site, these employees are required to follow practices and rules of the contractor establishment.

Q1. In a typical week, approximately what percent of employees at this business location currently telework the following amounts?

Enter whole numbers only and zero or blank if none. Total should add to 100 percent.

Telework all the time %

Telework some of the time, but not all %

Telework rarely or never %

Finding: Participants reported having a clear understanding of what this question was asking, but some difficulty with how to respond. Several participants stated they provided approximate percentages, with one stating they would have to go to human resources for an exact number. Response difficulties were related to the format, where a couple participants for small establishments provided counts of employees rather than percentages. These participants noted it would be helpful to have an indicator that percentages are required.

Participants in Hospital, Retail – Grocery and Consulting-Defense industries (e.g., those with essential workers) reported that most staff did not telework as their work could not be done remotely. For example, caring for patients, grocery store employees, or government contractors working on restricted activities (e.g., classified).

A couple participants have staff that are always remote, or work off-site. For example, an employee who works from another state, or staff who work on contracts that require work be performed only at the contracted work site. Participants were not clear whether they should be included as employed at the business location.

During the interviews, it was frequently required to remind participants the question was asking about “this business location” which resulted in adding this language to the question.

Recommendation: To address confusion about what employees to include, we recommend adding a Tool Tip to aid understanding:

Tool Tip: Include full-time and part-time employees; employees who are contracted to work at other locations for other companies; employees who have always been remote, but not at another business location; and temporary or seasonal employees. Do not include contract or subcontract workers. (Adapted from SOII).

Add “at this business location” to the end of the question text to help remind participants, especially multi-unit establishments of who they should be reporting for. A percent indicator should be added to clarify how participants should be reporting and wording to the instruction for clarity.

Q2. Since the start of the pandemic, did this business location increase telework for some or all employees?

o Yes

o No

Finding: All participants answered yes to this question. For participants with essential workers, or that conducted work that could not be done remotely, they had some staff who did telework. This was usually corporate staff or non-essential employees. All participants mentioned that these changes occurred at the start of the pandemic, increasing several weeks after the pandemic.

One participant (Consulting – Defense industry) noted that for staff that could not telework, they implemented a program where one-half the staff worked alternate weeks, with alternating weeks off opposite from the other half of staff. This was made possible through the CARES Act and helped to reduce the number of staff in the office at one time.

Recommendation: No changes are recommended for this question.

Q3. Do you expect increases in telework for some or all employees to continue after the pandemic?

o Yes, some or all changes will continue

o No

o There was no telework increase at this location

Finding: There were a mix of responses to this question, with four reporting “Yes, some or all” and two reporting “No.” All participants reported this question was clear, but there were two interesting observations when participants were asked to explain how they answered.

One participant reported “No”, but said that conversations were happening with management about this and a decision has not yet been made. For recruiting potential employees, telework is a good recruitment option for this business to consider.

One participant reported “Yes, some or all”, saying that staff are not required to come back, but many are choosing to do so.

For essential employees, there was no change in their status, but participants were clear in thinking about those who were able to telework.

Recommendation: No changes are suggested for this question, but it is clear that for establishments where a decision has not been made, “No” may be the response selected.

Q4. Since the start of the pandemic, has this business location STARTED any of the following employee flexibilities? (mark all that apply).

 Flexible or staggered work hours

 Compressed or alternative work schedule

 Voluntary reduction in hours worked (change to part-time or reduced hours)

 Job-sharing (two employees split hours/tasks of a full-time job)

 Paid leave for dependent care (additional paid leave to employees with dependent care responsibilities due to the pandemic)

 None of the above

Finding: Participants generally understood what this question was asking with several participants stating this question was clear. There were some observed difficulties with paid leave and state mandated changes that were not reported (not captured by current response options). All but one participant thought about changes that started as a result of the pandemic. One participant wanted to put the company in a good light and reported “Flexible or staggered work hours”, even though this was in place before the coronavirus pandemic.

There were two issues with the response categories for these questions that lead to flexibilities not being reported. Two participants reported there were state mandated sick leave increases (California and New York) that resulted in changes to their business’ sick leave policies. This was not reported as there is not a response option for this. One participant described a flexibility where staff were given additional vacation pay, but it was not called vacation pay and could be used for dependent care. This participant did not include this and reported “None of the above” instead.

Recommendation: Since participants may want to engage in socially desirable reporting, we recommend stressing “started” in the question text, or using different wording “newly implemented” (in place of “started”) to stress that only changes resulting since the coronavirus pandemic should be reported.

To address the issue of state mandated sick leave, add a response option for this:

• Increased sick leave, or state-mandated leave increases

We do not recommend any changes to address the atypical failure to report additional paid leave. As described by the participant this fits the definition provided in the response category. It may be helpful to clarify the parenthetical text to “additional paid leave of any type to employees…”.

Q5. Because of the pandemic, does this business location require some or all employees to routinely wear a face covering or any protective gear when they are on-site?

o Yes

o No

Finding: Most participants understood this question, with some stating this question is very clear, but there were indications of confusion over the reference period. Two participants did note that face coverings are not required in individual offices, but are required in common areas. This led to different responses from each (one “Yes” and one “No”).

There were some interesting reasons given for “Yes” responses (all participant’s reported “Yes”)

• One participant stated that insurance mandates this for the company or the company will be liable.

• One participant stated this was required starting in March, 2020, but discontinued the requirement after Labor Day of 2020. This participant still reported “Yes”.

• One participant stated the business would be discussing this issue on June 15^th, so the requirement may change.

As noted, some of the responses above indicate that some participants are reporting for “ever” versus “currently”. A participant who reported “No” stated the question appears to be asking about current requirements, but this was not clear. From the question text, it is unclear which is desired.

Recommendation: Participants appear to be interpreting this question as whether masks were ‘ever’ required. To be certain, this should be clarified. If the ‘current’ business activity is required, we recommend adding this to the question text. Because requirements may vary within areas of an establishment, we recommend adding “in any part of the building”.

• Because of the pandemic, does this business location currently require some or all employees to routinely wear a face covering or any protective gear when they are anywhere on-site in any part of the building?

• Because of the pandemic, did this business location ever require some or all employees to routinely wear a face covering or any protective gear when they are on-site in any part of the building?

Q6. Does this business location require employees working on-site to have a temperature screening prior to entering their place of work?

o Yes

o No

Finding: All participants described this question as easy to understand and all reported “No”. Two participants reported the establishment did this early during the coronavirus pandemic, but they no longer do any temperature screening. All but one participant interpreted this question as asking about the establishment’s current practice. One participant reported they were thinking about “during the pandemic” but the establishment never did any temperature screenings.

Recommendation: From the question text, it is not clear what the reference period should be. This should be clarified using “ever” or “currently”.

• Does this business location currently require employees working on-site to have a temperature screening prior to entering their place of work?

• Did this business location ever require employees working on-site to have a temperature screening prior to entering their place of work?

Q7. Has this business location required some or all employees to get a COVID-19 vaccination before coming to work on-site?

o Yes

o No

Finding: Participants did not appear or report any difficulty with this question, with several noting the question was clear. One participant did note confusion between their business and the business location – considering the requirements of the building they lease space from that they would have to follow. We do not recommend a change because of this as it is an idiosyncratic distinction and did not affect the response (if there were independent building requirements, it would be up to the business to enforce with employees).

Several participants offered that it is difficult to require this due to exemptions (e.g., religious); or concerns with the establishment’s insurance carrier (covering the establishment), stating that employees cannot be required; or concerns with HIPPA regulations. One participant noted that they helped employees get vaccinated but did not require vaccinations.

Recommendation: No changes are recommended for this question.

Q8. Has this business location offered any employees a financial incentive or paid time off to get a COVID-19 vaccination?

o Yes

o No

Finding: Only one participant reported “Yes” to this question. However, two participants, reporting “No”, indicated that employees can “stay on the clock” or allowing employees to use “work time” to get vaccinated. These participants stated they would have said “Yes” if this was specified in the question wording. Participants appear to be viewing this as additional time offered, separate from their workday in some cases specified as “PTO” (Paid Time Off).

Recommendation: If regular work hours that do not count as paid leave or “on the clock time” should be considered for this question, a revision is necessary.

• Has this business location offered any employees a financial incentive, paid time off, or permitted employees to use work time (e.g., remain on the clock) to get a COVID-19 vaccination?

Q9. Since the start of the pandemic, has this business location changed its square footage of physical space?

o Decreased

o Stayed about the same

o Increased

Finding: All participants reported “Stayed about the same” and probing indicated that physical space changes did not occur. One participant noted that their business’ lease agreements are long-term (e.g., 5-7 years) and changes require significant planning and time. Generally, this question was understood by participants, however one participant did report some confusion stating they changed how they use the space – redistributed desks for social distancing. This participant still appropriately reported “Stayed about the same”.

Recommendation: No changes are recommended for this question.

Q10. In the next 12 months, how do you expect this business location will change its square footage of physical space?

o Decrease

o Stay about the same

o Increase

Finding: Participants generally did not have any difficulty understanding this question as it is similar to the previous with a future thinking context. However, participants did express difficulty with being able to answer as many reported they do not know what may be planned or that it depends on how many employees plan to come back into the office. Participants generally reported their expectations. Two participants noted that with long-term lease agreements, the time frame may be too narrow. Their leases generally run 5-7 years so they would have to wait until their lease was expiring.

Recommendation: While many participants expressed that they were “not sure” what the establishment will do in the future, all were able to provide an answer based on their expectation. This may not occur in a production environment and many participants may leave the question blank if allowed. If skipping the question is possible, we recommend adding a “Don’t know” or “Not sure” response option since it would be a meaningful value.

Because of the effect of long-term lease agreements, many businesses that are planning to reduce their space may not report this as it will be outside the 12-month reference period. We do not recommend a change, but include it for consideration when analyzing data.

Q11. Since the start of the pandemic, has this establishment relocated?

o Yes, moved within the same city or county

o Yes, moved to a different city or county, but within the same state

o Yes, moved to a different state

o No, did not move since the start of the pandemic

Finding: Participants uniformly found this question to be clear or stated that it was straightforward. All but one participant reported “No”. The one participant reporting “Yes, moved within the same city or county”, said this was done at the start of the pandemic, but not because of the pandemic.

Recommendation: No changes are recommended for this question.

Q12. In the next 12 months, do you expect this establishment will relocate?

o Yes, moved within the same city or county

o Yes, moved to a different city or county, but within the same state

o Yes, moved to a different state

o No, relocation is not planned within the next 12 months

Finding: This question was also uniformly understood as it is similar to the previous question, with a future thinking context. As with previous questions related to moving or physical space changes, there was mention that this is influenced by lease agreements.

Recommendation: No changes are recommended for this question.

Q13. At any time during the pandemic, did this business location start or increase using ANY of the following types of workers? (mark all that apply)

 Independent contractors, freelancers, or consultants

 Temporary help agency workers

 Companies that provide contractors or subcontractors

 Online platform companies that arrange assignments for workers through an app and collect a commission from your establishment for each task/job workers do

 Did not start or increase use any of these

Finding: Participants generally did not have difficulty with this question. Three participants reported they did not use any of these, one selected “Independent contractor” where one was added to staff during the pandemic, but the participant would not classify this as a result of the pandemic. Two participants reported “Temporary help agency workers”, both in the healthcare industry (one was a large addition of nurses; the other temporary workers to help with temperature screening).

Many participants specifically stated the question was clear, understandable, or the response categories were “distinct”. There were two participants stating they were not familiar with the fourth option (Online platform companies) and were not certain what this would include. One participant reported the wording for the third option was confusing (they did not know what was meant) as in their business, they are usually a prime contractor.

Recommendation: Where there was some lack of familiarity with some worker groups, participants felt clear this was not used and a reason for the unfamiliarity. As a result, no changes are recommended for this question. The question appears to be clear with respect to whether or not the worker types are used.

Q14. When the pandemic is over, do you expect that this business location will use any of the following types of workers? (mark all that apply)

 Independent contractors, freelancers, or consultants

 Temporary help agency workers

 Companies that provide contractors or subcontractors

 Online platform companies that arrange assignments for workers through an app and collect a commission from your establishment for each task/job workers do

 No plans to use any of these

Finding: As with the previous questions, most participants found this question to continue to be clear and understand it was asking about practices for the next 12 months. Two participants did note that their planned use would not be a change resulting from the pandemic ending, but rather a continuation of business as normal – any use of these worker types would not be a result of the pandemic ending.

One participant noted that the phrase “When the pandemic is over” implies consideration of state and local government guidelines or restrictions related to safety or coronavirus pandemic related measures.

Recommendation: No changes are recommended for this question. However, caution will need to be taken when interpreting responses as some participants noted these uses were not increasing or a result of changes due to the end of the coronavirus pandemic.

Q15. Does this business location use of any of the following types of automation? (mark all that apply)

 Self-service kiosks (including kiosks to order and pay for goods)

 Voice-recognition-based customer service/automated online chats with customers

 Automated document analysis and review

 Industrial robots for building maintenance (including daily cleaning or disinfecting)

 Industrial robots for assembling goods (including robots that weld, and pick-and-place robots to assemble, select parts, or inspect products)

 Industrial robots or management systems for packing goods for shipment

 Automated provision of physical medical care (such as drawing blood) and physical rehabilitation

 Not using any of the above

Finding: All participants reported understanding what this question was asking, with only one participant reporting using any of the types listed. The one participant, representing a hospital, reported using “Self-service kiosks”. This participant at a later question also recalled that they introduced remote document scanning system and would consider this to fit under Option 3, however they did not include it as this question. The same participant noted they use “Telehealth” and considered this a form of automation that would not be captured by the current question.

Another participant in the Retail – Grocery industry said the company looked into automated kiosk (automated check-out) and automated cleaning robots. However, the automated kiosk was determined to be counter to the company’s culture and the automated cleaning robots were determined to be too expensive.

Recommendation: As with other questions, there was a lack of familiarity with some response options due to lack of relevance to their business or industry. This did not appear to affect reporting or understanding of what was asked. The only recommendation is to consider adding an open-ended option to capture automation types that may not be inclusive of the current response options.

Q16. Since the start of the pandemic, did this business location start or increase its use of any of the types of automation listed in the previous question (Question 15)?

o Yes

o No

Finding: Participants clearly understood this question. One participant noted that this is “self-evident” that since they were not using, they could not start or increase. Another participant stated that we are asking if use of any of the automations listed were started or increased after the start of the coronavirus pandemic.

Only one participant reported “Yes” to this question, which was an automation not reported in the previous question.

Recommendation: While one participant made a recall error affecting the prior question, this question helped the participant with recall. No changes are recommended to this question.

As a result of the pandemic, some businesses have had to temporarily suspend drug or alcohol testing of new applicants and current employees.

Q17. Is this business location currently drug testing or alcohol testing new applicants or current employees?

o Yes

o No

Finding: Participants were uniform in reporting a clear understanding of this question. Some participants noted that legal requirements did not change, so they were still doing any required testing. One participant reported they would only report “No” if the survey is confidential, otherwise they would misreport “Yes”.

Recommendation: No changes are recommended to this question, but it will be important to highlight confidentiality of the survey – if not already part of instrument, add to the introduction.

Q18. Since the start of the pandemic, has this business location reduced or delayed drug testing or alcohol testing?

o Yes

o No

Finding: There was no reported confusion with this question and participants demonstrated a clear understanding of “reduced or delayed testing”. All participants reported “No”, with some noting that legal requirements did not change. One participant did note that it was more difficult to get testing, but it was not reduced or delayed by the company.

Recommendation: No changes to this question are recommended.

Q19. Since January 1, 2021, did this business location receive a federal or state government Coronavirus-related loan or grant tied to re-hiring or maintaining employees on the payroll?

o Yes

o No

o Don’t know

Finding: Participants understood the types of loans referenced with several participants mentioning “PPP”, “EIDL loan”, and “CARES Act”. There was variety in responses with three participants answering “No”, one answering “Don’t know”, and two answering “Yes”. A couple participants answering “No” noted the reference period of January 1, 2021, however, the two participants answering “Yes” included loans that were received in the previous year.

The participant reporting “Don’t know” stated they would need to contact the finance office, which would be a quick email.

One participant noted that in the Tool Tip instructions, they recognize the loan as “PPP” and not the full text and suggest this may be easier to recognize if included.

Recommendation: There is evidence that some participants appear to be focusing on recognition of the loan type and not when it was obtained. To address this, we recommend added an instruction that is always visible below the question text:

Do not include PPP, EIDL, or CARES Act loans received BEFORE January 1, 2021.

Add (PPP) in the Tool Tip after “Paycheck Protection Program” as this may be more recognizable.

Tool Tip: Federal and state governments have created loans and grants to help businesses that have been hurt by the Coronavirus pandemic. For example, the Paycheck Protection Program (PPP) or Economic Injury Disaster Loans (EIDL) administered by the Small Business Administration.

Q20. If this business received any type of coronavirus-related loan since the onset of the pandemic, has this loan been converted to a grant?

o Yes

o No

o Don’t know

Finding: There was some difficulty with understanding what this question was asking by a few participants. For participants whose business never received any type of loan this was clear as the default was “No” since it did not apply to them. One of these participants noted they would like a “Not Applicable” response option feeling it would be more appropriate.

The difficulty was with the terminology of converting the loan to a grant. Participants were not familiar with this and instead said the loan was “forgiven”. One participant answered “Don’t know” because of this difficulty, another answered “Yes” assuming that is what was meant. Another participant whose business did not have any loans assumed this meant the loan was forgiven.

Participants were clear in thinking about any loans and not just loans received since January 1, 2021.

Recommendation: It is clear that relying on participant understanding of how loans were changed, so the business did not have to pay them back, by converting to a grant will lead to misreporting. While forgiving the loan may not technically be accurate to how the loan was modified, participants understand it in this way. Rather than directly using “forgiven” terminology, we recommend adding parenthetical text to reference this interpretation as shown below.

• If this business received any type of coronavirus-related loan since the onset of the pandemic, has this loan been converted to a grant (this generally means the loan was forgiven)?

General Comments About the Survey

At the conclusion of the cognitive interview, participants were asked a few general questions about improvements the survey. There were very few additional comments, but the most notable are provided below.

• An essential business noted that there were not any questions about what these businesses did for employees, for example offering: appreciation pay; increased hourly pay; bonuses; or bringing food trucks on-site.

• A California based business stated that “California is different” and there may be state-specific activities that are missed by these questions. One example is California offering 80 hours of “COVID leave pay” that may not be offered by other states.

14