Initial Cognitive Report on the 2022 BRS Pilot Test

2022 Business Response Survey (BRS)

Initial Cognitive Report on 2022 BRS Pilot Test

July 8, 2022

Background

For the 2020 and 2021 BRS, cognitive review consisted of expert survey review and 9 cognitive telephone interviews. This expedited cognitive review period provided valuable feedback and survey improvements but lacked live respondent use of the survey instrument. In 2021, some respondents had issues with question 3 on that survey and required follow up. For the 2022 BRS, BLS has greatly expanded cognitive review to include setting up a production version of the survey on the web system, email solicitation for pilot respondents, respondents filling out the survey questions in a production environment and adding 3 cognitive questions to gauge survey difficulty and respondent reaction. Using data collected from the pilot, BLS has been able to evaluate response rates, respondent willingness and ability to accurately answer questions, and usefulness of the responses. BLS also conducted follow up interviews to further understand how respondents were interpreting questions and providing responses.

2022 BRS Pilot Test Overview

Two emails were sent to 2,001 respondents for a cognitive pilot test of the 2022 BRS. The first email went out on June 7, 2022, with a follow up on June 21, 2022. There were 560 completed responses as of July 8, 2022. Respondents answered the questionnaire online and a select, representative group received follow-up phone interviews with BLS cognitive expert staff.

Undeliverable	Responses	Incomplete
54	560	17
2.7%	28%	3% of responses

The individual question response rates range from 95-99% with a response rate of 97% for question 1 and a response rate of 95% for the last questions 25. There was limited drop off from the first through final question and no individual question experienced high rates of respondent skips or refusals.

Respondents were also asked how difficult the survey was and why it was difficult. Most respondents did not find this survey to be difficult and the top reasons for difficulty were the need to consult with records or other company employees to answer questions. This is in line with expectations.

Q23 How easy or difficult was it to complete this survey?
Very easy	63.97%
Somewhat easy	22.35%
neither easy nor difficult	9.44%
Somewhat difficult	3.47%
Very difficult	0.77%

Q24 For what reasons was this survey difficult?
Consult records	31.91%
Consult with other company employees	14.89%
Questions required calculations	9.57%
Some questions were not clear	21.28%
Some other reason	22.34%

• Overall response patterns make sense and are in line with expectations

• Average time of completion was 4 ½ minutes, with a range of 25 minutes to 1 minute.

• Q25 - 86.2% think the survey is at least somewhat relevant for understanding current change in the labor market

• The “soft” skip patterns with instructions-only to “Skip to question X” are working well. Some respondents are still filling in questions that they could skip. Respondents are filling in these responses correctly (ex. They have no telework and should skip the telework percentages, but they fill that in as 100% No Telework). BLS plans to implement more intuitive skip pattern visuals in the production version.

• Respondents are answering questions correctly overall in terms of the numeric responses (ex. Overall hires > Hires that took <30 days)

Pilot test - Follow-up interviews

A total of 16 respondents from the cognitive test volunteered to participate in a follow-up debriefing interview about their experiences completing the survey. Respondents were from a mix of industries; 14 had hired employees in April of 2022 and 7 had employees that work remotely or telework. The two respondents who did not hire in April of 2022 were asked general probes to get at their comprehension of the hiring questions (e.g., how easy or difficult it would be to answer on a future survey, and how they would arrive at their answers). All respondents had rated the survey as “somewhat easy” or “very easy” to complete.

A summary of the main findings and recommendations from these interviews are outlined below:

• Introduction text. Respondents thought the introduction was clear and easy to understand, and most had no problems determining which location to report for. No changes are recommended.

Telework section

• Q1. Respondents generally understood the question about whether the location has employees who currently telework in any amount. A couple respondents displayed minor issues with the term “telework.” For example, one respondent thought the term was related to telemarketing only, and another felt the term is outdated and favored “remote” work. Several respondents noted that staff do some work tasks at home occasionally, such as payroll, responding to emails, or billing. For most respondents, this type of telework was not formally recorded or required by the company. Two respondents noted they allow ad-hoc telework if employees do not feel well, but that this is rare.

Recommendation: consider changing the question to “Do any employees at this location CURRENTLY telework or work from home in any amount?” Although the text “work at home” is in the telework definition provided, not all respondents will read this carefully or remember it while answering. Including it in the question may resonate more with respondents and help clarify the meaning. If this change is made, it could also apply to other questions that ask about telework as needed.

• Q2. The question on what percent of employees currently telework all the time, some of the time, or rarely/never, was clear and well understood by respondents. Most respondents could easily pull this information from company records or did some quick calculations to determine what percent was appropriate for each category. A couple respondents noted they do not formally track this information in company records, so they inferred their answers by job titles and position locations, estimating who typically teleworks or comes on-site. No question changes are recommended.

• Q3. The question about whether telework would increase, decrease, or stay the same in the next 6 months was well-understood and clear to respondents. Most respondents found this question easy to answer since they did not expect the current policy to change or because they would be gradually phasing employees to on-site work in the next 6 months. No question changes are recommended.

• Q4. The question about whether employees teleworked before the pandemic began was generally well-understood by most respondents and easy to answer since several had expanded telework during the pandemic. For businesses where telework was not possible, the question felt a little redundant to respondents, but they were able to provide an answer easily. No question changes are recommended.

Hiring section

• Q5. The question on whether respondents hired any new employees in April of 2022 was generally well understood and clear to respondents. Most respondents reported that they used records to answer the question, but some knew off the top of their head. Most respondents understood the italicized instructions to “include employees that were hired, even if they have not yet formally started,” to mean that job candidates were extended a job offer but their start date had not yet come, or they were not officially on payroll yet. A few respondents remarked that only employees that actually started at the job should be included because a lot of new hires “ghost them,” or just never show up on their first day. At least two respondents noted they excluded these new hires from their answers.

A few respondents questioned the short reference period or “snapshot” of just April 2022; they felt focusing only on hiring during this month was not representative of the hiring the company does throughout the year. One respondent suggested a three-month window would have been more representative. At least two respondents also seemed to telescope their responses – bringing in hires that occurred just before April of 2022. This type of issue may occur more at small or medium sized establishments where respondents may be more likely to rely on memory versus records to arrive at their responses.

Recommendations:

• Consider adding to the question instructions, “Include employees that were hired, even if they have not yet formally started or have left the position since [reference period /April 2022].” This instruction would help respondents know BLS want to collect all hires, even if they didn’t show up on their first day or left the company for any reason in the same month.

• Consider expanding the reference period beyond a single month if capturing more representative information about hiring is needed.

• Q6. The question on how many new employees the location hired was well understood and clear to respondents. Most respondents checked payroll records or knew this information since they are involved in hiring. One respondent noted their company records only shows the start date, which may not be the actual date employees were hired. No question changes are recommended.

• Q7. The question on how many new employees will telework all the time was clear and well understood. Most respondents checked payroll records or knew this information since they are involved in hiring. No question changes are recommended.

• Q8. The question on ways the business location tried to attract more applicants was generally well understood and clear to respondents. A couple of respondents noted if the reference period (April 2022) were expanded a couple of months they would have included additional strategies to attract candidates in their responses. Expanding the reference period might capture additional responses that would not otherwise be selected when collecting data for only one month. A few respondents mentioned it would have been helpful to have an Other, specify option to indicate other ways they attracted candidates. A few respondents also mentioned that employees want expanded telework or remote work opportunities and that this might be a response option to consider adding.

Recommendations:

• Consider expanding the reference period beyond a single month if capturing more representative information about ways of attracting candidates is desired.

• Consider adding a new response option, “Expanded telework or remote work.”

• Consider adding an “Other” or “Other, specify” response option if capturing additional ways of attracting candidates would be beneficial.

• Q9/Q10. The questions on whether any of the newly hired employees in April 2022 were for positions open more than 30 days (and how many), were well understood and clear to respondents. However, there was some ambiguity in this question for a few respondents. For example, several businesses had a high level of turnover within the 30-day period, and are constantly recruiting more employees, so it was hard to say whether any one particular position had been open more than 30 days. Two respondents noted that the positions are “fungible” in that if they interview a candidate for an open position but realize their skillset is better suited to a different area, they might create a new job for that individual. Because respondents understood the question well and were able to arrive at accurate responses, no changes are recommended, but note these potential sources of measurement error.

• Q11. The question on positions that took more than 30 days to fill requiring a license or certificate was generally well understood and clear to respondents. The question was easy for most respondents to answer, examples included engineering certificates, food handling certificates, and a mortgage license. Several respondents remarked some of these licenses or certificates are preferred or beneficial, but not required. Because of this, underlining the word “required” might help respondents distinguish between a preference versus requirement. No other question changes are recommended.

Recommendation: Underline the word “required” in the question.

• Q12. The question on positions that took more than 30 days to fill requiring a BA degree or higher was generally well understood and clear to respondents. Several respondents remarked that a BA is preferred, but not a requirement. Underlining the word “required” might help respondents distinguish between a preference versus requirement. No other question changes are recommended.

Recommendation: Underline the word “required” in the question.

• Q13. The question on whether the newly hired employees were for positions that had been open for 30 days or less was generally well understood and clear to respondents. No question changes are recommended.

• Q14. The question on positions that took 30 days or less to fill and how many required a license or certificate was clear and well understood, and no problems were observed. However, underlining the word “required” might help respondents distinguish between a preference versus requirement. No other question changes are recommended.

Recommendation: Underline the word “required” in the question.

• Q15. The question on positions that 30 days or less to fill and how many required a BA degree or higher was clear and well understood, and no problems were observed. However, underlining the word “required” might help respondents distinguish between a preference versus requirement. No other question changes are recommended.

Recommendation: Underline the word “required” in the question.

Vacancies section

• Transition to vacancy section. Respondents were asked for feedback on the transition text to the section on vacancies. Most respondents seemed to understand the language, understood what “open positions” referred to and found the text helpful. No changes are recommended.

• Q16. The question on whether the location has any open positions was well understood and clear to respondents. Most respondents knew they usually have openings and verified this by looking at company records or asking another co-worker to confirm. No question changes are recommended.

• Q17. The question on how many open positions the location currently has was well understood and easy for respondents to answer. Most indicated these are positions that are currently being recruited and advertised for. No question changes are recommended.

• Q18. The question on how many open positions are eligible for full-time telework was clear and well understood by respondents. It was also easy for most respondents to answer. A few respondents noted they do not track this information in company records, so they had to infer the answer based on the job titles of the open positions, or by the team or location they work for. No question changes are recommended.

• Q19. The question on how many of the positions have been open for more than 30 days without being filled was generally well understood for most respondents. Similar to the issues observed in Q9/Q10, some respondents had difficulty answering due to the large amount of turnover at the company, particularly if multiple positions turned over within the 30-day period. However, all respondents were able to provide an answer. No question changes are recommended.

• Q20. The question on whether the location had any open positions in the last year was generally easy for respondents to understand, and most thought about the previous 12 months as a reference period. However, one respondent said he thought of “only the current year, 2022” and gave his answer in terms of open positions since January 2022. In addition, several respondents noted that this question was redundant with the prior question on whether they did any hiring in April of 2022 (Q5) and suggested that only respondents who answered “no” to Q5 should question should receive Q20.

Recommendations:

• Consider changing the question wording to define the reference period (e.g., Has this location had any open positions in the last 12 months?”).

• If possible, implement a skip pattern so that only respondents who did not hire in the reference period (e.g., April 2022) are asked Q20.

• Q21/Q22. The questions asking respondents to report ways that they advertised positions requiring a BA degree or not was generally easy for respondents to understand and the difference between the two questions was obvious. When asked if they would add items to the list, a few respondents mentioned contacting temporary staffing agencies. A couple of respondents also mentioned separating out “Word of mouth” and “Social media” from response option number 5; some businesses only did one or the other and thought of them differently.

Recommendations:

• Consider changing response option 4 to “Recruitment firm or temporary staffing agency.”

• Consider separating out “Word of mouth” and “Social media” from response option 5 if separate estimates for these types of advertising is needed.

General feedback

• As mentioned above, many respondents expressed recent difficulties in identifying applicants, hiring, or retaining new hires, and how the COVID-19 pandemic may have affected these issues. Due to this, a new probe was added to the debriefing protocol to allow respondents to provide open-ended comments on this, which respondents appreciated during the interviews. This reinforces the need to add an optional open-ended item where respondents can provide this information.

• Most respondents identified themselves as the best person to answer these types of questions in the future. They were usually company owners, managers, CFOs, or HR staff.

• Respondents mentioned the questions that required them to look at records included the items on hiring (e.g., how many people were hired in the reference month), how long the positions had been open for, what percent of staff telework or work remotely, or educational requirements for positions. They usually looked at company records, including payroll, timekeeping records, recruitment software, or checked with other staff members to verify these answers.

Changes Resulting from the Cognitive Test

1. Telework Questions – No changes

BLS reviewed the telework questions text and decided to leave the questions in their original format. As noted, the telework description includes “work from home” and telework as a concept is pretty well known and understood to be “paid work,” as in employees are paid to work from an alternate location. It’s possible that by including ‘work from home’ in the body of the question we’d be deviating from the paid work concept and would unnecessarily open up the telework questions to be answered affirmatively, for every employee that might work 9-5 in the office and occasionally check their email from home. BLS cognitive lab agreed with this no change decision.

2. Question 6 - How many new hires in April 2022?

BLS looked at a ratio of average monthly employment to the number of new hires reported in April. Most respondents reported values that were in line with expectation. About 1% of respondents reported values that were higher than expected. One reason for this could be that they are providing a firm-level responses. BLS is adding text to the section introduction and question 6 reiterating “this location only, not including other locations of this company.”

BLS has concluded that asking for hires data for more than one month would be overly burdensome for large establishments having to review records to obtain the data and could also impact the smaller establishments relying on recall of the prior month to answer the questions about hires. The reference for hires will be July 2022.

To clear up any confusion concerning hires that that may leave a position shortly after being hired, BLS is updating the text to “Include employees that were hired, even if they have not yet formally started, or have left the position since being hired.

3. Question 8 – What did businesses do to attract more applicants

Added “Expanded telework or remote work” to response options.

4. Questions 11/12, 14/15 - Licensing and degree requirements

These questions ask about professional license or bachelor’s degree requirements for new hires that took both >30 and <30 days to hire. The original question gave respondents 3 options to select: All, Some or None. Initial results showed that most respondents with >5 hires are selecting “Some” for both >30 and <30 days, yielding results that will not be meaningful in making comparisons or distinctions.

These questions were updated to ask for approximate percentages, rather than the 3-level Likert scale options. The follow up email blast tested respondent ability to provide approximate percentages. These responses to updated online questionnaire revealed no difficulty in respondents being able to provide the approximate percentages.

Pending technical updates to the system, the word required will be underlined, as recommended.

5. Question 20 – Hires in the last year

BLS will change the question to hires in the last 12 months. Also changed reference in questions 21 and 22 to last 12 months.

6. Questions 21 and 22 – Response options.

No changes to the response options listed. Temporary help agency employees are not employees of the surveyed establishment. Hiring a temp would be in place of hiring at the establishment. Social media and word-of-mouth are generally considered the same. BLS cognitive lab agreed with this no change decision.

7. Option for more feedback

During cognitive interviews, many respondents reported that they wanted to provide additional information about the difficulties that they are having with hiring. BLS also received a detailed email back from a respondent with additional information about difficulties hiring. For this reason, the decision was made to add a line to the Thank-You page with a link for respondents to be able to provide additional feedback, if they so desire. BLS cognitive lab agreed with this change.

Attachment: Updated 2023 BRS screenshots

7