SBPS Phase 6 Cognitive Pretesting

Cognitive Interviewing Protocol

Small Business Pulse Survey – Phase 6

4.30.21 draft

Interviewer notes are in red.

Research Questions to Address:

• Are respondents able to answer the questions as intended?

• Are there any items that cause respondents problems?

  • What can be done to fix those problems?

• Estimate of average time to complete?

Introduction:

• Hello. Thank you for your time today. We are looking to obtain feedback on a revised version of a short survey that has been designed to learn more about the effects of the Coronavirus pandemic on small businesses.

• Our purpose for speaking with you today is to see how respondents answer the survey questions and to evaluate how the questions work or don’t work.

• We are not testing you– we only want to evaluate the questionnaire.

• Today, we will have you complete the short questionnaire, then we will review some of the questions with you to get your feedback.

• Email link to Business Pulse survey to respondent.

• I just sent you an email. Can you please click on the link to the Qualtrics survey and read over the first page? This study is being conducted under the authority of Title 13 USC. We plan to use your feedback to improve the survey questions and make sure they make sense to respondents like you. Do you agree to participate? Thank you. Please click on the checkbox at the bottom of the screen, but do not advance the screen just yet.

Before the Questionnaire:

• Before we begin, could you tell me a little bit about your business -- what types of goods or services does this business provide?

• Thank you. And what is your role in the business? (What kind of responsibilities do you have?)

• Since they were sampled before, should we ask if they ever responded to the survey in the past? Ask if they remember getting a request for this?

Have Respondent Complete the Questionnaire:

Now I’d like for you to go through the survey question by question. Please tell me your answer to each question as you go along. After you are done, we will go back and discuss some of the questions in more detail.

Document responses to each question and the time respondent took to complete survey.

___ minutes

Review the Questionnaire:

Thank you for completing the survey. On your screen should be a summary of your responses. You can use this as we talk a little bit more about some of the questions you have just answered.

First, what is your overall impression of this survey?

Item 8 (Difficulties hiring paid employees)

Now let’s go back and look at Item 8.

Q8. In the last week, did this business have difficulties hiring paid employees?

Select only one:

• Yes

• No

• Not applicable

• Please tell me in your own words what this question is asking.

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• What does “difficulties hiring paid employees” mean to you? What did you think about when answering this question?

• Would “face challenges hiring paid employees” be more or less clear to you?

• What does ‘In the last week’ mean to you? What timeframe were you thinking about?

• Do you have any suggestions on how to improve this question?

Item 10 (Change in number of paid employees)

Next let’s look at Item 10.

Q10. The White House declared a national emergency because of the Coronavirus pandemic on March 13, 2020.

Compared to what was normal before March 13, 2020, how has the number of paid employees at this business changed?

Select only one:

• Large increase in paid employees

• Moderate increase in paid employees

• Little or no change in paid employees

• Moderate decrease in paid employees

• Large decrease in paid employees

• Please tell me in your own words what this question is asking.

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• What does “Compared to what was normal before March 13, 2020” mean to you? What time frame were you thinking of?

• Do you have seasonality in your business related to the number of paid employees? Were you taking this into account in answering this question? How?

• What do the different response options mean to you? For example, what would you consider a large increase as opposed to a moderate increase? How about a large decrease as opposed to a moderate decrease?

• Do you have any suggestions on how to improve this question?

Now I would like you to think about a slightly different version of this question.

Alt 10. Compared to what was normal before March 13, 2020, has the number of paid employees at this business changed?

Select only one:

• Yes, increased

• Yes, decreased

• No change

• What is your reaction to this version of the question?

• In your own words, what is the difference between the two versions?

• Do you prefer this version, or the original version?

Item 11 (Change in percent of paid employees working onsite)

Now on to Item 11.

Q11. Compared to what was normal before March 13, 2020, how has the percent of this business’s paid employees working onsite changed?

Select only one:

• Large increase in percent of employees working onsite

• Moderate increase in percent of employees working onsite

• Little or no change in percent of employees working onsite

• Moderate decrease in percent of employees working onsite

• Large decrease in percent of employees working onsite

• Please tell me in your own words what this question is asking.

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• What does “percent of employees working onsite” mean to you? Who did you include in your response? Did you exclude anyone?

• What time period were you thinking about?

• Do you have seasonality in your business related to employees working onsite? Were you taking this into account in answering this question? How?

• What do the different response options mean to you?

• Do you have any suggestions on how to improve this question?

Now I would like you to consider a slightly different version of this question.

Alt 11. Compared to what was normal before March 13, 2020, has the percent of this business’s paid employees working onsite changed?

Select only one:

• Yes, increased

• Yes, decreased

• No change

• What is your reaction to this version of the question?

• Do you prefer this version, or the original version?

Item 12 (Change in customer base)

Next let’s look at Item 12.

Q12. Compared to what was normal before March 13, 2020, how has this business’s customer base changed?

Select only one:

• Large increase in customer base

• Moderate increase in customer base

• Little or no change in customer base

• Moderate decrease in customer base

• Large decrease in customer base

• Please tell me in your own words what this question is asking.

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• What does “this business’s customer base” mean to you? Is it different – or the same – as the number of customers over this time period?

• What do the different response options mean to you? What would you consider a ‘large’ increase or decrease as opposed to a ‘moderate’ increase or decrease?

• What time period were you thinking about?

• Do you have seasonality in your business related to your customer base? Were you taking this into account in answering this question? How?

• Do you have any suggestions on how to improve this question?

Now I would like you to look at a slightly different version of this question.

Alt 12. Compared to what was normal before March 13, 2020, how has the demand for this business’s goods or services changed?

Select only one:

• Large increase in demand

• Moderate increase in demand

• Little or no change in demand

• Moderate decrease in demand

• Large decrease in demand

• What is your reaction to this version of the question?

• What does “demand for this business’s goods or services” mean to you?

• Does the rewording make it easier or harder for you to answer this question?

• Do you prefer this version, or the original version?

Item 13 (Change in prices for goods from suppliers)

Now Item 13.

Q13. Compared to what was normal before March 13, 2020, how have the prices this business pays for goods from suppliers changed?

Select only one:

• Large increase in prices

• Moderate increase in prices

• Little or no change in prices

• Moderate decrease in prices

• Large decrease in prices

• Please tell me in your own words what this question is asking.

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• What does “prices this business pays for goods from suppliers” mean to you? What specific goods were you thinking of? What types of suppliers?

• What do the different response options mean to you? What would you consider a ‘large’ increase or decrease as opposed to a ‘moderate’ increase or decrease?

• What time period were you thinking about?

• Do you have seasonality related to the prices this business pays for goods from supplier? Were you taking this into account in answering this question? How?

• Do you have any suggestions on how to improve this question?

Now I would like you to look at a slightly different version of this question.

Alt 13. Compared to what was normal before March 13, 2020, how have the prices this business pays for goods or services changed?

Select only one:

• Large increase in prices

• Moderate increase in prices

• Little or no change in prices

• Moderate decrease in prices

• Large decrease in prices

• What is your reaction to this version of the question?

• What does “prices this business pays for goods or services” mean to you? How does it differ from “prices this business pays for goods from suppliers”?

• What types of “goods or services” were you thinking of?

• Does the rewording make it easier or harder for you to answer this question?

• Do you prefer this version, or the original version?

Item 14 (Increased use of technology)

Next let’s look at Item 14.

Q14. Since March 13, 2020, has this business permanently increased its use of technology in order to reduce its number of paid employees?

Select only one:

• Yes

• No

• Please tell me in your own words what this question is asking.

• [If yes] Can you tell me a little about what you did?

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• What does “permanently increased its use of technology” mean to you? What did you include here, and what did you exclude, if anything?

• How would you answer if you were not sure if it will be temporary or permanent?

• How would you answer if the “increased use of technology” was not “in order to reduce the number of paid employees”?

• Do you have any suggestions on how to improve this question?

Now I would like you to consider a slightly different version of this question.

Alt 14. Since March 13, 2020, has this business permanently increased its use of technology resulting in a reduction in its number of paid employees?

Select only one:

• Yes

• No

• What is your reaction to this version of the question?

• Does the rewording “resulting in a reduction in its number of paid employees” make it easier or harder for you to answer this question?

• Do you prefer this version, or the original version?

Item 15 (Received financial assistance)

Ok, now let’s look at Item 15.

Q15. On December 27, 2020, the Consolidated Appropriations Act 2021 was enacted providing a second round of Coronavirus pandemic relief. Recently the American Rescue Plan Act of 2021 was enacted providing further Coronavirus pandemic relief.

Since December 27, 2020, has this business received financial assistance from any of the following sources?

Select all that apply:
• Paycheck Protection Program (PPP)
• Paycheck Protection Program (PPP) loan forgiveness

• Economic Injury Disaster Loans (EIDL)

• Restaurant Revitalization Fund

• Shuttered Venue Operators Grants
• This business has not received financial assistance from the above sources since December 27, 2020.

• Please tell me in your own words what this question is asking.

• How did you come up with your answer to this question? Would you need to reach out to anyone else in your firm to answer this question? Who?

• Are you familiar with each of the response options?

• What time frame were you thinking of when answering this question?

• Did you receive financial assistance from any of these sources prior to December 27?

• Do you have any suggestions on how to improve this question?

Item 19 (Describe permanent changes to operation due to Coronavirus pandemic)

Now let’s look at Item 19.

Q19. Please describe any permanent changes this business has made to its operations due to the Coronavirus pandemic.

(1000 characters)

• What is this question asking you, in your own words?

• What does “permanent changes this business has made to its operations” mean to you?

• What types of operations are you thinking are covered by this question?

• How would you answer if you were not sure if the changes will be temporary or permanent?

• Do you have any suggestions on how to improve this question?

Other Items

Are there any other questions you would like to go back and talk about?

(Review other items that raised concerns during survey completion.)

Wrap Up:

We are almost done. I just have a few more questions.

• Overall, how easy or difficult was it to answer these questions?

• Do you have any other comments or suggestions about anything we have discussed today?

That is all the questions I have for you today. Thank you very much for your time today. We really appreciate your help.

Document length of cognitive interview.

___ minutes

19