Cognitive Interviewing Protocol

Cognitive Interviewing Protocol

Proposed AI Supplement for the Business Trends and Outlooks Survey

12.23.24 draft

Research Questions to Address:

• Are respondents able to answer the questions as intended?

• Are there any items that cause respondents problems?

  • What can be done to fix those problems?

• Estimate of average time to complete?

Introduction:

• Hello. Thank you for your time today. We are looking to obtain feedback on some potential new questions for a survey that the Census Bureau has been conducting to provide insight into the state of the economy by providing timely data for key economic measures and business expectations about future conditions.

• We plan to use your feedback to improve the survey questions and make sure they make sense to respondents like you. We are not testing you– we only want to evaluate the questionnaire.

• Today, we will have you complete the short questionnaire, then you will be asked some questions to get your feedback.

• Thank you for signing the consent form. As noted, this study is being conducted under the authority of Title 13 USC. (If they did not yet sign: I just sent you an email. Can you please click on the link to the Qualtrics survey and read over the first page? This study is being conducted under the authority of Title 13 USC. Do you agree to participate? Thank you. Please click on the checkbox at the bottom of the screen, but do not advance the screen just yet.)

Before the Questionnaire:

• Before we begin, could you tell me a little bit about your business? What types of goods or services does this business provide? How is the business organized?

• Thank you. And what is your role in the business? (What kind of responsibilities do you have?)

Have Respondent Complete the Questionnaire:

Now I’d like for you to go through the survey question by question. Please tell me your answer to each question as you go along and I will ask you some follow-up questions.

General probes that may be used for any question throughout the interview:

• In your own words, what is this question asking?

• Reflect back on respondent’s answer: “you said…”

• How did you arrive at this number/answer this question?

• What records (if any) would you look at?

• Specifically, what would you include in this answer? What would you exclude?

• Would you consult other people to obtain this answer?

• How easy or difficult is it to answer this question?

Individual Items

The U.S. Census Bureau is requesting your assistance to produce important data on current economic conditions, as well as the impact on our nation’s economy when natural disasters or other emergencies occur.

Your response is especially important as it helps policy makers, government officials, and businesses like yours understand these economic effects. Please complete this easy survey; it will take approximately 25 minutes.

The data you provide will be aggregated to create a monthly snapshot of economic activity within the U.S. This survey is not intended to replace more comprehensive surveys that typically require the use of company records.

• Estimates are acceptable and encouraged.

• The survey will take approximately 25 minutes.

• Please include all domestic operations of this consolidated business, including all locations, subsidiaries, and divisions

• Can you please read through this introduction to the survey and let me know what jumps out at you?

• Is there anything else you would want to know up front? Anything that could be deleted?

1. Overall, how would you describe this business’s current performance? Please include all domestic operations of this consolidated business, including all locations, subsidiaries, and divisions.

• Excellent

• Above average

• Average

• Below average

• Poor

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• Do you find the instruction after the question helpful or not helpful? Do you have any suggestions for improving this instruction?

2. How did this business’s operating revenues/sales/receipts change in January, compared to December? Estimates are acceptable.

• Increased

• Decreased

• No change

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What time frame were you answering for?

• What does "in January, compared to December" mean to you?

• Do you have any suggestions on how to improve this question?

3. How did this business’s number of paid employees change in January, compared to December?

• Increased

• Decreased

• No change

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What does "paid employees" mean to you?

• Do you have any suggestions on how to improve this question?

4. Did this business use Artificial Intelligence (AI)¹ in producing goods or providing services in January? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.)

• Yes

• No

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What does Artificial Intelligence mean to you?

• Would you like a more detailed definition? Other examples?

• What time period were you thinking about when answering this question? Would a different time period have been clearer or easier for you?

• Do you have any suggestions on how to improve this question?

5. In the last six months, in what business functions did this business use Artificial Intelligence (AI)? Select all that apply.

• Production of goods or provision of services (OR Core operations related to production of goods or provision of services?)

• Strategy and business development

• Finance and accounting

• Sales and marketing

• Customer service

• Research and development

• Information technology

• Human resources

• Public relations and communication

• Management and administration

• Sourcing, supply chains, and purchasing

• Quality management and control

• Distribution

• Legal and compliance

• Other (please describe: ___________________)

• This business did not use AI in any business functions in the last six months

• Do not know

[If No or Do not know to 4 AND “This business did not use AI in any business functions in the last six months” or Do not know to 5, go to 12:]

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What does “business functions” mean to you in this question?

• Are the response options clear or unclear? Are there any answer choices potentially missing from this list?

• What time period were you thinking about when answering this question? Would a different time period have been clearer or easier for you?

• Do you have any suggestions on how to improve this question?

6. In the last six months, did this business use Artificial Intelligence (AI) to perform tasks previously done by employees?

• Yes

• No

• Do not know

[If No or Do not know to 6, go to 8:]

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What types of tasks were you thinking of?

• Do you have any suggestions on how to improve this question?

6. In the last six months, how many tasks previously done by employees were instead performed by Artificial Intelligence (AI)?

• A small number

• A moderate number

• A large number

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What do "a small number, a moderate number and a large number" of tasks mean to you?

• Do you have any suggestions on how to improve this question?

6. In the last six months, did this business create new tasks for employees as a result of using Artificial Intelligence (AI)?

• Yes

• No

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What does “create new tasks for employees” mean to you in this question?

• What types of tasks were you thinking of?

• Do you have any suggestions on how to improve this question?

6. In the last six months, did this business use Artificial Intelligence (AI) to perform operations previously performed by existing equipment or software?

• Yes

• No

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What types of operations were you thinking of?

• Do you have any suggestions on how to improve this question?

6. In the last six months, how did the use of Artificial Intelligence (AI) affect this business’s total employment?

• Increased

• Decreased

• Did not change

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• Do you have any suggestions on how to improve this question?

6. In the last six months, to use Artificial Intelligence (AI), what changes did this business make? Select all that apply.

• Trained current staff to use AI

• Hired staff trained in AI

• Purchased computing power or specialized equipment or software

• Purchased cloud services or cloud storage

• Changed data collection or data management practices

• Developed new workflows

• Used vendors or consulting services to install or integrate AI

• Other (please describe: ___________________)

• This business did not make any changes to use AI in the last six months

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• Reading through the answer categories, what do each of them mean to you? Are there any obvious categories missing? Any that could be deleted?

• Do you have any suggestions on how to improve this question?

6. In the last six months, did this business’s employees use Artificial Intelligence (AI) to assist in any work-related tasks?

• Yes

• No

• Do not know

[If No or Do not know to 12, have them review 13:]

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What does “work-related tasks” mean to you?

• What types of tasks were you thinking of?

• Are these tasks the same as, or different from, the tasks you reported earlier?

• How do you think this question differs from previous questions asking about AI use in your business?

• Do you have any suggestions on how to improve this question?

6. In the last six months, did this business’ employees use Generative AI to assist in any work-related tasks? (Generative AI is a type of Artificial Intelligence that uses prompts or other inputs to create text, images, music, videos, or code. Some examples of Generative AI include ChatGPT, Gemini, Copilot, Claude, Dall E, MidJourney, and GitHub Copilot.)

• Yes

• No

• Do not know

[If No or Do not know to 13, go to 15:]

• [For those who said No or Do not know to Q12] If you had answered Yes to the last question, you would see this question next. Can you please review this question and give me your feedback? Seeing this question, would that change your answer to Q12?

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What does “Generative AI” mean to you?

• What types of Generative AI were you thinking of?

• Do you think the definition and examples are helpful or not helpful? Do you have any suggestions to improve the definition or examples?

• Do you have any suggestions on how to improve this question in general?

6. In the last six months, what work-related tasks did this business’s employees use Generative AI to assist with? Select all that apply.

• Information processing, paperwork, or filing

• Writing or editing documents, emails, or communications

• Interpreting, analyzing, translating, or summarizing documents

• Customer support

• Developing or researching new projects, processes, or products

• Searching for information or technical help

• Software coding or debugging

• Data analysis or visualization

• Tutoring, training, or learning

• Other tasks (please describe: ___________________)

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• Reading through the answer categories, what do each of them mean to you? Are there any obvious categories missing? Any that could be deleted?

• Do you have any suggestions on how to improve this question?

6. During the next six months, do you think this business will be using Artificial Intelligence (AI) in producing goods or providing services? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.)

• Yes

• No

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What time period were you thinking about when answering this question?

• How confident are you in your answer to this question?

• Do you have any suggestions on how to improve this question?

6. During the next six months, in what business functions do you think this business will be using Artificial Intelligence (AI)? Select all that apply.

• Production of goods or provision of services (OR Core operations related to production of goods or provision of services?)

• Strategy and business development

• Finance and accounting

• Sales and marketing

• Customer service

• Research and development

• Information technology

• Human resources

• Public relations and communication

• Management and administration

• Sourcing, supply chains, and purchasing

• Quality management and control

• Distribution

• Legal and compliance

• Other (please describe: ___________________)

• This business will not be using AI in any business functions in the next six months

• Do not know

[If No or Do not know to 15 AND “This business will not be using AI in any business functions in the next six months” or Do not know to 16, ask 17]

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• What time period were you thinking about when answering this question? Would a different time period have been clearer or easier for you?

• Do you have any suggestions on how to improve this question?

6. Why does this business not plan to use Artificial Intelligence (AI) during the next six months? Select all that apply.

• Too expensive

• AI is not a mature enough technology yet

• Lack of knowledge on the capabilities of AI

• Concerns about privacy/security

• Concerns about bias

• Lack of skilled workforce

• Lack of required data

• Laws and regulations prevent or restrict use of AI

• Previous or current use of AI did not meet expectations

• Other (please describe: ___________________)

• AI is not applicable to this business

• Do not know

• How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?

• Reading through the answer categories, what do they each mean to you? Are there any answer categories missing?

• Where would you get information to factor into the decision not use AI?

• Do you have any suggestions on how to improve this question?

General Questions

First, what is your overall impression of these questions?

How comfortable were you in answering these questions for your company – very comfortable, somewhat comfortable, not very comfortable or not at all comfortable?

• Very comfortable

• Somewhat comfortable

• Not very comfortable (Can you say more about this?)

• Not at all comfortable (Can you say more about this?)

How confident are you in your answers to these questions – very confident, somewhat confident, not very confident or not at all confident?

• Very confident

• Somewhat confident

• Not very confident (Can you say more about this?)

• Not at all confident (Can you say more about this?)

If you had been answering these questions without being on the phone with me, would you likely have consulted records or others within your company?

• Yes, records (What records would you consult?)

• Yes, others within company (Who would you consult with?)

• No

About how long do you think it would have taken you to complete this survey (if we hadn’t been on the phone)?

_____ minutes

Thinking back through all the questions you answered for your overall company, do you feel you were the best person to answer these questions or would someone else at the company have been a better fit?

• Yes, I was the best person

• No, someone else would have been a better fit (What is that person’s job title?)

Overall, how easy or difficult was it to answer these questions– very easy, somewhat easy, neither easy nor difficult, somewhat difficult or very difficult?

• Very easy

• Somewhat easy

• Neither easy nor difficult

• Somewhat difficult (Can you say more about what was difficult?)

• Very difficult (Can you say more about what was difficult?)

Wrap Up:

Wrapping up now!

• Which of the questions we reviewed do you think would be most difficult to answer?

• Do you have any other comments or suggestions about anything we have discussed today?

Thank you very much for your time today.

16