Attachment A
DRAFT Cognitive Interview Protocol - 2018 Annual Business Survey
Research Questions to Address:
Are respondents able to answer the questions as intended?
Are there any items that cause the respondents problems?
What can be done to fix those problems?
Are there any order/context effects?
Materials:
Copies of questionnaire testing draft
Recorder, batteries
Consent form
Procedure:
Go through the Introduction and Before the Questionnaire questions.
Then provide R with the draft questions (Attachments B, C, and D) and ask R to fill it out. Once R completes each question, ask the general probes and any item-specific follow-up probes.
Note how R navigates the questionnaire and if they appear to be having any difficulty with the questions. If R appears to have any problems (utterances like “Hmm”, confused looks, etc.), probe on this.
Go through probes about Wrap-up Questions
Explain purpose of meeting: to obtain feedback on reporting to the 2017 ABS, to understand the process of answering questions, and to evaluate how new questions work or don’t work.
Some questions I will ask may seem odd and/or obvious, but I don’t want to assume I know what you are thinking.
Explain that we are not testing the respondent – we only want to evaluate the questionnaire
Structure of meeting: Understand your business and your role in it, then review the questionnaire and ask you questions as you complete it.
Permission to record discussion for note taking purposes? This study is being conducted under the authority of Title 13 USC. We plan to use your feedback to improve the design and layout of the form for future data collections. Only staff involved in this product design research will have access to the recording. Have R sign consent form.
Before the Questionnaire:
What types of goods or services does this business provide?
What is your role in the company? What kind of responsibilities do you have?
Are you one of the owners of the business?
Have you completed other surveys from the Census Bureau or other agencies? If so, which ones?
What was your role in the process for responding to this questionnaire? (Gather data? Enter data? Consult with data providers? Etc.)
Questionnaire test
Now let’s take a look at the draft questions. I would like you to go through and answer the questions as if you were filling out an actual form. If you don’t have answers to any particular questions, you can just make a best guess. I will ask you some questions as we go through them.
These questions are intended for [name of business as listed in recruiting file].
General Probes:
What does this question mean to you?
How did you go about answering this question?
Would you be able to answer this question yourself? (If no) Who would be the appropriate person to answer it?
(If R appears to have problem/confusion) What are you thinking about?
Reflect R’s specific answer: You said ___. Can you tell me more about that?
If R notices skip instructions: The electronic system would skip automatically, but we will go over every question to make sure they are all clear.
How do you feel about the order or flow of the questions being asked in the survey?
Item-Specific Probes:
Company Information
10% or More Ownership
What do you think this question is asking?
Are the instructions clear?
Number of Owners
For respondents that selected No to the 10% or more ownership question, how did you come up with your response?
Does the March 12, 2017 date affect your ability to answer the question? If yes, what are the reasons?
Number of Paid Owners
What’s the difference between this question and the previous (number of owners) question?
Are you easily able to report this?
If response to number of owners and this question is the same, how would you respond?
Number of Employees
What’s the difference between the previous (number of paid owners) question and this question?
Total Sales and Revenues
Are you easily able to report this value for the entire business?
Do you believe that the question is asking about worldwide sales and revenue (outside US and domestic) or only domestic sales and revenue?
Does your company have foreign sales and revenues?
In what dollar unit did you report?
How do you feel about being asked to report dollar the amount in thousands?
Domestic Sales and Revenues
How did you come up with your response?
In what dollar unit did you report?
What do you think this question is asking about? Do you think it is asking about the domestic portion of the value reported in Total Sales and Revenue question?
If response to Total Sales and Revenues and this question is the same (meaning only domestic sales), how would you respond?
How do you interpret the phrase “attributable to or originated from domestic operations”? Does that help you understand that this question is asking for only domestic sales and revenue?
Please consider the phrase “Include sales and operating revenues to foreign customers, including foreign subsidiaries,” Does that help you understand that this question is asking for only domestic sales and revenue?
Research and Development Module
R&D Activities
Did all of the response categories make sense to you? If not, which categories did you experience difficulty with and why?
R&D Costs
How did you come up with your response?
Were the directions clear on what to include and not to include?
In what dollar unit did you report?
If respondent has costs for R&D undertaken outside of the U.S., ask the following questions:
Did the respondent include any foreign R&D expenses in this question?
If yes, can the respondent split out the foreign vs. domestic R&D expense?
If no, what is the reason the foreign R&D expense was not included?
Does your business’ foreign R&D expenses include any of the following:
Salaries, wages, fringe benefits
Plant, machinery, and equipment, except that which was capitalized because it had an alternative future use Materials, supplies, software
Rent, utilities
Consultants, contractors
Depreciation expense from plant, machinery, and equipment that was capitalized because it had an alternative future use
Types of R&D Costs
How did you determine your response for each option?
Do you need to check records?
Do you prefer providing percentages or dollar amounts?
Does the request for percentages or dollar amounts impact how the respondent prepares their response?
Note if R provides a percentage and R’s numbers don’t add up to 100 percent. If so, probe as to why.
Funding Sources for R&D Activities
How did you determine your response for each option?
Did you need to check records?
Do you prefer providing percentages or dollar amounts?
Does the request for percentages or dollar amounts impact how the respondent prepares their response?
Note if R provides a percentage and R’s numbers don’t add up to 100 percent. If so, probe as to why
Percent Paid For R&D Conducted by Others
How did you determine your response for each option?
Did you need to check records?
Do you prefer providing percentages or dollar amounts?
Does the request for percentages or dollar amounts impact how the respondent prepares their response?
Note if R reports a percentage and R’s numbers don’t add up to 100 percent. If so, probe as to why.
R&D Categories (*to be answered in dollar amounts instead of percentages)
How did you determine your response for each option?
Do you need to check records?
Do you prefer providing percentages or dollar amounts?
Does the request for percentages or dollar amounts impact how the respondent prepares their response?
In what dollar unit did you report?
BUSINESS SALES
What do you think this question is asking about?
How did you come up with your answers?
Did you experience any confusion when answering this question? If so, what could we do to improve how we are asking about this item?
Do you have any suggestions on how we could ask these questions in order to make them easier for you to answer?
R&D Employees
What do you think this question is asking about?
How did you come up with your response?
Did you experience any confusion when answering this question? If so, what could we do to improve how we are asking about this item?
What time period did you have in mind when answering this question?
Do you have any suggestions on how we could ask these questions in order to make them easier for you to answer?
R&D Employee Occupations
What do you think this question is asking about?
How did you come up with your response?
Did you experience any confusion when answering this question? If so, what could we do to improve how we are asking about this item?
What time period did you have in mind when answering this question?
Do you have any suggestions on how we could ask these questions in order to make them easier for you to answer?
Technology Module
Production Technology for Goods and Services
Can you tell me a little more about how you answered this question?
What time period were you referencing when answering this question?
Did the response categories make sense to you?
In your own words, please define what the following mean to you:
Testing, but not in production or service
Low Use
Moderate Use
High Use
In your own words, can please define what the following mean to you:
Artificial Intelligence
Cloud-based Computing Systems and Applications
Specialized Software
Robotics
Specialized Equipment
(Show them definitions) Now that you see the definitions, can you tell me if there are any that didn’t match your definition? Why?
What are the types of artificial intelligence (AI) that you use or are familiar with? Are there examples of AI that are not provided in the definition that you would recommend adding? Does your company do R&D that applies artificial intelligence to a problem? Does your company do R&D that is intended to advance the state of artificial intelligence?
What are the types of specialized software that you use at your company?
In your own words, what is the difference between AI and specialized software?
(If R marked Robotics) Tell me about the robotics that you have at your company.
What does the phrase “carrying out a complex set of actions” mean to you?
What are the types of specialized equipment that you have at your company? Tell me about them. Are there any examples of specialized equipment that aren’t listed that you would recommend that we add?
In your own words, what is the difference between robotics and specialized equipment?
Motivation for Technology Adoption and Utilization (Process and Methods)
What time period were you referencing when answering this question?
Did the response categories make sense to you?
Can you tell me a little more about how you answered this question for each of the categories listed?
Are there other reasons why you used the technologies in this question other than the ones listed?
Impact of Technology on Workforce (Process and Methods)
A. How did you decide on a response?
B. How did you come up with your answer?
(If R marked yes for Q3-B) How did the skill level of your workers change?
Impact of Technology on Worker Types (Process and Methods)
A. How did you chose an answer for this question? What do you think is meant by the term “production workers?”
B. How did you come up with your answer for this question? What types of workers did you think about when thinking about “non-production workers?”
C. How did you come up with your response?
D. What were you thinking about when answering this question?
E. What do you think is meant by “skilled workers?” How did you come up with your response?
Technology Based Goods and Services
How did you decide on a response?
How did you come up with your answer?
Motivation for Technology Adoption and Utilization (Goods and Services)
What time period were you referencing when answering this question?
Did the response categories make sense to you?
Can you tell me a little more about how you answered this question for each of the categories listed?
Impact of Technology on Workforce (Goods and Services)
A. How did you decide on a response?
B. How did you come up with your answer?
Impact of Technology on Worker Types (Goods and Services)
A. How did you choose an answer for this question? What do you think is meant by the term “production workers?”
B. How did you come up with your answer for this question? What types of workers did you think about when thinking about “non-production workers?”
C. How did you come up with your response?
D. What were you thinking about when answering this question?
E. What do you think is meant by “skilled workers?” How did you come up with your response?
Factors Adversely Affecting Technology Adoption and Utilization
What is this question asking about?
How did you come up with your answers?
Probe R about each of the column headers.
Wrap-up
For businesses with a proxy respondent during the interview:
Now that you have seen this survey, would you typically be the one who would be responsible for answering this survey?
If so, would you involve the owner(s) in answering these questions? Which questions?
How would you go about asking the owner(s) to answer these questions? Would you provide them with a copy of the question/survey?
Would you have the owner(s) review the completed form before it was submitted?
Would you have any concerns or hesitancy about answering any of the questions about the owners without their input?
All businesses:
Are there any other comments or suggestions you would like to make about anything we have gone over today?
Debriefing Questions
What were your thoughts on completing the 2017 Annual Business Survey?
About how long did it take you to complete the survey? Were others involved in completing the survey? If yes, was their time included in your estimate?
How many owners are there at the company?
(if more than 1) How did you go about completing the question for the other owners? Did they review the questions and provide answers? What was your process for doing this?
The survey asked questions about Research and Development activities. Did you have any issues answering questions in this section?
Was there anything on the survey that stood out to you as difficult to answer or confusing?
Do you have any recommendations for improving the survey?
Thank the respondent for their time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jennifer L Beck |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |