Phase 7 Cognitive Testing Report

Cognitive Testing Summary for Phase 7

Friday, September 10, 2021

Focus: Test Q17 below

Author: Melissa Cidade (ESMD)

In total, we spoke to six respondents representing small businesses in three sectors over a three-day period.  We used retrospective probing wherein respondents complete the survey and then interviewers ask a series of follow up questions to learn more about the response cognitive process.  In this round of testing, we probed on one new question based on a survey question from the UK Office for National Statistics (Business Insights and Conditions Survey): 

Q17. Comparing now to what was normal before March 13, 2020, has this business done any of the following? 

Select all that apply: 

• Adopted or expanded use of digital technologies 

• Changed management practices or business strategies 

• Introduced new products or services 

• Improved existing products or services 

• Improved methods of logistics, delivery, or distribution 

• Improved methods of producing goods and services 

• Invested in innovation activities 

• This business has not made any of these changes 

Question Meaning:  Generally, respondents understood this question to be asking about changes, and especially ‘positive’ changes (improvements).  Overall, they found the question easy to understand. 

Response Options

Initial response option	Finding	Recommendation	Final response option
Adopted or expanded use of digital technologies	Respondents thought of technical capabilities, including expanded internal use (like Skype and laptops) and external use (like websites and online transactions).	Retain as worded	Adopted or expanded use of digital technologies
Changed management practices or business strategies	Business strategies was seen as broad, and related to the delivery of products or services to customers.  Management practices related more to the day-to-day internal operations of the company.  These are two separate concepts warranting two separate response options.	Break into two response options.	Changed management practices
			Changed business strategies
Introduced new products or services	Response option performed as expected.	Update wording to "goods or services"	Introduced new goods or services
Improved existing products or services	Response option performed as expected.	Update wording to "goods or services"	Improved existing goods or services
Improved methods of logistics, delivery, or distribution	One respondent wondered if we meant deliveries to the business or deliveries by the business, but no others had any issues.	Retain as worded	Improved methods of logistics, delivery, or distribution
Improved methods of producing goods and services	Response option performed as expected.	Update wording to "goods or services"	Improved methods of producing goods and services
Invested in innovation activities	Respondents struggled more with this category than others.  One noticed the word ‘invested’ and pointed out that any innovations are not using ‘new’ money, but rather expanding already in place costs.  Another said that their innovation has all been digital, which is captured in the earlier response option.  Still a third just didn’t know what the category was referring to.	Drop this category.
This business has not made any of these changes	Response option performed as expected.	Retain as worded	This business has not made any of these changes

FINAL QUESTION

Q17. Comparing now to what was normal before March 13, 2020, has this business done any of the following?

Select all that apply:

• Adopted or expanded use of digital technologies

• Changed management practices

• Changed business strategies

• Introduced new goods or services

• Improved existing goods or services

• Improved methods of producing goods or services

• Improved methods of logistics, delivery, or distribution

• This business has not made any of these changes