Cognitive Testing of the Types of Workers and Management Practices questions for 2015 Annual Survey of Entrepreneurs

MEMORANDUM
To:

Patrice Norman, Economy-Wide Statistics Division
Beth Goldberg, Economy-Wide Statistics Division
Naomi Blackman, Economy-Wide Statistics Division

From:

Dave Tuttle, Data Collection Methodology & Research Branch, Economic Statistical
Methods Division
Aryn Hernandez, Data Collection Methodology & Research Branch, Economic Statistical
Methods Division

Date:

January 28, 2016

Re:

Cognitive Testing of the Types of Workers and Management Practices questions for
2015 Annual Survey of Entrepreneurs

This memorandum summarizes the research conducted by the Data Collection Methodology & Research
Branch on the Types of Workers and Management Practices questions being considered for the 2015
ASE. We conducted cognitive interviews with 20 companies in the Dallas, Texas and Kansas City,
Missouri metropolitan areas. The distribution of companies by size (number of employees) and NAICS
sector is summarized in the tables below. Data for number of employees were included in the recruiting
files provided by EWD based on prior responses either to the 2012 SBO or the 2014 ASE, and may not
reflect the most accurate data for the companies we interviewed.
Number of employees
0 employees
1 to 4 employees
10 to 19 employees
20 to 49 employees
100 to 249 employees
250 to 499 employees
500 to 999 employees
2,500 to 4,999 employees
Total

Number of companies
2
4
3
5
3
1
1
1
20

Sectors
23: Construction
31-33: Manufacturing
42: Wholesale Trade
44-45: Retail Trade
52: Finance and Insurance
53: Real Estate and Rental and Leasing
54: Professional, Scientific, and Technical Services
55: Management of Companies and Enterprises
56: Administrative and Support and Waste
Management and Remediation Services
61: Educational Services
72: Accommodation and Food Services
Total

Number of companies
1
3
1
2
2
1
5
2
1
1
1
20

We summarize the findings of the research and provide recommendations below. The interview
protocol and test questionnaire used for the interviews are included as appendices to this
memorandum.
Types of Workers section
The “types of workers” questions (1-7) were generally interpreted by respondents as intended. All
respondents correctly understood the tasks identifying the types of workers used by their businesses
and reporting the percentage of each type of worker. The main problems with this section have to do
with confusion between how the categories relate to what they consider to be their employees, some
variations in interpretation of certain categories, and appropriate methods for calculating percentages.
Finding #1: Full-time vs. part-time paid employees
Some respondents asked for clarification on the distinction between full-time and part-time employees.
We also observed some variation in how respondents distinguished between full-time and part-time
employees. In addition, some respondents had more than one way of distinguishing between full- and
part-time employees depending on the application. Several respondents mentioned the Affordable Care
Act (though all of them referred to it as “Obamacare”).
Recommendation: We recommend adding some text to the effect of “however your company
defines full-time/part-time” at the end of each category, to reassure respondents that it is
acceptable to apply their own definitions.
Finding #2: Temporary vs. leased employees
A few respondents were not clear as to the distinctions between temporary and leased employees.
When asked to describe the difference, most could not cite any distinguishing features. A few
respondents were familiar with the difference as they had some direct experience with or other
knowledge specifically of leased employment. One respondent’s company has only leased employees;
he selected full-time, part-time, and leased employees, and commented that he was not sure whether
the categories were mutually exclusive.
The respondents who understood leased employment tended to define it as a way of reducing or
eliminating the costs for administering payroll and benefits, and/or as a way of creating larger pools of
employees in order to offer them better benefits. In contrast, they tended to define temporary
employment as having one or a small number of employees work for their business on a short-term
basis, usually low-skilled tasks.
Recommendations: We recommend adding some clarification of the difference between the
two concepts.
With regard to the issue of respondent who selected more than one category when the
company only has leased employees, this is addressed by the recommendation for Finding #4.

Finding #3: Contractors, subcontractors, independent contractors, or outside consultants
Most respondents thought of this category in terms of other firms doing work for them (e.g., grounds
maintenance, construction sub-contractors, contracting manufacturing or other processes associated
with production, etc.). “Contractors” and “sub-contractors” are widely-used terms for work that is
outsourced to other firms, and do not appear to be associated with individual workers who join
businesses’ workforces under contract. The terms appear to be so salient that the additional terms
“independent contractors” and “outside consultants” do not appear to be effective in affecting the
scope of respondents’ interpretations of the item. Based on the first two terms, respondents arrive at an
interpretation of what they are being asked, instantly and with high confidence, and do not need to
expend any more effort in understanding what the category means.
A few respondents mentioned that they have contract employees and defined them as receiving a 1099
form. One respondent made the distinction that on one occasion, employees from a contract firm
worked at the respondent business’ site and took instruction from the business’ staff.
Recommendation: We recommend adding some clarification of what this category is intended
to collect. The sponsors’ addition of “workers who received a 1099” may be sufficient, but we
recommend consideration of additional criteria, such as:
•
•
•
•
•

Contract employees who become part of the business’ workforce
Contract employees who work at the business’ own locations
Contract employees who work alongside the business’ own staff
Contract employees who take orders directly from the business’ staff
Exclude contract arrangements in which work is outsourced to another firm

Finding #4: Calculating percentages
During our interviews, all respondents reported “top-of-head” estimates of their headcounts, but said
they would use their records to report precise figures. They generally appeared to think of their
workforces only as those on the payroll, namely full-time and part-time employees. They first reported
an estimate for full-time employees, and those who had part-time employees reported the difference of
full-time from 100%. This was easy for them to do. However, when confronted with the task of reporting
percentages for the other categories, some respondents were not sure how to go about it. The reason
appears to be that the presumed denominator, which includes all the types of workers, is not intuitive to
respondents. As they tended to think only in terms of their actual payroll employees, it was not natural
for them to think of the other types as their employees. Those who marked any category besides fulltime or part-time would have to reconsider how they calculated full-time and part-time employees in
order to report percentages for the other types.
In addition, some respondents were unsure how to derive percentages for particular situations. For
example, one respondent considered the business’ employee turnover and wondered whether he
should count the number of different people they employed throughout the year, or the number of
positions. Similarly, another respondent occasionally subcontracts installation of his business’ products

work for short periods throughout the year: “We contract out installation of our product. Typically 2-4
guys on a job, 2-3 weeks for entire project, a few times a year. It would be hard to put a percent on that.
I would have to do some kind of hours calculation.”
Some respondents noticed that the questions ask for an average, and said they would instead choose
the headcount on a particular date such as March 12.
We observed that respondents who were owners of small businesses tended not to include themselves
as employees of their companies.
Recommendations: The sponsors’ creation of a new item that asks for percentages for the full
list of categories may be effective in conveying the idea that all types of workers selected should
be considered the denominator. We recommend adding “100%” at the bottom of the column of
response fields to further reinforce this idea. It may also be helpful to include an instruction to
the effect of “For the purposes of this survey, we consider any workers working for your
business to be part of your workforce, even if they are not payrolled employees.” Include an
instruction for owners to count themselves as employees.
We strongly recommend that the questions that follow the screener do not include any
categories not selected in the screener. Reactions to being presented with categories which
were not selected can range from minor irritation to concerns that the instrument does not
work, and may cause some respondents to go back to the previous page to ensure that they
reported correctly on the screener. As the ASE is a fairly lengthy survey, the possibility that the
additional burden and frustration might lead to break-offs should not be ignored.
On the basis of such observations as noted above, we conclude that respondents may be willing
to go to burdensome lengths to calculate precise figures. Such burden should be considered in
light of the intended uses of these data and the level of precision needed. If precise figures are
not needed, we recommend the use of closed-ended response options with intervals instead of
open-ended fields asking for exact figures. At a minimum, we recommend that addition of a
prominent instruction indicating that estimates are acceptable for these items.
Finding #5: Tasks performed by workers
The list of tasks was generally interpreted correctly by respondents, with a few exceptions. "Operations”
was unclear to a few respondents, who found it too vague, and did not associate it with the core
revenue-generating activities of their businesses. A few respondents also were not sure where to report
employees in accounting and finance roles.
Recommendations: To clarify “Operations”, we recommend adding text to expand on and clarify
the intended meaning, such as “related to the primary revenue-generating activity of this
business” or “related to providing this business’ products or services.” During a recent meeting
it was suggested that BLS’ definition of “operations” be used, but we do not recommend that
the definition be used in its entirety as it was too lengthy, technical, and manufacturing-focused.

Consider revising the seventh option as something to the effect of “General management, for
example, administration, accounting, human resources, infrastructure, etc.”

Management Practices section
Overall, most of the questions in this section performed fairly well, and were interpreted and answered
correctly. Some over-arching findings follow, and then question-specific findings.
Finding #6: Focus on manufacturing and other process-oriented industries
One over-arching finding is that most respondents observed that the questions and response options
seem oriented toward manufacturing or other process- or logistics-oriented types of businesses. They
often struggled to find meaningful categories, especially at service sector businesses.
Recommendations: Specific findings and recommendations are offered below.
Finding #7: Double-barreled and complex questions
Several of the questions in this section are double-barreled to some degree, that is, they presume the
presence of the activity or characteristic at the focus of the question. Several respondents commented
on this with regard to some questions. Although all of the questions have response options to indicate
“not applicable,” they appear at the end of the options. All respondents were eventually able to find the
“not applicable” options, though in some cases they attempted to make their interpretation of the
questions fit such that they could provide a positive answer. This is a common risk in survey research,
and best practice is to add screener questions to ascertain the existence of the conditions before asking
respondents to provide qualifying information.
In addition, the combination of unfamiliar concepts with additional qualifying dimensions results in
complex questions. We observed respondents having to re-read questions and consider them at length,
but often they were not confident that they had the right understanding and asked us to confirm their
interpretations. As respondents must consider multiple dimensions within the same question
simultaneously, they may not expend the effort necessary to understand fully what the question asks,
and the risk of misinterpretation and false-positive responses increases. Further, we observed instances
in which respondents appeared to lose track of the reference period for specific questions and may have
been thinking in general terms and/or including prior events (a phenomenon known as “telescoping”)
rather than thinking specifically about what happened in the reference year.
Recommendations: We recommend consideration of measures to address the complexity of
questions in this section, including creating screener questions to first ascertain the existence of
each condition in the reference year before asking for more information.

Although it is best practice to place the reference period at the beginning of a question, in this
case it leads to somewhat awkward structure due to the multiple clauses (“what best describes”
and “what happened at this business”). “In 2015” modifies “what happened” but is removed
from the latter by its placement at the beginning of the sentence. We suggest consideration of
restructuring these questions so that “in 2015” follows “what happened at this business.”
Additional question-specific recommendations follow.
Finding #8: Service or Production Problems
Most respondents chose one of the first three options, and some of them indicated that there is a
“right” answer, suggesting social desirability and/or other similar biases in the way answers are chosen.
Also, as noted above, some respondents appeared to be thinking in general terms rather than about
actual, specific problems and how they were addressed.
We observed some misinterpretation of “continuous improvement process.” Some respondents applied
this concept in a general way, and not in the sense of a formal program of continuous monitoring and
improving efficiency. Respondents without formal processes tended to consider various kinds of
improvements large and small, whether formal or informal.
The example following the question appeared to narrow respondents’ interpretation in an unintended
way. Respondents often focused on the example of “equipment breaking down” to the exclusion of
considering other specific instances where problems arose. The example also caused some respondents
to remark that this item appears to be focused on manufacturing and did not seem to apply to service
sector businesses.
Recommendations: Clarify the third option to mean “formal continuous improvement
process/program” if appropriate. Replace one of the examples with one or more examples
related to the service sector.
Finding #9: Number of Key Performance Indicators
As with “continuous improvement” in the prior question, most respondents interpreted “key
performance indicators” broadly and not in the sense of formal business performance metrics.
Respondents at larger businesses appeared more likely to consider and report formal metrics than those
at smaller businesses. One respondent described what this question is looking for as “measurable and
quantifiable” metrics.
Also like the prior question, the examples following the question affected how respondents interpreted
the question and the types of metrics they thought of and considered. Respondents, especially at
smaller companies, often considered only the types of KPI’s presented as examples and did not consider
other types not on the list.

Recommendations: We recommend adding some clarification of what is meant by “key
performance indicator,” such as that they are formal, quantifiable, planned, regular, etc., as
appropriate.
We also recommend that the number of examples be limited to three and that they include
examples that are easily applicable to service sector businesses.
We recommend consideration of a screener question. Comprehension of the concept of KPI may
be improved by creating a screener that first attempts to ascertain whether the business uses
any KPI’s, which would focus respondents’ attention on the concept by itself with no other
dimension.
Finding #10: Frequency of Key Performance Indicators
Most respondents selected more than one response, but a few wondered whether they were supposed
to select only one.
Recommendations: Add instruction to “select all that apply.”
Finding #11: Business Targets
The inclusion of examples following the question affected some respondents the same way as with the
Key Performance Indicators question, i.e., filtered and limited the types of targets they considered.
With regard to the time frames in the response options of the first part, everyone appeared to interpret
them correctly.
In the second part, several respondents read the second response option (“with some effort”)
colloquially and interpreted it to mean a normal or more than normal level of effort, and selected that
option.
Recommendations: We recommend limiting the number of examples and including service
sector examples.
Per the discussion at a recent meeting, we recommend changing the response options in the
second part. We also suggest removing “Possible to achieve” from each option and placing it on
the line above the options, to reduce the amount of text respondents must read in each
response option. In addition, we recommend adding an option for “Did not achieve production
targets.” Here is the recommended revision to the response options in full:
Possible to achieve…
…with minimal effort
…with less than normal effort
…with a normal amount of effort
…with more than normal effort
…with extraordinary effort

Did not achieve production targets
Finding #12: Employee Promotion
Respondents generally interpreted this item as intended, though a few expressed disapproval at the
inclusion of “family connections” in the response options. Other respondents expanded on the idea of
“other factors” to include things like office politics and poor management as factors affecting
promotion.
Recommendations: No recommendation needed, but be aware of the possibility of bias due to
the socially unacceptable perception of family connections with regard to promotion.
Finding #13: Underperforming Employees
No problems were detected with this question, although a few respondents pointed out that it is
awkwardly constructed. This awkwardness is related to its double-barreled nature, as it presumes that
the business had under-performing employees.
Recommendations: Consider creating a screener question to ascertain whether the business had
to deal with underperforming employees in the reference year.
Finding #14: Record-keeping and Decision-making
These questions were generally understood as intended. The main problem we observed was that a few
respondents did not limit consideration of the first question to record-keeping and decision-making just
to sales and purchase. When they got to the second question they realized that they had also been
considering records for other purpose and included, e.g., accounting records in their consideration of
the first question. Some reported that they consider sales and purchases records to be a subset of the
broader category of budget and finance records.
Recommendations: We recommend that the order of these two questions be reversed. Placing
the broader category of budget and finance records first should eliminate the issue of “scope
creep” in the narrower category of sales and budget records.
Finding #15: Time to complete the draft questions
Respondents were asked to estimate the amount of time it would take to complete the Types of
Workers and Management Practices questions. They reported a range of 5 to 60 minutes, with an
average of 24 minutes.
Recommendations: No recommendation needed.
Finding #16: Web-only reporting option
Respondents were asked how the Economic Directorate’s move toward electronic-only reporting would
affect their ability to complete a survey like the draft questions we reviewed. All respondents said they
prefer online reporting. Most said they could complete the survey all online, but some said they would

prefer to have a downloadable PDF version for various reasons – because they prefer to read the
questions and write their answers on paper before entering them online, or they may need to take a
copy to someone else for information they may not have. Some said they would also want to download
a copy of the survey with their reported data for their records.
Recommendations: Continue to offer a downloadable PDF version of the survey to support
offline response preparation and documentation of submitted responses.

2015 Annual Survey of Entrepreneurs
Part I – Types of Workers
1. Types of Workers
In 2015, which of the following types of workers were used by this business?
Select all that apply.








Full-time paid employees
Part-time paid employees
Paid by day laborers
Temporary staffing obtained from a temporary help service
Leased employees from a leasing service or a professional employer organization
Contractors, subcontractors, independent contractors, or outside consultants
None of the above

The following questions would only apply if you checked the corresponding options in question 1.

2. Full-Time Paid Employees
a. In 2015, on average what percent of the total number of workers was accounted for by full-time paid
employees?
Round to the nearest whole percent. Your best estimate is fine. If none, report “0”.
____%
b. In 2015, what types of tasks did full-time paid employees perform?
Select all that apply.











Procurement, logistics, and distribution
Operations
Marketing, sales, and customer accounts
Customer and after sales service
Product or service development
Technology and process development
General management and firm infrastructure
Human resources management
Strategic management
None of the above

3. Part-Time Paid Employees
a. In 2015, on average what percent of the total number of workers was accounted for by part-time paid
employees?
Round to the nearest whole percent. Your best estimate is fine. If none, report “0”.
____%
b. In 2015, what types of tasks did part-time paid employees perform?
Select all that apply.











Procurement, logistics, and distribution
Operations
Marketing, sales, and customer accounts
Customer and after sales service
Product or service development
Technology and process development
General management and firm infrastructure
Human resources management
Strategic management
None of the above

4. Paid by Day Laborers
a. In 2015, on average what percent of the total number of workers was accounted for by paid by day
laborers?
Round to the nearest whole percent. Your best estimate is fine. If none, report “0”.
____%
b. In 2015, what types of tasks did paid by day laborers perform?
Select all that apply.











Procurement, logistics, and distribution
Operations
Marketing, sales, and customer accounts
Customer and after sales service
Product or service development
Technology and process development
General management and firm infrastructure
Human resources management
Strategic management
None of the above

5. Temporary Staffing Obtained from a Temporary Help Service
a. In 2015, on average what percent of the total number of workers was accounted for by temporary staff
obtained from a temporary help service ?

Round to the nearest whole percent. Your best estimate is fine. If none, report “0”.
____%
b. In 2015, what types of tasks did temporary staff obtained from a temporary help service perform?
Select all that apply.











Procurement, logistics, and distribution
Operations
Marketing, sales, and customer accounts
Customer and after sales service
Product or service development
Technology and process development
General management and firm infrastructure
Human resources management
Strategic management
None of the above

6. Leased Employees from a Leasing Service or Professional Employer Organization
a. In 2015, on average what percent of the total number of workers was accounted for by leased employees
from a leasing service or a professional employer organization?
Round to the nearest whole percent. Your best estimate is fine. If none, report “0”.
____%
b. In 2015, what types of tasks did leased employees from a leasing service or a professional employer
organization perform?
Select all that apply.











Procurement, logistics, and distribution
Operations
Marketing, sales, and customer accounts
Customer and after sales service
Product or service development
Technology and process development
General management and firm infrastructure
Human resources management
Strategic management
None of the above

7. Contractors, Subcontractors, Independent Contractors, or Outside Consultants
a. In 2015, on average what percent of the total number of workers was accounted for by contractors,
subcontractors, independent contractors, or outside consultants?
Round to the nearest whole percent. Your best estimate is fine. If none, report “0”.

____%
b. In 2015, what types of tasks did contractors, subcontractors, independent contractors, or outside
consultants perform?
Select all that apply.











Procurement, logistics, and distribution
Operations
Marketing, sales, and customer accounts
Customer and after sales service
Product or service development
Technology and process development
General management and firm infrastructure
Human resources management
Strategic management
None of the above

Part II – Management Practices
1. Service or Production Problems
In 2015, what best describes what happened at this business when a service or production problem arose? For
example, finding a quality defect in a product or piece of equipment breaking down.
 We fixed it but did not take further action
 We fixed it and took action to make sure that it did not happen again
 We fixed it and took action to make sure that it did not happen again, and had a continuous
improvement process to anticipate problems liked these in advance
 No action was taken

2. Number of Key Performance Indicators
In 2015, how many key performance indicators were monitored at this business? For example, metrics on sales,
production, cost, quality, inventory, customer satisfaction, absenteeism and deliveries on time.





1-2 key performance indicators
3-9 key performance indicators
10 or more key performance indicators
No key performance indicators

3. Frequency of Key Performance Indicators
During 2015, how frequently were the key performance indicators reviewed at this business?







Yearly
Quarterly
Monthly
Weekly
Daily
Hourly or more frequently

 Never

4. Business Targets
a. In 2015, what best describes the time frame of business, service, or production targets at this business?
Examples of business, service, or production targets include number of customers, production, quality,
efficiency, waste, on-time delivery.





Main focus was on short term (less than one year) targets
Main focus was on long term (one year or more) targets
Combination of short-term and long-term targets
No targets – Skip to Employee Promotion

b. In 2015, how easy or difficult was it for this business to achieve its service or production targets?






Possible
Possible
Possible
Possible
Possible

to achieve without much effort
to achieve with some effort
to achieve with normal amount of effort
to achieve with more than normal effort
to achieve with extraordinary effort

5. Employee Promotion

In 2015, what was the primary way employees were promoted at this business?
 Promotions were based solely on performance and ability
 Promotions were based partly on performance and ability and partly on other factors (for
example, tenure or family connections)
 Promotions were based mainly on factors other than performance and ability (for example,
tenure or family connections)
 Employees are not normally promoted

6. Under-Performing Employee

In 2015, when was an under-performing employee reassigned or dismissed?





Within 6 months of identifying employee under-performance
After 6 months of identifying employee under-performance
Rarely
Never

7. Record-keeping and Decision Making for Sales and Purchases Activities

a. In 2015, how did this business handle its record-keeping for sales and purchases activities?
Select all that apply.





Kept paper records
Kept electronic records
Records handled by another business
Records not kept for sales and purchases activities – Skip to Record-keeping and Decision
Making for Budgeting and Financing Activities

b. In 2015, were data from records for this business’s sales and purchases activities used in decisions
regarding each of the following?
Select all that apply.









Design of new products or services
Forecasting demand for products or services
Ordering supplies or materials
Scheduling or managing deliveries
Financial planning
Targeting potential customers
Preparing this business’s taxes
Other (specify) – _________________________________________

8. Record-keeping and Decision Making for Budgeting and Finance Activities
a. In 2015, how did this business handle its record-keeping for budgeting and finance activities?
Select all that apply.





Kept paper records
Kept electronic records
Records handled by another business
Records not kept for budgeting and finance activities – Skip next question

b. In 2015, were data from records for this business’s budgeting and finance activities used in decisions
regarding each of the following?
Select all that apply.









Design of new products or services
Forecasting demand for products or services
Ordering supplies or materials
Scheduling or managing deliveries
Financial planning
Targeting potential customers
Preparing this business’s taxes
Other (specify) – _________________________________________

Draft 12-11-15

2015 Annual Survey of Entrepreneurs – Cognitive Interview Protocol
Research Questions to Address:
• Are respondents able to answer the questions as intended?
• Are there any items that cause the respondents problems? What can be done to fix
those problems?
• Are there any order/context effects?
• How much time do respondents estimate they will need to complete the survey?
• How does the web-only reporting option affect the response process? What substitute
for a mailed form should be provided, if any?
Materials:
• Copies of questionnaire testing draft
• Recorder, batteries
• Consent form
Procedure:
• Note whether R responded to the 2014 ASE.
• Go through the Introduction and Before the Questionnaire questions.
• Then provide R with the draft questionnaire and ask R to fill it out. Once R completes
each question, ask the general probes and any item-specific follow-up probes.
• Note how R navigates the questionnaire and if they appear to be having any difficulty
with the questions. If R appears to have any problems (utterances like “Hmm”, confused
looks, etc.), probe on this.
• Go through probes about Web-Only Reporting and Wrap-up Questions
Introduction:
• Explain purpose of meeting: to understand the process of answering questions, and to
evaluate how new questions work or don’t work.
• Some questions will seem odd and/or obvious, but I don’t want to assume I know what
you are thinking.
• Explain that we are not testing the respondent – we only want to evaluate the
questionnaire
• Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
• Permission to record discussion? Solely for note-taking purposes. Have R sign consent
form.
Before the Questionnaire:
• What types of services does this business provide?
• (If employer) Approximately how many employees does this business have?
• What is your role in the company? What kind of responsibilities do you have?
• Have you completed other surveys from the Census Bureau or other agencies? If so,
which ones?
1

Draft 12-11-15

•

(If previous ASE respondent) What was your role in the process for responding to this
questionnaire? (Gather data? Enter data? Consult with data providers? Etc.)

Questionnaire test
Now let’s take a look at the draft questions. I would like you to go through and answer the
questions as if you were filling out an actual form. If you don’t have answers any particular
questions, you can just make a best guess. I will ask you some questions as we go through
them.
These questions are intended for [name of business as listed in recruiting file].
General Probes:
• What does this question mean to you?
• How did you go about answering this question?
• Would you be able to answer this question yourself? (If no) Who would be the
appropriate person to answer it?
• (If R appears to have problem/confusion) What are you thinking about?
• Reflect R’s specific answer: You said ___. Can you tell me more about that?
• If R notices skip instructions: The electronic system would skip automatically, but we will
go over every question to make sure they are all clear.
Part I – Types of Workers
Q1 – Types of workers
• What does each category listed here mean to you?
• Is “leased employee” a concept that is familiar to you? (If yes) What term do you use for
“leased employee”?
Q2a – Full-time paid employees
• How did you arrive at this figure?
Q2b
•
•

How did you decide which options to select?
What does each category listed here mean to you?

Q3 – Q7
• For each percentage: How did you arrive at this figure?
• For the task list: How did you decide which options to select?
Q7a
•

Do you recall seeing the text below the question? (“Round to the nearest…” etc.)

Q7b
•

Do you recall seeing the text below the question? (“Select all that apply.”)
2

Draft 12-11-15

Part II – Management Practices
Q1 – Service or production problems
• How did you decide which option to select?
• What does “service or production problem” mean to you?
• What are some examples of such problems encountered at your business, if any?
• What does each option mean to you?
• What does “continuous improvement process” mean to you? Is that a concept you are
familiar with? (If yes) What term do you use for this concept?
• Do you recall reading the example after the question? Do you find this example helpful
or not helpful?
Q2 – Number of key performance indicators
• What does “key performance indicator” mean to you?
• (If one or more selected) What are some examples of the indicators you were thinking
of?
• Do you recall reading the example after the question? Do you find this example helpful
or not helpful?
Q3 – Frequency of key performance indicators
• (For any responses besides Never) Can you tell me more about the type(s) of review you
had in mind when you selected your response(s)?
Q4a – Business targets
• What does “business, service, or production targets” mean to you?
• What are some examples of such targets used at your business, if any?
• What does “time frame” mean to you?
• Do you recall reading the example after the question? Do you find this example helpful
or not helpful?
• Note whether R indicates any confusion between “targets” and “performance indicators”
in Q2-3.
Q4b
•

What does each category mean to you?

Q5 – Employee promotion
• How did you decide which answer to select?
Q6 – Under-performing employee
• What does “under-performing employee” mean to you?
Q7a – Record-keeping and decision making for sales and purchase activities
•
•

What does “record-keeping” mean to you?
What does “sales and purchases activities” mean to you?
3

Draft 12-11-15

Q7b
•
•
•

Please tell me in your own words what each of these answer categories means to you.
(For each category selected) What were you thinking about when you selected that
category? How does your business use data to affect decisions about that category?
If ‘Other’ is selected but R does not write in Specify blank: What other sales and
purchases activities that are not listed do you keep records of?

Q8a – Record-keeping and decision making for budgeting and finance activities
• What does “budgeting and finance activities” mean to you?
Q8b
•
•
•

Please tell me in your own words what each of these answer categories means to you.
(For each category selected) What were you thinking about when you selected that
category? How does your business use data to affect decisions about that category?
If ‘Other’ is selected but R does not write in Specify blank: What other budgeting and
finance activities that are not listed do you keep records of?

Web-Only Reporting
• Adjust the tense of these questions according to whether R responded to the 2014 ASE.
• This survey only has an internet reporting option. How [would/did] that work for you?
• [Will/did] it make any difference that we [will/did] not send a paper form?
• Thinking about the questions we have been going over today, would you have any need
for something to download or print to assist you in preparing your responses?
o (If yes) What sort of downloadable/printable document would you want to have?
What would it look like? What would you do with it? What level of detail should it
contain?
• Do you want to keep copies of your response to this survey for your records?
o (If yes) What format would you prefer?
• What has been your experience with using internet reporting systems for surveys or
other reporting?
Wrap-up Questions
• (If previous R) About how long did it take you to complete the survey before?
• How long do you think it would take you to complete these questions?
• Any other feedback you would like to give us on these questions or (If 2014 R) your
experiences with the survey?
Thanks for your time!

4