Attachment F - Cognitive Testing Report

ATTACHMENT F

COGNITIVE INTERVIEW REPORT

Cognitive interview Report 12-051

National Survey of Business Competitiveness

September 2012

Prepared for

Timothy Wojan

Resource and Rural Economics Division

Economic Research Service, USDA

(202) 694-5419

[email protected]

Prepared by

Danna L. Moore, Ph.D.

Principal Investigator

Yi Jen Wang, M.A.

Study Director

and

Kent Miller, M.A.

Study Director

on behalf of SESRC

SESRC

Social & Economic Sciences Research Center (SESRC)
PO Box 644014
Washington State University
Pullman, Washington 99164-4014
Telephone: (509) 335-1511
Fax: (509) 335-0116

National Survey of business competitiveness

September 2012

ERSR10

Submitted by

Danna L. Moore, Ph.D.

Principal Investigator

Yi Jen Wang, M.A.

Study Director

and

Kent Miller, M.A.

Study Director

Social & Economic Sciences Research Center

PO Box 644014; Wilson-Short Hall 133

Washington State University

Pullman, WA 99164-4014

509-335-1511

509-335-0116 (fax)

[email protected]

SESRC Project Profile

Title: National Survey of Business Competitiveness

Objectives: The primary purpose of this report is to summarize the findings from the cognitive interview testing conducted on September 17 – 18, 2012 in the Pullman, WA and Moscow, ID area.

Method: Using both retrospective and concurrent interview methods to test the paper questionnaire and the telephone script.

Timeframe: September 17 and 18, 2012

Sponsor: Timothy Wojan

Resource and Rural Economics Division

Economic Research Service, USDA

355 E Street SW

Washington, DC 20024

(202) 694-5419

[email protected]

Principal Investigator: Danna L. Moore, Ph.D.

Study Directors: Yi Jen Wang, M.A. and Kent Miller, M.A

SESRC Acronym: ERSR10

Deliverables: Brief summary, updated questionnaire, recruitment letter, and a copy of the voice recording consent form.

Contents

SESRC Project Profile i

I. Cognitive Interview Testing 1

Table 1. Characteristics of the Six Interviews 2

II. Interview Summary 2

Addendum I. Updated Questionnaire after the Interviews 18

Addendum II. Recruitment Letter Used for Cognitive Interviews 34

Addendum IIII. Audio Recording Consent Form 35

1. Cognitive Interview Testing

On September 17^th and 18^th, 2012, a total of six cognitive interviews were conducted to test the National Survey of Business Competitiveness questionnaire for the USDA Economic Research Service’s REIS project.

Four businesses were recruited to complete the paper questionnaire on September 17^th. The first two interviews used a retrospective interview method, and the last two used a concurrent interview method. Two more businesses were recruited to complete a phone interview and a paper questionnaire on September 18^th. A retrospective interview method was used for the phone interview and a concurrent interview method was used for the paper questionnaire. A retrospective method is when the respondents completes the questionnaire and then answers questions about it. A concurrent interview method is when the respondents answer the question while at the same time “thinking aloud” about the process s/he is going through to answer the questions.

All respondents (excluding the phone interview respondent) were given the recruitment letter which tells them the purpose of this interview and were asked to sign an audio recording consent form prior to the interview. They received a cash incentive of $40 dollars as a token of appreciation after the interview was complete. The telephone interview respondent was read the confidentiality statement at the beginning of the phone interview and understood that the whole interview was recorded. His $40 cash was mailed to him the day after he completed the interview.

Table 1 shows the characteristics of the six interviews.

Table 1. Characteristics of the Six Interviews

Interview #	Date	Length (minutes)	Interview Type	Mode of questionnaire	Respondent Gender
R1	9/17/12	27:09	Retrospective	Paper	Male
R2	9/17/12	27:00	Retrospective	Paper	Male
R3	9/17/12	19:25	Concurrent	Paper	Male
R4	9/17/12	Over 40	Concurrent	Paper	Male
R5	9/18/12	39:34	Retrospective	Telephone	Male
R6	9/18/12	47:18	Concurrent	Paper	Female

2. Interview Summary

Interview #1

Type of business: Cabinetry

9/17/12 11:00 AM

Respondent gender: Male (referred to as R1 below)

Cognitive Interview Type: Paper Questionnaire & Retrospective

Interview length: 27:09 minutes

Comments and findings from the interview

1. Q5: only area-oriented. Most of the factors not applicable to R1.

2. R1 identified the branching error on Q6.

3. The term “workforce” is understandable to R1.

4. For R1’s business, no education is required for anyone. Work experience is crucial to them though—especially woodworking and equipment use.

5. Q14 is directed for different types of business. R1 didn’t know some of the options. They did use special kind of software but it was all about machinery operations and design layout. They did nothing over the internet except for buying stuff. Their business has a website but never tracks it. It shows customers work done for others.

6. Q24: They saw what happened regarding their work practices but they didn’t really document them. No written procedures for if people get hurt. R1 felt that he and his guys knew how to do things although they didn’t just give out a piece of paper as instructions. No customer satisfaction format form- it’s all personal. If the customer was not satisfied, they would just come in/call and complain. They would work with customer.

7. Q26: Sometimes it did happen but that didn’t come up very often and every complaint is responded to.

8. Q27: Question sounds ambiguous to R1. Due to the nature of their business (cabinetry), their products are not new as a concept. But R1 likes to think that they are getting better. They make modifications, so he thought that’s improved from customers’ point of view. However, whether he thinks it’s significantly improved or not may not make any difference to the customers. Use processes and finish materials and finishes that are new.

9. Q28: Question doesn’t really apply to R1’s business. They are always building something that doesn’t work out well. It’s not like they were going into a product line and then abandoned it if it didn’t work out. They got a new machine but that’s not innovation.

10. Q41: R1 felt that the direction of the scale on this one is not consistent with others in the survey. Suggested change: A major problem-> A minor problem-< No problem.

11. Q42: Suggested to add a Don’t know option.

12. Q44: R1 felt like this question was targeted for larger business.

13. Q46: Again, question not related to R1’s business and he didn’t answer any on this question. Suggested if answered all NOs on Q45, skip to Q47.

Overall impression:

The questionnaire wasn’t hard but some questions were found to be ambiguous. R felt his business was “small potatoes” and the questionnaire is more directed to bigger business. If the survey comes in the mail, it would take an incentive for R1 to do the survey. Also on the contact letter, R1 needs to know right away what the survey is trying to do for his business environment otherwise he probably won’t answer the general questionnaire. R1 needs a reason to do it; otherwise it would be too time-consuming. Suggests moving benefit appeal or reasons to first few sentences in letter and introduction.

Questionnaire changes identified

1. Q5 Break out the single list of 16 factors related to business location into two questions: Q5 asks about 10 factors important for business location and Q5a asks about 6 factors making the community an attractive place to work.

2. Q6 branching change- now if Q6a=2 No, it skips to Q8.

Interview #2

Type of business: Auto Body Repair and Paint Services

9/17/12 1:15 PM

Respondent gender: Male (referred to as R2 below)

Cognitive Interview Type: Paper Questionnaire & Retrospective

Interview length: 27:00 minutes

Comments and findings from the interview

1. Overall impression: No questions are hard to answer but some don’t apply to his business.

2. Q8a: For R2’s business, body men and painters are considered professionals. Have specific skills, experience, and certifications to do the work.

3. Q13: They have written job description for every position even for janitors.

4. Q14: R2’s business has a website that people can have a price check and give feedback on the website. They also spent a couple hundred dollars so the website would appear on top of Google search. The website also serves as advertising over the internet. They don’t buy anything from the internet in terms of auto parts. They do subscribe monthly to use estimate software which is $400 a month.

5. Q17e: question doesn’t apply.

6. Q27: In the last 3 years, they are doing the same thing they’ve always done. They are not producing new products.

7. Q28: Question probably applies to larger companies.

8. Q34: R2 doesn’t have any debt. If he had some, he would probably put down repay debt. And if he was younger, he would probably try innovation. But right now, he is just trying to hold on in this economy.

9. Q38: R2 said that they didn’t see the recession until now. They think it’s because we (this area) are on the “tail of the dog” here. [Interpreted as ”Experience recession later than the rest of the nation”]. This last year has been down [for sales]. Kind of the same thing for businesses at this side. He also thinks that the vitality of the local economy is a big problem right now.

10. When asked what keeps your business competitive, R2 said: reputation, honesty, and “we do the job well”, and good relationship with insurance companies.

Overall impression:

Survey experience was good. But if it comes in the mail, there’s only 50% chance that he’ll do the survey. If there are some perceivable benefits, R2 might consider doing the survey. The length was not too long particularly; however, a lot of the questions don’t apply. Their business doesn’t have cutting edge technology but he certainly likes to be competitive and have been trying things throughout the years.

Questionnaire changes identified

1. Decided to add to Q14j: "Computer software specifically designed for your business or industry."

Interview #3

Type of business: Environmental Engineering

9/17/12 4:20 PM

Respondent gender: Male (referred to as R3 below)

Cognitive Interview Type: Paper Questionnaire & Concurrent

Overall length: 19:25 (for completing the questionnaire)

Comments and findings from the interview

1. Q2: Although their business has been operating since 1981, they just moved into the current building. So R3 was unsure as to “what year” he should put here. The year in the area or the year he moved into this physical location.

2. Q7b, R3 needs a definition for the term “retirement plan.”

3. Q14e: R3 wasn’t sure if business website qualifies as web advertising.

4. Q19: R3 has customers in and outside of the community and he was unsure if he should check both. (Even if customers were not identified as very valuable in Q18, he ended up not checking any answers for the customer.)

5. Q44: R3 asked if he should still respond to “how important is this program” for each program if he didn’t use it.

6. Q45: R3 asked if SBA qualifies for federal government.

7. Q8a: R3 identified employees with engineering/environmental/inspection licenses as professionals.

8. Q10: R3 thought it would be a lot easier to respond to if the question asks the number of employees instead of %.

9. Q12c&d sound redundant to R3.

10. Q14: R3’s business is just getting into “iPad/tablet applications.”

11. Q19, R3 has multiple answers for suppliers and customers but the survey only allows single choice. Also answered the survey wrong due to not reading the question closely.

12. Q27e: R3 wasn’t sure what it meant but sure it didn’t apply. (He still checked No though).

13. Q33: R3 didn’t know the difference between d and g. For him, their business purchased designed apps for Pads, accounting system, etc.

14. Q42c: R3 needed definition. He hasn’t seen any of them.

15. Q45: R3 clarified if personal credit card referred for business use.

Overall impression:

It’s highly likely that he’ll respond to this survey if it comes in the mail. He likes to respond to this kind of survey and thinks it is a help to the business. He stated the opinion that “The survey time was well spent on this questionnaire”.

Questionnaire changes identified

1. Changed Q44 question to: "The final section of the survey is to assess which types of government or government-sponsored programs are the most helpful to businesses. If you used, please rate how important each program has been for this business in the last 3 years." Also added wording "If used," just before the wording of “how important is this program?” on top of the column as part of the header.

Interview #4

Type of business: Airport Services

9/17/12 5:30 PM

Respondent gender: Male (refer to as R4 in below)

Cognitive Interview Type: Concurrent

Overall length: unknown (over 40 minutes when stopped because R4 had service call)

Completed later and then picked up.

Comments and findings from the interview

1. R4 was thoughtful and thought out loud as he answered the questionnaire.

2. Q5h: marked somewhat important as they provide transportation services so they are not reliant for most transportation. Views all these items as very important for locating his business.

3. Q9: Did not know whether to include his pilots and flight instructors who are professionals in this average or not. Not included.

4. Q14: commented on the unreliability of broadband internet service at their location. Commented on the use of Quickbooks as important business software for them as a small business.

5. Q17 & Q18: mentioned that his young employees have taught them and encouraged them to increase technology use and information use. Now have an on-line way for pilots to schedule airplanes for rent and use from his business.

6. Q19: commented that he needed to check more than one of the items for customers and it was not clear from the questionnaire if he could do that for distances of sources. He had customers in all of the distances that were very valuable.

7. Q31: indicated he did not monitor his competitors.

8. Q34: asked if he was to interpret question as: “if money were to drop from the sky—how would I use it”. That was how he answered it.

9. Q41: until just very recently availability of high speed reliable internet was a problem, had to use DSL.

10. Indicated in margins that Aviation fuel did not have a sales tax on it in Idaho however he rated state and local tax rate as a major problem.

11. Q42: did not understand what was meant by “activity level” . Indicated in margins that Banks were a pain to deal with and rated them as not active and not present.

Overall impression:

Easy to complete questionnaire. Answered all questions. Found questions relevant.

Questionnaire changes identified

Asked for definitions in some places and whether to include professional pilots in hourly wage or not.

Definitions needed:

1. Who should be categorized as “professional”. Include pilots and flight instructors. These are large portion of his workforce.

2. Activity level on Q42.

Overall comments on innovation at end of questionnaire: “We have had innovation that would sell in aviation industry, but have not had the time or training to carry through with the development. Sometimes you need to hire the job done to see it get done.”

Interview #5

Type of business: Bolt-on hybrid technology for trucks

9/18/12 11:30 AM

Respondent gender: Male (refer to as R5 in below)

Cognitive Interview Type: Phone & Retrospective

Overall length: 39:34 minutes

Comments and findings from the interview

1. Q8b: Need definition for “professional” for the phone interview.

2. Q34: wordings sound awkward on the phone.

3. R5’s business hasn’t gone to the operation yet. It is still in development phase although they are spending money, buying knowledge, hiring lawyers, etc. However, due to their unique situation, the part that they are not producing anything yet is hard to answer.

Overall impression:

Questionnaire sounds good, no question was invasive. It was a bit too long.

Questionnaire changes identified

1. Changed "your business" to "this business" throughout the questionnaire.

2. Q10: added a "Number of employees" column.

3. Q20: added "Check here if you did not have any sales in 2011."

4. Q21a: added "Check here if you did not have any sales in 2007."

5. Q25: Reversed categories.

6. Q26: Reversed categories.

7. Q42: Reversed categories.

8. Q42: Added a "Don't know" column.

Interview #6

Type of business: Mechanical Engineering

9/18/12 3:30 AM

Respondent gender: Female (referred to as R6 below)

Cognitive Interview Type: Paper Questionnaire & Concurrent

Overall length: 47:18 minutes

Comments and findings from the interview

1. R6 needed more space on Q1 and Q3.

2. Q5: R6 suggested make DK and NA gray out or more standout on the side. Response categories should be reversed so it reads Very important->Somewhat important->Not important. Also suggested to collapse choices by their categories to separate the choices more. Increase separation between legitimate answers and the “missing data options”.

3. Q5d: Question needs to emphasize “locating” more. Underline or make larger bolder font.

4. Q7: R6 suggested add paternity leave to Q7d since most employees in their company are male. Some men also take time off when child is born under family leave act.

5. Q9: R6 wasn’t sure how to answer this since their workforce were all managers and professionals (licensed professional Engineers). The only one was not was the secretary. That was the only person considered for her answer on the hourly rate of workforce. Need definition as to what is considered to be professional and what to do if professionals are your primary workforce.

6. Q12: R6’s answer was actually referred to their managers and professionals.

7. Q12ab: Question was unclear to R6 so she didn’t answer.

8. Q17d: Question unclear. Lack of knowledge of what?

9. Q18: Question unclear.

10. Q20 & Q21a: For many rural area, you don’t get anywhere within one hour drive. R6 suggested that we consider increasing the local distance measure to 2-3 hour drive.

11. Q34: R6 had no debt but no NA category to check. She checked Not at all likely and worried that it might be interpreted as she had debt but didn’t want to repay it.

12. Q36a: R6’s business didn’t apply for patents itself but did collaborate with other companies and helped them with obtaining patents. Suggested to add words like “provide substantially for another company to apply.”

13. Q41b: Vitality of local economy is a minor problem for their company. However, finding a job for their professional engineer workers’ spouses has been a major problem, which doesn’t show in the choices.

14. Q44: R6 wanted more information on these programs. Suggested we offer a web link to find out more about all of these programs for businesses. She suggested that might be an incentive to participate in the survey.

15. Q45 & Q46: Suggested to add instructions if all NOs on Q45, they should skip to Q47.

Overall impression:

Questionnaire is well organized and looks professional.

Questionnaire changes identified

1. Q7 item d. expanded to “Offer paid maternity or paternity leave”

2. Q22: added "Check here if you did not have any sales in 2011."

3. Q45/Q46: Added instruction to skip Q46 if all answers in Q45 are "no."

4. Q9: Changed question to read For 2011, what is your best estimate of the average hourly wage for your non-salaried workers at this location?

5. Q21a Check box should line up with the abc options below.

Overall findings from the testing:

1. A particular focus of the cognitive interviewing was to assess the workability of the auxiliary questions (Q13-Q13b, Q14, Q24-Q26, Q28, Q34, and Q37) that will be used to differentiate substantive innovators from nominal or claimed innovators. Q34 is a hypothetical question that has never been used before, asking about potential uses of surplus funds. All of the Rs answered this question easily. The pilot study will need to assess how easily this question is answered by Rs in very large establishments. The other questions on data driven decision-making related to management practices (Q13-Q13b, Q24-Q26) and technology (Q14) were also easily understood and answered. In this very small sample, responses to the auxiliary questions effectively distinguished the substantive innovators (R3, R5 and R6) from the nominal innovators (R1, R2, and R4).

2. The attempt to define an equivalent to production workers in manufacturing by defining “workforce” as all workers excluding managers and professionals is not appropriate for many service-producing industries. The definition of “workforce” is dropped. Q9 now asks about average wages of non-salaried workers regardless of occupation type. Q10 now asks about the occupational composition of the business and Q10a asks about the minimum educational requirement by OMB occupational group. This will provide better information that should be easier to answer as the response is now regarding qualities by well-defined class of employee rather than qualities of a potentially ambiguous single group of employees.

3. Respondents did not pay close attention to Q19 and all answered the survey question incorrectly. To make “very valuable” stand out more. the response categories are now qualified as “providing very valuable information.”

4. Also for Q19, some respondents have checked multiple answers for one source and our current questionnaire (for phone and web) only allows single response. We changed this into a multiple response category on the phone and web.

5. Questions 36-36b that originally asked about patents the establishment applied for is modified to include participation in patent applications.

6. Several Rs had problems understanding Q42 which used a general “indicate the activity level” construct to correspond with two different types of activities in the question items. To resolve this problem, the question was separated into two questions, the first asking “How involved in promoting businesses” and the second asking “How much civic leadership does THIS business provide in the community?”

7. To reduce burden on the substantial number of respondents with no debt or no borrowing in the past 3 years, Q45 now screens borrowers who go on to complete the list of sources of funds from non-borrowers who skip to Q47.

8. We need to give the respondent a reason to do the survey. Incentives are always welcome and if we want to convince the respondents to do it, it must be presented in the first paragraph in the contact letter. They are interested in knowing how the survey will benefit them if they choose to do it.

9. To accommodate establishments with employees that are still in the development stage, branching in the phone and web survey will avoid questions about sales or shipments after the first occurrence. In the mail survey, all questions regarding sales or shipments will provide a check box if the establishment had no sales or shipments.

Addendum I. Updated Questionnaire after the Interviews

Addendum II. Recruitment Letter

Dear Business Owner/Manager,

You are part of a vital but often unrecognized part of the economy – the small business community – and the USDA's Economic Research Service (ERS) wants to hear from you.

ERS, an independent government agency, is contacting small businesses throughout the United States to understand the challenges that businesses such as yours currently face.

ERS is tasked with understanding the linkages of what keeps businesses vital and thriving with available resources. That information is then used by USDA to develop programs to help businesses. Therefore it is very important to learn about the challenges you face and what is needed for businesses to stay effective.

ERS is a vital source of information that helps communities, businesses, and state and local governments invest, provide services and plan for the future. With statistics and information on rural businesses the ERS enables decision makers and political officials to appropriately fund and develop programs that further small business development in rural communities and take other actions that lead to the long term health of the economy.

The Social and Economic Sciences Research Center (SESRC) is conducting the survey for ERS and plans to have an in-depth interview with a small number of local businesses first to identify any problems or issues in questions, wordings or format visual design.

If you agree to participate, you will be asked to complete the survey questionnaire. Observers from the SESRC will also be there to take notes and interview you further after the interview in order to get your input on how the questionnaire could be improved. The interview will between 30 to 60 minutes, and we will give you $40 as a token of our appreciation for your help in this important study.

Danna L. Moore, Ph.D.

Interim Director

Social and Economic Sciences Research Center

Washington State University

Contact us at 509-335-4169 or by e-mail at [email protected].

Thank you for contributing to this study of our nation’s small businesses.

Addendum III. Audio Recording Consent Form

Washington State University Voice Recordings Consent Form

Social and Economic Sciences Research Center

hereby grant permission to the Social and Economic Sciences Research Center (SESRC) at Washington State University (WSU) to have voice recordings made for SESRC Research purposes to understand comments and questions related to business.

Any other use of images and/or recordings, my name, and/or interview comments requires advance permission.

We understand that we can revoke this consent at any time upon notice to SESRC, at which time either or both of us will sign a copy of the denial (below) for use of images or voice recordings.

We agree to use of voice recordings as set forth above:

________________________________ ________________

Signature of Participant Date

________________________________ ________________

Signature of Witness (required) Date

We do not agree to use of digital images or voice recordings as set forth above:

________________________________ ________________

Signature of Participant Date

________________________________ ________________

Signature of Witness (required) Date

Social & Economic Sciences Research Center
Washington State University

P.O. Box 644014
Pullman, Washington 99164-4014
Telephone: (509) 335-1511 Fax: (509) 335-0116

http://www.sesrc.wsu.edu

[email protected]