Various Economic Area Pretesting Activities

Draft 11-21-13

2012 Survey of Business Owners

Respondent Debriefings

Draft Protocol

Heidi M. St.Onge

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)

Before the interview, document the following:

• Respondent/Nonrespondent [R , NR]

• Response mode(s) [W, P , OP, BOTH]

• Number of Follow-ups, if any [FU1L, FU1P, FU1L , FU2P]

• Received/requested paper form [RQF , RF]

• Attempted web reporting but not successful [BO , OR]

• Reported on paper and is a “Special case” [SCY , SCN]

• Called into the Help desk [HY , HN]

• Reported to Economic Census (paper or Web) [ECP , ECP]

Expected Length of Interview: 15-20 minutes

This protocol is divided into the following sections:

• Section A: Questions for all respondents and nonrespondents

• Section B: Questions for online and paper respondents

• Section C: Questions for respondents who used a combination of online and paper (breakoffs and “Over-respondents”)

• Section D: Questions for all nonrespondents

• Section E: Additional questions for respondents and nonrespondents (time permitting)

Research Questions to Address:

1. What effect, if any, did the web-push strategy have on response and nonresponse?

2. Which aspects of the initial mailing and follow-up materials were helpful or detrimental and what could be improved (i.e., messages, envelopes, etc.)?

3. What role does the paper form play in response to the Survey of Business Owners (SBO)?

4. Why do respondents choose to report electronically or by paper form?

5. Why do respondents break off after attempting to use the electronic instrument? What usability issues within the electronic instrument lead to nonresponse, breakoffs, failure to submit, etc.? Do respondents break off because of the sensitivity or difficulty of specific questions?

Instructions:

Prior to initiating a call, please review details on the response mode and nonresponse status of the target respondent. All of the respondents and nonrespondent being interviewed will answer Section A. Respondents and nonrespondents will also be asked questions from one subset of one additional section of the protocol (Sections B-D) based on their response mode or nonresponse status. Additionally, we may ask respondents a third set of questions from one additional subset of questions from Section E, providing respondents fit the criteria for that section and time permits. Please identify the sections you will be asking each respondent and print these sections prior to beginning the phone call.

Introduction:

This year we asked you to participate in our Survey of Business Owners. In order to improve the survey, we would like to get your feedback on the materials we sent you and our survey website. . Would you mind taking a few minutes to share your experience with our survey request?

Section A: Questions for all Respondents and Nonrespondents

1. Do you remember the letter/s we sent?

(If respondent answers “Yes” continue with the following questions.)

• (If R received more than one letter) How many letters do you recall receiving?

• What was your reaction to the letter/s? Would you say your reaction was positive, negative, or neutral? What information from the letter/s, if any, do you recall?

• Was there anything you found useful about the letter? Anything not useful?

• Did you have any concerns about the letter/s? (If yes) What were they?

• What were your overall impressions of the envelope the letter was in? Did it look the same as other mail you typically get, or did it look different?

• How is the mail generally handled in your company? How did the letter reach you? Did it come directly to you, or did it change hands before it reached you?

• Is this type of envelope handled differently from a larger package?

• If we had sent you a paper form first, do you think you would have filled out the form and mailed it back or completed the survey online?

(If respondent answers “No” continue with the following questions.)

• How is mail generally handled in your company?

• How are letters from the government handled within your company?

• Would a letter from the Census Bureau come directly to you, or would it change hands before it reached you?

• Would a standard sized envelope be handled differently than a larger package?

• If we had sent you a paper form first, do you think you would have filled out the form and mailed it back or completed the survey online?

Section B: Questions for Online and Paper Respondents

1. For respondents who completed the survey online (ask the same questions of respondents who requested a paper form):

Our records indicate that you completed the form online. What made you choose to respond online instead of by paper?

• Did you have any concerns about reporting online? Did you have any concerns about the security of the website?

• Did you experience any issues entering the user ID and password into the system? (If yes) what were they?

• What were your overall impressions of the online survey?

• How easy or difficult was it for you to complete the online form?

• Did you encounter any difficulties that you remember? (If yes) What were they?

• Any suggestions for improving the online survey?

• Did you use a copy of the paper form to help you complete the survey? (If yes) How did you use this paper form? What would you have done if we did not send a paper form?

• Did you keep a copy of the paper form for your records?

• Did you have to contact the Census Bureau in order to complete the form online?

2. For paper respondents who completed the survey:

(If R answered “Yes” to Question 1 in Section A)

I see from our records that you completed and returned the paper form; what made you choose to respond that way instead of reporting online?

• Did you attempt to go to the website listed in the letter?

(If yes) What happened when you tried to go to the website? Did you encounter any problems?

(If no) Do you have access to the internet? (If yes) How easy or difficult did you expect it would be to respond online? How would you describe the quality of your internet access?

• Did you have any concerns about reporting online? Did you have any concerns about the security of the website?

• Did you have to contact the Census Bureau in order to complete your form?

• Did you keep a copy of the paper form for your records? What did you use it for?

• What would you have done if we did not send a paper form?

(If R answered “No” to Question 1 in Section A)

• If you had seen the letter first, asking you to report online, would you have reported online or would you prefer to send in the paper form?

• Do you have access to the internet? (If yes) How would you describe the quality of your internet access?

• How easy or difficult did you expect it would be to respond online?

• Did you have to contact the Census Bureau in order to complete your form?

• Did you keep a copy of the paper form for your records? What did you use it for?

• Do you have any suggestions for letting people know that they have the option of filling out the survey online?

3. For paper respondents who completed the survey by printing off the website “special” cases:

I see from our records that you completed and returned a copy of a paper form (and you seemed to have printed a screenshot/old version of the form):

• How did you get the paper copy that you returned?

• Which reader did you use to open the PDF? Were you using Adobe Acrobat Reader? Adobe Acrobat Pro? Pdfonlinreader.com? Something else?

  • Which version of that reader were you using?

• Are you using Microsoft Windows, Mac OS, or something else?

  • Which version are you using?

• What browser did you use? Which version?

• Did you view the document in your browser or did you save to disk?

• What kind of printer were you using?

• Did you distribute the PDF to other users before printing and submitting?

(If R answered “Yes” to Question 1 in Section A)

I see from our records that you completed and returned the paper form; what made you choose to respond that way instead of reporting online?

• Did you attempt to go to the website listed in the letter?

(If yes) What happened when you tried to go to the website? Did you encounter any problems?

(If no) Do you have access to the internet? (If yes) How easy or difficult did you expect it would be to respond online? How would you describe the quality of your internet access?

• Did you have any concerns about reporting online? Did you have any concerns about the security of the website?

• Did you have to contact the Census Bureau in order to complete your form?

• Did you keep a copy of the paper form for your records? What did you use it for?

• What would you have done if we did not send a paper form?

(If R answered “No” to Question 1 in Section A)

• If you had seen the letter first, asking you to report online, would you have reported online or would you prefer to send in the paper form?

• Do you have access to the internet? (If yes) How would you describe the quality of your internet access?

• How easy or difficult did you expect it would be to respond online?

• Did you have to contact the Census Bureau in order to complete your form?

• Did you keep a copy of the paper form for your records? What did you use it for?

• Do you have any suggestions for letting people know that they have the option of filling out the survey online?

Section C: Questions for Respondents Who Used a Combination of Online and Paper (Breakoffs and “Over-respondents”)

1. For online respondents who switched to and completed via paper after starting online:

Our records indicate that you responded online for part of the survey but then reported on paper. How did you find responding online?

What were the reasons you did not complete the form online?

• Did you encounter any problems? (If yes) what were the problems you had?

• Did you experience any issues entering your ID/password? (If yes) what were they?

• Did you visit any additional pages on the website, like the instructions or other information about the survey?

• Did you contact the Census Bureau at any point while completing the form?

Do you have any suggestions for improving our online instrument?

2. For respondents who reported on both online and paper:

I see from our records that you responded online. Did you encounter any difficulties with reporting online? (If yes) what were they?

• Did you experience any issues logging into the system? (If yes) what were they?

• Did you experience any issues navigating through the electronic instrument? (If yes) what were they?

• Did you visit any additional pages on the website?

• Do you have any suggestions for improving the online survey?

Our records indicate that you also responded on paper. Can you tell me what made you also choose to report via paper?

• Do you prefer reporting online or by paper?

• What could we do to encourage online response in the future?

Section D: Questions for all Nonrespondents

1. For nonrespondents who requested a paper form:

Our records indicate that you requested a paper form. What made you choose to request a paper form?

• How did you request your paper form? Online? Over the phone? What were your overall impressions of this process?

• Did you ever receive the paper form?

• Did you encounter any difficulties with reporting that you remember? (If yes) what were they?

• The Census Bureau is considering not sending out paper forms. How would you feel about getting a letter pointing you toward reporting online instead of form in the mail?

2. For online respondents who breakoff after starting online and become nonrespondents:

Our records indicate that you responded online for part of the survey but then reported on paper. How did you find responding online?

• What were the reasons you did not complete the form online?

• Did you encounter any problems? (If yes) what were the problems you had?

• Did you experience any issues logging into the system? (If yes) what were they?

• Did you experience any issues navigating through the electronic instrument? (If yes) what were they?

• Did you visit any additional pages on the website?

• Did you contact the Census Bureau at any point while completing the form?

  • What could we do to improve our online instrument?

3. For all other nonrespondents:

• What can we do to ensure someone will receive and open our mail in the future?

Section E: Additional Questions for Respondents and Nonrespondents (Time Permitting)

1. For all respondents and nonrespondents who requested assistance:

I see from our records that you called into the Census Bureau to request help? What issues were you calling about?

• What was your experience contacting the Census Bureau?

• Was your problem resolved?

• Did you have to call more than once to get your questions answered?

• What were your overall impressions of this process?

2. For paper respondents that received or requested paper for SBO and reported online for the Econ Census:

Our records indicate that you reported online to the Economic Census survey. What made you choose to respond on paper for the Survey of Business Owners?

• Did you consider responding online for the Survey of Business Owners? Why or why not?

• Where there any specific reasons you chose to report online for the Economic Census?

• What were your overall impressions of the Survey of Business Owners?

• What are thing things you like about reporting online? What are some things you do not like about reporting online?

• What could we do to encourage online response in the future?

• Any suggestions for improving our online surveys?

3. For online respondents to SBO who reported by paper for the 2012 Economic Census:

Our records indicate that you reported on paper to the Economic Census survey. What made you choose to respond online to the Survey of Business Owners?

• Did you consider responding on paper for the Survey of Business Owners? Why or why not?

• Where there any specific reasons you chose not to report online for the Economic Census?

• What were your overall impressions of reporting online for the Survey of Business Owners?

• Did you encounter any difficulties with the online version of the Survey of Business Owners? (If yes) what were they?

• What could we do to encourage online response in the future?

• Any suggestions for improving our request for participation in the online survey?

d. For SBO nonrespondents who reported online to the Econ Census:

• Are you responsible for all government surveys? Do other people in the organization complete other Census Bureau forms?

• Did you complete the Economic Census in 2012? (If yes) what are your reactions to reporting online to something like the Economic Census versus something like the SBO?

• Does the time of year influence your response to survey requests? How about the due date?

  • Which times of year are best for you?

• What could we do to encourage online response in the future?

Thank you for your time and assistance today!

SBO Draft Protocol

11