Various Economic Area Pretesting Activities

4/18/2014

Current Surveys Letter-Only Tests – Respondent and Nonrespondent Debriefings Protocol

Prepared by Dave Tuttle, Response Improvement Research Staff

This debriefings protocol is to be used for the following surveys:

• 2013 Annual Retail Trade Survey (ARTS)

• 2013 Annual Wholesale Trade Survey (AWTS)

• 2013 Annual Capital Expenditures Survey and Information and Communications Technology Survey (ACES/ICT)

Before conducting the debriefing, review the values for the following variables from the recruiting and paradata files:

• Respondent or nonrespondent

• Mode – Electronic (ER), self-print paper form, requested paper form, paper form follow-up

• Prior year mode – Electronic or paper form

• Apparent usability issues – Logged in but failed to submit, submitted both electronic and paper, etc.

• Check-in date

• Mailing type to which R responded – Initial mailing, F1, F2, etc.; Letter-only or paper form follow-ups (if applicable)

Sections of protocol

• Part I – Respondents

  • Part Ia. – All respondents

  • Part Ib. – Electronic respondents

  • Part Ic. – Paper respondents

• Part II – Nonrespondents

Debriefing procedures

• For each debriefing case, review the values for the above variables from recruiting and paradata files, to determine which sections of the protocol to use.

• Instructions for variable-specific questions are found at the beginning of each section. Ask variable-specific questions as appropriate.

• Part Ia. should be asked of all respondents, but not nonrespondents. Parts Ib., Ic., and II should each only be asked of electronic respondents, paper respondents, and nonrespondents, respectively.

Introduction (all debriefings)

• My name is ____ from the United States Census Bureau. I am calling in regard to the [name of survey].

• In order to reduce the cost of our program, we decided to mail the survey request without a form this year. So now we are following up with companies that received the request in order to get an idea of how that plan worked and how we might need to improve for next year.

Screener for respondents

• I have you listed as the person who completed the [survey], is that correct?

• (If yes) I would like to ask you a few questions, which should take about five minutes. Would that be ok?

• (If no) Would it be possible for you to put me in touch with the person who completed the survey?

Screener for nonrespondents is included in Part II.

Part Ia. – All respondents

1. Do you recall the letter(s) we sent you about the survey?

If no, skip to Q2.

If yes:

1a. What do you recall about it/them, if anything?

1b. Does anything stand out to you about the content of the letter? Anything positive? Negative?

1c. Does anything stand out to you about the appearance of the letter? The envelope? Anything positive? Negative?

1d. In your opinion, did the letter contain adequate information? If no, what additional information would you want to find in a letter like that?

1e. What did you think about the fact that we asked you to go to our website instead of sending a form?

2. How did you decide whether or not to respond to the survey?

3. How would you feel about getting email notifications about surveys?

3a. What information should they contain?

3b. When should we send them?

Part Ib. – Electronic respondents

Note whether the following applies and ask as appropriate:

• Submitted ER after paper form FU, request, or Self-print – Q9-9c

• Submitted both ER and paper – Q10-10b

4. How did you decide to complete the survey electronically?

5. While completing the survey electronically, did you also use any of the following?

5a. Print outs of screen from survey web pages? If yes, how?

5b. PDF version of survey form from web site? If yes, how?

5b1. If PDF – Paper, electronic, or both?

5c. Spreadsheet If yes, how?

6. Do you have any suggestions for other types of documents or materials that we could provide to help you in completing the survey?

7. Did you encounter any problems while using the electronic reporting website for this survey? If yes, what kinds of problems?

8. Did you have any concerns about reporting information to a website?

If yes – What kinds of concerns? Does it make a difference if it is a government website?

Submitted ER after receiving paper form FU, requesting paper form, or paper form Self-Print

9. Our records indicate that you submitted your data through the website after you [received/requested/printed] a paper copy of the form. Did the paper form play a role in the process of completing the survey?

9a. If yes, how did you use the paper form?

9b. Would it have made a difference if no paper form were available?

9c. If yes, what kinds of information would you need if no paper form were available? How should we make that information available to you?

Submitted ER and paper

10. Our records indicate that we received your data through the website as well as via a completed paper form. Did you intend to submit the data both ways?

If yes

10a. What made you decide to submit paper and electronic surveys?

If no

10b. Do you have any suggestions for making it clearer that [you only need you to submit once/your data were received via the website]?

Part Ic. – Paper respondents

Note whether any of the following apply and ask as appropriate:

• Received a form follow-up – Q15

• Requested a paper form – Q16-16b

• Returned a self-print form – Q17-18

• Prior ER – Q19

11. How did you decide to complete the survey using the paper form?

12. Did you attempt to use the electronic reporting website?

If R did NOT attempt to use ER:

12a. Did you consider using the electronic reporting system? If yes – What made you decide not to use it?

If R attempted to use ER:

12b. Did you encounter any problems with it? If yes – What kinds of problems?

13. Did you have any concerns about the security of the website? If yes – What kinds of concerns?

14. Did you use the paper form to assist you in…

14a. …Consulting with other people? If yes, how?

14b. …Gathering data from other people? If yes, how?

Received paper form follow-up

15. We sent you a copy of the form in a follow-up mailing; if we had not sent a copy of the survey form, do you think you would have completed the survey?

Requested a paper form

16. Were you aware of the option to print out a copy of the form to use to complete and return the survey?

If R was aware of self-print:

16a. How did you decide whether or not to use it?

If R was NOT aware of self-print:

16b. Is that something you would use in the future?

Returned a self-print form

17. What did you think about the option to print out the form and use it to complete the survey?

18. How did you decide to use that option?

Prior electronic reporters

19. In the past, we received electronic responses for the [NAME OF SURVEY] from your company. Is there any particular reason you decided to send in a paper form this year?

All

20. If in the future the [NAME OF SURVEY] did not provide a paper form, how would that impact your reporting process?

21. Are there any tools we could provide that could take the place of the paper form?

Part II – Nonrespondents

Note whether any of the following apply and ask as appropriate:

• Break-offs – Q23-23d

• Letter-only follow-ups – Q25

• Requested a paper form – Q26

22. According to the most recent records I have, we have not received a response to the [NAME OF SURVEY] from your company. I have you listed as the contact person for the [survey] – Do you recall receiving the survey request, which would have been around [about one week after mailout]?

If R does not recall the request:

22a. Is it possible our letter went to someone else in the company?

If yes – Would it be possible for me to speak to that person?

(Obtain name and phone number and terminate call.)

If no – Is there anything we can do in the future to ensure that the survey reaches the right person in the company?

If the request went to someone else:

22b. Would it be possible for me to speak to that person?

(Obtain name and phone number, thank them for their time, and terminate call. Recommence question sequence with other person.)

Once the correct recipient is contacted, introduce reason for call and proceed with interview:

22c. What do you recall about the letter(s)?

22d. Does anything stand out to you about the content of the letter? Anything positive? Negative?

22e. Does anything stand out to you about the appearance of the letter? How about the envelope? Anything positive? Negative?

22f. In your opinion, did the letter contain adequate information? If not, what additional information would you want to find in a letter like that?

22g. What did you think about the fact that we asked you to go to our website instead of sending a form?

22h. How did you decide whether or not to complete the survey?

Note – DO NOT ask Q23 for Break-offs – skip to Q24.

23. Did you attempt to complete the survey online?

If yes, skip to Q24b.

If no, continue with next applicable group.

Break-offs

24. Our records indicate that you logged in to the website, but did not submit your completed survey. Did you intend to complete the survey, or did you decide to stop before completing it?

If R intended to stop

24a. What made you decide to stop completing the survey?

If R did not intend to stop

24b. Do you recall having any problems with the electronic reporting web site?

If yes – What kinds of problems?

Both

24c. Did you attempt to get help by calling the Census Bureau?

If yes – Was the call helpful or not helpful?

24d. Did you look for information to assist you on the website?

If yes – Were you able to find any helpful information, or were you not able to find any information relevant to the problem at hand?

Letter-only follow-ups

25. In terms of your decision whether to respond, would it have made a difference if we had sent a paper copy of the survey form? If yes, how would that have affected your decision?

Requested a paper form

26. Did you receive the paper form that you requested?

If yes – How did you decide whether to complete the survey?

All nonrespondents

27. Did you have any concerns about reporting information to a website? If yes – What kinds of concerns? Does it make a difference if it is a government website?

28. Did you have any concerns about answering any specific questions online? If yes – What kinds of concerns?

29. Were you aware that you could print out a copy of the form and use that to complete the survey? If no, would that have affected your decision whether to respond?

Closing – Thank you for your time, and your assistance with our survey.

8