Protocol

2017 PA, PRA, FCEA Language Testing

DRAFT PROTOCOL

Aryn Hernandez and Krysten Mesner

July 19, 2017

This protocol is a guide – the questions presented here will not necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.

Note: As much information about interview participants and enterprise/establishments should be obtained prior to the interview as possible.

Part A – Introduction

1. Introduce observers and their background

2. Purpose of visit:

Thank you very much for your time. Let me start by telling you a little about what we will be doing today. We revising the content of our mailed survey requests. We are in the process of redesigning portions of this letter in order to better meet our data users’ needs. We would like to get your input on the different versions of the letter, in order to make them as informative and easy to read as possible.

3. Permission to audio record discussion? Have R sign consent form.

Before we get started, I would like to audio record this interview so I don't have to rely on my memory later. This session is confidential. Only persons connected with this project will have access to your recording. If that is all right with you, please sign this consent form. It also tells you about the confidentiality of this session.

Part B – Background Information

Before we start with more specific questions, I would like to ask you a few questions about this company and your role in it.

1. What is your formal job title?

1. Could you tell me a little about your role and responsibilities?

2. How does your company typically handle survey requests that arrive in the mail?

   1. Do you typically receive the survey request letters?

Part C – Evaluation of Draft Letters

Note: Each respondent should only receive either mandatory letters or voluntary letters, these should be alternated between every other respondent. The order in which each respondent views the letter versions (control vs proposed) should alternate with each respondent. Please note which version the respondent views first.

Now, I would like you to review a draft copy of the revised letter. Please take your time in looking it over. (Note whether respondent flips to back of letter and reads it.)

1. Without looking back at the letter, do you recall reading a section in the letter labeled ‘Authority and Confidentiality?’ (Note: if respondent does not recall the section, allow them to look it over.)

2. Can you tell me in your own words what the ‘Authority and Confidentiality’ section is trying to convey? What is your overall impression of this section?

3. Would you need anyone else in your company to review this information before you were able to respond to the survey?

Specific probes for the Control letters only. If the respondent gets the control letter second, ask these probes when noted later in the protocol.

I’m going to show you some cards. Each card is labeled with a sentence or two from the letter we just looked at. There are no right or wrong answers.

For each card ask:

1. Can you tell me in your own words what this sentence means?

2. Is this sentence clear or unclear?

3. Optional probes for specific cards, only if necessary: Can you tell me in your own word what that phrase means in the context of this sentence?

   1. “can use your responses only to produce statistics?”

   2. “disclosure of this information is permitted?”

   3. “work-related purposes?”

   4. “subject to all of the published routine uses?”

   5. “Privacy Act System of Record Notice titled “COMMERCE/CENSUS-4, Economic Survey Collection”?”

Now, let’s look at another version of this letter. The only difference is the ‘Authority and Confidentiality’ section. Please take a moment to read this section.

1. Is one version more clear than the other?

Note: if the Control version was second, ask specific probes here.

Part D – Card Sort of Letter Content

Note: cards should be shuffled before being given to respondent.

I’m going to give you a small stack of cards. Each card is labeled with a section of the letters we just looked at. Please take a few minutes to put the cards in the order in which you think the information should be presented in the letters. Please talk through your thought process as you arrange the cards. There are no right or wrong answers. (Note the order in which the respondent lists the cards.)

If the respondent has not already stated the answers, ask these probes after they have finished sorting the cards:

1. Why is [item at top of list] most important in your opinion?

2. Out of the items at the bottom of your list, do you feel like any do not belong in the letter?

3. When you receive these letters, do you typically read the entire letter or just parts of the letter?

   1. (If yes) Which parts?

      1. Do you ever find the need to refer back to the parts you didn’t read originally?

Part E – Card Sort of Persuasive Appeals

Note: cards should be shuffled before being given to respondent.

Now, I have another set of cards for you to sort. These cards are labeled with some of the things the statistics that are produced with you data get used for. Please sort these cards into two lists: one list of facts you already knew and one list of facts you did not know. (Note the order in which the respondent lists the cards.)

If the respondent has not already stated the answers, ask these probes after they have finished sorting the cards:

1. In the list of facts you knew, do you think any of these facts are interesting? Why (not)?

2. In the list of facts you didn’t know, did you find any that were interesting? Why (not)?

3. Where would you expect to find the information presented on these cards?

   1. Examples if needed: in the letter, on a website, on a fact sheet included with the letter…

I’d like you to do one last card sort. Please take a few minutes to put these cards in order from most motivational reason to complete the survey to least. (Note the order in which the respondent lists the cards.)

If the respondent has not already stated the answers, ask these probes after they have finished sorting the cards:

1. Why is [item at top of list] most motivational in your opinion?

Part F – Wrap Up

1. What other survey requests do you get from the government – that is, from other organizations besides the Census Bureau?

2. How often do you receive survey requests? (What is the timing?)

3. Do any survey requests come at the same time, or within a couple of weeks of one another?

4. What distinguishes the different survey requests from one another? What do you do to determine if they are the same or different survey requests?

5. How do you sort survey request to keep them organized?

6. How do you prioritize survey requests that you receive around the same time?

7. Do you have any other comments, questions or suggestions about anything we discussed today?

If the respondent is interested, give the CBB: Small Business Edition flyer to small business respondents or the CBB: Regional Analyst Edition flyer to larger business respondents. Also, give them the America’s Economy Mobile App info sheet.

Thank you for your help today!

4