Protocol

Cognitive Testing of Privacy & Confidentiality Questionnaire

PARTICIPANT ID #: _________________________ DATE: ____ / ____ / 2018

INTERVIEWER’S NAME: _________________________________

START TIME: ______: ______ AM / PM

SECTION 1. INTRODUCTION

Hello, I’m [NAME OF INTERVIEWER]. I work for the Census Bureau. Thanks for agreeing to help us today. Let me start by telling you a little about what we will be doing today. The United States Census Bureau counts the population in the U.S and also conducts various kinds of surveys.

Today we are going to look at a questionnaire similar to the one that will be used to count the population in 2020 (which many people just call the “census”) and also a brief survey about how census data are used and protected. First I’m going to ask you to fill out a version of a census questionnaire, then I’ll give you the brief follow-up survey to fill out, and then I’ll ask you some questions about the survey questions: what they mean to you, and what your reactions to them are. Do you have any questions about what we’ll be doing?

SECTION 2. INTERVIEW CONSENT

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. Our session today is completely confidential, so all information you provide is protected and we won’t use your name in any report. Your participation in this study is completely voluntary. You may decline to answer any particular question. This form also asks for your permission to have this session audio recorded. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes or having to rely on my memory later. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS FORM THEN SIGNS AND DATES FORM

INTERVIEWER SIGNS AND DATES FORM

IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the audio recorder.

SECTION 3. Decennial Questionnaire

PLACE THE DECENNIAL QUESTIONNAIRE IN FRONT OF PARTICIPANT

Please take a few minutes to fill out this questionnaire. As I mentioned earlier, this is similar to the questionnaire that will be used to count the U.S. population in 2020. Although we’ll talk about the questions in this survey a little bit later on, I won’t be able to answer any questions about it until later, so please fill out the questionnaire as you would if I wasn’t here.

ALLOW UP TO 25 MINUTES FOR INTRODUCTION AND DECENNIAL QUESTIONNAIRE ADMINISTRATION. IF RESPONDENT IS TAKING TOO LONG, ENSURE THAT THEY HAVE AT LEAST ANSWERED ALL QUESTIONS FOR PERSON 1 AND 2, THEN SAY: In order to make sure we end this interview on time, we are going to move to the next part of the interview. CONTINUE TO “SECTION 4. PRIVACY & CONFIDENTIALITY QUESTIONNAIRE” INSTRUCTIONS BELOW.

SECTION 4. Privacy & Confidentiality Questionnaire

THINK ALOUD PRACTICE

For the next set of questions, I’d like you to do something a little different. Continue to read the questions and write down your answers, but I’d also like you to think aloud as you answer the questions. Tell me everything you are thinking about as you go about answering each question. I may remind you to think aloud as we go through the survey questions.

Let’s begin with a practice question. Remember to try to think aloud as you answer.

Practice Question 1. How many windows are there in the place where you live? _____________

IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

PROBE IF NEEDED:

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking.

Okay, let’s move on.

Privacy & Confidentiality Questionnaire Items

PLACE THE PRIVACY & CONFIDENTIALITY QUESTIONNAIRE IN FRONT OF PARTICIPANT

Please take a few minutes to fill out this questionnaire, and remember to think aloud as you answer the questions.

[WAIT FOR RESPONDENT TO ANSWER QUESTIONS 1 – 24]

Section 1 Probes:

1. Now let’s switch gears and talk about the set of questions you just answered. First, let’s talk about some of the words and phrases that were used in the questions.

1. All of the questions in this section asked whether it would “concern you if someone was able to find out” some information about you or your household. What does the word concern mean to you in the context of these questions?

2. What kinds of concerns were you thinking of when you answered these questions?

3. [IF NEEDED] If different questions brought up different concerns for you, it would be helpful for you to talk about each question separately.

1. [IF NEEDED] Some of these questions asked about concerns you have about your own information, while others asked about concerns for information of other people in your household. Were the concerns that you thought about different when you were thinking about your own information and the information of other people in your household? Was it any different when you were thinking about children in your household?

2. [IF R ANSWERED “NO” FOR ALL BINARY Qs IN 1 THROUGH 24] It looks like you didn’t have any concerns about someone finding out the types of information asked about in these questions. Can you say more about why you are not concerned? What concerns, if any, do you think other people might have?

4. Would any of your answers be different if we asked you, “Would it worry you if someone was able to find out your age, race, etc?”

1. [IF YES] Which items would you give a different answer for?

2. Can you tell me more about why you would give a different answer to [that/those] items?

3. [IF NEEDED] What is the difference between something causing you concern and something causing you to worry?

1. All of the questions in this section also referred to “someone” who might find out information about you or your household. Were you thinking about any type of person in particular when these questions talked about “someone” finding out your information? If so, who?

6. Some of the questions asked about “other people in your household”. Who came to mind when you were thinking about the people in “your household”? Did you think about both adults and children?

[IF R REPORTED ANY CONCERNS] Next, let’s talk about some of the specific questions.

7. Looking back at question [INSERT QUESTION NUMBER] [READ QUESTION], could you tell me more about the concerns you thought you might have if someone was able to find out [INSERT INFORMATION TYPE]?

[REPEAT I.7 FOR EACH QUESTION WHERE R INDICATED SOME CONCERNS]

8. [IF R ANSWERED “YES” FOR (question about all the information)] You answered that you would be concerned if someone was able to find out ALL OF THE INFORMATION included in the Census Bureau questionnaire, but you might have been thinking about some specific combinations of information (for example, your name and home ownership status) when you decided on your answer. I’d like to know more about your concerns for someone finding out particular combinations of information. Here is a list of all the different types of information you were asked about [GIVE R HANDOUT]. Please look over the list and then check off the top 3 or 4 pieces of information that would concern you if someone was able to combine them.

1. [IF NEEDED] Tell me more about how you decided which pieces of information would be the most concerning to you if someone was able to combine them.

2. Are there any other combinations of information from this list that would be especially concerning to you?

That’s all the questions I have for this section of the survey. Please continue answering the questions, and I’ll ask you some more questions when you get to the end of the next section. Remember to think aloud while you read the questions and come up with your answers.

[WAIT FOR RESPONDENT TO ANSWER QUESTIONS 25 – 30]

Section 2 Probes:

2. Now let’s switch gears and talk about the set of questions you just answered. First, let’s talk about some of the words and phrases that were used in the questions.

1. First, please tell me in your own words what the phrase “data breach” means to you.

[INTERVIEWER NOTE: For the next few probes, we are interested in the diversity of real-world examples that the respondent has heard of.]

2. [IF R DID NOT GIVE ANY REAL-WORLD EXAMPLES] Sometimes it’s easier to explain a concept with examples than with a definition. When you think of the phrase “data breach,” do any real-world examples come to mind?

3. [IF R ALREADY GAVE REAL-WORLD EXAMPLES] Can you think of any other real-world examples of a “data breach”?

4. Next, please tell me in your own words what the term “hacking” means to you.

5. [IF R DID NOT GIVE ANY REAL-WORLD EXAMPLES] When you think of the term “hacking,” do any real-world examples come to mind?

6. [IF R ALREADY GAVE REAL-WORLD EXAMPLES] Can you think of any other real-world examples of “hacking”?

[INTERVIEWER NOTE: Although we do not explicitly mention this in our definition or examples, when someone combines an anonymous dataset with insider/personal information to identify an individual (known as a “motivated individual attack”) this is also a type of re-identification.]

7. Next, please tell me in your own words what the term “re-identification” means to you.

8. [IF R DID NOT GIVE ANY EXAMPLES] When you think of the term “re-identification,” do any examples come to mind?

9. [IF R ALREADY GAVE EXAMPLES] Can you think of any other examples of “re-identification”?

10. [IF R DOESN’T SEEM TO UNDERSTAND THE CONCEPT OF RE-IDENTIFICATION] The paragraph introducing questions 25 through 27 gives a definition and an example of re-identification [READ LAST TWO SENTENCES OF PARAGRAPH ALOUD]. Looking back at the definition and example, does it change your understanding of what re-identification means? [IF NEEDED] Is there anything about the definition or the example that is unclear or confusing?

Next, let’s talk about some of the specific survey questions.

11. [IF NEEDED] Looking back at question 27 [POINT TO QUESTION], please tell me a little more about how you came up with your answer.

12. Let’s look at question 28. [READ QUESTION ALOUD] Now that you’ve looked at the question again, please tell me in your own words what you think this question is asking.

13. [IF NEEDED] Looking back at question 30 [POINT TO QUESTION], please tell me a little more about how you came up with your answer.

That’s all the questions I have for this section of the survey. Now there is only one section of the survey left. Please finish answering the questions, and I’ll ask you some more questions when you get to the end of the survey. Remember to think aloud while you read the questions and come up with your answers.

Section 3 Probes:

2. Let’s switch gears one more time and talk about the set of questions you just answered. First, let’s talk about some of the words and phrases that were used in the questions.

1. Looking back at question 31 [POINT TO QUESTION], please tell me a little more about how you came up with your answer.

[INTERVIEWER: WE WANT TO SEE WHETHER THE SHORT RESPONSE OPTIONS MADE THEM LOSE THE REFERENCE TO THE TRADE-OFF]

1. [IF NEEDED] Was it easy or difficult for you to come up with an answer to this question? What made it [easy/difficult]?

2. Were you thinking of specific risks when answering this question? [IF NEEDED] What risks were you thinking of?

3. Were you thinking of specific benefits of statistics when answering this question? [IF NEEDED] What benefits were you thinking of?

[IF R REPORTED ANY CONCERNS IN 32-35] Next, let’s talk about some of the specific questions.

2. Looking back at question [INSERT QUESTION NUMBER] [READ QUESTION], could you tell me more about the concerns you thought you might have someone was able to find out [INSERT INFORMATION TYPE]?

[REPEAT III.2 FOR EACH QUESTION WHERE R INDICATED SOME CONCERNS]

3. [IF R ANSWERED “NO” TO ALL IN 32-35] It looks like you didn’t have any concerns about someone finding out your social security number or income. Can you say more about why you are not concerned? What concerns, if any, do you think other people might have?

Section 4 Probes:

2. Looking back at question 36 [POINT TO QUESTION 36], in your own words, can you tell me what the question is asking?

   1. Tell me more about how you came up with your answer.

2. Let’s talk about question 39. Can you tell me in your own words what this question is asking?

   1. Tell me more about how you came up with your answer.

3. What about question 42? Can you tell me in your own words what this question is asking?

   1. Tell me more about how you came up with your answer.

4. And question 43, can you tell me in your own words what this question is asking?

   1. Tell me more about how you came up with your answer.

5. What about question 44. Can you tell me in your own words what this question is asking?

   1. Tell me more about how you came up with your answer.

Changing the focus from the Census Bureau to Social Media,

IV.6. Do you think that Facebook does a good job of providing information to the public about how it keeps data confidential?

□ Yes

□ No

□ Don’t know/Refuse

IV.6.A Tell me about how you came up with your answer.

IV.7. How much of the time do you think you can trust Facebook to do what is right?

□ Just about always

□ Most of the time

□ Only some of the time

□ None of the time

□ Don’t know/Refuse

IV.7.A Tell me about how you came up with your answer.

IV.8 Overall, would you say that you trust the Census Bureau or Facebook more with your personal data?

5. I just have a couple of final questions to wrap up.

1. Is there anything else you want to add that you haven’t had a chance to talk about yet?

2. [IF R SEEMED TO HAVE COMMENTS ABOUT OR DIFFICULTY WITH DECENNIAL FORM] Is there anything you wanted to discuss about the census questionnaire you filled out at the beginning? Was there anything about it you found confusing or difficult to answer?

SECTION 5. DEBRIEFING

6. Interviewer observations:

1. What were your general impressions of this respondent and interview?

2. Did the respondent seem to have any language difficulties that may have impacted the interview?

3. Did you notice anything else that may have affected the respondent’s understanding of the main concepts?

4. Is there anything you would like us to know about this interview for the analysis?

9