SCOPE Confidentiality Language Testing

DRAFT In Person Letter Protocol

Protocol: Letter variant (with eye tracking)

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. I will be working with you today on a usability study. Our group evaluates how easy Census products are to use. When potential users, such as you, have difficulty with using the product, we have an opportunity to fix it before it is released to the public. We will use comments and experiences from you and other participants to help make improvements. We appreciate your help with this study.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session for usability evaluation. The recording will take video of the screen as you perform the task and capture audio of the session to get an accurate record of your feedback. It will be used solely for research purposes, and only those of us involved in the project will have access to the recording. Your name will not be associated with the recording or any of other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date.]

Thank you.

Start Recording with Camtasia

I now have a short demographic questionnaire that I’d like you to fill out.

Have the participant (P) fill out the demographic questionnaire. Remember to move away from where P is answering so P doesn’t feel like we are looking over P’s shoulder.]

Thank you.

Today you will be helping us evaluate some new language that we are considering using on our census materials – including on our website. After you complete the task, I will ask some questions about your experience.

[Eye tracking]

While you are performing the task, we will record where you look on the screen. In order to accurately record where your eyes look, we would like you to do a simple task that will allow the computer to find your eyes. To get the best possible data, first please situate yourself in a position that is comfortable for you. It will be important to keep your head and chair as stable as possible during the session. On the next screen, [gesture] you will see a red dot moving across the screen. Without moving your head, please follow the red dot with your eyes as it moves and keep your eyes on it as it pauses in each corner of the screen.

Ready?

[Calibrate the P’s eyes, check the calibration quality, repeat calibration until the calibration quality is satisfactory.]

Great.

So that we can make sure the calibration was successful, we will do a quick task with you at two points during the session. One will be right now, and the second time will be in just a few minutes after the next part of the session. I’m going to bring up an image. [bring up reading task] Please read out loud what appears on the screen.

Thank you.

[Run the first reading task – will be used to gauge how accurate the ET data is.]

[TASK 1: User is asked to read the paper letter or the .pdf version of the letter]

Make sure CAMTASIA is recording]

Now we will begin the main portion of the session. I’m going to open up /give you a letter that is sent to typical respondents to our surveys. I would like you to take a few minutes to read it over, getting familiar with the information in the letter. Let me know once you are finished.

[Start STOPWATCH right after the webpage is opened.]

[Stop STOPWATCH right when they hit Login.]

[Hit Escape right after they P tells you they are finished reading.]

[Log the reading time.]

Satisfaction question 1: Have P answer satisfaction statement

(Use separate piece of paper. Answer categories range from easy to difficult -see attached satisfaction question)

This material is __________ to understand.

Free Recall: Have P share anything they remember

Now I have some questions for you.

1. Can you tell me about everything you learned from the letter?

2. Anything else? (or Tell me more.)

3. Anything else? (or Tell me more.)

4. Anything else? (or Tell me more.)

Word recognition test: Identify words you saw on the screen

Here is a list of words. I would like you to identify those you saw in the letter / in the online version of the letter by circling them on this paper. [Hand P card with the 10 words – 5 from the reading passage 5 not on the reading passage but somewhat similar]

Forced Choice test: User answers a forced choice sentence.

Here is a statement I’d like you to read over and select the one that is true:

1. The information I provide will be seen and shared with other government agencies

2. The information I provide will be seen and used only by the Census Bureau.

Can you tell me how you arrived at that answer?
(if necessary) Tell me more.

Alternative forced choice statements include:

1. My personal information will not be shared with others

2. My personal information will be shared with other government agencies

Can you tell me how you arrived at that answer?
(if necessary) Tell me more.

1. Federal agencies will monitor online traffic and data that goes in and out of the servers.

2. Only the Census Bureau will monitor online traffic and data that goes in and out of the servers.

Can you tell me how you arrived at that answer?
(if necessary) Tell me more.

OTHERS??

In own words

1. I’d like you to tell me in your own words what the letter you just read is saying?

2. (if necessary) What’s the main idea? What would you walk away with after reading that…. Any gut reaction or impressions?

Recall - Targeted probe:

1. Do you recall any information about confidentiality? What specifically?

2. Do you recall any information about monitoring of your information? What specifically?

[TASK 2: User is asked to read condition specific language –targeted piece of reading]

Make sure CAMTASIA is recording]

Now we will do another reading of some of the same material that you may already have read so that I can ask you additional questions about the information. It contains some new language that Census is considering using. Please read it over and let me know when you are finished. When you are finished I will ask you questions about what you have read.

[Start STOPWATCH right after the webpage is opened.]

[Stop STOPWATCH right when they tell you they are finished.]

[Hit Escape once P tells you they are finished reading.]

[Log the reading time.]

Satisfaction question 1: Have P answer second satisfaction statement

(Use separate piece of paper. Answer categories range from easy to difficult -see attached satisfaction question)

Free Recall: Have P share anything else they remember

Now I have some questions for you.

1. Can you tell me about anything else you learned from the reading?

2. Anything else? (or Tell me more.)

3. Anything else? (or Tell me more.)

4. Anything else? (or Tell me more.)

Debriefing Probes

1. If you read this and then were expected to answer some survey questions from the Census would you be more or less likely to respond?

2. Do any of the words in the paragraph you read give you concerns?

3. What concerns do you have?

4. What do you think will happen to the information you enter into the survey?

5. (Depending on condition)

6. What about if it was electronic?

7. What about if you decided to fill out the paper survey – do you think this information would apply to you?

[Open it up again so they can look at it as they answer]

(If needed Note these may differ by condition – depending on the final wording and may not need to ask all the questions if user has already mentioned this earlier: )

1. Did you see the words “security monitoring” in the paragraph? Take some time to read that sentence or that paragraph if you like.]

2. What does “security monitoring” mean in your own words?

3. How about the words “information systems” Take some time to read that sentence or that paragraph if you like.]

4. What does “information systems” mean in your own words? Take some time to read that sentence or that paragraph if you like.]

5. How about the words “transmit your data”

6. What does “transmit your data” mean in your own words?

7. (condition specific) Did you notice where it said “Department of Homeland Security”? What did you think about as you read that? Any concerns/thoughts when you read that?

(If yes [bring up the paragraph again]: ) Can you show me where you saw it?

if no [bring up paragraph and point out where “XXXX” is] What does “XXXXX” mean to you? [If needed:] Take some time to read that sentence or that paragraph if you like.]

1. (condition specific) Another version of this paragraph might end with the sentence "Your data are further protected by Department of Homeland Security employees and contractors through security monitoring of the systems that transmit your data and may be used for any lawful government purpose." (give respondent a slip of paper with this new sentence) What does "any lawful government purpose" mean in your own words?

2. Do any of the words in this new sentence give you concerns?

3. What concerns do you have?

[Run the second reading task – will be used to gauge how accurate the ET data is.]

Okay, now we will look at another reading passage this time again to ensure our ET equipment is working accurately. Please read this passage out loud [point].

That’s all for today.

Thank you very much for your time.