Public Protocol Public Protocol

Study Protocol (general public)

Administrative Records Messaging Study

Thank you for your time today. My name is XX and I work with the Human Factors and Usability Group at the U.S. Census Bureau. Today, we will work together to evaluate how easy or difficult it is for a user like you to use a new webpage design in the Census Bureau’s website, and to make recommendations based on your and others’ experiences to improve the website. So, your feedback is very important to us. I appreciate your participation.

Before we start, there is a form I would like you to read and sign. It explains the purpose of this study and your rights as a participant. It also informs you that we would like to take a video of the device as you use it and record the audio from this session to get an accurate record of your feedback. Only those of us involved in the project will review the recording. The recording will solely be used for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand consent form; give time to read and sign; sign own name and date, start recording.]

Thank you.

Ok great.

I would also like you to fill out two short questionnaires, one is demographic questionnaire and the other is an internet experience questionnaire.

[Have the participant first fill out the demographic questionnaire—simply move away from where they are answering so they don’t feel like we are looking over their shoulder.

Ask the participant to fill out the Internet experience questionnaire]

Thank you.

Think Aloud: As we go through the tasks today I want you to act exactly the way you would if you were at your home, but with one major difference. I would like you to think aloud while you are performing the tasks. I am interested in the process you go through in your mind as well as what you are thinking and feeling as you go about the activities.

Since people usually are not familiar with Think Aloud protocol, let’s do a practice before we start. Think aloud as you answer the question, “how many windows are in your home?”

[Help the participant to complete the practice if needed.]

Great that’s what I want you to do throughout our session. If you fall silent during the session, I will remind you to think aloud by saying something like “tell me what you’re thinking.” I don’t mean to interrupt your train of thought, but I simply want to remind you that we are interested in what you are thinking.

At the end of the session, I will have some questions for you about your experience as you worked on specific screens and how satisfied you were with the survey. We may open up some of the screens and talk about them.

[Eye tracking]

We are also going to use an eye tracker to record where you look on the screen as you work on the tasks. Now we will do a simple task that will allow the eye tracker to find your eyes. You will see a red dot moving around on the screen, please follow the dot with your eyes.

[Calibrate the participants’ eyes]

Today, we will ask you to perform a few tasks using the website. While you are performing the tasks, you may run into difficulties, please don’t blame yourself. Any difficulties are the result of the design, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the website. I did not create the website, so don’t feel like you have to hold back on your thoughts to be polite to me. Any thoughts are welcome and we appreciate your help so we can make the website better.

[Hand participant the mailed letter]

Now, let’s start the tasks. Here is the scenario: You have been selected by the Census Bureau to complete a survey, and received this invitation letter. Take a moment to look over it.

[Give participant a moment to look/read through the letter.]

After reading this letter, you noticed that the third paragraph on the back talked about how the Census Bureau would combine the personal data I gave to other government agencies with my responses to this survey. [Point to the third paragraph] You are wondering why the Census Bureau does that. You decided to go to the Census Bureau’s website, census.gov, and to find out more about how or why the Census Bureau gets data from other agencies.

Task 1: Navigation

Now, you are seeing this website, www.census .gov. Please go ahead to find the information you are looking for. Remember to think aloud!

[Show participant www.census.gov]

[Note which path/s were taken and number of steps to get to main page (in a separate log sheet). Offer help after ONE minute.]

Task 2: Technical Pages

Ok, great! Now you have read some general information, and want to know some more detailed information.

First, you want to find out, what type of information the Census Bureau is collecting about you from other agencies [This info is on the "CBLI Community" page].

Second, you want to find out what kind of research has been being done at the Census Bureau with administrative data? [This info can be found by clicking on the "Working Papers" left-navigation link --OR-- through the "research papers" link on the General Info page]

[End of the tasks]

Retrospective Probes

Ask all the probes using a power point slide with those screens.

Screen 1: census.gov website

If you observed an issue on this screen, then probe:

I noticed that you [hesitated/had trouble] with this screen. Can you tell me what was going on for you then? Probe any specific behavior you noticed when they were on this screen.

How easy or hard was it to find the information we asked you to look for? If necessary: Can you tell me more?

Did you think that you could find the information by clicking anywhere else?

If yes: Where did you consider clicking?

Do you have any other comments on this screen?

Screen 2: Access point 1/2/3/4/5/6

If you observed an issue on this screen, then probe:

I noticed that you [hesitated/had trouble] with this screen. Can you tell me what was going on for you then? Probe any specific behavior you noticed when they were on this screen.

[Point to the “Data linkage/reuse” link]

Tell me about your reaction to this link.

Do you think the phrase “Data linkage and reuse” is appropriate here? Tell me more about that.

If you could change that wording, what would you call it instead?

Screen 3: General Data Linkage/Reuse or FAQ Page

In your own words, what did you think was the main point of this page?

Were there any parts of the page that were confusing?

If necessary: Can you tell me more about that?

Were there any parts of the page that were particularly helpful?

If necessary: Can you tell me more about that?

Was there any information that you would have expected to see here that was missing?

What do you think you might do if you could not find the General Data Linkage/Reuse website?

Census is considering including a few more messages on this website. [Show respondent Additional Message Sheet]

What do you think about these messages?

Do you think either of these would be helpful on the website?

If necessary: Tell me more about your answer.

Screen 4: CBLI Homepage

If you observed an issue on this screen, then probe:

I noticed that you [hesitated/had trouble] with this screen. Can you tell me what was going on for you then? Probe any specific behavior you noticed when they were on this screen.

In your own words, what did you think was the main point of this page?

Were there any parts of the page that were confusing?

If necessary: Can you tell me more about that?

Were there any parts of the page that were particularly helpful?

If necessary: Can you tell me more about that?

How easy or difficult was it to find the more technical CBLI content?

If necessary: Can you tell me more about that?

Debriefing Questions:

Overall, did you find the website easy or difficult to navigate?

Did you find the content easy or difficult to understand?

Have you looked at websites with similar information in the past? If yes: How did that experience compare to this one?

Is there anything else you would like to mention that you have not had a chance to mention yet?

[Administer satisfaction questionnaire.]

4