Various Demographic Area Pretesting Activities

Version No.	Date	Revision Description	Author
0.01	5/02/2012	Initial Draft	Megan Iverson
0.02	5/02/2012	Internal Peer Review	Megan Iverson, Mimi Chen, Matt Prorok
0.03	5/02/2012	Follow up Peer Review	Megan Iverson, Mimi Chen, Matt Prorok
0.04	5/10/2012	Removed unemployment scenario, updated numbering and table of contents	Megan Iverson
0.05	5/25/2012, 5/28/2012	Removed Scenarios 2, 8b, and 11. Updated scenarios 9, Added page breaks between the scenarios and the wrap up questions Updated Scenario 4 to reflect voter turnout instead of population	Megan Iverson
0.06	5/29/2012	Updated Consent Form	Megan Iverson

test Preparation

Configure the mobile usability lab and start the Morae software to record the session
Set the prototype to the homepage (Census.gov)

Notes for the Facilitator

All text that is in italics is for the facilitator only and is not to be read aloud to the participant.

The following are tips the facilitator should keep in mind during the testing process:

Context of the website redesign effort:

The Census.gov website has grown organically over the last 20 years and the content is stove piped, based on organizational structure. The purpose of the redesign effort is to group and integrate content from across the site into easy to find and logical places, as well as to highlight the vast information and data Census produces.

Key areas to probe:

Navigation (grouping of content and labels)
Expectations of content when navigating through the site

Research objectives:

Can the user find desired content?
Can casual users successfully navigate the new menu and contextual navigation to find content?

Metrics that are essential to record:

Time on task – General time estimates will be recorded to give an insight into how quickly a user interprets the process for the following
- Finding specific information, for example: population of a county
- Looking for tables/data
- Downloading data to manipulate
- Gather a subjective satisfaction of the user’s acceptance to the system
Number of errors per task
Success rate per task
Navigational paths taken to information
System Usability Scale (SUS) survey, a simple, ten-item scale giving a global view of subjective assessments of usability. The SUS is an inexpensive, yet effective tool for assessing the usability of a website. It provides an easy-to-understand score from 0 (negative) to 100 (positive) based on participants responses to 10 statements on a five point scale with 1 being Strongly Disagree to 5 being Strongly Agree. A score of 70 indicates a well received website.

Overall session agenda:

(5 Min.) Introduction/Instructions

(2 Min.) Consent Form

(3 Min.) Pre-Test Questionnaire

(75 Min.) Scenarios

(5 Min.) Post-Test Questionnaire

General Introduction

Good [morning/afternoon/evening]. My name is _______________, and I will be working with you in today’s session. Thank you for your help today. What we will be showing you is a prototype of the redesigned Census.gov website.

What is the purpose of usability testing?

The purpose of this test is to learn how users interpret and use the User Interface. In the end, your input will help us improve the product to its users.

What is the general testing procedure?

We will start with instructions on how you will participate, then you will fill out a pre-test survey about your education level and computer experience. The main part of the test will consist of scenarios, and we will wrap up with a post-test survey. The session will take 1 ½ hours.

Instructions

Scenarios & Tasks:

Today we will be giving you a few different scenarios to work through. This presents a series of common tasks that you might attempt while using the Census.gov website. As part of the study, you may be asked to fill in various forms or enter comments. Please do not worry about errors in spelling for the purposes of this test.

Think Out Loud:

While you are working on the tasks, I’d like you to behave as you normally would in the work environment, with one major exception: I’d like you to think out loud as you perform the scenarios. By ‘thinking out loud,’ I mean that I’d like you to briefly tell me what you’re doing, why you are doing it and your reaction when you did it. I’ll be over here taking notes, trying to keep up with what you’re saying. On occasion, I may ask you to clarify something you’ve said. Also, feel free to ask questions. I may not be able to answer them all right away, since we’re interested in what people do when they don’t have someone sitting next to them, but I will try to answer any questions you still have when we’re done.

Keep on Task:

You might see things on the screen that you would like to investigate; but since this is a prototype, not everything works like it will in a completed system.

Hand Waving:

Since what we are doing today is working with a prototype, we may have to do some hand waving and say, “well if you did that . . . the system would have done this. . . .”. This is the downside to what we are doing today – that it isn’t finished yet. The upside is that it isn’t finished yet, so your feedback can have a large impact on how the final system looks and works. So please let me know your thoughts as we work through this series of scenarios today.

In-Progress:

The system you are about to see is a work in-progress. The end result may not appear exactly as it does today. You may encounter errors during this session. If you encounter an error, please notify the facilitator. This error is not the result of something you did but rather an indication of the early phase of design the website is in.

Don’t Assume:

Each scenario is different from the others. So if some link or button doesn’t work with one scenario, don’t assume it won’t work for the next. Go ahead and try the things you think should work for each task. Again, each scenario is unique so what you may do in a previous scenario will not affect the data in another scenario.

Last Reminders:

Remember to:

Think out loud.
Give feedback on what you see and ask questions – what makes sense, what doesn’t, etc.
You are the expert – that’s why we’re here, to learn from you.
This is not testing you, but the computer system, and its “look and feel”.

Before we get started, do you have any questions?

Consent Form

You have been recruited as a usability participant to help provide U.S. Census Bureau with feedback on the new Census.gov website. The session will be run by members of Accenture’s User Experience Group, who will gladly answer any questions you may have about the session. We will observe you and record information about how you work with the system. We will also ask you to answer questions in regards to your thoughts on the system.

By signing this form, you give your permission to the U.S. Census Bureau to use your recorded verbal statements to be used for the purposes of U.S. Census Bureau and Accenture’s internal evaluation. We will not use your name when using your verbal statements.

You will be asked to comment on a variety of subjects relating to the system and will be given a survey to complete for additional research information. It is important to note that you are not being evaluated in any way. We expect the session to last about 1 ½ hours. Your participation is voluntary and your answers will remain strictly confidential.

This usability study is being conducted under the authority of Title 13 USC. The OMB control number for this study is 0607-0725. This valid approval number legally certifies this information collection.

There are no known risks associated with this evaluation.

Your rights as a participant are as follows:

You have the right to withdraw from the session at any time for any reason.
At the conclusion of your session, you may withdraw any of your data. Please inform the team immediately if you desire to withdraw your data.

Finally, we greatly appreciate your time and effort for participating in this session. Remember, you cannot fail any part of this session, and there are no right or wrong answers to the issues on which you are asked to comment. Your signature below indicates that you have read this consent form in its entirety and that you voluntarily agree to participate. Thank you in advance for your time and feedback.

Your Signature: ____________________________________________________________

Printed Name: ____________________________________________________________

Date: ________ / _________ / ________

Phone Number: (________) _______ - ________

Pre Session Survey

All answers are confidential.

Demographics

Age:

___Under 18

___18-29

___30-39

___40-49

___50-59

___60 and older

Gender:

___Male

___Female

Highest Education Level Completed:

___High School

___Some College

___Associates Degree

___Bachelors Degree

___Masters Degree

___Doctorate Degree (PhD, MD, JD)

Computer Experience

Where do you use a computer? (Check all that apply.)

___Home

___Work

___Somewhere else, such as school, library, etc.

How many hours per week do you use a computer (including home and work)?

Hours per week: [ ] 0-10 [ ] 10-20 [ ] 20-40 [ ] 40+

How would you describe your level of experience using a computer?

[ ] Novice [ ] Beginner [ ] Intermediate [ ] Advanced

How would you describe your level of experience using the Internet – and websites in general?

[ ] Novice [ ] Beginner [ ] Intermediate [ ] Advanced

For the following questions, please circle one number.

Not Comfortable Comfortable

a. How comfortable are you in learning to navigate new Websites?	1 2 3 4 5
b. Computer windows can be minimized, resized, and scrolled through. How comfortable are you in manipulating a window?	1 2 3 4 5
c. How comfortable are you using, and navigating through the Internet?	1 2 3 4 5

Never Very Often

d. How often do you work with any type of data through a computer?	1 2 3 4 5
e. How often do you perform complex analyses of data using a computer?	1 2 3 4 5
f. How often do you use the Internet or Websites to find information? (e.g., reports, news articles, data tables, blogs, etc.)	1 2 3 4 5
g. How frequently are you exposed to Census information (terms, data, etc.)?	1 2 3 4 5

participant Script

Scenario 1: Find the population of San Diego County, California

Scenario 1 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to find the population of San Diego County, California.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I feel confident I could find the population of a different county easily.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 2: Find and Download a Native American tribal area map and find demographic information about Native Americans.

Scenario 2 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to download the Native American tribal area map.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I feel confident I could find other maps easily.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 3: Find the Hispanic voter turnout data tables for the 2010 election at the state level in Excel format.

Scenario 3 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to view the data tables of Hispanic voter turnout at the state level for the 2010 election.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I feel confident I could find other race and ethinic voter turnout by state for the 2010 election easily.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 4: Find Redistricting Data based on the 2010 Census.

Scenario 4 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to find redistricting data based on the 2010 Census.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 5: Find the Schedule B product classification code for golf clubs.

Scenario 5 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to find the product code for golf clubs in Schedule B.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 6: Find and download Poverty Data at the national level and a publication related to Poverty.

Scenario 6 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to find poverty data at a national level.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

It was easy to find a publication related to Poverty at a national level.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I feel confident I could find Poverty data for other levels (states, counties, etc).

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 7: Find building permit data at the state or metropolitan level over the last five years.

Scenario 7 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to find building permit data at a state or metropolitan level for the last five years.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 8: Find a working paper on “uninsured medical expenses” written by Kathleen Short and the date she used. Then find her related research.

Scenario 8 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to find a working paper on “uninsured medical expenses” written by Kathleen Short.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

It was easy to find Kathleen Short’s related research.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

It was easy to find the data Kathleen Short used to write her working paper.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Scenario 9: Download international demographic data including age structure, fertility rate, and urbanization for the purpose of running different scenarios to understand the impact on population projections.

Scenario 9 Wrap-Up

Taking all the screens you saw into consideration, would you give these screens a Thumbs Up, Neutral, or Thumbs Down? Why?

Ratings

It was easy to download international demographic data.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

I felt unsure about where to start looking.

Strongly Strongly

Disagree 1 2 3 4 5 Agree

Open Ended Questions/Comments

Is there anything in particular that you liked/disliked about the screens we just spent time discussing?

What do you think of the wording used on the screens? Do they make sense to you?

Post Session Questions

The following post-test questions will be asked once all scenarios have been completed:

What is your general reaction to the site?
What is the purpose of this website?
What organization is responsible for this website? How can you tell?
Is it clear to you whom this website is designed for?
What kinds of information would you go to this website to find?
What did you like best about the site?
What did you like least about the site?
Is there anything that you feel is missing on this site?
If you were to describe this site to a co-worker, what would you say?
Does the information provided on this site make you trust government more or less?
What is your level of trust in government after reviewing this site? (Scale of 1 to 10, with 1 being totally untrustworthy and 10 being completely trustworthy).

Following the launch of the website, the Census bureau is planning to add additional content and information to the website.

What you would like to see on the Census website in the future?
Are there particular aspects of the Census Bureau that you want to learn more about?
Would you consider using social media / collaboration features to provide feedback to the Census Bureau?
Any other questions or comments about what you have done today?

Post Session Survey

Please respond to the following statements in regards to the Census.gov website you used today by marking an “X” in the appropriate column.

	Strongly Agree	Agree	Neutral	Disagree	Strongly Disagree
I think that I would like to use this website frequently.
I found the website unnecessarily complex.
I thought the website was easy to use.
I think that I would need the support of a technical person to be able to use this website.
I found the various functions in this website were well integrated.
I thought there was too much inconsistency in this website.
I would imagine that most people would learn to use this website very quickly.
I found the website very cumbersome to use.
I felt very confident using the website.
I needed to learn a lot of things before I could get going with this website.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	figue315
File Modified	0000-00-00
File Created	2021-02-02

Various Demographic Area Pretesting Activities

Generic Clearence for Questionnaire Pretesting Research

omb1219census.govtestingenc2_rev2