Study Plan

Usability Evaluation Plan for census.gov Refresh

Draft: February 12, 2018

Last revision: February 12, 2018

Version 1.0

Study Team

Erica Olmsted-Hawala (project lead)

Xxx xxx

Lin Wang

Sponsors

ADCOM

Human Factors and Usability Research Group

Center for Survey Measurement

U.S. Census Bureau

1. Introduction

The census.gov Refresh is an initiative to enhance census.gov to “CREATE A CLEAR PATH TO DATA AND INFORMATION SO THAT PEOPLE RECOGNIZE CENSUS.GOV AS THE MOST ACCURATE AND USER-FRIENDLY SOURCE OF INSIGHT INTO THE PEOPLE, PLACES, AND ECONOMY OF THE USA.” According to the planning document, the project duration is from September 2017 to December 2019. The project will be carried out in phases. This Usability Evaluation Plan documents the usability evaluation work in FY 2018 (from now to September 30, 2018)

The purpose of this project is to optimize the application’s usability through iterative usability evaluations during the entire lifecycle of the product design and development. Usability will be evaluated in terms of respondent’s effectiveness and efficiency in survey completion, and satisfaction with the experience of interacting with the application.

2. Project Objectives

This project is to accomplish the following objectives:

1. Minimize potential usability problems during application design phase.

2. Identify and address usability problems during application development phase.

3. Assess application’s usability upon the completion of development.

3. Project Scope

According to census.gov Refresh’s tentative testing schedule, there will be five releases in FY18 as listed in Table 1. Each release has about 20 workdays for testing including user acceptance, usability, and Section 508 compliance (Table 1). Based on the project scheduling, we propose the following usability evaluation:

1. UI design review.

2. Quick formative usability testing of wireframe prototype at the completion of user interface design but before coding.

3. Comprehensive formative testing at each release as listed in Table 1.

4. Accessibility testing at each release.

The usability testing and accessibility testing will be conducted on both desktop computer and smartphone.

Table 1. Comprehensive Usability Testing Schedule

Testing	Start	End
Components & Editable Templates Testing	3/5/18	3/30/2018
Header/Footer/Local Navigation/Banner Testing	4/16/18	5/17/18
Homepage Testing	5/21/18	6/8/18
Templates/Pages for Subjects, Sub-topics, Explore Data, Audiences Testing	7/23/18	8/17/18
Group C Testing	8/20/18	9/14/18

4. UI design review

The usability evaluation team will participate in the census.gov Refresh weekly working session, to conduct UI design reviews with the sponsor and development team. The usability evaluation team will provide their opinions on the design prototypes.

5. Quick formative usability test

Upon the completion of each wireframe prototype, a quick formative usability test will be conducted on both desktop and smartphone versions of the application, on a small sample of potential users. The purpose of the test is to discover obvious usability problems promptly so that the development team can address the problems in a timely fashion. During the test, the participants’ performance will be investigated using a think-aloud protocol.

5.1. Methods

For each quick test, a set of test cases will be designed according to the functionalities developed for the release. The participant will be required to complete tasks designed in the test cases. During the task performance, the participant will be asked to think aloud (verbalizing what he/she is thinking) and will be probed as needed. After the completion of the tasks, the participant will be debriefed about his/her experience with the application.

Participants. Four individuals will participate in the testing, 2 for desktop version and the other 2 for smartphone. Participants will be recruited primarily based on their demographic compatibility with what is required for target user population. The following approaches will be used in recruitment:

1. Word of mouth, craigslist ads, flyers posted in libraries, and the recruiting database, where appropriate

2. Sponsor’s assistance with recruiting by contacting their sources

3. Notifying the sponsor of any difficulty recruiting by the 2nd week of testing

Test design. The test will consist of 3-5 usability tasks based on what the UI design is.

Data collection methods. The following methods will be used to collect participants’ performance data:

1. Think-aloud protocol with minimal probing such as “Keep Talking,” “What are you thinking?” and acknowledgement tokens (linguists refer to this as backchannels) such as “Um-hum?”

2. Observation notes

3. Retrospective Debriefing

4. Audio and video recording

Data analysis.

Qualitative

1. Review behavioral observations, spontaneous verbalizations and answers to debriefing questions in order to identify problems

2. Produce gaze patterns on PC to investigate whether participants attended to or ignore important parts of the screens. We will investigate whether we can produce gaze patterns on smartphones.

Testing procedure. The session will be conducted one-on-one, i.e., one participant and one test administrator (TA), with one note taker. The test will be carried out in the following sequence:

1. Participant’s consent

2. TA start audio and video recording

3. Participant complete tasks, while thinking-aloud

4. Participant complete satisfaction questionnaire

5. TA conduct retrospective Debriefing

Testing facility. A testing session will be carried out in either the usability lab or a community library.

One session is estimated to last about 60 minutes. This task is expected to be completed in 4 work days.

6. Comprehensive usability test

Upon each release, a comprehensive formative usability test will be conducted on both desktop and smartphone versions of the application, on a relatively large sample of potential users. The purpose of the test is to assess the overall usability of the release and to identify usability problems that need to be addressed.

6.1. Methods

For each comprehensive test, a suite of test cases will be designed according to the functionalities developed for the release. The participant will be required to complete tasks in the test cases. During the task performance, the participant will be asked to think aloud and will be probed as needed. In addition, the eye-tracking technique will be employed to record the participant’s visual scan and gaze pattern, when it is feasible. After the completion of the tasks, the participant will be debriefed about his/her experience with the application.

Participants. 18 individuals will participate in the testing, 9 for desktop and the other 9 for smartphone. The same inclusion criteria and recruitment strategies as that in the quick formative testing will be applied.

Equipment. The participants will use the Census-Bureau-provided laptop/desktop computer or smartphones for the testing.

Test design. The test will consist of 3-5 usability tasks based on what the UI design is.

Data collection methods. Same as that of quick formative testing.

Data analysis. Same as that of quick formative testing.

Testing procedure. Same as that of quick formative testing.

Testing facility. Same as that of quick formative testing.

One user testing session is estimated to last about 60 minutes. This task is expected to be completed in 20 work days.