Original Memo

Generic Information Collection Request

CEDSCI Usability Evaluation (Phase I)

Request: The U.S. Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We propose to conduct a usability evaluation of the CEDSCI, the Census Bureau’s main data dissemination outlet. We are seeking approval for this project that is to be conducted jointly by Census Bureau staff and the contractor.

Background: The CEDSCI is an initiative to enhance the U.S. Census Bureau’s data dissemination capability, and consequently to foster a user-focused and cost-effective data dissemination environment. The goal of CEDSCI is to provide users with credible statistics, easy-to-use tools, and standardized data elements. The CEDSCI will be carried out in phases. In Phase I, an application will be developed for users to search for certain demographic and economic information collected by the Census Bureau. The application will have two versions, one for desktop and the other for smartphone. This usability evaluation project is to assess the usability of the Phase I application.

Purpose: The purpose of this project is to optimize the application’s usability through iterative usability evaluations during the entire lifecycle of the product design and development. Usability will be evaluated in terms of respondent’s effectiveness and efficiency in survey completion, and satisfaction with the experience of interacting with the application. This project is to accomplish the following objectives:

1. Minimize potential usability problems during application design phase.

2. Identify and address usability problems during application development phase.

3. Assess application’s usability upon the completion of development.

4. Establish performance benchmarks for the application.

Population of Interest: The general public.

Language: The evaluation will be conducted in English only.

Timeline: The evaluation will be conducted in parallel with the systems development, from October 1, 2016 to September 30, 2017.

Project scope: The usability evaluation will include four aspects, to be conducted in order:

1. Baseline summative testing of the existing AFF application.

2. Usability review of the high-level information architecture.

3. Rapid formative usability testing at the completion of each sprint.

4. Comprehensive formative testing at each release.

5. Summative testing at the application release.

The CEDSCI system will be developed using the Agile methodology. From October 1, 2016 to September 30, 2017, there will be 4 major releases at a 90-day interval. Within each 90-day interval, there will be 6 sprints of 15 days each.

Due to the nature of Agile development, detailed functionalities of the application will be designed in sprints. In general, the application will have the following functional components: Navigation, data search, text display, graphical display, data download, and etc.

Table 1 outlines estimated project schedule for the aforementioned tasks.

Table 1. Estimated Project Schedule

Activity	Start	End	Duration (day)
Baseline summative testing of the existing AFF application			20
Information architecture review			5
90-day Interval I
Rapid formative test, Sprint 1			2
Rapid formative test, Sprint 2			2
Rapid formative test, Sprint 3			2
Rapid formative test, Sprint 4			2
Rapid formative test, Sprint 5			2
Rapid formative test, Sprint 6			2
Comprehensive formative testing			20
90-day Interval II
Rapid formative test, Sprint 1			2
Rapid formative test, Sprint 2			2
Rapid formative test, Sprint 3			2
Rapid formative test, Sprint 4			2
Rapid formative test, Sprint 5			2
Rapid formative test, Sprint 6			2
Comprehensive formative testing			20
90-day Interval III
Rapid formative test, Sprint 1			2
Rapid formative test, Sprint 2			2
Rapid formative test, Sprint 3			2
Rapid formative test, Sprint 4			2
Rapid formative test, Sprint 5			2
Rapid formative test, Sprint 6			2
Comprehensive formative testing			20
Final Release
Final summative testing – a test of the overall product			30

General Protocol: Participants will first be asked to complete a demographics questionnaire and Internet experience questionnaire. Then participants will perform a list of tasks designed to access certain functionalities for the particular development sprint, using a think-aloud protocol (except for the summative test). After completing the tasks, each participant will be asked to complete a satisfaction questionnaire, and asked debriefing questions about different screens.

Sample: 10 males and 10 females will participate in the baseline assessment. For each rapid formative usability test, there will be 3 male and 3 female participants to be tested. There will be total 54 males and 54 females participate in the rapid formative test. 9 males and 9 females will participate in each comprehensive formative testing. 20 males and 20 females will participate in the final summative testing. The total number of participants will be 168 participants. Participants inclusion criteria:

1. speak fluent English,

2. are age between 18 years to 70 years old,

3. have at least one year of Internet experience,

4. have school education of at least high school diploma.

Recruitment: Participants will be recruited using a combination of word-of-mouth, flyers, online and paper advertisements.

Use of Incentive: Due to the length and complexity of the task as well as the necessity to travel to test locations, we plan to offer an incentive of $40 to offset the costs of participation, such as travel and parking.

Below is a list of materials to be used in the current study.

1. Study plan including possible test cases (Enclosure 1)

2. Sample testing protocol (Enclosure 2)

3. Demographic questionnaire (Enclosure 3)

4. Internet and mobile experience questionnaire (Enclosure 4)

5. Debriefing questionnaire (Enclosure 5)

Length of interview: We estimate 60-90 minutes per participant. (Respondents will be screened using the Census Bureau’s generic screener; thus the burden hours for screenings are covered under a separate request.) The total estimated participant burden for this request is 252 hours or less.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Lin Wang

Center for Survey Measurement

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-9069

[email protected]

9