Study Plan

Usability Evaluation Plan for the Online FNS Application Website

Draft: February 12, 2016

Last revision: February 27th, 2016

Version 2.0

Study Team

Brian Falcone

Lucia Lykke

Elizabeth Nichols

Anna Sandoval Giron

Lin Wang

Patricia Goerman

Sponsor

Food and Nutrition Services (USDA)

Human Factors and Usability Research Group &

Language and Cross-Cultural Research

Center for Survey Measurement

Associate Directorate for Research and Methodology

U.S. Census Bureau

1. Project Objectives

This project is to accomplish the following objectives:

1. Identify usability problems using qualitative data.

2. Collect quantitative data-points requested by FNS.

2. Project Scope

The usability evaluation of the online FNS application website prototype will include formative and summative testing to be carried out in a single phase.

3. Assumptions

1. An external link to the online instrument will be available and the prototype will be fully operational during the test period.

2. Sponsors will let CSM know if they want to be on the invite list to observe sessions.

4. Constraints

1. Low sample size per group should limit the use of the quantitative results beyond an exploratory scope

5. Comprehensive Testing of the current online FNS prototype application website

A comprehensive test, which will include both formative and summative testing, will be conducted in order to identify usability problems and to assess usability.

5.1. Methods

In the comprehensive usability testing, the participant will be required to complete the online FNS online prototype application for school meals. During this task, the participant will be asked to think aloud (speaking out what he/she is thinking) and to respond to several retrospective probe questions following the completion of the application. In addition, an assisting researcher will manually log quantitative data points of interest for summative testing while the participant performs the task.

Participants. A total of 58 individuals who have either applied for school meal benefits previously or who may be eligible will participate in the testing. Of this, 36 participants will be tested in English and the remaining 22 participants will be tested in Spanish. The following approaches will be used in recruitment:

1. Word of mouth, craigslist ads, flyers posted in libraries and community centers, and the recruiting database where appropriate

2. Sponsor’s assistance with recruiting by contacting their sources

Test design. We adopted a 2 (language) x 3 (device) between-subject design. The 58 participants will be split into the following conditions:

		DEVICE
		PC/Laptop	Tablet	Phone
LANGUAGE	Spanish	12	5	5
	English	12	12	12

Test case. The participant will complete the online FNS application uninterrupted to simulate how they would complete the application if they were at home.

Data collection methods. The following methods will be used to collect participants’ performance data:

Qualitative:

• Think-aloud protocol with minimal probing such as “Keep Talking,” “What are you thinking?” and acknowledgements such as “Um-hum?”

• Observation notes

• Retrospective Debriefing

• Types of documentation participants brought (if any), and the extent to which they consulted it

• What is searched in help searchbar and on what page (addressed by observation notes)

• Feedback on why users move quickly or slowly (addressed by think-aloud)

Quantitative:

• Satisfaction Ratings administered after the participant completes the application

• Manually Logged Data-Points of Interest:

  • Number of applicants that use help and on which pages

  • Number of applicants that use tooltips and on which pages

  • Number of applicants that use “check it out” (on the “Before you Begin…” page)

  • Duration on “Before you Begin…” screen for those that do and do not select “check it out”

  • Number of applicants that return to attestation page to change name and how the return is completed (back buttons (from where); link on adult income section; summary screen)

  • Number of participants that used progress bar navigation

  • Rate of backward navigation from the summary screen and to what screen

  • Number of times applicants attempt to use browser back button and on what screens

Data analysis.

Qualitative

1. Review behavioral observations and answers to debriefing questions in order to identify problems

Quantitative

1. Compute the overall satisfaction ratings

2. Compute descriptive statistics for manually logged quantitative metrics from data-points of interest

Testing facility and procedure. A testing session will be carried out in either the usability lab or a community library. The session will be conducted one-on-one, i.e., one participant and one test administrator (TA), with one quantitative data-logger. The test will be carried out in the following sequence:

1. Participant’s consent

2. TA start audio and video recording

3. Participant complete the FNS online prototype application, while thinking-aloud

4. TA conduct retrospective Debriefing

5. Participant completed satisfaction questionnaire

One session is estimated to last about 60 minutes.

6. Accessibility Evaluation

Accessibility evaluations will be performed concurrently with the comprehensive usability testing. For desktop and laptop screens, Section 508 regulations will be followed. For smartphone screens, WCAG 2.0 guidelines will be followed. The online application prototype will be evaluated for accessibility using assistive and native software. Specifically, desktop/laptop versions will be evaluated with JAWS 17 and the IE 11 Web browser, running on Windows 7 for conformance to Section 508. The Smartphone versions will be evaluated with the Google Talkback speech synthesizer and Chrome browser running on an Android operating system for conformance to WCAG 2.0.