Testing Plan

Plan for Usability Testing of the

Supplemental Nutrition Assistance Program (SNAP) Online Application

(English)

(Spanish)

Draft: April 29, 2019

Version 2.0

Study Team

Erica Olmsted-Hawala

Anna Sandoval Giron

Dave Tuttle

Jessica Graber

Sponsor

FNS

Center for Behavioral Science Methods

Associate Directorate for Research and Methodology

U.S. Census Bureau

1. Objectives

1. Find usability problems that interfere with an applicant’s ability to complete an online SNAP application accurately, efficiently and with satisfaction while they are using a smartphone, tablet or laptop. Focus on design of the application with respect to:

   1. Questions, instructions, and response options

   2. Link labels

   3. General layout of form on the page

   4. Button location size and labels

   5. Ability to login/out and have their answers saved (if application has this functionality)

2. Find cognitive issues with user comprehension of the instructions, questions, or response categories

2. Assumptions

1. The test period will begin October 2019 and run through October 2020 with approximately 1 State agency data collection completed a month, with some time in-between data collection as needed for analysis or recruitment, etc.

2. The online application will be stable (that is, no updates or changes) during each individual State agency test.

3. State agencies will provide access to their online application – likely through a mirror site/testing environment. (Potential to host test site internally to census – needs to be looked into).

4. State agencies will provide access to their online application 2 to 4 weeks before testing begins (this could be in the form of an external link to the online form).

5. The online application will be fully operational during the test period.

6. Half of the participants will complete the online application using a Census-provided laptop. If an external link is available then eye tracking will be collected; if a VPN connection is used, then it’s less likely that eye tracking will be able to be collected (eye-tracking equipment and the test site would need to be pre-tested typically in the 2-4 weeks prior to testing).

7. Half of the participants will complete the online application with their own personal smartphone– they will need to use their own cellular data or Wi-Fi provided by local libraries or community centers (if the location where test occurs has free Wi-Fi).

3. Participants

1. Obtain at least 10 English speaking (5 laptop and 5 mobile) and, depending state, 10 Spanish speakers (5 laptop and 5 mobile) to participate in testing, per state.

   1. Participants will not have participated in previous testing of the online application.

   2. Participants will have at least one year’s experience with using the Internet for things other than email.

   3. Participants will not

4. Recruiting

1. Use word of mouth, craigslist ads, flyers posted in libraries, use of contacts made through Community Based Organizations (CBOs) that help individuals seeking state assistance, and if necessary from in-person intercept methods.

2. Sessions will take place either at a local SNAP office, or at locations more convenient for participants (what we refer to as off-site testing) at local libraries and community centers.

5. Test Methods

1. Demographic and Internet usage questionnaire results

2. Think-aloud (Participants are asked to describe what they are thinking and feeling while completing the online application)

3. Satisfaction questionnaire

4. Results of vignettes

5. Retrospective debriefing

6. Sessions (audio and video of the screen) - recorded in Camtasia.

7. Sessions will be one-on-one (one participant to one Test Administrator).

8. Eye-tracking on laptops (if an external website is available.)

6. Participant Tasks

1. Answer online application questions as they pertain to real life

   1. Use think aloud protocol to collect any spontaneous feedback on the application

2. Complete a satisfaction questionnaire about the online application

3. Answer Vignettes (Pretend situation) Tasks 1, 2, 3

4. Answer Debriefing questions.

Vignettes

Task 1 (during session)

Imagine you had to go somewhere and were not able to finish answering the questions at this time. What would you do?

[Once they are exited, then ask them to resume.]

Okay now I’d like you to imagine that you are ready to continue working on the online form. What would you do?

Task 2

Okay now I would like you to imagine you had a question about this online form. You wanted to learn more about why the form is asking for your social security number. What would you do?

Task 3

Placeholder vignette – if there is a question from team about a less traveled path in the form – we will ask them to imagine a scenario and answer the question based on this scenario

For example: You have some concerns about whether the information you enter into the form will be kept safe. What would you do to learn more about how the information will be kept secure?

OR:

You have some money that you get from local handy man work that you do in your neighborhood (e.g., mowing lawns, etc.) you never deposit this money in the bank and it varies by the week with how much it is. Right now you have $175 in cash at home from your recent handyman jobs. What would you do on this question?

7. Test Design English – 10 participants / Spanish 10 participants (if appropriate to state)

1. Half the participants will use mobile half will use laptop

2. 2 State agencies will use a stand-alone online application for SNAP, that is, SNAP is the only program the application is used for.

3. 2 State agencies will use an online application that incorporates both SNAP and another Federal benefits program, such as TANF or Medicaid.

8. Measures

Objective 1: Observation and oral comments from participants collected in real-time and by reviewing video and audio of the session.

Objective 2: Comments during the interview and comments and observations during vignettes and debriefing probes by reviewing video and audio of the session.

Objective 3: If the survey is available externally, and works with the TOBII studio we can collect eye-tracking data. Eye tracking metrics would look at number of fixations in a given area, number of times participants fixated on a given area, duration of fixations on a given area.

9. Data collection

October 2019 - March 2020

60-90-minute interview

Data are collected at remote locations across the US. Initial five state agencies proposed include Oklahoma, Indiana, Maryland, Massachusetts, and Virginia. See Table 1 for a region and time of when first contact (made by FNS) will be made with the state about the upcoming usability study. States chosen are from different regions of the country, and were not involved in any significant changes to their online instrument (including not involved in other ongoing studies that FNS is running). Within a state, user testing may include rural, urban, or suburban areas.

Selection criteria used to select State agencies:

• Each state representing a different region

• States had online applications

• There was a diversity of online application take up rates.

Table 1. Region and Time of First Contact with States

State	Region
1 – Maryland (FNS contact in August)	MARO
2 – Massachusetts (FNS contact in September)	NERO
3 – Virginia (FNS contact in October)	MPRO
4 – Oklahoma (FNS contact in November)	SWRO
5 – Indiana (FNS contact in January)	MWRO

10. Remuneration

$40 as approved by OMB

11. Instrument

External site: URL FOR SNAP by state

SNAP is aprox 19-minute survey