OMB Memo

Usability testing of the Supplemental Nutrition Assistance Program
The Census Bureau plans to conduct additional survey work under the Generic Clearance for
Internet Nonprobability Panel Pretesting (OMB number 0607-0978).
Purpose of the Study
The Census Bureau’s Center for Behavioral Science Methods Usability Lab is planning a study
to understand user interface design and cognitive issues with the online forms that different states
use when people apply for the Supplemental Nutrition Assistance Program SNAP (formerly
known as food stamps). The SNAP offers nutrition assistance to millions of eligible, lowincome individuals and families and provides economic benefits to communities. SNAP is the
largest program in the domestic hunger safety net. In order to receive SNAP benefits, individuals
and families must meet certain requirements. The form is designed to identify eligibility for the
program. The form varies state to state but collects such information as a household’s gross
monthly income, household mortgage or rent payments, utility costs, dependent care expenses,
along with information on household members including social security number and
citizenship/immigration status, among other information.
The study will identify areas of the online form that cause users to struggle or have confusion
with the goal being to identify ways to modify or design the form that could make for a better
user experience and data quality outcomes. As the forms vary by state the usability team will
look for commonalities across the forms and identify what design or question wording works
better than others across the different states where the testing occurs. There will be five different
state forms that will be tested for usability. The five states are Maryland, Virginia,
Massachusetts, Oklahoma and Indiana. States were chosen by the staff at SNAP and based on
diversity of area, online form usage rates, ability to accommodate testing in the state area. The
feedback from this study will help the team to better understand how participants currently
interact with the online form and what changes need to be made to improve the design.
The key research questions or substantive issues being examined are to 1) find usability
problems which interfere with a respondent’s ability to answer the online SNAP form accurately,
efficiently and with satisfaction while they are using a smartphone or laptop. 2) find cognitive
issues with the instructions, questions, and response categories of the online form.
Methodology


Population of Interest: The usability and cognitive evaluation will focus on assessing
and improving the user experience for people who are eligible to receive SNAP
benefits.



Timeline: Testing will be from October 2019 and go through October 2020.
Recruiting will start mid-September, 2019 once OMB approval is received.



Language: Testing at all five states will be conducted in English. Testing will also be
conducted in Spanish where relevant.



Devices: Respondents will participate using either smartphones or laptops. Devices
may be provided by Census Bureau or may be respondents’ own devices.



Sample: Participants will be low-income individuals who are likely to be eligible to
receive SNAP benefits but who have not yet filled out the form for benefits.
Participants will not have participated in previous testing of a census online
instrument. Participants will have at least one year’s experience with using the
Internet for things other than email. For those participants using their own smartphone
or tablet, they will know how to use that device to access the Internet.



Respondent recruitment: Participants will be recruited using a combination of wordof-mouth, in-person visits to soup kitchens and local food pantries, and food charities
to drop off letters/flyers, and conduct on the spot recruitment, online & paper
advertisements, and reaching out to other local non-governmental organizations that
are focused on helping lower-income families with food shortages, etc. Participants
will be recruited in the five different states that are selected to be representative of
other states in the U.S. We have additional screener questions that will be used to
obtain the needed participants. These are included in the package.



Interviews: Researchers from Center for Behavioral Science Methods (CBSM) will
conduct the interviews. We will ask participants to provide their answers in the online
form – as they pertain to his/her real life. During this time, the participant will be
asked to think aloud (verbalizing what he/she is thinking) and will be probed as
needed. After the completion of the online form, the participant will answer a
satisfaction questionnaire. We will use vignettes (pretend situations) to get
information on specific parts of the form that may not be answered by all or by the
majority of participants but that the sponsor would like to get feedback on. Finally, we
will conduct a debriefing by asking the participant to review selected screen shots of
the online form and answer targeted probes about those questions. Concurrent probes
will be used to focus on any questions the sponsor would like additional feedback on
as well as any area of the form that the usability lab team notice is confusing to
participants. The sponsor is invited to observe sessions as we can typically bring one
to two additional observers, as space permits.



Location of interviews: Sessions will take place at the Census Bureau Headquarters
for the states where this makes sense (Maryland and Virginia) as well as at local
community centers, and libraries. For the other states (Massachusetts, Oklahoma, and
Indian) usability testing will take place at local community centers or local libraries.



Recordings: The session will be audio recorded and the computer screen will be video
recorded. We may also employ eye-tracking technology.



Informed Consent: We will inform participants that their response is voluntary and
that the information they provide is confidential and will be accessed only by
employees involved in the research project. We will used the standard consent form




used at the Center for Behavioral Science Methods. This consent form indicates that
the respondent agrees that the interview can be recorded to facilitate analysis of the
results. Participants who do not consent to be video or audio-taped will still be allowed
to participate.
Incentive: Usability testing participants will receive $40 to offset the costs of
participation in this research, such as travel and parking.
Burden: We expect to screen 500 potential respondents at approximately 5 minutes per
call for 100 successful scheduled user sessions. Each usability session should last 60
minutes and there will be 10 user sessions per state in English in 5 states, and 10 user
sessions per state in Spanish for 5 states. The total respondent burden is expected to be
183 hours. See Table 1.

Table 1. Total Estimated Burden
Category of Respondent
Screening
Cognitive/Usability Interviews English
Cognitive/Usability Interviews Spanish
Totals

No. of
Respondents

Participation
Time

Burden

500
50
50

5 minutes
60 minutes
60 minutes

83 hours
50 hours
50 hours
183 hours

Attachments
1. Protocol used for the study (Enclosure 1) including possible debriefing probes and
vignettes
2. Paper questionnaire, from MD, VA, MA, OK, ID (Enclosure 2) (screenshots of the live
instrument are not available yet)
3. Demographic questionnaire (Enclosure 3) (Previously approved by OMB from earlier
usability studies)
4. Mobile experience questionnaire (Enclosure 4)
5. Satisfaction questionnaire (Enclosure 5) (Previously approved by OMB from earlier
usability studies)
6. Additional screener questions for SNAP eligible recipients (Enclosure 6)
7. FNS test plan (Enclosure 7)
For further information about this study, please contact Erica Olmsted-Hawala at 301-763-4893
or [email protected].