Original OMB Letter approved oct 18 2019

Usability testing of the Supplemental Nutrition Assistance Program

The Census Bureau plans to conduct additional survey work under the Generic Clearance for Internet Nonprobability Panel Pretesting (OMB number 0607-0978).

Purpose of the Study

The Census Bureau’s Center for Behavioral Science Methods Usability Lab is planning a study to understand user interface design and cognitive issues with the online forms that different states use when people apply for the Supplemental Nutrition Assistance Program SNAP (formerly known as food stamps). The SNAP offers nutrition assistance to millions of eligible, low-income individuals and families and provides economic benefits to communities. SNAP is the largest program in the domestic hunger safety net. In order to receive SNAP benefits, individuals and families must meet certain requirements. The form is designed to identify eligibility for the program. The form varies state to state but collects such information as a household’s gross monthly income, household mortgage or rent payments, utility costs, dependent care expenses, along with information on household members including social security number and citizenship/immigration status, among other information.

The study will identify areas of the online form that cause users to struggle or have confusion with the goal being to identify ways to modify or design the form that could make for a better user experience and data quality outcomes. As the forms vary by state the usability team will look for commonalities across the forms and identify what design or question wording works better than others across the different states where the testing occurs. There will be five different state forms that will be tested for usability. The five states are Maryland, Virginia, Massachusetts, Oklahoma and Indiana. States were chosen by the staff at SNAP and based on diversity of area, online form usage rates, ability to accommodate testing in the state area. The feedback from this study will help the team to better understand how participants currently interact with the online form and what changes need to be made to improve the design.

The key research questions or substantive issues being examined are to 1) find usability problems which interfere with a respondent’s ability to answer the online SNAP form accurately, efficiently and with satisfaction while they are using a smartphone or laptop. 2) find cognitive issues with the instructions, questions, and response categories of the online form.

Methodology

• Population of Interest: The usability and cognitive evaluation will focus on assessing and improving the user experience for people who are eligible to receive SNAP benefits.

• Timeline: Testing will be from October 2019 and go through October 2020. Recruiting will start mid-September, 2019 once OMB approval is received.

• Language: Testing at all five states will be conducted in English. Testing will also be conducted in Spanish where relevant.

• Devices: Respondents will participate using either smartphones or laptops. Devices may be provided by Census Bureau or may be respondents’ own devices.

• Sample: Participants will be low-income individuals who are likely to be eligible to receive SNAP benefits but who have not yet filled out the form for benefits. Participants will not have participated in previous testing of a census online instrument. Participants will have at least one year’s experience with using the Internet for things other than email. For those participants using their own smartphone or tablet, they will know how to use that device to access the Internet.

• Respondent recruitment: Participants will be recruited using a combination of word-of-mouth, in-person visits to soup kitchens and local food pantries, and food charities to drop off letters/flyers, and conduct on the spot recruitment, online & paper advertisements, and reaching out to other local non-governmental organizations that are focused on helping lower-income families with food shortages, etc. Participants will be recruited in the five different states that are selected to be representative of other states in the U.S. We have additional screener questions that will be used to obtain the needed participants. These are included in the package.

• Interviews: Researchers from Center for Behavioral Science Methods (CBSM) will conduct the interviews. We will ask participants to provide their answers in the online form – as they pertain to his/her real life. During this time, the participant will be asked to think aloud (verbalizing what he/she is thinking) and will be probed as needed. After the completion of the online form, the participant will answer a satisfaction questionnaire. We will use vignettes (pretend situations) to get information on specific parts of the form that may not be answered by all or by the majority of participants but that the sponsor would like to get feedback on. Finally, we will conduct a debriefing by asking the participant to review selected screen shots of the online form and answer targeted probes about those questions. Concurrent probes will be used to focus on any questions the sponsor would like additional feedback on as well as any area of the form that the usability lab team notice is confusing to participants. The sponsor is invited to observe sessions as we can typically bring one to two additional observers, as space permits.

• Location of interviews: Sessions will take place at the Census Bureau Headquarters for the states where this makes sense (Maryland and Virginia) as well as at local community centers, and libraries. For the other states (Massachusetts, Oklahoma, and Indian) usability testing will take place at local community centers or local libraries.

• Recordings: The session will be audio recorded and the computer screen will be video recorded. We may also employ eye-tracking technology.

• Informed Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. We will used the standard consent form used at the Center for Behavioral Science Methods. This consent form indicates that the respondent agrees that the interview can be recorded to facilitate analysis of the results. Participants who do not consent to be video or audio-taped will still be allowed to participate.

• Incentive: Usability testing participants will receive $40 to offset the costs of participation in this research, such as travel and parking.

• Burden: We expect to screen 500 potential respondents at approximately 5 minutes per call for 100 successful scheduled user sessions. Each usability session should last 60 minutes and there will be 10 user sessions per state in English in 5 states, and 10 user sessions per state in Spanish for 5 states. The total respondent burden is expected to be 183 hours. See Table 1.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	500	5 minutes	83 hours
Cognitive/Usability Interviews English	50	60 minutes	50 hours
Cognitive/Usability Interviews Spanish	50	60 minutes	50 hours
Totals			183 hours

Attachments

1. Protocol used for the study (Enclosure 1) including possible debriefing probes and vignettes

2. Paper questionnaire, from MD, VA, MA, OK, ID (Enclosure 2) (screenshots of the live instrument are not available yet)

3. Demographic questionnaire (Enclosure 3) (Previously approved by OMB from earlier usability studies)

4. Mobile experience questionnaire (Enclosure 4)

5. Satisfaction questionnaire (Enclosure 5) (Previously approved by OMB from earlier usability studies)

6. Additional screener questions for SNAP eligible recipients (Enclosure 6)

7. FNS test plan (Enclosure 7)

For further information about this study, please contact Erica Olmsted-Hawala at 301-763-4893 or [email protected].