Testing Protocol for Reauthorization Application

Supplemental Nutrition Assistance Program

FNS-252R (re-authorization application) Cognitive Testing Protocol

PARTICIPANT ID #:

DATE:

/

/

INTERVIEWER NAME:

START TIME:

:

AM/PM (circle one)

Section 1. Introduction and Informed Consent

Hello, I’m [INTERVIEWER NAME]. I work for the Census Bureau. We are working with the Food and Nutrition Service to help them improve the SNAP application. Thanks for agreeing to meet with me today.

INTRODUCE OBSERVERS, IF ANY

We are talking to store owners or their staff about the process of applying for SNAP authorization. I’ll ask you to access the online application and walk me through the steps you took in completing the application. It’s ok if you don’t remember all the details.

Please be assured that our meeting today will not affect your status with regard to the SNAP program. The sole purpose of this meeting is to understand how the application process works for applicants and to look for any ways the application may need to be improved to make it easier to use.

PLACE THE CONSENT FORM IN FRONT OF THE RESPONDENT

Before we start, I would like you to read over the document in front of you. It explains a little bit about this interview and provides information about your rights as a participant. It also asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading, please sign it.

PARTICIPANT READS AND SIGNS FORM

IF PARTICIPANT CONSENTED TO AUDIO TAPING, READ:

I will now turn on the audio recorder.

TURN ON AUDIO RECORDER

Section 2. Background Information

I’d like to ask you a few questions before we open the application.

Please describe your business.

What is your role in the business?

(If owner) How long have you owned the business?

(If not owner) How long have you been working for this business?

What do you remember about completing the SNAP application?

Were there any aspects you found easy or straightforward? What were they?

Were there any aspects you had problems with? What were they?

Did you need to ask anyone else in your business to provide any information for the application?

• What information did you need to get from others?

• What is their position/title? How did you go about getting the information from them?

• How did you communicate the request?

• Did they have any questions, and if so, what did you tell them?

• Did they seem to have any problems with getting you the requested information, if so, what kinds of problems, and how did you handle the problems they presented?

• Did they express any concerns about providing you the requested information?

• In your opinion, were you able to get the requested information in what you consider a timely manner?

• Did you have to make any changes to the information they provided in order to make it fit the application, if so, what kinds of changes?

How long did it take you to complete the application, including the time you may have spent consulting records and other persons?

Did you contact the Food and Nutrition Service for assistance at any point? If so, what did you contact them about? Were they able to help with your problem? If so, how did they solve it?

Did you keep any notes or documents that you may have used or created while you were completing the application? If so, please show them to me and describe what they are.

Section 3. Start the application

Ok, now we’re ready to start the application.

THINK ALOUD PRACTICE

As you go through the application, I’d like you to do something called thinking aloud. I’d like you to tell me everything you are thinking about as you complete each section. This will help us to understand your process for completing the application. This isn’t something that we do every day, so let’s do a practice before we start: Please think aloud as you answer the question, how many windows are there in the place where you live?

IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

THINKALOUD PROBES:

• What are you thinking?

• Tell me more about that. Why did you say [ANSWER]?

• How did you come up with that answer?

• What are you including?

• I noticed that you hesitated. Tell me what you were thinking.

• Keep talking.

• Mm-hm?

Great, that’s what I want you to do as we go through the application. I may remind you to think aloud if you get quiet as we go along. Now we’re ready to get started.

IF USING THE APPLICANT’S COMPUTER:

I will need to access the test site myself, as I am not allowed to share the log-in information with anyone. Can you please open a browser for me? Then I will turn the computer back over to you.

IF USING YOUR OWN COMPUTER, OPEN THE APPLICATION AND LOG IN, THEN PLACE THE COMPUTER IN FRONT OF THE APPLICANT.

ONCE LOGGED IN:

This is the starting point for the application. This is just a testing site for the application, not the real application. I’d like you to go through and complete the application as if it were the real thing. Please enter data for all relevant items. If you need to access records and you have them handy, please do so, to help me understand your process better. If you need records that are not readily available, you can just describe them to me. Please be assured that any and all information you enter will be permanently deleted when we are finished.

IF APPLICANT INDICATES THEY ARE NOT COMFORTABLE ENTERING REAL DATA: You can enter approximate data or make up data.

Please go ahead and complete the application, and remember to think aloud as you go through it. As you finish each page, please pause before moving on to the next page so I can ask some questions.

Page 1 – Application Log In

Please tell me in your own words what the purpose of this page is.

What would/did you do on this page?

Where did you get the information you entered here?

When you completed your actual application, did you have any issues with entering the customer number or password?

Do you recall seeing the text in the shaded boxes? What do these mean to you?

Thank you for that information. Please continue.

Page 2 – Documents and Eligibility

Please tell me in your own words what the purpose of this page is.

When you completed your actual application, do you recall whether you read the information on this page?

Please take a moment to review the information.

How well do you think this information describes the process you went through?

Do/did you find this information useful or not useful?

What does the text in the shaded box mean to you?

What would you do next? (If needed) Please continue.

Page 3 – Acknowledgement Agreement

Pages 4-8 –Acknowledgement Agreement details, appearance of save button

Please tell me in your own words what the purpose of this page is.

What would/did you do on this page?

While completing the real application, do you recall whether you clicked on any of the small triangles? If so, do you recall whether you found that information helpful or not helpful?

What do the “accept/decline” choices mean to you?

What does the green button mean to you? Did you ever use it?

Page 9 – Basic Information

Please tell me in your own words what the purpose of this page is.

What would/did you do on this page?

Did you know all the information you needed for this page, or did you have to use any records? If so, what information did you have to get from records? Please describe the records.

POINT TO THE FIRST ITEM, “IF DIFFERENT…” – Please tell me in your own words what this item is asking for.

POINT TO BLUE BOX “ALTERNATE TELEPHONE…” – Please tell me in your own words what this item is asking for.

Do you recall seeing the note in the blue shaded bar at the top? What does this information mean to you? Did you ever have to call the number to make a correction?

Page 10 – Ownership Information Part 1

Please tell me in your own words what the purpose of this page is.

How did you go about completing this page?

When you completed the actual application, do you recall seeing the shaded box? What does that mean to you? Did you have to add any owners?

(IF ADDED) How did you get the information needed to complete this page? Do you recall seeing the text above “Additional Owner 1”? Did that instruction affect how you completed this page? If so, how?

Do you recall seeing the red button on the right? What does that button mean to you? When you completed the actual application, did you have to remove any owners? If so, why did you have to remove them?

Are there any additional owners, partners, etc. that you did not list on this page? If so, how did you decide not to list them?

Did you find any of these questions sensitive or difficult to answer? If so, please tell me about that? What made them sensitive or difficult?

Page 11 – Ownership Information Part 2

Please tell me in your own words what the purpose of this page is.

How did you go about completing this page?

Did you know all the information you needed for this page, or did you have to use any records? If so, what information did you have to get from records? Please describe the records.

Page 13 – Sales Information

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these items.

What sources of information did you use to complete these items?

How easy or difficult was it to provide the requested data?

FOR “SALES CATEGORIES” AS NEEDED: How did you go about getting the information you reported for each of the “Sales Categories”? Please tell me in your own words what each category means to you. How does each category relate to your store? In your opinion, are these categories clear or not clear? Do you have any suggestions for clarifying these categories?

This section asks you to report percentages. In your opinion, was this difficult or not difficult? Are there any categories for which you had to estimate a figure? If so, how did you come up with your estimate? In your opinion, was this difficult or not very difficult?

NOTE WHETHER RESPONSES TO SALES CATEGORIES TOTAL TO 100%.

Do you recall seeing the text under “Total Retail Sales”? What is this text telling you? Did this text affect how you completed this page? If so, how?

Does your store have wholesale sales? If so, did you include or exclude those sales from what you reported? How easy or difficult is it to exclude them?

Do you recall seeing the text under the “Tax Year” answer choices? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Did you refer to any instructions while completing this page? If so, what instructions did you refer to? Did you find them helpful or not helpful for understanding what this page is asking you for?

Page 13 – Inventory Information

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these items.

What sources of information would/did you use to answer this page?

How easy or difficult is it to provide the requested data?

Do you find the examples helpful or not helpful?

First Section

Do you recall seeing the text in bold print near the top of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

What does “varieties for each category” mean to you? How did you go about getting the information to report for each staple foods category?

What does “on a continuous basis” mean to you? What would you do if you were currently sold out of a type of inventory that you usually have on a continuous basis?

Second Section

Do you recall seeing the text in bold print in the middle of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

How did you decide how to answer for each category?

What does “stocking units” mean to you? Please tell me in your own words what the difference is between “varieties” and “stocking units.”

Third Section

Do you recall seeing the text in bold print toward the bottom of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

How did you decide how to answer for each perishable foods category?

Page 14 – Supplemental Information

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these items.

How easy or difficult is it to provide the requested data?

What sources of information would/did you use to complete each item?

Page 15 – Review and Submit

Please tell me in your own words what the purpose of this page is.

What did you do on this page when completing the real application?

What does the text below “Review and Submit” mean to you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

What does the table mean to you? Did you do anything with this table? (IF NEEDED) Did you click any of the links in this table? If yes, what did you do with the document? Did you print or save a copy?

What does the text below the table mean to you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

POINT TO PENALTY WARNING STATEMENT: Do you recall whether you read the text in the bottom of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

What do the “accept” and “decline” answer choices mean to you? In your opinion, are these choices clear or not clear?

Page 16 – Confirmation

Please tell me in your own words what the purpose of this page is.

What did you do on this page when completing the real application?

Do you recall seeing the shaded box? Please tell me in your own words what this text means to you. Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Did you download a copy of your application?

POINT TO TEXT BELOW “SAVE APPLICATION”: Please tell me in your own words what this text means to you. Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Did you print or save a copy of this page?

Section 4. Wrap-up

In your opinion, was this site easy or difficult to use?

What did you like about this site, if anything?

What did you not like, if anything?

Do you have any suggestions for making the site easier to use?

Did an FNS representative contact you about your application? If so, what did they contact you about? Did they try to get clarification about or correct errors on your application? If so, what did they ask you about? If there were errors, were they able to explain clearly the nature of the errors?

Did an FNS representative visit your store? If so, what did they do? Did they find any apparent errors on your application? If so, were they able to explain clearly the nature of the errors?

Thank you for your time and assistance today.

FNS-252R Reauthorization Application CI Protocol 11