Testing Protocol for Initial Application

Supplemental Nutrition Assistance Program

FNS-252 (initial application) Cognitive Testing Protocol

PARTICIPANT ID #:

DATE:

/

/

INTERVIEWER NAME:

START TIME:

:

AM/PM (circle one)

Section 1. Introduction and Informed Consent

Hello, I’m [INTERVIEWER NAME]. I work for the Census Bureau. We are working with the Food and Nutrition Service to help them improve the SNAP application. Thanks for agreeing to meet with me today.

INTRODUCE OBSERVERS, IF ANY

We are talking to store owners or their staff about the process of applying for SNAP authorization. I’ll ask you to access the online application and walk me through the steps you took in completing the application. It’s ok if you don’t remember all the details.

Please be assured that our meeting today will not affect your status with regard to the SNAP program. The sole purpose of this meeting is to understand how the application process works for applicants and to look for any ways the application may need to be improved to make it easier to use.

PLACE THE CONSENT FORM IN FRONT OF THE RESPONDENT

Before we start, I would like you to read over the document in front of you. It explains a little bit about this interview and provides information about your rights as a participant. It also asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading, please sign it.

PARTICIPANT READS AND SIGNS FORM

IF PARTICIPANT CONSENTED TO AUDIO TAPING, READ:

I will now turn on the audio recorder.

TURN ON AUDIO RECORDER

Section 2. Background Information

I’d like to ask you a few questions before we open the application.

Please describe your business.

What is your role in the business?

(If owner) How long have you owned the business?

(If not owner) How long have you been working for this business?

What do you remember about completing the SNAP application?

Were there any aspects you found easy or straightforward? What were they?

Were there any aspects you had problems with? What were they?

Did you need to ask anyone else in your business to provide any information for the application?

• What information did you need to get from others?

• What is their position/title? How did you go about getting the information from them?

• How did you communicate the request?

• Did they have any questions, and if so, what did you tell them?

• Did they seem to have any problems with getting you the requested information, if so, what kinds of problems, and how did you handle the problems they presented?

• Did they express any concerns about providing you the requested information?

• In your opinion, were you able to get the requested information in what you consider a timely manner?

• Did you have to make any changes to the information they provided in order to make it fit the application, if so, what kinds of changes?

How long did it take you to complete the application, including the time you may have spent consulting records and other persons?

Did you contact the Food and Nutrition Service for assistance at any point? If so, what did you contact them about? Were they able to help with your problem? If so, how did they solve it?

Did you keep any notes or documents that you may have used or created while you were completing the application? If so, please show them to me and describe what they are.

Section 3. Start the application

Ok, now we’re ready to start the application.

THINK ALOUD PRACTICE

As you go through the application, I’d like you to do something called thinking aloud. I’d like you to tell me everything you are thinking about as you complete each section. This will help us to understand your process for completing the application. This isn’t something that we do every day, so let’s do a practice before we start: Please think aloud as you answer the question, how many windows are there in the place where you live?

IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

THINKALOUD PROBES:

• What are you thinking?

• Tell me more about that. Why did you say [ANSWER]?

• How did you come up with that answer?

• What are you including?

• I noticed that you hesitated. Tell me what you were thinking.

• Keep talking.

• Mm-hm?

Great, that’s what I want you to do as we go through the application. I may remind you to think aloud if you get quiet as we go along. Now we’re ready to get started.

IF USING THE APPLICANT’S COMPUTER:

I will need to access the test site myself, as I am not allowed to share the log-in information with anyone. Can you please open a browser for me? Then I will turn the computer back over to you.

IF USING YOUR OWN COMPUTER, OPEN THE APPLICATION AND LOG IN, THEN PLACE THE COMPUTER IN FRONT OF THE APPLICANT.

ONCE LOGGED IN:

This is the starting point for the application. This is just a testing site for the application, not the real application. I’d like you to go through and complete the application as if it were the real thing. Please enter data for all relevant items. If you need to access records and you have them handy, please do so, to help me understand your process better. If you need records that are not readily available, you can just describe them to me. Please be assured that any and all information you enter will be permanently deleted when we are finished.

IF APPLICANT INDICATES THEY ARE NOT COMFORTABLE ENTERING REAL DATA: You can enter approximate data or make up data.

Please go ahead and complete the application, and remember to think aloud as you go through it. As you finish each page, please pause before moving on to the next page so I can ask some questions.

Page 1 – Start new application / continue saved application

Page 2 – “Check status of previous application” expanded

Please tell me in your own words what the purpose of this page is.

What did you do on this page?

After you completed your actual application, did you click on any of the options for submitted applications? If yes, what did you do?

ASK APPLICANT TO EXPAND BOTH ARROWS IF NOT ALREADY DONE: In your opinion, what is the difference between these two tables?

Thank you for that information. Please go ahead and start a new application.

Page 3 – Select application type

Page 4 – Shows PA/PRA text

Please tell me in your own words what the purpose of this page is.

What did you do on this page?

Did you have any difficulty in deciding what to select?

When you completed your real application, did you click on the triangle for information about the Privacy Act and Paperwork Act?

Page 5 – Before you Begin

Please tell me in your own words what the purpose of this page is.

What did you do on this page?

Did you notice this information when you completed the real application? Did/do you find it useful or not useful?

Do you recall seeing the note in the shaded bar at the top? What does this information mean to you?

Do you recall seeing the note in the blue shaded bar at the bottom? What does this information mean to you?

Page 6 – Acknowledgement Agreement

Page 7 – Acknowledgement Agreement details

Please tell me in your own words what the purpose of this page is.

What would/did you do on this page?

While completing the real application, do you recall whether you clicked on any of the small triangles? If so, do you recall whether you found that information helpful or not helpful?

What do the “accept/decline” choices mean to you?

Page 8 – Basic Information

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these items.

How easy or difficult is it to provide the requested data?

What sources of information would/did you use to answer this question?

While completing the real application, do you recall whether you clicked on the green button at the bottom? What does this button do?

Page 9 – Ownership Information – non-profit

Page 10 – Ownership Information – government-owned

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these items.

How easy or difficult is it to provide the requested data?

What sources of information would/did you use to answer this question?

Do you recall whether you read the paragraph at the top of the page? Please tell me in your own words that that is saying.

Do you recall seeing the blue text in the middle of the page? Did that instruction affect how you completed this page?

What does the blue button on the right mean to you? When you completed the real application, did you add any additional owners?

Are there any additional owners, partners, etc. that you did not list on this page? If so, how did you decide not to list them?

Did you find any of these questions sensitive or difficult to answer? If so, please tell me about that? What made them sensitive or difficult?

Page 11 – Sales Information (actual figures)

Page 12 – Sales Information (estimated figures)

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these items.

How easy or difficult is it to provide the requested data?

What sources of information would/did you use to answer this question?

FOR “SALES CATEGORIES” AS NEEDED: How did you go about getting the information you reported for each of the “Sales Categories”? Please tell me in your own words what each category means to you. How does each category relate to your store? In your opinion, are these categories clear or not clear? Do you have any suggestions for clarifying these categories?

This section asks you to report percentages. In your opinion, was this difficult or not difficult? Are there any categories for which you had to estimate a figure? If so, how did you come up with your estimate? In your opinion, was this difficult or not difficult?

NOTE WHETHER RESPONSES TO SALES CATEGORIES TOTAL TO 100%.

Do you recall seeing the text under “Total Retail Sales”? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

When you submitted your real application, did you submit estimated or actual figures?

IF ESTIMATED: How did you come up with your estimate?

Do you recall seeing the text under the “Estimated” and “Actual” answer choices? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Does your store have wholesale sales? If so, did you include or exclude those sales from what you reported? How easy or difficult is it to exclude them?

Do you recall seeing the text under the “Tax Year” answer choices? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Page 13 – Inventory Information

Page 14 – duplicate with data entered

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these spaces/fields.

How easy or difficult is it to provide the requested data?

What sources of information would/did you use to answer this question?

Do you find the examples helpful or not helpful?

Do you recall seeing the text in bold print near the top of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

What does “varieties for each category” mean to you?

What does “on a continuous basis” mean to you? What would you do if you were currently sold out of a type of inventory that you usually have on a continuous basis?

Do you recall seeing the text in bold print in the middle of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

What does “stocking units” mean to you? Please tell me in your own words what the difference is between “varieties” and “stocking units.”

Do you recall seeing the text in bold print toward the bottom of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Page 15 – Supplemental Information

Page 16 – duplicate showing expanded days/hours

Please tell me in your own words what the purpose of this page is.

How did you go about completing these items?

Please tell me in your own words what is being asked for in each of these spaces/fields.

How easy or difficult is it to provide the requested data?

What sources of information would/did you use to answer this question?

Page 17 – Review and Submit

Please tell me in your own words what the purpose of this page is.

What did you do on this page when completing the real application?

Do you recall seeing the text in the shaded box near the top? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Do you recall seeing the text in the shaded box in the middle of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Do you recall whether you read the text in bold print in the bottom half of the page? What is this text telling you? Did you find it helpful or not helpful? In your opinion, is this text clear or not clear?

Please tell me in your own words what the blue button is for. When completing your application, did you click that button? What did you do with the PDF after downloading it?

What do the “accept” and “reject” answer choices mean to you?

Page 18 – Documents to Submit

Please tell me in your own words what the purpose of this page is.

What did you do on this page when completing the real application?

Please tell me in your own words what each of these paragraphs mean to you.

The text at the top mentions an “FNS number.” What is that? Did you do anything with that number?

What is the purpose of the blue button “Print required certification…”? Did you use this button when you completed your real application?

Please tell me in your own words what the green button is for. Did you use this button when you completed your real application? Please click this now and select an unimportant file and upload it. IF USING CENSUS LAPTOP: Please browse to the desktop and select the file labeled “fill in name” and upload it.

When you completed your real application, what did you upload? Did you already have these files or did you have to create them for this application? If you created them, how did you do it? Did you have any difficulty with this process?

What is the purpose of the blue button at the bottom? Did you use this button when you completed your real application?

Do you recall seeing the text at the bottom that starts with “Important”? What does that text mean to you?

In steps 2 through 4 there are links to examples. When you completed your real application, did you click on any of these links? If so, did you find the examples helpful or not helpful?

On the left side of some pages is a list of the sections of the application. Did you use this when you completed the application? If so, how did you use it? What did you use it for?

At the bottom of most pages is a previous button. Did you use this when you completed the application? If so, what did you use it for?

At the bottom of most pages is a print page button. Did you use this when you completed the application? If so, what did you use it for? What made you decide to print some pages? Did you retain them after your submitted your application?

Section 4. Wrap-up

In your opinion, was this site easy or difficult to use?

What did you like about this site, if anything?

What did you not like, if anything?

Do you have any suggestions for making the site easier to use?

Did an FNS representative contact you about your application? If so, what did they contact you about? Did they try to get clarification about or correct errors on your application? If so, what did they ask you about? If there were errors, were they able to explain clearly the nature of the errors?

Did an FNS representative visit your store? If so, what did they do? Did they find any apparent errors on your application? If so, were they able to explain clearly the nature of the errors?

Thank you for your time and assistance today.

FNS-252 Initial Application CI Protocol 11